continuous security testing
TRANSCRIPT
Who am I?
• Agile, TDD Coaching, Ugly Code Cleaning Dude
• I love coding - Java, C#, Javascript, C/C++, PHP, Perl, and some weird ones
• I speak English, Cantonese, and Mandarin
2
Odd-e Pte. Ltd.Steven Mak 麥天志Agile CoachHong KongEmail: [email protected]: www.odd-e.comTwitter: stevenmak
Sunday, 15 December, 13
Is that what you feel?
4
Script Unreadable?
Keep Changing?
Time consuming to write?
Sunday, 15 December, 13
Technical Activity
Workflow
Specification pyramid
5
RuleClarity
Stability
Specification
Users can understand
AutomationTechnical
Sunday, 15 December, 13
Use Examples
6
With 3 judges giving scores 4, 20, and 18, the displayed score should be 42.
When the first 2 judges have given their scores, e.g. 10 and 5, the intermediate score of 15 should be displayed already.
No scores displayed as a dash (–), not zero.
Maximum score from a judge is 20 points!
Sunday, 15 December, 13
Examples, Tests, and Spec
7
Examples Tests
Requirements
can become
elaborate verify
Sunday, 15 December, 13
More ideas from• Threat Modelling• Session-Based Test Management / Exploratory Testing• Product Requirement• Experts
8
Sunday, 15 December, 13
Avoid imperative• login• enter username• enter password• enter homepage• click category• choose product• put it on shopping cart• click generate order• .....
10
Sunday, 15 December, 13
Avoid imperative• login• enter username• enter password• enter homepage• click category• choose product• put it on shopping cart• click generate order• .....
11
Given I selected a doll in shopping cartWhen I generate orderThen the order should contain dolland the price is 83.55
Sunday, 15 December, 13
Avoid imperative• login• enter username• enter password• enter homepage• click category• choose product• put it on shopping cart• click generate order• .....
12
Given I selected a doll in shopping cartWhen I generate orderThen the order should contain dolland the price is 83.55
This “Given When Then” is a common pattern called Gherkin
Sunday, 15 December, 13
Good ones• Focus on business, not software design• Not coupled with code• Not coupled with UI• Concise• Use domain languages
13
Getting us towards Living Documentation and can be executed against existing
system
Sunday, 15 December, 13
Robot Frameworkwww.robotframework.org
14
Sunday, 15 December, 13
Test Tools
Robot Architecture
15
Test Data (Tables)
Robot Framework
Test Libraries
System Under Test
Test Library API
application interfaces
Robot comes with a number of built-in test libraries and you can (should!) add your own.
Test libraries can use any test tool necessary to interact with the system under test.
Sunday, 15 December, 13
Test Cases are composed of keyword-driven actions
17
!"#$%&'()*+%),'-./()0
this is the name of a test case
Sunday, 15 December, 13
Test Cases are composed of keyword-driven actions
17
!"#$%&'()*+%),'-./()0
this is the name of a test casethese keywords form the test case
Sunday, 15 December, 13
Test Cases are composed of keyword-driven actions
17
!"#$%&'()*+%),'-./()0
this is the name of a test casethese keywords form the test case
keywords receive arguments
Sunday, 15 December, 13
2 types of keywords
18
We can import keyword libraries for a test case
...and libraries may be configured, too.
Sunday, 15 December, 13
2 types of keywords
18
We can import keyword libraries for a test case
...and libraries may be configured, too.
This keyword comes from the imported library.
Sunday, 15 December, 13
2 types of keywords
18
We can import keyword libraries for a test case
...and libraries may be configured, too.
This keyword comes from the imported library.
This is a user keyword, implemented in table format.(Think macros composed of other macros.)
Sunday, 15 December, 13
19
Data-driven test cases
this is the name of a test casethese keywords form the test case
keywords receive arguments
Sunday, 15 December, 13
20
using Template
*** Test Cases ***Email Delivered Acceptance Rule [Template] Confirm Email Delivered Workflow [email protected] [email protected] 3asyp3asy 1 [email protected] [email protected] 3asyp3asy 0
*** Keywords ***Confirm Email Delivered Workflow [Arguments] ${sender} ${recipient} ${password} ${number_of_emails_expected} Open Mail Box ${MAIL_SERVER} ${recipient} ${password} Count Mail Received ${sender} ${number_of_emails_expected}
Keyword used as template
test data feed as arguments
Sunday, 15 December, 13
Given-when-then (BDD)
21
*** Test Cases ***Addition Given calculator has been cleared When user types "1 + 1" and user pushes equals Then result is "2"
*** Keywords ***Calculator has been cleared Push button C
User types "${expression}" Push buttons ${expression}
User pushes equals Push button =
Result is "${result}" Result should be ${result}
this is the name of a test case
these keywords form the test case
Sunday, 15 December, 13
Variables
22
!"#$"%&'(
)#*+,-*++"./,&$.'0
!"#$%&'"(()*+,*%-."/012345167&89:&."(()*+,*%-.";400<=2>6?@89>@."A$B'.C'CD8'A-Sunday, 15 December, 13
25
*** Settings ***Resource resource.txt
*** Test Cases ***Checking Opened Ports [Template] Only these ports are opened 22 25 80 135 139 445
*** Keywords ***Only these ports are opened [Arguments] @{expected_ports} @{actual_ports_opened}= Scan with Fast Mode ${HOST} List Should Contain Sub List ${actual_ports_opened} ${expected_ports}
*** Settings ***Library nmapLibraryLibrary Collections
*** Variables ***${HOST} www.scrumprimer.org
import nmap
class nmapLibrary: def scan_with_fast_mode(self, host): nm = nmap.PortScanner() nm.scan(str(host), arguments="-F") return [str(port) for port in nm[str(nm.all_hosts()[0].encode())].all_tcp()]
resource.txt
port_scanning.txt
nmapLibrary.py (with python-nmap)
Sunday, 15 December, 13
26
pybot -d output nmap.txt ==============================================================================Port Scaning ==============================================================================Checking Openned Ports | PASS |------------------------------------------------------------------------------Nmap | PASS |1 critical test, 1 passed, 0 failed1 test total, 1 passed, 0 failed==============================================================================Output: /Users/stevenmak/Work/robotframework/securityTests/2013.12.14VXCon/output/output.xmlLog: /Users/stevenmak/Work/robotframework/securityTests/2013.12.14VXCon/output/log.htmlReport: /Users/stevenmak/Work/robotframework/securityTests/2013.12.14VXCon/output/report.html
run the test:
report: (also available in xml format for Jenkins integration)
Sunday, 15 December, 13
Acceptance Test Driven Development
29
Discussin workshop
Developin concurrence
Deliverfor acceptance
Sunday, 15 December, 13
30
Discussin workshop
Developin concurrence
Deliverfor acceptance
Focus on customer collaboration and user
engagement. Try to get as many of these people attend
as you can.
Product OwnerDev Team
Users
IT operations
Help DeskTech Writers
?
Sunday, 15 December, 13
31
Discussin workshop
Developin concurrence
Deliverfor acceptance
With 3 judges giving scores 4, 20, and 18, the displayed score should be 42.
When the first 2 judges have given their scores, e.g. 10 and 5, the intermediate score of 15 should be displayed already.
No scores displayed as a dash (–), not zero.
Maximum score from a judge is 20 points!
Sunday, 15 December, 13
31
Discussin workshop
Developin concurrence
Deliverfor acceptance
With 3 judges giving scores 4, 20, and 18, the displayed score should be 42.
When the first 2 judges have given their scores, e.g. 10 and 5, the intermediate score of 15 should be displayed already.
No scores displayed as a dash (–), not zero.
Maximum score from a judge is 20 points!
Robot tests are written in tables so that computers can read them
Sunday, 15 December, 13
Collaboration is key
33
team gets feedback earlier
scope of work is clear and
understood by all
team understands what they're implementing
shared language and vocabulary is
built
team collaborates closely with
product owner
Sunday, 15 December, 13
CITCON Hong Kong
34
• When: Apr 11 & 12, 2014• Cost: Free• Registration: contact me• Sponsorship Welcome!
http://citconf.com/hongkong2014/
Sunday, 15 December, 13
Thank you for spending time with me this evening.More feedback can be sent to:
35
Odd-e Hong Kong Ltd.Steven Mak 麥天志Agile CoachHong KongEmail: [email protected]: www.odd-e.comTwitter: stevenmak
Sunday, 15 December, 13