testing automaton - cfsummit 2016
TRANSCRIPT
Who am I
• Luis Majano
• Computer Engineer
• Imported from El Salvador
• CEO of Ortus Solutions
• Creator of all things Box
Bugs cost money
Cost To Fix Time detected Requirements Design Building Testing Post-Release
Time Introduced
Requirements 1x 3x 5-10x 10x 10-100x
Design --- 1x 10x 15x 25-100x
Building -- -- 1x 10x 10-25x
^ Kaner, Cem; James Bach, Bret Pettichord (2001). Lessons Learned in Software Testing: A Context-Driven Approach. Wiley. p. 4. ISBN 0-471-08112-4.
• Just do it!
• You will get dirty
• It can hurt (a little)
• Learned behavior
NO MORE EXCUSES
IT WILL ACCELERATE YOUR DEVELOPMENT
TestBox : What you get
API Docs
Core
Test Browser
Test Harness
Test Runner
Samples
install testbox —saveDev
install testbox@be —saveDev
Test Harness
Automated test results!
xUnit/BDD Test Bundles
Harness bootstrap
HTML Runner
ANT Runner
Test Bundle CFCs• Inherits from testbox.system.BaseSpec
•TesBox will then execute all tests/specs/suites within 1 or more bundles
RUN TESTS
• Execute bundles via the URL• http://localhost/tests/bundle.cfc?method=runRemote
• SOAP Runner• HTTP/REST Runner• ANT Runner• NodeJS Runner• CommandBox Runner• Custom Runner
TESTBOX REPORTERS
•ANTJunit : Variant of JUnit for Jenkins•Codexwiki : Mediawiki syntax•Console : Server console•Doc : Semantic HTML•Dot : Awesome Dots•JSON•JUnit•Raw : CFML Struct•Simple : A basic HTML reporter•Text : Back to the 80’s, Text•MinText : Great for CommandBox Runners•XML •Tap : A test anything protocol reporter•Min : Minimalistic Report
CommandBox Runnerpackage set testbox.runner = "http://127.0.0.1:57538/tests/runner.cfm"
testbox run
1
2
{ "runner": "http://127.0.0.1:57538/tests/runner.cfm", "directory": "/tests/specs", "recurse": true}
NodeJS Test Runnernpm install -g testbox-runner
create .testbox-runnerrc
1
2
Run it : testbox-runner2
DEVELOPER AUTOMATION
1.Grunt Watchers + LiveReload2.Grunt Watchers + testbox-runner3.Grunt Watchers + commanbox4.ColdBox Elixir
Jenkins + CommandBoxwget http://mirrors.jenkins-ci.org/war-stable/latest/jenkins.war
box server start warpath=jenkins.war
1
2
Accounts
Activate For Repositories
1
2
Create .travis.yml3
Commit to repository4
Travis CI - Getting Started
TRAVIS + ANTlanguage: javabranches: only: - developmentsudo: requireddist: trustybefore_install:- sudo apt-key adv --keyserver keys.gnupg.net --recv 6DA70622- sudo echo "deb http://downloads.ortussolutions.com/debs/noarch /" | sudo tee -a /etc/apt/sources.list.d/commandbox.listinstall:- sudo apt-get update && sudo apt-get --assume-yes install commandbox- box install- box server startbefore_script:# Startup the app- curl http://localhost:49616/script: - ant -f workbench/build.xml
TRAVIS + COMMANDBOXlanguage: javasudo: requireddist: trustybefore_install:- sudo apt-key adv --keyserver keys.gnupg.net --recv 6DA70622- sudo echo "deb http://downloads.ortussolutions.com/debs/noarch /" | sudo tee -a /etc/apt/sources.list.d/commandbox.listinstall:- sudo apt-get update && sudo apt-get --assume-yes install commandbox- cd tests- box install- box server startbefore_script:# Startup the app- curl http://localhost:49615/script: > testResults="echo $(box testbox run)"; echo "$testResults"; if grep -i "\[Failures: [1-9][0-9]\?[0-9]\?\]\|\[Errors: [1-9][0-9]\?[0-9]\?\]\|<t[^>]*>\|<b[^>]*>" <<< $testResults; then exit 1; fi
TRAVIS + MULTIPLE ENGINESlanguage: javasudo: requiredjdk:- oraclejdk8env: matrix: - [email protected] - ENGINE=lucee@5 - ENGINE=adobe@2016 - ENGINE=adobe@11 - ENGINE=adobe@10before_install:- sudo apt-key adv --keyserver keys.gnupg.net --recv 6DA70622- sudo echo "deb http://downloads.ortussolutions.com/debs/noarch /" | sudo tee -a /etc/apt/sources.list.d/commandbox.listinstall:- sudo apt-get update && sudo apt-get --assume-yes install commandbox- box installbefore_script:- box server start cfengine=$ENGINE port=8500script: > testResults="$(box testbox run runner='http://127.0.0.1:8500/tests/runner.cfm' )"; echo "$testResults"; if grep -i "\[(Failures|Errors): [1-9][0-9]\?[0-9]\?\]\|\[Errors: [1-9][0-9]\?[0-9]\?\]\|<t[^>]*>\|<b[^>]*>" <<< $testResults; then exit 1; finotifications: email: false
Resources
• TestBox : ortussolutions.com/products/testbox
• CommandBox: ortussolutions.com/products/commandbox
• Slack: boxteam.herokuapp.com
• CFML Slack: #box-products
• travis-ci.org
• jenkins.io