12/14/2015 concept of test driven development applied to embedded systems m. smith university of...
TRANSCRIPT
![Page 1: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/1.jpg)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
1
Automated Testing Environment
Concepts required for testing embedded systems adopted in this course
(quizzes, assignments and laboratories)
![Page 2: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/2.jpg)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
2 / 28
To be tackled today
Why test, and what kinds of tests are there? Difference between
TLD – Test Last Development TDD – Test Driven Development
(Test First Development) How do you add tests? More details on the testing process E-TDD and
the testing toll E-UNIT Other kinds of available tests – time and access Design of custom TESTs and ASSERTS
![Page 3: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/3.jpg)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
3 / 28
Why test?
Unit test I AM CURRENTLY BUILDING A NEW FUNCTION –
does it work the way I expect? Testing my and my partner’s understanding of the
Design WE ARE DESIGNING A NEW PRODUCT
-- If we can’t explain (to ourselves via a test) the product ideas – how do we know when product is working?
Regression testing MY PARTNER ADDED A NEW FEATURE to a product –
how do I know it did not “break something” I wrote? System testing
How do I PROVE TO THE CUSTOMER that the product (Lab. or assignment) works the way that was requested?
![Page 4: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/4.jpg)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
4 / 28
What are the problems with this sort of “Human intervention” test?
This code would be placed
inAssignments1
directory with the rest of the
customer code
![Page 5: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/5.jpg)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
5 / 28
VisualDSP development environment(IDDE) – Communication overhead
DebuggingInterface
“JTAG” boundary scan operation
Processor
Peripherals
BF533 Evaluation Board
VisualDSP++
program
Debugging Interface
HPPCIce
Lab. station (ICT318 / 320)
Each time we want to send a message (via printf( ) or cout) we must “stop” the processor from doing what we want it to do, and then send messages over the interface – Slow, plus the testing can impact on peripheral performance.
E.g. no sound while doing printing in Assignment 2 and Lab. 1
THIS PROBLEM IS COMMON TO ALL EMBEDDED SYSTEM DEVELOPMENT NOT JUST THE BLACKFIN USED IN THE LAB.
![Page 6: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/6.jpg)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
6 / 28
More issues with this sort of test!
Tests are mixed up “with production code”, in many places Must remove tests before release to customer
Unreliable – may result in test messages at wrong time in released code
Difficult to “add” and “remove tests” from production code without introducing new errors
Results can be checked automatically – therefore unlikely to have tests run often during development – leads to unreliable product
![Page 7: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/7.jpg)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
7 / 28
E-TDD Tool
E-TDD – embedded test driven development Build-the-tests-first (during design) ideas for product
development has been made popular by the “Agile” approach
Requires an automated testing approach (test framework) In this class, we are going to use the E-Unit tool for
its automated behaviour Will use in both “test last” development (what most people
currently do) and in “test first” development (trying to encourage).
I (the customer) will provide tests so that you can check that you have developed the “right” stuff at the “high” level
You will build additional tests to check your own code at the “low level”
![Page 8: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/8.jpg)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
8 / 28
Test Environment
Can download “LabTestEnvironment.zip” file from the web (Check the September webpages for exact name of file)
LabX directory is where you keepyour “project” code LabXcode_main.cpp LabX_start.cpp LabX_initializeASM.asm
LabXTests is where you keep the tests for LabX
LabXtests_main.cpp (built by GUI) Tests_for_LabX_start.cpp Tests_for_LabX_initializeASM.cpp written in C++) E_UnitConnect.cpp (built by GUI)
![Page 9: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/9.jpg)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
9 / 28
Build a new VDSP project “Assignment1Test” and place in “Assignment1Tests” directory
Now add certain standard files to the project Same set of files for all assignments and
laboratory Use the automated features of the GUI
to “convert” VDSP project to test project
![Page 10: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/10.jpg)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
10 / 28
Now add your Assignment1 subroutines files to the Assignment1Test project
Don’t move or copy the files into the test directory – just “add” them to the project
Note the use of “Assignment1.h” which has the prototypes for all “C++” and “assembly code” functions used (from AssignmentX dir.)
This files are “added” using VDSP to the“Test” project and NOT copiedfrom the Assignment1 directory
![Page 11: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/11.jpg)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
11 / 28
The GUI has automatically added the “Tests_main.cpp”
![Page 12: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/12.jpg)
The GUI has automatically added a “TestGroup” file for you
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
12 / 28
![Page 13: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/13.jpg)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
13 / 28
Modify “Assignment1Tests.cpp” to perform the needed testing
![Page 14: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/14.jpg)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
14 / 28
Tests and Test Groupshave a standard format
TESTGROUP NAME
TESTGROUP NAME,
TEST NAME
Add further tests at this location
![Page 15: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/15.jpg)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
15 / 28
Tests then need to be customizedA lot of this can be automated by GUI
![Page 16: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/16.jpg)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
16 / 28
Test ComponentsRemember format for “exams” and “quizzes”
The ACTUAL Test (This is a TEST MACRO)CHECK(result1 == result2) – If an error occurred what
was ‘result1’ being used for?Error result1 != result2
Would get better test error messages if we “self-documented the code”
int resultCPPCode = ; int resultASMCode = CHECK(resultCPPCode == resultASMCode);Error resultCPPCode != resultASMCode
Test CONTROL statements (ALL MACROS) TEST_CONTROL( testgroup_name) TEST_FILE_RUN_NOTIFICATION( testgroup_name)
![Page 17: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/17.jpg)
Have added stub versions of functions
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
17 / 28
Note “code DEFECT”
There is NO return valueplaced in R0
![Page 18: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/18.jpg)
Now build source files to make executable
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
18 / 28Note the format of the error messages with both C / C++ and name-mangled ASM name shown
![Page 19: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/19.jpg)
Fix the code. Review the code for syntax errors and “logical” errors
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
19 / 28
DON’T IGNORE WARNINGSFIX THE PROBLEM, e.g. add .extern or fix type
![Page 20: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/20.jpg)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
20 / 28
Now “build and run” the testsAll the tests MUST fail (no code written) otherwise using bad tests
Quick test failure visualization
Detailed test failure information
Something wrong – how did the test pass with no code?
![Page 21: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/21.jpg)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
21 / 28
Now “build and run” the testsAll the tests should fail (no code present)
Quick test failure visualization
Detailed test failure information
Clicking on the “detailed test info”automatically takes you to the “test” that failed
![Page 22: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/22.jpg)
The test that failed
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
22 / 28
Clicking on the “detailed test info”automatically takes you to the “test” that failed
![Page 23: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/23.jpg)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
23 / 28
Test information modes arecontrolled in “Test_main” Show Failures and Expected Failures only
Show Failures and Successes
![Page 24: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/24.jpg)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
24 / 28
Add a separate test to show that result2(from the ASM code) is equal to 42
Add further tests at this location
![Page 25: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/25.jpg)
Gui automatically adds new test
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
25 / 28
![Page 26: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/26.jpg)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
26 / 28
Write the tests for “self-documenting” messages(What does “result1 == result2” mean if 200 tests present?)
HAVE – What is a ‘result1’ and a ‘result 2’
WANT
![Page 27: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/27.jpg)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
27 / 28
Write the “CPP” code to get to workbut “Write the ASM for speed”
FAIL
SUCCESS
SUCCESS
“../E-UNITIncludes/E-UNIT_Tests1Sept07.h”
![Page 28: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/28.jpg)
Get a “Volun-tell” to come to the front and write the “better” test
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
28 / 28
![Page 29: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/29.jpg)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
29 / 28
Available ASSERTS More can be “customized”
CHECK(expected == actual) – Success if passes
XF_CHECK(expected == actual) – Success if “fails”
CHECK_EQUAL(expected, actual)
XF_CHECK_EQUAL(expected, actual) – Success if fails
CHECK_CLOSE(expected, actual, tolerance)
CHECK_ARRAY_EQUAL(expected, actual, length)
CHECK_ARRAY_CLOSE(expected, actual, length, tolerance)
REPORT(“character message”);
Special changes to editor .ini file makes this ASSERTS into highlighted key-words
![Page 30: 12/14/2015 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1 Automated Testing Environment Concepts](https://reader035.vdocuments.site/reader035/viewer/2022062322/5697bfa31a28abf838c969ef/html5/thumbnails/30.jpg)
04/21/23 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada
30 / 28
Handled today – some easy “non-programming” quiz and midterm questions
Why test, and what kinds of tests are there? Difference between
TLD – Test Last Development TDD – Test Driven Development
(Test First Development) How do you add tests for Assignment 1? More details on the testing process E-TDD and
the testing tool E-UNIT Other kinds of available tests – time and access Design of custom TESTs and ASSERTS