12 May 2015
Testing Communications in the Connected Car
Chris Wellens
InterWorking Labs
iwl.com
+1.831.818.7963
InterWorking Labs
Saving the World from
Network Failure!
12 May 2015
Testing Communications in the Connected Car
● InterWorking Labs helps engineers:
➢ Understand how well a device or app will perform under adverse network conditions....
➢ With network emulators and protocol testers …
➢ To optimize device or app performance.
12 May 2015
Testing Communications in the Connected Car
Tonight's Topic:
How do connected car
[manufacturers, suppliers, aftermarket players]
test, verify, and validate the proper
[implementation, interoperability, robust operation]
In the automotive network?
12 May 2015
Testing Communications in the Connected Car
Agenda
1. What components require testing?
2. What industry standards are used?
3. How are those standards implemented?
4. How do you test the communications?
– What types of testing should be applied?
5. ECU and related modules testing
6. Infotainment and Driver Assistance testing
12 May 2015
Testing Communications in the Connected Car
1. What vehicle components require communications testing?
Any component that exchanges data with:
another component another vehicle other external communication nodes (vehicle-to-
infrastructure communication).
Depending on the industry standard communication protocols
12 May 2015
Testing Communications in the Connected Car
2. What Industry Standards Are Used?
AUTOSAR: The AUTomotive Open System Architecture
See www.autosar.org
12 May 2015
Testing Communications in the Connected Car
12 May 2015
Testing Communications in the Connected Car
➔ But AUTOSAR only applies to ECUs!
➔ Infotainment needs more bandwidth!
➔ TODAY FUTURE
AUTOSAR “Classic” Platform
“Adaptive” AUTOSAR Platform
For ECUs For ECUs, Infotainment, Driver Assistance,And more
12 May 2015
Testing Communications in the Connected Car
3. How are those industry standard solutions implemented?
● You can always read the standards specifications and do the implementation yourself.
● Or, you can license an AUTOSAR implementation from an embedded system supplier:
– Our host: Mentor Graphics
– Others: Wind River, QNX, Electrobit, Vector, etc.
● Or, you can look for an open source solution.
12 May 2015
Testing Communications in the Connected Car
4. Now we address communications testing.
12 May 2015
Testing Communications in the Connected Car
What are the different types or kinds of testing that should be applied?
Functional Tests include:
‣ Negative Testing
‣ Inopportune Testing
‣ Conformance/Compliance Testing
‣ Interoperability Testing
‣ Deep-path Testing
And ...
12 May 2015
Testing Communications in the Connected Car
Stress and Reliability Tests include:
‣ Load Testing
‣ Stress Testing
‣ Performance Testing
‣ Line Speed Testing
‣ Robustness (Security) Testing
‣ Endurance Testing
12 May 2015
Testing Communications in the Connected Car
12 May 2015
Testing Communications in the Connected Car
AUTOSAR requires the TCP/IP family.
TCP is 40 years old –
Isn't it thoroughly debugged at this point?
So glad you asked that question ...
12 May 2015
Testing Communications in the Connected Car
➢ InterWorking Labs used its Maxwell PRO TCP/IP Test Suite to test Windows 7 with all updates applied.
➢ Windows 7 passed the IPv6 Ready Logo program
➢ Windows 7 passed at the highest level.
AND here's what we found ...
12 May 2015
Testing Communications in the Connected Car
● Out of 396 TCP/IP Protocol Tests:
✗ 327 tests passed✗ 69 tests failed✗ 10 IPv6 protocol-specific tests failed✗ 18 tests of other protocols using IPv6 failed
(that passed over IPv4)● For the IPv6 failures, the most serious ones
involved incorrect checksums and IPv6 fragments delivered out of order
12 May 2015
Testing Communications in the Connected Car
Of the non-IPv6 protocol tests that failed, the ones that could cause problems to customers are:
✔ incorrect checksums
✔ incorrect issuance of ICMPv4 redirects
✔ lack of support for overlapping IPv4 fragments
✔ dropping of some IPv4 datagrams containing unknown options
✔ poor handling of out-of-order TCP segments
✔ poor handling of lost TCP SYN connections.
12 May 2015
Testing Communications in the Connected Car
CONCLUSION:
The TCP/IP protocol family requires testing on all products.
12 May 2015
● Or
12 May 2015
Testing Communications in the Connected Car
The TCP family is used in AUTOSAR.
How do we test for robustness, security, interoperability and conformance/compliance?
✔ We will use a test suite program to send pathological TCP segments to the device under test.
✔ The device will have to respond appropriately.
✔ It should not hang or crash.
✔ It should exhibit the proper behavior as defined in the standards document.
So how does that work conceptually?
12 May 2015
Testing Communications in the Connected Car
12 May 2015
Testing Communications in the Connected Car
Conformance/Compliance Fuzz Tests
Sends pathological segments Sends pathological segments
Reports all errors Monitor to check for DUT crash
Grades each test: Pass/Fail/etc DUT crashed or did not crash
Reports cause of the error Human tester must analyze and find the code causing the crash
Junior engineer Senior engineer required
12 May 2015
Testing Communications in the Connected Car
12 May 2015
Testing Communications in the Connected Car
Here's an example of a negative test:
Negative testing verifies that the device under test (DUT) responds correctly to error conditions or unacceptable input conditions.
SUMMARY: Append unknown option with a length of 41 to all segments.
DESCRIPTION: This test adds an unknown option to all TCP packets unless there is already an option in the packet or the packet is so large that adding an option would cause fragmentation.
The option to be added here consists of 6 octets. However the length octet in the option has a value of 41, which is too large to fit into the TCP header of any TCP packet (and doesn't match the actual number of octets actually in the option).
The option type is 31 which has not been allocated by IANA.
12 May 2015
Testing Communications in the Connected Car
Negative Test Example continued.
...
ASSUMPTIONS: None.
EXPECTATIONS: The affected segments must be discarded and the connection may be reset.
REQUIREMENT LEVEL: Must
REFERENCES: RFC 793: section 3.1 RFC 1122: section 4.2.2.5 {Cope with illegal option length}
NOTE: Linux fails this test! Linux accepts the packet, and sends it up the protocol stack. The wrong layer responds.
12 May 2015
Testing Communications in the Connected Car
Here's an example of a robustness / security test:
Robustness testing is the process of subjecting a device under test to particular input streams in an attempt to cause the device to fail. The input streams may be one of three types:
‣ Random input streams
‣ Valid input streams, and
‣ Invalid input streams.
Our aim is a precise, controlled, repeatable input stream.
12 May 2015
Testing Communications in the Connected Car
Robustness/ Security Test (A variation on the “Ping of Death” test) …
Enough fragments are sent to create 65536 bytes of data in any destination buffer (one more than is allowed.) The MTU is 512 + header length. A final fragment of 8 bytes occurs at the end. Fragments are transmitted in order of increasing segment offset.
Expected Outcome:
The fragments must eventually be discarded before anything is delivered to upper protocol layers. The device under test must not crash!
This test is also an example of a Conformance/Compliance test, because the requirements are defined in the following specifications:
RFC 791, pages 8, 13, 24, 25
RFC 1122, section 3.3.2.
12 May 2015
Testing Communications in the Connected Car
Let's look at how this works in the lab ...
12 May 2015
Testing Communications in the Connected Car
12 May 2015
Testing Communications in the Connected Car
12 May 2015
Testing Communications in the Connected Car
6. Infotainment and Driver Assistance Testing
For Infotainment and Driver Assistance
how can we assure that
performance will be acceptable
under all conditions?
12 May 2015
Testing Communications in the Connected Car
Defining the performance metrics of mobile networks:
Mobile networks perform more poorly than their wired counterparts on the following four performance metrics:
1) Throughput
2) Round Trip Time (RTT)
3) Jitter
4) Packet Loss
Evaluate the performance of our app or device under worst case conditions ...
12 May 2015
Testing Communications in the Connected Car
See References on last page
12 May 2015
Testing Communications in the Connected Car
12 May 2015
Testing Communications in the Connected Car
✔ Why use a network emulator?
✔ I can do ad hoc testing with some 3G/4G phones!
Goal is precise, deterministic test results for all boundary conditions.
Accuracy. Repeatability.
12 May 2015
Testing Communications in the Connected Car
12 May 2015
Testing Communications in the Connected Car
Consider 4G/LTE worst case conditions...
● You must tune your app or device to perform at these limits
● What if you cannot accommodate the limits?
12 May 2015
Testing Communications in the Connected Car
Consider Apple's Siri...
● “I'm really sorry, Chris, but I cannot answer that right now.”
12 May 2015
Testing Communications in the Connected Car
Infotainment and driver assistance applications and a geosynchronous satellite
12 May 2015
Testing Communications in the Connected Car
The signal from the ground station to the geosync satellite travels about 22,000 miles up and then 22,000 miles back down.
Geosync satellite communication automatically introduces a network delay of about 0.24 seconds or 240 milliseconds.
Other variables include the link rate, min and max payload size, header overhead, queue length, and so on.
How will your app perform in your moving car when data is coming from the satellite?
12 May 2015
Testing Communications in the Connected Car
12 May 2015
Testing Communications in the Connected Car
Summary ...
1. Components that require testing
2. Industry standards used and implemented:
AUTOSAR, TCP/IP family, 3G, 4G/LTE
3. Types of testing: stress, reliability, functionality ...
4. Testing of ECU related modules
6. Performance testing of Infotainment and Driver Assistance
12 May 2015
Testing Communications in the Connected Car
References 1 of 2:A Close Examination of Performance and Power Characteristics of 4G LTE Networks, 2012,
http://www.cs.columbia.edu/~lierranli/coms6998-7Spring2014/papers/rrclte_mobisys2012.pdf
One standard deviation added to median from the document Characterizing 4G and 3G Networks: Supporting
Mobility with Multi-Path TCP, UMass Amherst Technical Report UM-CS-2012-022,
http://people.cs.umass.edu/~yungchih/publication/12_mtcp_4g_tech_report.pdf
http://iwl.com/white-papers/emulating-3g-4g-networks – “Emulating 3G and 4G Networks”
Explains how data from the above academic research is incorporated into emulations
http://iwl.com/white-papers/network-protocol-testing – “Network Protocol Testing Overview”
Categories of network testing
12 May 2015
Testing Communications in the Connected Car
References 2 of 2:
http://iwl.com/white-papers/network-impairments/causes-and-correlation – “Causes and Correlation of Network Impairments”Explains the cause of phenomenon like excessive fragmentation and its impact on the network
http://www.eetimes.com/document.asp?doc_id=1280616&page_number=1 – “Automotive Ethernet: Evolution in the Fast Lane”Overview of where the standards and technologies are going, with emphasis on EthernetAny questions or comments, send a note to [email protected]
NOTE: © Copyright InterWorking Labs, Inc. ALL RIGHTS RESERVED.Distribute freely among colleagues; contact us to reuse any material in other published works.
12 May 2015
Testing Communications in the Connected Car
References continued:
http://iwl.com/white-papers/network-impairments/causes-and-correlation – “Causes and Correlation of Network Impairments”Explains the cause of phenomenon like excessive fragmentation and its impact on the network http://www.eetimes.com/document.asp?doc_id=1280616&page_number=1 – “Automotive Ethernet: Evolution in the Fast Lane”Overview of where the standards and technologies are going, with emphasis on EthernetAny questions or comments, send a note to [email protected]
NOTE: © Copyright InterWorking Labs, Inc. ALL RIGHTS RESERVED.Distribute freely among colleagues; contact us to reuse any material in other published works.