June 17, 2020 Sam Siewert

CEC450Real-Time Systems

RT System Design Project - Methods and Resources

The Standard Project

RT Standard ProjectHarder than it first appearsProblem is multiple clock domains– Similar to a software PLL (Phase

Lock Loop) - phase difference is a time difference

– Transition time of indicator(s) also a challenge

– Clock drift, jitter a challenge– You can get lucky, but reliability a

challengeFast digital clocks not allowed for 1 HzDigital clock must be used for 10 HzOpenCV of V4L2? Sam Siewert 2

8-powerful-quotes-from-yoda-the-og-wellness-guru/

Key Starter code:

1) Diff Interactive2) Simple-capture-18003) Sequencer generic

Demonstration of Clock Observer (Video)Sync to External Clock on TickSync only on a major period (e.g. every 1 minute, 1 hour)Or, sync once, but this is unrealistic, although good enough for class projectTake N frames after tick detectionAlways take N+fps after thatUse sequencer or write your ownOver-sample frames (10 Hz or better)Use “eom” to previewUse ffmpeg to encodeWatch out for bugs!

Sam Siewert 3

Simple Tick Detection Method to Lock Phase

Eye of Mate Method of Verification on Linux

Some BackgroundSynchronizing clocks important to industryE.g. Shortt SynchronomePhase-Locked Loops (USO’s)– Integrated circuits with quartz oscillators

(1970’s)– Can generate clock multiples (e.g. as we

did in CEC322)– Clock dividers (also used in CEC322)– Clock distribution on chip, on a board

UTC (broadcast radio, television)GPSNISTTAI

Sam Siewert 4

Synchronome_clockA clock that synchronizes

a clock with DSP - sound familiar? 3824-chip-scale-atomic-clock-csac

Chip ScaleAtomic Clocks

1922-56200 microsec/day

accuracy

V4L2 vs. OpenCVOpenCV– Advantages

1. cvPutText (nice way to mark frames)2. imshow (nice way to interactively debug)3. imwrite (nice way to save frames in any format)

– Disadvantages1. Requires C++ compile (C can be simpler) or deprecated C API2. OpenCV 2.x vs. 3.x vs. 4.x (portability issues)3. Can take a while to learn and master4. C++ not universally advised for RT systems (constructors, destructors, dynamic

binding, mangled namespace, etc. makes it harder verify)

V4L2– Advantages

1. Simple C API and Linux system calls2. Lower-level (efficient), but higher level than custom driver (UVC/USB)3. OpenCV uses it anyway, so less overhead

– Disadvantages1. Lower-level, so can be harder to understand2. No interactive debug features (imshow)3. Must write PPM/PGM with custom dump

Sam Siewert 5

RT Design – Software and HardwareSA/SD (Structured Analysis and Design) is fine for C programming– Block Diagrams (showing Hardware and Software Elements)– State Machines– Flowcharts (Control Flow)– Data Flow and Control Flow Diagrams– Entity Relation Diagram– Module Decomposition

Simple Design models that focus on RM are sufficientFocus on RMA with SA

Sam Siewert 6

RT Design - OOA/OOD/OOP - ReferencesCaspersen, Michael E. "A C++ framework for active objects in embedded real-time systems-bridging the gap between modeling and implementation." tools. IEEE, 1999.

Caal, G., and A. Divin. "Implementing real-time applications with concurrent objects." Real-Time Systems, 1994. Proceedings., Sixth Euromicro Workshop on. IEEE, 1994.

Pereira, Carlos. "Real time active objects in C++/real-time UNIX." Proc. of ACM SIGPLAN Workshop on Languages, Compiler, and Tool Support for Real-Time Systems. 1994.

Arjomandi, Eshrat, Ivan Kalas, and William O'Farrell. "Concurrency abstractions in a C++ class library." Proceedings of the 1993 conference of the Centre for Advanced Studies on Collaborative research: distributed computing-Volume 2. IBM Press, 1993.

Harbour, Michael González. "Programming real-time systems with C/C++ and POSIX." (2006): 1-9.

Sam Siewert 7

Learning Objectives – RT Design EffortResearch RT System Application Domain

– Define Goals and Objectives – e.g. real-time tracking of faces of interest in a crowd– Research how others have implemented similar real-time designs

RT Systems Analysis– Requirements– C, T, D parameters for each service S– Simulation and Worst-Case analysis with Cheddar Tool– Overall resource analysis of SW/HW (CPU, memory, I/O, power)

RT System Design– System level block diagram

Key hardware elements (Linux MCU, Camera, GPIO Sensor, Stepper Motor, etc.)Key software reuse (OpenCV, Linux OS with POSIX RT Extensions, etc.)

– System level service diagramService hardware interfaces (drivers), shared memory buffers and queues, synchronization (semaphores)

– Services decomposed into state machines

RT System Verification and Validation– Simple timestamp logging (syslog or file)– Kernelshark with Ftrace – Compare to Expectation for Cheddar Simulations– Sysprof

Presentation of Results (Report and Final Oral Exam) Sam Siewert 8

Notes on Research for SystemsRT Systems are Often Integrated in Specific Domains (Requires Research of the application and possibly citations to explain)

– Any topic your group chooses – Alternative to Standard Project (approved by instructor)– Topic ideas to research might include (but are not limited to):

Aviation, AerospaceMedical SystemsEnergy Systems (Process Automation)Transportation, ShippingRT Security, Remote Sensing, DefenseRT Information Storage and Retrieval, Enterprise SystemsRT Decision Support Systems (Business Intelligence)

Research can Start with Simple Google Search or Wikipedia, but should follow pointers to peer-reviewed resources [citations and references]

– IEEE Xplore, ACM Digital Library– Google Scholar, Microsoft Academic Search– Research Gate [research social network]– ERAU Library Resources– Industry Specifications, Journals, Developer Papers, White Papers [IBM, Intel, NVIDIA,

Texas Instruments, …]– Non-profit research institutes and agencies [CERT, SEI, NDIA, FAA, FIPS, CSRC,…]– Who are you referencing and citing and why?

Sam Siewert 9

Standards, Guidelines, PracticesAs Applicable to your RT System

– Key RT Domain Specific StandardsE.g. FAA DO-178B, FAA DO-178C and ARINC 653 for AvionicsE.g. for Intelligent transportation, IEEE ITSSEnterprise Data Systems – Commercial Standards and IV&V Labs (E.g. Medusa Labs Test Tools), Standards for Protocol (SCSI T10 and SAS, Fiber Channel T11, FCoE, iSCSI) and Performance Test Standards (TPC, SPC), SNIA RAIDAerospace – NASA Standards (https://standards.nasa.gov/documents/nasa ), Military Standards (MIL-STD-498), FAA Launch and Reentry, FAAMedical Systems – IEC 62304, E.g. Commercial SupportEnergy – NIST Testing, DoE STD-1172Transportation – Software Assurance, V&V, NHTSA Policies on Automated Vehicles, FAA Publications

– General Software Engineering Book of Knowledge – IEEE SWEBOK– SEI Software Standards – Cyberphysical Systems

Citations for text and notes are OK, but I encourage external such as Tools (Cheddar), Standards, and Domain Specific

Citations from academic (peer reviewed) papers are fantastic (similar to papers we read such as Liu and Layland) – Use library or google.com/scholar to find by topic

1. Liu, Chung Laung, and James W. Layland. "Scheduling algorithms for multiprogramming in a hard-real-time environment." Journal of the ACM (JACM) 20.1 (1973): 46-61.

2. Sha, Lui, Ragunathan Rajkumar, and John P. Lehoczky. "Priority inheritance protocols: An approach to real-time synchronization." IEEE Transactions on computers 39.9 (1990): 1175-1185.

Sam Siewert 10

Traditional Mission Critical SystemsAerospace and Aviation – DoD and NASA => HRT, HA/HR cross-strapping, Radiation tolerant/hardened, MIL-STD/NASA packaging

Medical Systems – FDA, US Dept. of Health => HRT/SRT, HR, FDA packaging

Energy Systems – DoE => HRT, HA/HR

Security Systems (Physcial and Cyber) – NIST => SRT, HA, Secure Data Centers

Transportation – FAA, DOT => HRT, HA/HR, ASEE

Defense – DoD (Varies by customer), Defense Acquisition Standards

Sam Siewert 11

Infamous System Software DefectsInfamous & Famous Field Defects

1. Boeing MCAS - Boeing, FAA, Press2. Toyota ABS Software Recall, 20103. Windows Genuine Advantage Outage, 20064. Mars Climate Orbiter Units Interoperability,

Loss, 19995. Mars Pathfinder Priority Inversion, Recovered,

19976. Ariane-5 501 Cluster Launch Software Failure,

19967. Pentium FPU Bug – Software Acceleration Co-

Processor Recall, 19948. ATT 4ESS Upgrade, Outage, 19909. Therac-25 Radiation Therapy Patient

Overdosing, 1985-8610. NORAD False Alarms, User Error for Test

Mode, Device Failures, 1979/198011. Apollo 11 Guidance Computer Overload, 1969 -

Recovered

Sam Siewert 12

http://en.wikipedia.org/wiki/Software_bug

RCA, Patch, Test, Fix!

http://history.nasa.gov/SP-350/ch-11-4.html

Rear Admiral Grace Hopper

DiscussionDo SEI CMM Quality Ratings or ISO 9000 Guarantee Results? Improve Probability of Success?

Are Standards and Guidelines Sufficient?

What Can be Done to Improve?

What Analysis are you or your Team Planning?

What are individual Team Member Roles / Contributions?

Coding Practices– Coding Standards– Linux Programming Top Errors and RT programming idioms (Service)– Design Principles – E.g. Cohesion and Coupling, Static Metrics (Cyclomatic

Complexity, McCabe, Fan-in, Fan-out)– Papers on SE Metrics – What’s up with software metrics?

Sam Siewert 13