Cost of highly safety critical software

It's always interesting to see data on industry software costs. I recently came across a report on software costs for the aviation industry. The context was flight-critical radio communications, but the safety standards discussed were DO-178B and DO-254, which apply to flight controls as well.

Here's the most interesting picture from the report for my purposes:

(Source: Page 28 https://www.eurocontrol.int/sites/default/files/content/documents/communications/29012009-certification-cost-estimation-for-fci-platform.pdf.pdf )

Translating from DO-178B terminology, this means:

• DAL A  (failure would be "catastrophic"):  3 - 12 SLOC/day
• DAL B  (failure would be "hazardous"): 8 - 20 SLOC/day
• DAL C (failure would be "major"): 15 - 40 SLOC/day
• DAL D (failure would be "minor"): 25 - 64 SLOC/day

Worth noting is that, in my experience, really solid mission critical but NOT life-critical embedded software can be done at up to 16 SLOC per day for well-run experienced teams, so it tends to line up with DAL B costs.

For interpretation, "DAL" expresses a criticality level (a "Development Assurance Level"), with more critical software requiring more rigorous processes.  The document has quite a lot to say about how the engineering process works, and is worth a read if you want to see how the aviation folks do business.  (I'm aware that DO-178C is out, but this paper talks about the older "B" version.)    Note that there are other cost models in the paper that are less pessimistic in that report, but this is the one that says "industry experience."

Have you found other cost of software data for embedded or mission critical systems?

Read More

Different types of risk analysis: ALARP, GAMAB, MEMS and more

When we talk about how much risk is enough, it is common to do things like compare the risk to current systems, or argue about whether something is more (or less) likely than events such as being killed by lightning. There are established ways to think about this topic, each with tradeoffs.

The next time you need to think about how much risk is appropriate in a safety-critical system, try these existing approaches on for size instead of making up something on your own:

ALARP: "As Low As Reasonably Practicable"  Some risks are acceptable. Some are unacceptable. Some are worth taking in exchange for benefit, but if that is done the risk must be reduced to be ALARP.

GAMAB: "Globalement Au Moins Aussi Bon"  Offer a level of risk at least as good as the risk offered by an equivalent existing system. (i.e., no more dangerous than what we have already for a similar function)

MEM: "Minimum Endogenous Mortality"  The technical system must not create a significant risk compared to globally existing risks. For example, this should cause a minimal increase in overall death rates compared to the existing population death rates.

MGS: "Mindestens Gleiche Sicherheit"   (At least the same level of safety) Deviations from accepted practices must be supported by an explicit safety argument showing at least the same level of safety. This is more about waivers than whole-system evaluation.

NMAU: "Nicht Mehr Als Unvermeidbar"  (Not more than unavoidable)  Assuming there is a public benefit to the operation of the system, hazards should be avoided by reasonable safety measures implemented with reasonable cost.

Each of these approaches has pros and cons.  The above terms were paraphrased from this nice discussion:
Kron, On the evaluation of risk acceptance principles,
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.455.4506&rep=rep1&type=pdf

There is an interesting set of slides that covers similar ground here, and works some examples. In particular the graphs involving whether risks are taken voluntarily for different scenarios is thought provoking:
http://agse3.informatik.uni-kl.de/teaching/suze/ws2014/material/folien/SRES_03_Risk_Acceptance.pdf

In general, if you want to dig deeper into this area, a search on
    gamab mem alarp 
will bring up a number of hits

Also note that legal and other types of considerations exist, especially regarding product liability.

Read More

Activity on my Safe Autonomy Blog

For those who might have missed it, most of my recent blogging has been on my Safe Autonomy Blog: https://safeautonomy.blogspot.com

Recent post topics include slides and paper preprints such as:

• Robustness Testing of Autonomy Software (ICSE 2018)
• Safety Validation and Edge Case Testing for Autonomous Vehicles 
• Heavy Tail Ceiling Problem for AV Testing
• Toward a framework for Highly Automated Vehicle Safety Validation

Read More

Embedded System Safety Lecture Video Series

I've posted the full series of embedded system safety lecture videos on YouTube.  These are full-length narrated slides of the core set of safety topics from my new course.  They concentrate on getting the big picture about failures, redundancy management, and safety integrity.

• Safety Overview     (slides)
• Dependability     (slides)
• Critical Systems     (slides)
• Safety Plan     (slides)
• Safety Requirements     (slides)
• Single Points of Failure     (slides)
• Redundancy Management     (slides)
• Critical System Isolation     (slides)
• Safety Architectural Patterns     (slides)

Each of the videos is posted to YouTube as a playlist, with each video covering one substantive slide. The full lecture consists of playing the entire play list, with most lectures being 5-7 videos in sequence. (The slide download has been updated for my Fall 2017 course, so in general has a little more material than the original video. They'll get synchronized eventually, but for now this is what I have.)

Obviously there is more to safety than just these topics. Supporting topics such as code quality and development approaches are currently just available as slides.  You can see the full set of course slides here:
  https://users.ece.cmu.edu/~koopman/lectures/index.html#642

Read More

Embedded Software Course Notes On-Line

I'm just wrapping up my first semester teaching a new course on embedded system software. It covers code quality, safety, and security. Below is table of lecture handouts.

NOTE: there is an update here:
     https://users.ece.cmu.edu/~koopman/lectures/index.html#642
which includes newer course notes and quite a few YouTube videos of these lectures.
You should use that URL instead of this blog post, but I've left this post as-is for Fall 2017.

18-642 Embedded System Software Engineering
Prof. Philip Koopman, Carnegie Mellon University, Fall 2017

	Slides	Topics
1	Course Introduction	Software is eating the world; embedded applications and markets; bad code is a problem; coding is 0% of software; truths and management misconceptions
2	Software Development Processes	Waterfall; swiss cheese model; lessons learned in software; V model; design vs. code; agile methods; agile for embedded
3	Global Variables	Global vs. static variables; avoiding and removing globals
4	Spaghetti Code	McCabe Cyclomatic Complexity (MCC); SCC; Spaghetti Factor (SF)
5	Unit Testing	Black box testing; white box testing; unit testing strategies; MCDC coverage; unit testing frameworks (cunit)
6	Modal Code/Statecharts	Statechart elements; statechart example; statechart implementation
7	Peer Reviews	Effective code quality practices, peer review efficiency and effectiveness; Fagan inspections; rules for peer review; review report; perspective-based reviews; review checklist; case study; economics of peer review
8	Code Style/Humans	Making code easy to read; good code hygiene; avoiding premature optimization; coding style
9	Code Style/Language	Pitfalls and problems with C; language use guidelines and analysis tools; using language wisely (strong typing); Mars Climate Orbiter; deviations & legacy code
10	Testing Quality	Smoke testing, exploratory testing; methodical test coverage; types of testing; testing philosophy; coverage; testing resources
11	Requirements	Ariane 5 flight 501; rules for good requirements; problematic requirements; extra-functional requirements; requirements approaches; ambiguity
12	System-Level Test	First bug story; effective test plans; testing won't find all bugs; F-22 Raptor date line bug; bug farms; risks of bad software
13	SW Architecture	High Level Design (HLD); boxes and arrows; sequence diagrams (SD); statechart to SD relationship; 2011 Health Plan chart
14	Integration Testing	Integration test approaches; tracing integration tests to SDs; network message testing; using SDs to generate unit tests
15	Traceability	Traceability across the V; examples; best practices
16	SQA isn't testing	SQA elements; audits; SQA as coaching staff; cost of defect fixes over project cycle
17	Lifecycle CM	A400M crash; version control; configuration management; long lifecycles
18	Maintenance	Bug fix cycle; bug prioritization; maintenance as a large cost driver; technical debt
19	Process Key Metrics	Tester to developer ratio; code productivity; peer review effectiveness
33	Date Time Management	Keeping time; time terminology; clock synchronization; time zones; DST; local time; sunrise/sunset; mobility and time; date line; GMT/UTC; leap years; leap seconds; time rollovers; Zune leap year bug; internationalization.
21	Floating Point Pitfalls	Floating point formats; special values; NaN and robots; roundoff errors; Patriot Missile mishap
23	Stack Overflow	Stack overflow mechanics; memory corruption; stack sentinels; static analysis; memory protection; avoid recursion
25	Race Conditions	Therac 25; race condition example; disabling interrupts; mutex; blocking time; priority inversion; priority inheritance; Mars Pathfinder
27	Data Integrity	Sources of faults; soft errors; Hamming distance; parity; mirroring; SECDED; checksum; CRC
20	Safety+Security Overview	Challenges of embedded code; it only takes one line of bad code; problems with large scale production; your products live or die by their software; considering the worst case; designing for safety; security matters; industrial controls as targets; designing for security; testing isn't enough Fiat Chrysler jeep hack; Ford Mytouch update; Toyota UA code quality; Heartbleed; Nest thermostats; Honda UA recall; Samsung keyboard bug; hospital infusion pumps; LIFX smart lightbulbs; German steel mill hack; Ukraine power hack; SCADA attack data; Shodan; traffic light control vulnerability; hydroelectric plant vulnerability; zero-day shopping list
22	Dependability	Dependability; availability; Windows 2000 server crash; reliability; serial and parallel reliability; example reliability calculation; other aspects of dependability
24	Critical Systems	Safety critical vs. mission critical; worst case and safety; HVAC malfunction hazard; Safety Integrity Levels (SIL); Bhopal; IEC 61508; fleet exposure
26	Safety Plan	Safety plan elements; functional safety approaches; hazards & risks; safety goals & safety requirements; FMEA; FTA; safety case (GSN)
28	Safety Requirements	Identifying safety-related requirements; safety envelope; Doer/Checker pattern
29	Single Points of Failure	Fault containment regions (FCR); Toyota UA single point failure; multi-channel pattern; monitor pattern; safety gate pattern; correlated & accumulated faults
30	SIL Isolation	Isolating different SILs, mixed-SIL interference sources; mitigating cross-SIL interference; isolation and security; CarShark hack
31	Redundancy Management	Bellingham WA gasoline pipeline mishap; redundancy for availability; redundancy for fault detection; Ariane 5 Flight 501; fail operational; triplex modular redundancy (TMR) 2-of-3 pattern; dual 2-of-2 pattern; high-SIL Doer/Checker pattern; diagnostic effectiveness and proof tests
32	Safety Architecture Patterns	Supplemental lecture with more detail on patterns: low SIL; self-diagnosis; partitioning; fail operational; voting; fail silent; dual 2-of-2; Ariane 5 Flight 501; fail silent patterns (low, high, mixed SIL); high availability mixed SIL pattern
34	Security Plan	Security plan elements; Target Attack; security requirements; threats; vulnerabilities; mitigation; validation
35	Cryptography	Confusion & diffusion; Caesar cipher; frequency analysis; Enigma; Lorenz & Colossus; DES; AES; public key cryptography; secure hashing; digital signatures; certificates; PKI; encrypting vs. signing for firmware update
36	Security Threats	Stuxnet; attack motivation; attacker threat levels; DirectTV piracy; operational environment; porous firewalls; Davis Besse incident; BlueSniper rifle; integrity; authentication; secrecy; privacy; LG Smart TV privacy; DoS/DDos; feature activation; St. Jude pacemaker recall
37	Security Vulnerabilities	Exploit vs. attack; Kettle spambot; weak passwords; master passwords; crypto key length; Mirai botnet attack; crypto mistakes; LIFX revisited; CarShark revisited; chip peels; hidden functionality; counterfeit systems; cloud connected devices; embedded-specific attacks
38	Security Mitigation Validation	Password strength; storing passwords & salt/pepper/key stretching; Adobe password hack; least privilege; Jeep firewall hack; secure update; secure boot; encryption vs. signing revisited; penetration testing; code analysis; other security approaches; rubber hose attack
39	Security Pitfalls	Konami code; security via obscurity; hotel lock USB hack; Kerckhoff's principle; hospital WPA setup hack; DECSS; Lodz tram attack; proper use of cryptography; zero day exploits; security snake oil; realities of in-system firewalls; aircraft infotainment and firewalls; zombie road sign hack

Note that in Spring 2018 these are likely to be updated, so if want to see the latest also check the main course page:  https://www.ece.cmu.edu/~ece642/   For other lectures and copyright notes, please see my general lecture notes & video page: https://users.ece.cmu.edu/~koopman/lectures/index.html

Read More

Highly Autonomous Vehicle Validation

Here are the slides from my TechAD talk today.

Highly Autonomous Vehicle Validation from Philip Koopman

Highly Autonomous Vehicle Validation: it's more than just road testing!
- Why a billion miles of testing might not be enough to ensure self-driving car safety.
- Why it's important to distinguish testing for requirements validation vs. testing for implementation validation.
- Why machine learning is the hard part of mapping autonomy validation to ISO 26262

Read More

Challenges and Solutions in Autonomous Vehicle Validation

Here are the slides from my AV17 Presentation on self-driving car safety:

Challenges and Solutions in Autonomous Vehicle Validation from Philip Koopman

Read More

SCAV 2017 Keynote: Challenges in Autonomous Vehicle Validation

Challenges in Autonomous Vehicle Testing and Validation from Philip Koopman

Challenges in Autonomous Vehicle Validation

Keynote Presentation Abstract

Philip Koopman

Carnegie Mellon University; Edge Case Research LLC

ECE Dept. HH A-308, 5000 Forbes Ave., Pittsburgh, PA, USA

koopman@cmu.edu

Developers of autonomous systems face distinct challenges in conforming to established methods of validating safety. It is well known that testing alone is insufficient to assure safety, because testing long enough to establish ultra-dependability is generally impractical. That’s why software safety standards emphasize high quality development processes. Testing then validates process execution rather than directly validating dependability.

Two significant challenges arise in applying traditional safety processes to autonomous vehicles. First, simply gathering a complete set of system requirements is difficult because of the sheer number of combinations of possible scenarios and faults. Second, autonomy systems commonly use machine learning (ML) in a way that makes the requirements and design of the system opaque. After training, usually we know what an ML component will do for an input it has seen, but generally not what it will do for at least some other inputs until we try them. Both of these issues make it difficult to trace requirements and designs to testing as is required for executing a safety validation process. In other words, we’re building systems that can’t be validated due to incomplete or even unknown requirements and designs.

Adaptation makes the problem even worse by making the system that must be validated a moving target. In the general case, it is impractical to validate all the possible adaptation states of an autonomy system using traditional safety design processes.

An approach that can help with the requirements, design, and adaptation problems is basing a safety argument not on correctness of the autonomy functionality itself, but rather on conformance to a set of safety envelopes. Each safety envelope describes a boundary within the operational state space of the autonomy system.

A system operating within a “safe” envelope knows that it’s safe and can operate with full autonomy. A system operating within an “unsafe” envelope knows that it’s unsafe, and must invoke a failsafe action. Multiple partial specifications can be used as an envelope set, with the intersection of safe envelopes permitting full autonomy, and the union of unsafe envelopes provoking validated, and potentially complex, failsafe responses.

Envelope mechanisms can be implemented using traditional software engineering techniques, reducing the problems with requirements, design, and adaptation that would otherwise impede safety validation. Rather than attempting to prove that autonomy will always work correctly (which is still a valuable goal to improve availability), the envelope approach measures the behavior of one or more autonomous components to determine if the result is safe. While this is not necessarily an easy thing to do, there is reason to believe that checking autonomy behaviors for safety is easier than implementing perfect, optimized autonomy actions. This envelope approach might be used to detect faults during development and to trigger failsafes in fleet vehicles.

Inevitably there will be tension between simplicity of the envelope definitions and permissiveness, with more permissive envelope definitions likely being more complex. Operating in the gap areas between “safe” and “unsafe” requires human supervision, because the autonomy system can’t be sure it is safe.

One way to look at the progression from partial to full autonomy is that, over time, systems can increase permissiveness by defining and growing “safe” envelopes, shrinking “unsafe” envelopes, and eliminating any gap areas.

ACM Reference format:
P. Koopman, 2017. Challenges in Autonomous Vehicle Validation. In
Proceedings of 1st International Workshop on Safe Control of Connected
and Autonomous Vehicles, Pittsburgh, Pennsylvania, USA, April 2017
(SCAV 2017), 1 page.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is  granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Copyright is held by the owner/author(s).
SCAV'17, April 21-21 2017, Pittsburgh, PA, USA
ACM 978-1-4503-4976-5/17/04.
http://dx.doi.org/10.1145/3055378.3055379

Read More

Challenges & solutions for Embedded Software Security, Safety & Quality (Full Tutorial Video)

This is a full-length video that talks about embedded software security, safety and quality: why it matters. What to do about it.


Embedded Software Quality Safety and Security [ECR]

The purpose of this video is to help you understand why safety and security are such a big deal for embedded systems, tell some war stories, and explain the general ways available to reduce risk when you're creating embedded and IoT products.

Topics covered include:

• Case studies of safety and security problems
• How to design for safety
• How to design for security
• Top 10 embedded software warning signs
• How to create high quality embedded software

(27 Slides / 45 minutes)

Slides Only: 

Embedded Software Security Safety & Quality from Philip Koopman

Read More

Safety Architectural Patterns (Preview)

Here's a summary video on Safety Architectural Patterns:


Safety Architecture Patterns Preview [ECR]

Other pointers on this topic (my blog posts unless otherwise noted):

• Monitor actuator pattern

For more about Edge Case Research and how to subscribe to our video training channel, please see this Blog posting.

Read More

Critical System Isolation (Preview)

Here's a summary video on Critical System Isolation:


Critical System Isolation Preview [ECR]

Other pointers on this topic (my blog posts unless otherwise noted):

• Self-monitoring doesn't mitigate single points of failure

For more about Edge Case Research and how to subscribe to our video training channel, please see this Blog posting.

Read More

Redundancy Management for System Safety (Preview)

Here's a summary video on Redundancy Management:


Redundancy Management for Critical Systems Preview [ECR]

Other pointers on this topic (my blog posts unless otherwise noted):

• No single points of failure

• Self monitoring can't mitigate a single point of failure

• Monitor Actuator pattern

For more about Edge Case Research and how to subscribe to our video training channel, please see this Blog posting.

Read More

Critical System Design (Preview)

Here's a summary video on Critical System Design techniques.


Critical Systems Preview [ECR]

Other pointers on this topic (my blog posts unless otherwise noted):

• Go beyond system functional testing to ensure safety
• How often will random faults kill someone if un-mitigated?
• Random hardware faults in general

Other pointers:

• Extensive list of techniques that can be used depending upon the SIL and safety standard that paplies to your system (IEC 61508-7)

For more about Edge Case Research and how to subscribe to our video training channel, please see this Blog posting.

Read More

Safety Requirements for Embedded Systems (Preview)

Here's a summary video on Embedded System Safety Requirements.


Safety Requirements Preview [ECR]

Other pointers on this topic (my blog posts unless otherwise noted):

• Don't require perfection
• Testing failsafe mechanisms
• Doer/Checker (monitor/actuator) design pattern

For more about Edge Case Research and how to subscribe to our video training channel, please see this Blog posting.

Read More

Embedded Software Safety Plan (Preview)

Here's a summary video on creating an embedded Software Safety Plan.   (See additional pointers below.)

Safety Plan Preview [ECR]

Other pointers on this topic (my blog posts unless otherwise noted):

• How a Risk Table works
• Go beyond just system testing to ensure safety
• Why creating a software safety plan is challenging for an autonomous vehicle
• Your embedded system might be safety critical

Other pointers:

• Medical device regulatory environment (Micrium blog)
• Software safety standards and guidebooks system-safety.org
• International safety standards via law.resource.org (e.g., look for IEC 61508 in their industrial safety standards) -- very useful for teaching access to actual international standards, although sometimes older versions

For more about Edge Case Research and how to subscribe to our video training channel, please see this Blog posting.

Read More

Autonomous Vehicle Safety: An Interdisciplinary Challenge

Autonomous Vehicle Safety: An Interdisciplinary Challenge

By Phil Koopman & Mike Wagner

Abstract:
Ensuring the safety of fully autonomous vehicles requires a multi-disciplinary approach across all the levels of functional hierarchy, from hardware fault tolerance, to resilient machine learning, to cooperating with humans driving conventional vehicles, to validating systems for operation in highly unstructured environments, to appropriate regulatory approaches. Significant open technical challenges include validating inductive learning in the face of novel environmental inputs and achieving the very high levels of dependability required for full-scale fleet deployment. However, the biggest challenge may be in creating an end-to-end design and deployment process that integrates the safety concerns of a myriad of technical specialties into a unified approach.

Read the preprint version here for free (link / .pdf)

Official IEEE version (subscription required):
http://ieeexplore.ieee.org/document/7823109/  
DOI: 10.1109/MITS.2016.2583491

IEEE Intelligent Transportation Systems Magazine (Volume: 9, Issue: 1, Spring 2017, pp. 90-96)

Correction:
"This would require a safety level of about 1 billion operating hours per catastrophic event. (FAA 1988)" should be
"This would require a safety level of about 1 billion operating hours per catastrophic event due to the failure of a particular function. (FAA 1988)"  (Note that in this context a "function" is something quite high level such as the ability to provide sufficient thrust from the set of jet engines mounted on the airframe.)

Read More

Embedded System Safety Overview (Preview)

Here's a summary overview video on Embedded System Safety.  (See additional pointers below.)


Embedded Software Safety Preview [ECR]

Other pointers on this topic (my blog posts unless otherwise noted):

• Safety culture
• Toyota Unintended Acceleration case study
• Layered defenses for safety critical systems
• Therac 25 summary

On-line resources:

• UK MOD, An Introduction to System Safety Management in the MOD, January 2011.

• RISKS Digest (archive of discussions on computing risks)
• Safety critical mailing list (international safety experts discuss almost everything; includes on-line archives)

John Knight's book: Fundamentals of Dependable Computing for Software Engineers (2012) is an excellent current book on software dependability and safety.

Nancy Leveson has some great publications in the area of software safety, and is credited for developing this as an academic field. Anyone doing software safety should read at least these:

• Software Safety: why, what, and how (a shorter, earlier version of the material in the Safeware book); pay to download from ACM, or sometimes you can find a free copy on-line if you don't have a subscription (1986)
• The Therac-25 Accidents (1993)
• High-Pressure Steam Engines and Computer Software (1992)
• Safeware (book on amazon; 1995) 
• Engineering a safer world (free on-line book) (2011)

For more about Edge Case Research and how to subscribe to our video training channel, please see this Blog posting.

Read More

A Driver Test For Self-Driving Cars Isn't Enough

I recently read yet another argument that a driving road test should be enough to certify an autonomous vehicle as safe for driving. In general, the idea was that if it's good enough to put a 16 year old on the road, it should be good enough for a self-driving vehicle.  I see this idea enough that it's worth explaining why it it's a really bad one.

Even if we were to assume that a self-driving vehicle is no different than a person (which is clearly NOT true), applying the driving test is only half the driver license formula. The other half is the part about being 16 years old. If a 12 year old is proficient at operating a vehicle, we still don't issue a drivers license. In addition to technical skills and book knowledge, we as a society have imposed a maturity requirement in most states of "being 16." It is typical that you don't get an unrestricted license until you're perhaps 18. And even then you're probably not a great driver at any age until you get some experience. But, we won't even let you on the road under adult supervision at 12!

The maturity requirement is essential.  As a driver we're expected to have the maturity to recognize when something isn't right, to avoid dangerous situations, to bring the vehicle to a safe state when something has gone wrong, to avoid operating when the vehicle system (vehicle + driver) is impaired, to improvise when something weird happens, and to compensate for other drivers who are having a bad day (or are simply suffering from only being 16). Autonomous driving systems might be able to do some of that, or even most of it in the near term. (I would argue that they are especially bad at self-detecting when they don't know what's going on.) But the point is a normal driving test doesn't come close to demonstrating "maturity" if we could even define that term in a rigorous, testable way. It's not supposed to -- that's why licenses require both testing and "being 16."

To be sure, human age is not a perfect correlation to maturity. But as a society we've come to the situation in which this system is working well enough that we're not changing it except for some tweaks every few years for very young and very old drivers who have historically higher mishap rates. But the big point is if a 12 year old demonstrates they are a whiz at vehicle operation and traffic rules, they still don't get a license.  In fact, they don't even get permission to operate on a public road with adult supervision (i.e., no learners permit at 12 in any US state that I know of.)  So why does it make sense to use a human driving test analogy to give a driver license, or even a learner permit, to an autonomous vehicle that was designed in the last few months?  Where's the maturity?

Autonomy advocates argue that encapsulating skill and fleet-wide learning from diverse situations could help cut down the per-driver learning curve. And it could reduce the frequency of poor choices such as impaired driving and distracted driving. If properly implemented, this could all work well and could improve driving safety -- especially for drivers who are prone to impaired and distracted driving. But while it's plausible to argue that autonomous vehicles won't make stupid choices about driving impaired, that is not at all the same thing as saying that they will be mature drivers who can handle unexpected situations and in general display good judgment comparable to a typical, non-impaired human driver. Much of safe driving is not about technical skill, but rather amounts to driving judgment maturity. In other words, saying that autonomous vehicles won't make stupid mistakes does not automatically make them better than human drivers.  

I'd want to at least see an argument of technological maturity as a gate before even getting to a driving skills test. In other words, I want an argument that the car is the equivalent of "being 16" before we even issue the learner permit, let alone the driver license. Suggesting that a driving test is all it takes to put a vehicle on the road means: building a mind-boggling complex software system with technology we really don't understand how to validate, doing an abbreviated acceptance test of basic skills (a few minutes on the road), and then deciding it's fine to put in charge of several thousand pounds of metal, glass, and a human cargo as it hurtles down the highway. (Not to mention the innocent bystanders it puts at risk.)

This is a bad idea for any software.  We know from history that a functional acceptance test doesn't prove something is safe (at most it can prove it is unsafe if a mishap occurs during the test).  Not crashing during a driver exam is to be sure an impressive technical achievement, but on its own it's not the same as being safe!  Simple acceptance testing as the gatekeeper for autonomous vehicles is an even worse idea. For other types of software we have found that in practice that you can't understand software quality for life-critical systems without also considering the rigor of the engineering process. Think of good engineering process as a proxy for "being 16 years old." It's the same for self-driving cars. 

(BTW, "life-critical" doesn't mean perfect. It means designed with sufficient engineering rigor to be suitable for its intended criticality. See ISO 26262 or your favorite software safety standard, which is currently NOT required by the government for autonomous vehicles, but should be.)

It should be noted that some think that a few hundred million miles of testing can be a substitute for documenting engineering rigor. That's a different discussion. What this essay is about is saying that a road test -- even an hour long grueling road test -- does not fulfill the operational requirement of "being 16" for issuing a driving license under our current system. I'd prefer a different, more engineering-based method of certifying self-driving vehicles for use on public roads. But if you really want there to be a driver test, please tell me how you plan to argue the part about demonstrating the vehicle, the design team, or some aspect of the development project has the equivalent 16-year-old maturity. If you can't, you're just waving your hands about vehicle safety.

Read More

Better Reboot Your Boeing 787 Every Three Weeks

Crashing after a prolonged up time due to a counter rollover or other problem is a classic mistake in computer software.  And, it just bit the Boeing 787. Again.

The Problem:

The Boeing 787 aircraft has three Flight Control Modules (FCMs) that are the subject of a new FAA Airworthiness Directive.  Based on that sentence alone, you want to make sure whatever that involves gets fixed before you fly on a 787!

The FAA says there is "a report" that all three FCMs can fail at the same time 22 days after they have been rebooted.  If you don't reboot the FCMs the FAA says this "could result in flight control surfaces not moving in response to flight crew inputs for a short time and consequent temporary loss of controllability."  This is FAA-speak for the airplane could crash. The're telling airlines to reboot the plane every 21 days to avoid this. Hope nobody forgets to do that! (In fairness, I understand it is likely that most planes get rebooted more often than this anyway. But this is not something you want to leave to chance.)

At this point we can only guess at the cause, but the usual guess is that it is a timer overflow problem. Let's hypothesize a 32-bit signed integer is counting the passing of time in milliseconds.  So a value of 32700 in that counter is 32.700 seconds.

How long until it overflows 31 bits of counting into the 32nd bit, which is the sign bit?

0x7FFFFFFF = 2147483647 ==> 2147483.647 seconds
2147483.647 seconds * (1 min/60 sec) (1 hr/60 min)(1 day/24 hr) = 24.9 days

Hmm, a bit longer than the 22 days the FAA reports.  Some time spent playing with various multipliers didn't seem to give a likely candidate.  Possible factors if it is a timer rollover would include fixed point math (e.g., time keeping in 256ths of a second) or scaling from a 400 Hz aircraft AC frequency. Or there could be some divided-down crystal oscillator frequency on the FCM that is involved.

Or, it could be something completely different.  Maybe there is memory that records operating parameters periodically and the system crashes when that fills up that memory (for example, logs that get downloaded every maintenance interval, with an expectation that the maintenance interval is more like a few days than a few weeks).

For now the cause is a bit of a mystery to us.  I'll bet the FCM engineers have a pretty good idea at this point. No doubt they'll issue a fix as fast as they can get the FAA to review it.

But the big news is that for the second time, the FAA is telling is telling the airlines they have to do a maintenance reboot of their planes.  Last time it was every 248 days. This time it's every 21 days.

It's bad enough that they have to reboot the infotainment systems once in a while.  For flight controls, this is not good news. This is the kind of problem that should be caught in design reviews.  Always think about what happens if any counter, timer, or data structure overflows.

Other Examples:

This is not the first time a problem with long-running software has happened beyond the usual memory leaks in everyday applications.  Some examples are:

Timer rollover bugs:

• B787 needs to be rebooted every 248 days due to a likely timer overflow bug [Blog][NY Times] [FAA]
• Air Traffic control loses contact with 400 aircraft due to a 32-bit time rollover in 2004 [IEEE Spectrum]
• IBM: Interface adapters hang after 497 days of uptime [IBM]
• Windows 95: hang after 49.7 days without reboot, counting in milliseconds [Microsoft]  (I met the engineers who found that one. And congratulated them on the significant feat of actually getting Windows 95 to run that long without crashing for some other reason!)

There are also plenty of date roll-over bugs:

• NASA Deep Impact Comet Mission terminated unexpectedly when at 2**32 seconds after Jan 1, 2000 (a time rollover bug). [IEEE Software]
• Y2K: on 1 January 2000 (overflow of 2-digit year from 99 to 00)   [Wikipedia]
• GPS: 1024 week rollover on 22 August 1999 [USCG]
• Year 2038: Unix time will roll over on 19 January 2038 [Wikipedia]

There are also somewhat related capacity overflow issues such

• 911 outage caused by database record limit [Washington Post]
• 512K day for IPv4 routers
• Lightsail spacecraft dies when log exceeds 32 MB [Space.com] 

And floating-point roundoff issues (thanks to Dan for reminding me of this one):

• Patriot Missile mishap after operating for 100 hours without a maintenance reboot [GAO]

If you want to dig further, there is a "zoo" of related problems on Wikipedia:  "Time formatting and storage bugs"

Read More

Single Point of Failure Overview (Preview)

Here's a summary video on Single Points of Failure for safety- and mission-critical systems.

Single Point of Failure Preview [ECR] 

Other pointers on this topic (my blog posts):

• In-depth discussion of single points of failure
• Why self-monitoring doesn't mitigate single points of failure
• Example probability for why this matters so much
• How redundant input processing should work
• Doer/checker (monitor/actuator) pattern

For more about Edge Case Research and how to subscribe to our video training channel, please see this Blog posting.

Read More