Autonomous Vehicles case study

Promise vs Reality

Self-driving companies publish what their vehicles are designed to do. They also file crash reports with NHTSA. This checks whether the second ever contradicts the first & finds that for almost every company, the question cannot be asked at all.

What could be checked

crashes were reported to NHTSA by companies. Checking one against a promise requires the company to have published a promise specific enough to check.

Ask the audit agent

Answers come from the same tables shown below. Verdicts are looked up, never inferred; the narrative around them is retrieved from crash reports and safety documents, and every claim carries a citation you can check.

What the audit found

Not assessed

These are not clean records. A company with no extractable promise was never checked, and reporting that as zero gaps would turn an unknown into a pass.

Every company

Sorted by crashes reported, showing the companies that could actually be audited. Switch to all companies for the other 34 and the last column, which says why each one could not be checked.

company crashes promise auditable why not

How it works

Two sources, brought to a common vocabulary. NHTSA's crash reports give an observed condition per incident: road type, weather, lighting, speed. The companies' own safety self-assessments give a stated boundary, extracted with a language model that must quote the sentence it relied on. A gap is a crash whose observed condition falls outside the stated boundary.

The extraction is checked against 75 human labels. On those, it invents no promises and misses twelve i.e. it errs toward silence, which is the safe direction for an audit that names companies. Full method in the walkthrough.

What this is not

It is not a safety evaluation and cannot be read as one. Crash counts without miles driven are not crash rates, and a company with no gaps may simply have promised nothing. The measure here is consistency between what was said and what was reported -- nothing more.