The Gap Between Simulation and Real-world Mileage Data

AVs are poised to revolutionize transportation, promising safer roads and more efficient travel. At the heart of this transformation lies data—specifically, mileage data. While real-world testing is essential, it is simulation data that should hold the key to unlocking the industry's full potential at scale. The ability to simulate billions of miles in diverse and challenging scenarios is unmatched in the real world. Yet, despite the incredible strides in simulation, simulation data is not translating into real-world results as efficiently as hoped and it is holding back the industry.

The Promise of Simulation

On paper, simulation is a dream for autonomous vehicle development. It allows companies to test their systems across countless scenarios, including the most difficult and dangerous driving conditions—without the risks, logistics, or costs of on-road testing. Simulation gives AVs access to rare edge cases—situations that might only occur once in hundreds of thousands of miles in real life. Simulating these difficult situations helps AV developers refine how vehicles react to unpredictable drivers, bad weather, or other unusual conditions. In contrast, most on-road driving consists of uneventful, straight-line commuting, which adds limited value when training AVs to handle complexity.

Why Isn't Simulation Enough?

Despite its potential, current simulation efforts are falling short of delivering the desired results. While companies like Waymo boast impressive numbers of simulated miles (20 billion simulated miles to date), the real-world performance of AVs remains a significant hurdle. The effectiveness of simulation in producing real-world performance is proving to be lower than expected. Current simulations have limitations in replicating the nuances of the real world. Real-world driving conditions are constantly changing in ways that are hard to predict or replicate in a digital environment. Variables like road surface conditions, obscure human behaviors, and intricate traffic patterns are difficult to simulate at high fidelity. The gap between virtual testing and the unpredictable, uncontrolled environment of real roads is proving difficult to bridge.

Real-world miles are much more valuable than simulation miles in terms of bringing an AV product to commercialization. However, the operational, safety, and financial challenges limits the ability for real-world miles to scale to the magnitudes needed. AV companies must achieve safety at scale to gain public trust and regulatory approval. Waymo’s 20 million rider-only miles pale in comparison to the 3.2 trillion miles driven annually by Americans. While simulation can accelerate testing, it cannot yet replicate the conditions that come with real-world experience.

A Simulation-First Future?

So, how does the industry get simulation efforts to better reflect real-world driving and make AVs a viable commercial solution?

• Better Simulations: The AV industry needs more sophisticated simulation environments that replicate the subtleties of human behavior and environmental conditions in greater detail. This involves advancing how AI models generate and interact with simulated worlds, as well as the industry collectively developing the edge cases that are fed into simulations.
• Integration of Real-World Data: There’s an ongoing need for real-world driving data to calibrate and validate the simulation efforts. By integrating more supervised and driverless miles, AV companies can use real-world data to improve the accuracy and realism of their simulations. These real-world results can then be re-introduced into the simulation environment to sharpen the vehicle’s performance further.
• Regulatory and Industry Collaboration: Industry standards for simulation testing, validation, and deployment need to evolve to include better guidelines for how simulations can truly represent real-life driving. As AV companies collaborate with regulators, it will become clearer how simulations can be enhanced to meet the necessary safety and performance benchmarks for commercial deployment.

Conclusion

While simulation is a powerful tool that allows AV companies to rack up massive amounts of data, it is not yet a substitute for real-world driving. Until simulation data better translates into real-world results, the AV industry’s growth will remain slower and more expensive than expected. Simulation is still the key to unlocking the true potential of autonomous vehicles, but those virtual miles need to better reflect the unpredictable, complex, and ever-changing realities of the road.

• Musings