Tesla’s “Shadow” Testing Offers A Useful Advantage On The Biggest Problem In Robocars
By Brad Templeton
Apr 29 2019
For some time, the biggest problem in robocars has been proving that you have done it — that you have built a car that is safe enough to deploy. If you’re making one, you must first convince yourself, along with your lawyers and board of directors, and you must convince the public. When you have accidents, you will need to convince the courts. In time, you will need to convince regulators too.
In light of Tesla’s recent opening up on their approaches, it is worth examining their approach, in contrast with that of others.
Everybody starts on closed roads and test tracks, but very quickly that becomes minimally useful and you need to try the more varied and risky situations of public streets.
Almost everybody agrees that on-road testing and development is essential. Nobody is going to ride in a car that’s not been extensively tested in the real world. The primary approach taken by most teams has been to rack up lots of real world miles and situations in different conditions, with a human safety driver, or rather two, overseeing the prototype testing to keep it safe. Waymo has now done 15 million miles of this.
Almost everybody serious also is testing in simulator. Simulation is not the real thing, but it offers some important advantages:
• It’s cheap — no need to even have a car, let alone a road or safety drivers.
• It’s fast — you can drive a million miles in simulator in a night.
• It’s safe — you can test dangerous situations you could never try in the real world, like pedestrians running in front of you or being in the middle of accidents
• It’s complex — you can drive all your simulated miles in interesting and challenging conditions, not wasting time on simple and boring ordinary roads.
• It’s repeatable — you can do the same test again and again.
• It’s variable — you can try millions of tiny variations of any situation. You can try every type of weather and lighting. You can try every type of road and traffic.
• It’s strange — you can test extremely rare situations, like what your car might do in a tsunami, earthquake, fire, flood, tornado or dust storm.
• It’s software — you can create complex failures of your simulated hardware, mixed in with every situation, and know how you deal with them.
It’s a very important technique, and listing all of its abilities would take a much longer document, but everybody should be doing it. Even so, it is not the real world.
Tesla showed off one of its testing techniques at its April 23 autonomy day, one it had hinted at before but not clearly described, namely “shadow mode testing.” This technique is one where Tesla has an edge over other players because hundreds of thousands of customers are paying to drive their cars around and unknowingly help with this testing.
In shadow testing, a car is being driven by a human, or a human with autopilot. A new revision of the autopilot software is also present on the vehicle, receiving data from the sensors but not taking control of the car in any way. Rather, it makes decisions about how to drive based on the sensors, and those decisions can be compared to the decisions of a human driver or the older version of the autopilot. If there is a decision — the new software decides to zig where the old one zags, or the new software cruises on when the human hits the brakes, an attempt can be made to figure out how different the decisions were, and how important that difference is. Some portion of those incidents can be given to human beings to examine and learn if the new software is making a mistake. If there is a mistake, it can be marked to be fixed, and the testing continues.
Tesla’s new hardware chip, which has vastly more processing power than the current generation they use, will allow them not only to run more complex neural networks in the autopilot. It also allows them to do more shadow testing, since it’s easy to have more than one autopilot running at a time.
Tesla touts their advantage here — they can shadow test many millions of miles every day, while companies using safety drivers test at a much slower rate. Before they release an update to autopilot, they can be almost sure it would not have caused a crash in millions of miles of driving situation. I say almost sure because shadow testing has one flaw. Once the shadow autopilot had decided to deviate from the real path of the car, both for important reasons like braking to avoid an accident, and more mundane reasons like changing lanes or adjusting itself in the lane, it no longer receives sensor data that reflect that change. It wants to change lanes, but the test is now over, as it can’t see what the sensors would have seen if it had done so, and other cars on the road who might react to that lane change will not be reacting to it. In important situations, the scenario can be converted over to a simulator which can test what would happen — in simulator, not reality — if the car actually obeyed the new software.