background
there are two very different teams in AD. one kind is people from top Tier1 and popular OEMs, the benefit of these guys is they are very matured at the product process management, e.g. V&V and the corresponding R&D process, the other pons of these guys, they have lots of engineering know-how/experience to make things work. mostly we know how the system/tool work, but compared to build the sytem/tool to work, the first knowledge is about 10%.
the mature R&D process is, as I see, is very valuable in a long time, which we’d say engineers getting richer as they getting older. till now, German and Janpanese top companies still have strong atmosphere to respect engineers, which keeps their engineers and their industry process management growing more and more mature. that’s a very good starting point for fresh young men, if they can join these top companies in early time.
the other team is from Internet companies, they are the kind of people with philosophy: as you can image out, I can build it up. while the philosophy always become true in IT and Internet service companies, and looks promising in industry fields, as the IT infra, like nowadays, service to cloud, office work in cloud, which derives lots of cloud infra, but the core like robot operation, making vehicle driving automatically, or computer aided engineering e.t.c. requires lots of know-how, far beyond the infra.
Internet teams are popular in China and US, but US still have strong engieer atmosphere like Germany, which is not in China. basically I want to say, there is no apprentice process to train fresh men to matured engineers in either companies or training organizations in China, which makes engineers here show low credit.
even Internet knowledge and skills in China, often we’d say programming work are youth meals, as programmer getting older they getting poor
that’s the tough situation for many young people, if the men got neither good engineering training, nor got smart and young enough to coding, he get doomed in his career. but that’s not a problem for the country neither for the companies, still Chinese workers are cheap, there are always lots of young people need bread beyond a promising career.
physical or virtual
the first-principal of writing a program:
- make it run ok
- make it run correct
- make it run performance-well
- make it extensible
the following is to understand why we need HiL, mil, sil, to ViL during verification and validation of an ADS product, from the physical or virtual viewpoint.
physical or virtual world
case 1) from physical world to physical sensor
namely, physical road test, either in closed-field, or open roads.
this is the case when we need ground truth(G.T.) system to validate the device under test(DUT) sensor, to evaluate or test the DUT’s performance boundary, failure mode e.t.c.
the people can come from sensor evaluation teams, or system integration teams.
case 2) from virtual world to physical sensor
the virtual world come either from digital twins, or replay from a data collector. this is hardware in loop(HiL) process, which used a lot in validate either sensor, or MCUs. when using HiL to validate MCUs, the virtual world is mimic signals e.t.c
virtual sensor
case 3) from virtual world to virtual sensor
virtual sensor has three kinds basically:
ideal (with noise) sensor model
statisticsly satisfied sensor model
physical sensor model
of course, the costing is increasing from ideal to physical sensor model.
ideally, the downside fusion module should be no senstive to sensors, either phyiscal sensors or virtual sensors(sensor models). the benefits of capturing virtual world by virtual sensor is so cheap, which makes it very good to training perception algorithms, when physical sensor data is expensive, which is often the reality for most companies now.
there is local maximum issues with virtual sensor data to train AIs, so in reality, most team used to mix 10% real-world sensor data to improve or jump out from these local maximums.
of course, virtual sensor is one fundamental element to close loop from virtual world to vehicle moving, in a sim env.
physical or virutal perception to fusion
here is how to test fusion, does it work correctly, performance well, how to handle abnormal(failure) case.
case 4) from physical sensor perception data to fusion
physical perception data comes in two ways:
- the sensor system in vehicle
- ground truth system(G.T.)
during RD stage, the sensor system in vehicle is treated as device under test(DUT), whose result can compare with the labeling outputs from G.T. system, which help to validate DUT performance, as well as to evaluate fusion performance.
in phyiscal world, ground truth sytem is usually equipped with a few higher precision Lidars, of course the cost is more expensive.
another pro of physical perception data is extract edge cases, which used to enrich the scenario libs, especially for sensor perception, and fusion.
during massive production stage, when the data pipeline from data collector in each running vehicle to cloud data platform is closed-loop, which means the back-end system can extract and aggregate highly volume edge case easily, then these large mount of physical perception data can drive fusion logsim.
question here, edge case scenarios are these abnormal perception/fusion cases, but how to detect edge cases from highly volume data in cloud(back-end), or to design the trigger mechanism in vehicle(front end) to only find out the edge cases, is not an matured work for most teams.
another question here, to evaluate fusion, requires objective ground truth, which is not avaiable in massive production stage. an alternative option is using a better performance and stable sensor as semi ground truth, such as Mobileye sensor.
case 5) from virtual sensor perception data to fusion
when in sim world, it’s easily to create a ground truth sensor, then it’s easily to check fusion output with the g.t. sensor, which is great, the only assuming here, is the sim world is highly vivid of the physical world. if not, the ground truth sensor is not useful, while obviously to build a digital twin as phyiscal as possible is not easy than create the ADS system.
on the other hand, if the sim world is not used to evaluate fusion, for example, used to generate synthetic images, point cloud to train perception AI modules, which is one benefit.
in summary, when evaluate and validate fusion, it requires ground truth labelling, either from physical g.t. system, or virtual g.t. sensor. 1) the g.t. system is only used for during R&D stage, with a small volumn of g.t. data; for massive release stage, there is no good g.t. source yet; 2) g.t. sensor in virtual world is easy to create, but requires the virtual worls is almost physical level.
second opnion, fusion evaluation is deterministic and objective. so if the fusion can validated well during RD stage, considering its performance robost, stable, there is no need to validate fusion module in massive lease. when perception/fusion edge case happens, we can study them case by case.
third opinon, for anormal case, e.g. sensor failure, sensor occupied cases, also need validate during RD.
planning
the evaluation of planning good or not is very subjective, while in fusion, ground truth is the criteria. so there are two sols:
- to make subjective goals as much quanlity as possible
- define RSS criterias to bound the planning results
ideally, planning should be not sensitive to fusion output from physical world, or virtual world. and when come to planning verification, we asumme the fusion component is stable and verified-well, namely should not delay find fusion bugs till planning stage.
case6) from sim fusion output to planning
SiL is a good way to verify planning model. previously, planning engineers create test scenarios in vtd, prescan, and check does the planning algorithms work.
for a matured team, there are maybe hundreds and thousands of planning related scenarios, the should-be verification process requires to regress on all these scenarios, so to automatically and accelerate this verification loop, gives the second solution: cloud SiL with DevOps, like TAD Sim from Tencent, Octopus from Huawei e.t.c.
another benefits of SiL in cloud is for complex driving behavior, real vehicle/pedestrains to vehicle interactions, and especially when the scenario lib is aggregating as the ADS lifecycle continues.
if the planning algorithm is AI based, then to mimic the real-human drivers driving behaivor and assign these driving behaviors to the agents in the virtual world, will be very helpful to train the ego’s planning AI model.
here are two models: imitation learning and reinforcement learning. firstly we train the agent/ego driving behaivor using physical world driving data by imitation learning in a training gym; and transfer these trained model to agents and ego in the sim world, they continue driving and interaction with each other, and keep training itself to do better planning.
for corner/edge cases, as planning is very subjective, and lots of long-tail scenarios are there, so physical world situations collection is very valuable.
case 7) from physical fusion output to planning
Tesla looks be the only one, who deployed the shadow mode trigging mechanism to get planning corner cases from physical driving, and the data close loop.
the large volumn of physical driving data is very useful for verification and iteration of planning:
aggregate more real planning scenarios, by detecting edge cases
train better driving behavior, as close as possible to humans
control
control is the process from planning output, e.g. acc, decel, turning-angle to physical actuator response, brake force, engine torque e.t.c
there are a few common issues :
nonlinear characteristic, mostly we don’t get a perfect control output as expect. e.g. from decel to brake force.
the actuator has response delay, which need professional engineers train to get a good balance, but still doesn’t work as expect at all the scenarios
the actuators as a whole is very complex, tuning requires.
case 8) from planning to virtual vehicle model
this is where sim requires a high-precise vehicle dynamic model, which affects the performance. but a simple vehicle dynamic does work for planing, if not requires to cosist with the physical performance.
case 9) from planning to physical vehicle
ViL, which is another big topic