system validation in l2 adas

background

from system requirements, the functional models can be created. the bottom line of ADAS is safety, which defines the safety requirements. during ADAS system validation, both function modules and safety modules should be covered.

my experience lay in SiL a lot. a question recently is how much percentage between SiL and HiL is a good choice?

a few years early, SiL, especially world-sim, e.g. LGSVL, Carla are highly focused. the big companies said, to achive a L3+ autonmous driving system, there needs at least 10 billion miles driving test, which is almost impossible in traditional vehicle desing lifecycle, which gives a life for virtual sim driving validation.

which even not consider the iteration of ADAS algorithms. as if there is no mechanism to gurantee the previous road data is valid to validate the new version of algorithms, they have to collect another 10 billion miles for each algorithm version iteration.

but now most OEM choose the gradually evaluation way from ADAS to AD. during ADAS dev and test, Bosch has mature solutions, they don’t need 10 billion miles to validate AEB, ACC, LK kind of L2 ADAS functions, usually 1500 hrs driving data is enough, and they have ways to re-use these driving hrs as much as possible during their ADAS algorithms iteration.

I am thinking the difference is about function-based test and scenario-based test.

ADAS functions always have a few limited if-else internally and state machines in system-level. for function-based test, to cover all these if-else logic and state machines jumping, with additional safety/failure test, the ADAS system can almost gurantee to be a safe and useful product.

there is no need to cover all scenarios in the world for L2 ADAS system. of course, once someone did find out all physical scenarios in the world, they can define new ADAS features. and add these new features to their products.

in a word, L2 ADAS is a closed-requirement system, the designers are not responsible for any cases beyond the designed features.

For L3 AD system, the designers is responsible for any possible scenarios, which is a open-requirements system, there is no gurantee the design features is enough to cover all scenarios in real world, so which need scenario-based test and keep evaluating through OTA.

For L2 ADAS, OTA is not a have-to option in some way, but L3 ADAS has to OTA frequently.

OTA for l2 adas is to improve the performance of known features, through analysis of big driving data, but it doesn’t discovery novel features.

OTA for l3 adas need to improve the performance of known features, as well as to discover new features as much as possible and as quick as possible.

which of course bring requirement of OEM’s data platform. Tesla is currently the only OEM, who can close the data loop, from customer’s driving data, to OEM’s cloud, and OTA new/better features back to customers.

there is no doubt, Tesla evaluation is scenario-based. collecting scenarios is not only for validate existing systems, or test purpose, but more for discover new features, e.g. from lane/edestrain/vehicle detection to traffic light detection e.t.c.

the traditional OEMs still don’t know how to use scenario data valuablely, only image the collected data can do validate or calibrate a better performance known features.

ADAS HiL

HiL helps to validate embedded software on ECUs using simulation and modeling teches. at the bottom of V model, there are software dev and test, the upper right is software/hardware integration test, namely HiL.

as I understand, the purpose of HiL is software and hardware integration test, and there are two main sub validation: safety requirements, and function requiements.

to validate safety in HiL, fault injection is used. safety module is very bottom-line, and any upper functional module is rooted from safety module. in a narraw view, fault injection only focus on known or system-defined faults, which is finite cases; in a wide view, any upper functional cases can reach to a fault injection test case, which is almost unlimited.

in the narraw view, coverage of functions(both function internnal state machine and across-function jump trig mechanism) is iterable, even though its total size may be hundreds of thousands.

ADAS massively production solution

1R1V(3R1V)

Level 2 ADAS system, some OEMs called it as highway Assist(HWA), mostly provided by Tier1, e.g. Bosch.

5R1V(5R5V)

Level 2+ ADAS system, some OEMs called it as HighWay Pilot(HWP), hidden meaning it’s L3, but in fact, not a fully L3.

8R1V(5R12V)

camera-priority solution, used in Tesla and Mobileye.

L2 ADAS system validation

from traditional vehicle system test and validation, there are finite cases, e.g. ESP interfaces. for ADAS system validation, which is scenario based, and mostly there are infinite cases.

there are different ways to cover ADAS system validation:

fault injection test, the vehicle powertrain, ADAS sensors, ADAS SoC/MCU, all these HW has random probability to fail.

Fault Injection

fault injection tests is used to increase the test coverage of Safety Requirements. Within the fault injection tests arbitrary errors are introduced into the system to proof the safety mechanisms that have been encoded in the software to examine. fault injection tests is only a subset of Requirements based testing.

ISO 26262-4 [System] describes the Fault Injection Test as follows:
The test uses special means to introduce faults into the item. This can be done within the item via a special test interface or specially prepared elements or communication devices. The method is often used to improve the test coverage of the safety requirements, because during normal operation safety mechanisms are not invoked.

The ISO 26262-4 [System] defines the fault injection test as follows: Fault Injection Test includes injection of arbitrary faults in order to test safety mechanisms (e.g. corrupting software or hardware components, corrupting values of variables, by introducing code mutations, or by corrupting values of CPU registers). arbitrary faults is actually from a known list of errors.

fault injection is good at testing known sorts of bugs or defect, but poor at testing novel faults or defects, which are precisely the sorts of defects we would want to discover. therefore, fault injection in ADAS is to verify the designed safety mechanism/responses.

to verfication of the system. if an ADAS system is designed to tolerate a certain class of faultss, then these faults can be directly injected into the system to examine their responses. for errors which is too infrequent to effectively test in the field, fault injection is powerful to acclerate the occurence of faults.

in summary, fault injection is more verification tool, rather than a tool to improve performance or find novel design faults.

software verification and validation

verification is about whether there is the functions as required. validation is about how good the functions are as required. verfication tech includes: dynamic testing and static testing. e.g. funcitonal test, random test; validation tech includes: (h/sw) fault injection, dependency analysis, hazard analysis, risk analysis e.t.c.

the software verification is usually a combination of review, analyses, and test. review and analyses are performed on the following components:

requirement analysis
software arch
source code
outputs of integration process
test cases and their procedures and results

for test is to demonstrate it satisfies all requirements and to demonstrate that errors leading to unacceptable failure conditions has safe strategies. usually including: