Towards safer and more reliable autonomous systems

Autonomous vehicles rely on precise deep machine learning algorithms (DML) for their proper functioning, but conventional software engineering methods are insufficient to ensure their correct behaviour. In the research program SMILE, QRTECH tackled this challenge by developing a safety cage and cutting-edge methods for identifying out-of-distribution inputs.

Machine learning based systems are becoming increasingly common in a wide range of applications, from chatbots to medtech. Ensuring the accuracy and reliability of such systems is crucial, especially in safety-critical solutions like autonomous vehicles. However, development of these methods differs significantly from traditional software engineering processes, making the verification and analysis a challenge.

QRTECH have been involved in the Vinnova funded research program SMILE, with the objective of developing methods allowing DML-based functions to be included into safety critical vehicular applications, with quality control requirements from industrial standards. The program included three projects, where the SMILE I project focused on researching verification and validation for DML systems as well as mapping of challenges faced by the automotive industry. SMILE I concluded that implementing a safety cage concept would be a crucial next step for future projects. A safety cage can be viewed as a supervisor that oversees the predictions made by a machine learning model to determine their reliability.

In the SMILE II project QRTECH conceptualized and developed a safety cage based on the statistical analysis of activations inside the neural network. A proof-of-concept demonstrator was developed for a perception system, trained to classify cars, motorcycles and trucks in highways during sunny weather. The demonstrator performance was then tested in city environment and in foggy weather. The safety cage intercepted several incorrect predictions made by the neural network.

During the phase of SMILE III, QRTECH investigated how to combine the safety cage into a system architecture, tested new state-of-the-art methods for detecting out of distribution inputs and implemented an uncertainty aware neural network that could be used in identifying outliers or edge cases. More investigation is required in developing such methods for implementation, especially in a real time system and in situations where training data is limited. Together with the project partners, QRTECH conducted a review of a methodology called Assurance of Machine Learning for use in Autonomous Systems (AMLAS) that is designed to ensure the reliability of machine learning components in autonomous systems.

In conclusion, the SMILE program has been successful in addressing the challenges of developing safe and trustworthy autonomous systems from both the software and standards perspectives. The development of the safety cage and the use of the AMLAS methodology are two examples of how to tackle these challenges. With continued efforts and collaboration, this can pave the way for the creation of more secure autonomous systems that meet the needs of society.

The SMILE program was based on a collaboration between industry, institutes and academia. It consisted of several consecutive research projects exploring the challenges while introducing machine learning based systems in automated driving applications. It also aimed to propose strategies to cope with those challenges. In addition to QRTECH, the program involved several partners, including RISE, Volvo Cars, Semcon, ESI Nordics and INFOTIV.

www.vinnova.com