CV and AI based Car Infotainment Panel Test Automation

The Customer was looking for automating the testing of their infotainment screens, as system testing and regression testing cycles were long as well expensive with sub-optimal level of test coverage.

Solution Description

Designed an automation test suit using technologies such as Robotic arm with camera, computer vision software that recognizes icons & texts for performing the test and then comparing the outcome against the stored expected screens.
The vision software combines real-time image recognition and text recognition models which enables the test automation system:
- To identify the on-screen elements and navigate through the menu for execution of test cases
- To validate the outcome screen content against expected content.
The system consists of Test automation suit running in a PC interfaced with a robotic arm instrumented with the camera, Infotainment unit (OAT) and the visual AI model.

Outcome and Benefits Delivered

The pre-trained convolutional neural network model has given high accuracy results for screen/icon recognition than traditional methods.
Recognition and localization of elements on the result screens.
Image recognition is performed in real-time from a live video stream.
Text localization and recognition.
LSTM and attention-based neural network for text detection
Real-time on-screen slider movement tracking using image processing.
Achieved high test coverage, high level of regression runs, along with accurate test results.

The Problem Statement

The Customer was looking for automating the testing of their infotainment screens, as system testing and regression testing cycles were long as well expensive with sub-optimal level of test coverage and test results.

The Infotainment screens are configurable according to localization and customer preference. In order to automate the testing of screens to ensure adequate test coverage along with accurate comparison of test results, the customer was looking for:

An Visual AI driven test automation suit that executes the test cases on different screens, and then compare and produce test results
Ensure targeted test coverage and regression test cycles

The Solution / System Description

Robotic Arm with Industrial grade camera

4 axis DOF robot with customized gripper.
- Payload 500g
- Max. Reach 320mm
- Position Repeatability(Control) 0.2 mm
- Communication USB / WIFI / Bluetooth

Vision AI based Solution

Industrial grade camera for image acquisition - 5MP, CMOS, 15.0 fps, 2592 x 1944, 1/4", Rolling Shutter
Neural network based image recognition for identifying icons and screens
Custom algorithms for measurements and position detection using OpenCV
Hand eye coordination of robotic arm with machine vision was implemented
Trained a custom neural network OCR on Google Cloud for high accuracy text detection for special fonts.
Automated tool for creating dataset for training OCR.