Real-Time Glass Damage Detection Using Edge Computing

Deep Learning and Edge Computing advances have enabled new applications of object detection in image processing across a variety of domains. An example of a novel use of image analysis is detecting external vehicle damage for insurance and glass repair purposes.

Various deep learning techniques are available for not only detecting damages on automobiles (such as broken glass, dents, scratches, and damaged body panels) but also for estimating their severity and estimated repair costs.

The insurance industry relies heavily on cost estimation. The use of automated damage detection and cost estimation techniques can reduce both the amount of time (these produce results much faster than a manual survey and inspection process) and the amount of money spent on the insurance survey.

As such techniques require only the images of damaged vehicles, they can be used remotely, without having the surveyor physically visit the location of the vehicle. This is where real-time edge detection comes into the picture.

Real-time Edge Detection

A real-time edge detection technique is beneficial for a number of reasons. And the following aspects motivated us to carry forward the real-time glass damage detection using edge computing.

Pandemic Related Restrictions: Adjusters would no longer need to physically visit vehicles in order to conduct a vehicle damage audit.
Fast Track Claims: This method can automate windshield claims. Thus, the claim will not require any manual intervention in a specific percentage of cases.
Real-time Response: By implementing a real-time machine learning model on the edge, users and insurers will get immediate responses. Furthermore, Edge Deployed ML models can run 10-15 times faster than Server Deployed ML models.

Collecting And Tagging Data For Training The ML Model

Identifying The Damage Tags For The Model

We tagged the damage on the windshield for our model as cracks. We then created two classes: crack and no crack to identify the damages. However, one drawback to this approach was that its output yield was low since the size and nature of the cracks vary.

But when we analyzed thousands of windshield images, we classified the windshield cracks further in the following categories:

Spider Core - A spider-like shape with a hole in the middle

Line Crack - A visible crack in a line

Small Chip Offs - Stone chips on the windshield

Hole - Windscreen glass showing larger stone chips

Spread - When many cracks are in close proximity to each other and cover a large area of the Windscreen.

The accuracy of the second method was much higher than the first method when images were tagged with the above-mentioned tags.

Segmentation Models To Identify Standardized Car Objects

We would be utilizing segmentation models to identify standardized car objects that often lead to false positives.

Following are a few examples where we use the segmentation (as opposed to a bounding-box method) to identify standard objects inside the vehicle, which are often mistaken for glass breakage, for example, the boundary of the seats, the boundary of the steering wheel, and the boundary of the window glass. By using this approach, the model would learn to distinguish between shatter and standard object reflections.

Combining Multiple Frames

To detect damages, processing several frames instead of individual frames is very important. This is especially crucial for two major error categories: External light reflection and damage merging with background.

Contrary to damages, false positives due to the above error categories are only present on a a limited number of frames - not on most frames. Our goal would be to reduce false positives due to external light reflections by using multiple frames.

Edge Deployment

Quantization

In Machine Learning, quantization is the process of converting FP32 (floating point 32 bits) to a smaller precision like INT8 (Integer 8 bit) and performing all critical operations such as convolution in INT8.

And finally, the output will be converted to higher precision in FP32. This step is crucial to deploying any model to an EDGE platform such as Android, IOS, Web, etc.

Advantages Of Quantization

There are several benefits of quantization, including:

Increasing power efficiency and reducing computation demand
Reducing the size of the model and the storage capacity required.

Quantization In Practice

There are two primary methods for doing quantization in practice.

Post-training: Use float32 weights and inputs to train the model, then quantitatively transform the weights. Its main advantage is that it is easy to use. The downside is that it may result in a loss of accuracy.
Quantization-aware training: Quantify the weights during training. Using this approach, even the gradients are calculated for the quantized weights. This is the most efficient way to use int8 quantization, but it is also the most time-consuming.

Camera Setup For Capturing Photo/Video Of Windshield

To capture the photo or video of a windshield, the right camera setups play a crucial role. Often, the windshield chips and cracks are not visible to the eye since glass is a highly reflective material.

For this reason, frames have to be placed in both the inside of the car, and at 180 degrees from the outside. This is again because glass is highly reflective and laminated.

This is how the camera setup can help when properly placed.

At Least 50% Of The Frame Should Be Covered By The Windshield

Minor damage to the windshield can be detectable if the windshield captures at least 50% of the frame for a few seconds.

Strong Reflections Should Be Avoided

The best way to minimize reflections is to have some shade on the top of the car. Due to reflections, it is difficult to see shatter on the windshield in the left photo. In the right photo, however, the windshield is clearly visible due to a non-transparent shade on top of the car.

Multiple Perspectives

A minimum of three cameras are required to capture cracks in windshields (the more the better, of course!). An example is the crack on the windshield, which is visible only in the first image (see the following four images).

Small Chip Offs

Small chip-offs may be unseen in photos and videos taken from certain angles. There is a small chip present in the photo below that can't be seen in the image. Therefore, the user is guided to also include the entire windshield in the zoom-in view.

Error Analysis And Potential Solutions

We analyzed errors and proposed the following potential solutions for each:

Rain Streaks

Raindrops on windshields often get misclassified as damage due to certain patterns.

Rain classifier

When raindrops cover a large area of a windshield, it is difficult for even human eyes to determine the degree of damage. Thus, a classifier can be used to reject cases when a car is covered with raindrops. The user can wipe the car and retake the video.

External Light Reflections

Light from the outside often creates reflections on windshields, which create the illusion of damage.

Combining multiple frames: Instead of processing individual frames, combine multiple frames for damage detection. Due to reflections of light from outside, false positives from external damage occur much less frequently. By using multiple frames, we would reduce false positives caused by external light reflections.
Frame selection optimization: Develop a more accurate model for frame selection optimization. Furthermore, we were able to host both our part detection model and shatter detection model on the EDGE to resolve this issue. This enabled real-time and accurate frame extraction for windshields.

Standard Objects Reflections From Car Components

There are certain car components that look like windshield chips or line cracks.

Segmentation models to identify standardized car objects: Use segmentation models to identify standardized car objects that often get confused with damage, like steering wheels, seat boundaries, etc.

Non-Standard Objects Inside The Car

There are some items in the car that don't meet the standards (for instance, a ballpoint pen on the chest) that may be detected as damages.

There are limitations to fully solve this issue. To resolve it we would use a combination of:

Camera setup: We can avoid some false positives if we can collect angles of the windshield with a very limited view of objects inside the car.
Further performance improvement may be achieved by using fast, high-resolution cameras. High-resolution high-speed cameras may not detect a crack in a chest ballpoint pen, for example.

Wire Reflections

Shattering of windshields due to overhead wire reflections is often mistaken for shattering.

ML training: Our models have been trained by studying a large set of windshield images with overhead wire reflections.

Tree Branch Reflections

The reflection of tree branches, specifically palm trees, on windshields is often misclassified as shatter.

ML training: We have already trained our models on a considerable amount of windshield images that reflect overhead trees.

Damage Merging With The Background

Car backgrounds are often visible behind the windshield (through the back glass and behind the window). Occasionally, car damage is not visible because the background merges with the car.

Combining multiple frames: Instead of processing each frame individually, combine them together.

Ensuring Low Latency And High Accuracy For The Model

Quantization reduces model size and improves CPU and hardware accelerator latency while also retaining model accuracy, with little loss of model precision.
Quantified models are difficult to achieve high accuracy, as they require a lot of experimentation. A number of different configurations must be tried to get the desired results.
A lightweight model architecture is required to achieve a balance between low latency and high enough accuracy.

Challenges in Vehicle Damage Detection

As real-world vehicle damage photographs are captured in an environment that is uncontrolled and because vehicles have metallic surfaces with reflective surfaces, reflections may appear that are misinterpreted as damage. Researchers used a joint semantic-instance segmentation approach to handle mirror reflections in one study.

There is a potential loss of broader contextual information with two-stage object detectors.Through this joint approach, instance segmentation is given a broader context. Through this approach, reflections are automatically identified without labeling them explicitly.

An AI model may sometimes identify an undamaged vehicle as damaged, which would be a false positive. By using a two-tier model, the number of false positives can be reduced to some degree.

In the first-tier model, only binary classification will be performed on images. The second-tier model identifies the type of damage in an image if it is predicted to be damaged.

Companies using Image-based Vehicle Insurance Processing: Image-based vehicle insurance processing is a valuable business use case that has a good deal of automation potential. Whenever a vehicle is damaged in an accident, the insurance company is responsible for the repair costs.

The assessment of the damage and the costs involved occurs in the automobile workshop. The process is slow, manual, and dependent on the subjective opinion of the surveyor. There are many delays and errors in this process. This can be solved by using artificial intelligence-based automatic damage detection.

By automatically detecting car damage based on photographs taken at the scene, claims processing can be sped up and costs reduced. Insurance companies can use automated AI-based systems to speed up estimation and claim processing, benefitting both them and their customers.

How Our Solution Brings Competitive Advantage

By transforming the claim value chain, the solution benefits insurers. By speeding up claims settlements, it can create a better customer experience, while also increasing efficiency. Incorporating fraud detection into the model would allow additional checks and verifications of the claim declaration image and the repair quotes collected.

A model output might prove useful for repair shops, car resellers, and other intermediaries, allowing for better-informed decision-making, more accurate pricing, and more effective staff planning. Since our solution is not brand-specific, it appeals to different car manufacturers as well as other large automotive companies.

The Way Forward

Car photo inspections are crucial for many players in the car ecosystem, including insurers, governments, rental companies, etc. At present, manual inspections are costly, slow, prone to fraud, and require a lot of people. Inspektlabs automates car ai inspections by taking car insurance inspection photos and videos of each car.

Using an EDGE deployed ML model, Inspektlabs built and deployed a robust web-based solution for the detection of windshield cracks in this project.