Car Damage Detection using AI: Understanding the methodology

Car damage inspections (a crucial process for companies in the car insurance, leasing, and rental industry) have historically been a manual process with human inspectors required to inspect and analyse the extent of damage on the vehicle.

As time progressed and technology got better, we have now moved to using AI and automation, which make the entire process a lot more efficient - right from reducing the effort involved, cutting down errors and false reports, to avoiding fraud altogether.

(Also read: The evolution of vehicle inspections - exploring the past, present, and future)

What types of vehicle damage can AI detect, and what is the methodology behind it? How do you ensure high accuracy to avoid false vehicle damage reports and tackle the challenges that arise in the process? Let's dive into the details in this blog.

Table of contents

1. Types of car damages
2. Car damage detection with AI models
   1. Object detection model
   2. Segmentation model
3. Common challenges in car damage detection using AI models
4. Which models work better for damage detection?
5. Importance of labelling and data quality for damage detection models
6. Car damage detection via tracking method
7. Metrics used for building a damage detection model
   1. mAP for object detection model
   2. IoU for segmentation model
8. Conclusion

Types of car damages

Most car damages that occur after an accident or some kind of impact are cosmetic and structural. While some of these damages are quite obvious, there are a few categories that might be harder to notice. The impact and extent of damage varies depending on how the car's body came into contact with the external factors that caused it.

Car damages mostly fall into three primary categories based on the component impacted - metal damage, glass damage, and miscellaneous damage.

Metal Damage

Metal damage occurs on the metallic parts of the car's body (often including the bumper, hood, doors, boot, or other metal parts) and can be categorized as dents, scratches, or tears.

What are dents?
Dents are the damages caused when the impact leads to a metal body getting pressed towards the inside, causing a concavity on the surface of the vehicle. A considerable amount of these occur when a car crashes.

What are scratches?
These are the most common types of damages that occur on a car's surface and are a result of the metallic parts rubbing against another hard of

What are tears?
Tears occur when the impact to a car leads to the metal parts splitting into two pieces. These usually occur either at the edge of the car part or inside it.

Glass Damage

Glass damages are usually impacts that occur on the glass components of the car body, such as the windshield, back glass, window, headlights, and tail lights. They can further be classified into cracks, chips, spider cracks, and large-range glass damage.

What is a crack?
A crack, also known as a fracture in automotive glass, is a structural break in the vehicle's windshield or window. They can result in a major safety hazard because even a minor crack can compromise the glass' integrity, reducing the ability to absorb and distribute impact, which in turn increases the risk of injury during accidents.

What is a chip?
A chip (technically known as a stone chip or pit) is superficial damage to a car's glass parts caused by the impact from small stones and debris displacing a small portion of the outer layer of the glass, forming a small indentation. This damage can vary in size based on the force of impact and the object's shape.

What is a spider crack?
A spider crack is a specific type of glass fracture that starts from a central point of impact and extends outwards with multiple cracks, resembling a spider web. This type of damage affects the structural integrity of the glass and also obstructs visibility while having the risk of growing rapidly if not treated at the right time.

What is large-range glass damage?
Large-range glass damage occurs when a major area of the car windshield/window is fractured or shattered due to impact. In case of large-range glass damage, the only solution is to replace the damaged glass to ensure both visibility and structural safety.

Miscellaneous Damage

The third category of damage that does not include metal or glass damage can be specified as miscellaneous damage. For example - a gap between car parts, dislocation, etc.

Car damage detection with AI models

Now that we have a better understanding of the types of damages occurring to a car, let's dive into how we can detect these damages using photos and videos.

There are a variety of machine learning models we can use to detect vehicle damage depending on the task. The most common ones used are

• Object detection model
• Segmentation model

Object detection model

Object detection is a primary task in the field of computer vision in machine learning with the aim to detect damage in an image fed to it using deep learning and its advances as the primary algorithm.

Some of the most common object detection models include YOLO (You Only Look Once), EfficientDet, and Faster R-CNN, which provide high accuracy and can detect multiple types of damage in an image.

There are a ton of networks that have been released in publications that we can utilize to tackle different problem statements in damage detection. These networks can be modified by changing the backbone used to extract these features. Each backbone has a different network complexity and processing time required to extract different features.

To see the best results from different networks, we first need to identify the right problem statements and then choose the network best suited for those problems.

Segmentation model

Segmentation models in machine learning are often used to partition or classify data into distinct segments or groups and work by dividing an image into meaningful parts or regions by assigning labels to each pixel.

This model typically relies on supervised learning, and in the case of damage detection, the model would learn to associate certain pixel patterns with specific objects by being trained on images with labelled regions.

We often use advanced models like Vision Transformers for this task, and the ideal type of vision transformer model depends on specific needs, such as the level of accuracy or the complexity of the damage.

Common challenges in car damage detection using AI

While there are different models that can help us detect damage using AI, we also need to keep in mind that these AI models are prone to errors, which can result in false positives or false negatives in vehicle damage reports.

• False positives are the error cases where the algorithm falsely identifies damages when none exist.
• False negatives are cases where the AI fails to detect damage even when the damage exists.

Some of the most common challenges that AI models face during damage detection are -

Reflective surface of the vehicle
Shiny or metallic surfaces are very prone to reflections, and this can often cause the AI model to misinterpret them as defects on the surface of the vehicle, resulting in false positives. For example - light bouncing off a car's curved surface might look like a dent or a scratch, which will in turn result in incorrect detections.

Perspective variations
While some damages might be easy to detect from a certain angle, the same damage might be invisible on camera from a different view. Similarly, it is also possible for a certain light effect to be mistaken as a dent or a crack when viewed from a certain angle.

Lighting issues (shadows and low light)
Poor lighting or shadows on the vehicle can either hide damage on a surface or make the surface look damaged. In the case of low light, capturing dents or scratches proves to be even more challenging, leading to false negatives, while shadows or uneven lighting can be mistaken as damage, resulting in false positives.

Damage overlap
In some cases, multiple damages might overlap, like a scratch covering a dent. The visibility of the dent may be reduced because the scratch distracts the algorithm (or in some cases, even the human eye) making it harder to detect the underlying damage.

Micro damages
Micro damages are tiny scuffs and indentations on the car's surface that aren't as easy to detect because of their size. In many cases, they might go undetected even during human inspections.

How to overcome these challenges

Yes, detecting damage via AI isn't as easy as it seems, and it comes with its own set of challenges. However, there are a few solutions to work around these challenges. For instance -

• Multi-angle imaging is used to capture the damage from different viewpoints which gives the AI model varied perspectives of the damage present on the vehicle, i.e. resulting in reduced false positives or negatives caused by the reflections and perspective variations.
• To avoid shadows and low light issues, shadow detection and processing techniques are applied to clear shadows from the images. In case where there is excessive darkness or shadows, the model is trained to flag the images and exclude them from the damage detection process to avoid false positives or negatives.
• Damage overlap is a rare scenario and is less frequent. To avoid this, specific visuals are collected for such cases, which are later used as data points to train the model to identify these cases where they occur.
• Micro damages can be captured by making sure you capture the entirety of the car for better quality analysis. Additionally, focusing on using a good camera, picking the right location, lighting, and background, and framing them the right way is the way to go.

(Also read: Guidelines for capturing micro damages through photos/videos of a damaged vehicle)

Which model works better for car damage detection?

Both object detection and segmentation models come with their own set of pros and cons. The efficiency of both models (as we discussed earlier) is purely based on the scenario it is being used in. However, both models will always pose a challenge of reporting false positives or negatives in certain scenarios.

The ideal solution to avoid this (also the best model for vehicle damage detection) would be to use both models together.

When we combine both models, we get an ensemble of models that combine the strengths of both models, covering up the drawbacks of each other and hence, resulting in fewer errors in the final prediction.

To better understand how this works, let's examine the models in action for scratch and glass crack detection.

Scratch detection

Check out this image below - where the results of scratch detection were generated using only the object detection method.

As we can see, the bottom detections are accurate, but the top detection is a false positive. Here, the model confuses the reflection for a scratch, leading to a wrong result.

In some cases, the false positives can outnumber the actual scratches, and the model may suggest that the car is severely damaged when it isn't.

Now, let's look at the same with the segmentation model overlaid on the current image.

As you can see, the model successfully localized the actual scratch damage with high precision despite there being a reflection on the car (the green patches highlight the prediction generated using the segmentation model).

Now, let's combine both models and build an ensemble between them.

As you can see, the boxes with overlapping segmentation pixels (here in green) are accurate predictions, while empty boxes are inaccurate.

From this, we can segregate the boxes into two categories - one that overlaps with the segmentation pixels (in blue) and the other with empty boxes (in yellow). Post this, we can then train the AI model to accept the blue boxes (with overlap) and reject the yellow ones (without overlap), thereby resulting in a more accurate and reliable prediction of damage on the vehicle.

Glass crack detection

Since glass is inherently highly reflective, using the object detection model can pose a bigger challenge in damage detection. This is where the ensemble technique we used earlier proves its worth.

Let's see how this model works to detect glass cracks using the image below.

Here are the results we get by running the object detection model on it.

As we can very clearly see, the object detection model, in addition to the cracks present, has also classified the reflections of the poles as glass cracks, i.e. several false positives generated.

Now, when we run the same through a segmentation model for analysis, we can see that it accurately predicts the crack damage visible in the image.

This prediction will be used in combination with the object detection model results to generate the final ensemble prediction.

After the segregation of the two categories of boxes (blue for overlaps and yellow for no overlaps), we will finalize only the overlapping ones.

The final ensemble prediction (as you can see) is highly accurate, with no false positives. This technique demonstrates the power of ensembling the two models and their impact on metrics and accuracy.

Importance of labelling and data quality for damage detection models

Labelling is a very important step to help improve the accuracy of your damage detection model. This is crucial because this data is used to "train" the model to recognize patterns and make predictions based on it.

Labelling, put simply, is the process of assigning specific tags or annotations to data (in this case, images and videos of different kinds of damages) and using precise labels ensures that the model learns correctly from the data being fed.

For example, when labelling a scratch, it is important to use multiple smaller bounding boxes that closely fit the shape of the damage rather than using one or two larger boxes that cover more area than necessary.

Additionally, data selection and analysis also plays a very important role in improving the accuracy of the damage detection model. Analysing the data helps us identify where the model is struggling so we can focus on improving training with more of those specific cases.

Both precise labelling and thoughtful data selection are important for training high-performing damage detection models.

Car damage detection via tracking method

In a lot of cases, it so happens that we've trained the model on a certain amount of data, but the data we've fed the model is still scarce. This could either happen because of a lack of images or because of irregularities in labelling the images used.

In such cases, we often end up with a model that's not robust enough and prone to reporting false positives or false negatives.

However, we can still detect damage by switching from images to videos of the car's damage. This method involves video analysis and tracking objects (here, damage) as a solution, which uses information from the neighbouring frames to accurately predict the damage and its correct location.

Let's test this method on a cracked windshield.

Let's break this down frame by frame.

In the video used above, we can see the car moving from left to right, and a red box on the windshield identifies the prediction of the crack detection model. As you can see, the model detects the crack accurately 5 out of 7 times except for frame 3 and frame 7 (which are false positives).

Tracking the red box and its position using different tracking algorithms can help us determine the box in future frames. The frequency and placement of the tracked boxes can help differentiate between true positives and false positives.

Metrics used for building a damage detection model

When building any machine learning or deep learning model, it is very important to set a metric that determines how well the model will perform in the test data. Evaluating the model's performance based on these metrics and picking the best results is what helps us train and improve the accuracy of a damage detection model.

There are two main metrics we focus on based on the two models we discussed earlier (object detection and segmentation).

mAP for object detection model

mAP of Mean Average Precision is a performance metric used to evaluate how well a model detects objects (in this case, damages) in images, giving you an idea of how accurate and thorough the model is at its job.

Imagine you have a model that's trying to find damages like dents or scratches in a set of images given to it. The mAP tells you how well it did the job based on two parameters -

• Precision: How many of the detected images were actually correct (the model didn't mistake something else for damage)
• Recall: How many of the real damages the model actually found (it didn't miss any).

The mAP score combines both of these parameters and gives you a number (between 0 and 1 or 0% to 100%) that reflects how well the model did overall. A higher score means higher accuracy of the model in detecting damage and vice versa.

IoU for segmentation model

IoU is a metric that computes the ratio of intersection between the ground truth reference box and the predicted box of the model to the union of two boxes, essentially telling you how much overlap the boxes have with each other.

Based on the amount of overlap, the IoU is assigned a score varying between 0 and 1, where 0 means no overlap and 1 means there's a complete overlap.

This is then compared to a threshold score, which is the opposite of the IoU score, i.e. 1 means no overlap, and 0 means complete overlap.

Based on this comparison, the model builds these inferences:

• IoU ≥ threshold: The detection is correct, i.e. a true positive (TP)
• IoU < threshold: The detection is incorrect, i.e. a false positive (FP)
• If no box is predicted, it is a false negative (FN)

We can then determine the precision and recall of the model based on this formula, which in turn will help us determine the mAP of the model that dictates the model's accuracy.

There is another metric for the evaluation and comparison of models. It's called the F1 score.

This is the harmonic mean of precision and recall and can be computed using:

The results from this formula helps us evaluate and compare models and finally pick the ones that perform the best.

Conclusion

As time passes and technology advances, it is necessary for us to move to smarter ways of vehicle inspections, which include AI and automation, which come with their own benefits.

(Also read: Humans vs. AI inspections: A comparison across 7 parameters)

While there are many models we can use for AI vehicle inspections, it is necessary to know that each model has its own set of drawbacks and picking the right ones based on different situations is crucial to ensure the right results.

Want to explore how you can automate vehicle inspections and improve efficiency at your company? Book a demo with Inspektlabs to learn more.