In this blog, we will cover the fundamentals of car damage, its types, and categorizations. Then, we will briefly discuss the challenges that car damage detection presents. We will further discuss the three approaches to achieve high accuracy during damage detection, viz. object detection, ensemble techniques, and tracking. Finally, we will discuss the metrics used to improve accuracy.
Cars can suffer from damages when they come into contact with sharp objects, road accidents, or any other incidents that harm the car's body, impacting its original shape and form. Automated car damage detection is becoming increasingly common, and many users are turning to these sophisticated systems. This technology is improving as vehicle damage detection using AI becomes more prevalent. Many different types of car damage detection techniques are available at our disposal.
Car insurance, car leasing, and rental car companies benefit from vehicle damage detection using AI. Image-based convolutional neural networks can accurately recognize car damage through photos and video. The car damage detection process is crucial to the motor industry and the related stakeholders. It identifies damages and estimates the repair cost. It is often impossible to manually inspect every part of a car, but vehicle inspections are efficient and feasible using machine learning algorithms. Furthermore, AI damage inspections can recognize and analyze multiple types of damage in seconds. This way, users can make more informed decisions regarding the condition of their vehicles.
Types of Car Damages
The most common car damages after an accident are cosmetic and structural. Some of these damages are obvious, while others may be harder to notice. Regardless of the cause of the damage, it is vital to understand the different types and how they can affect your car. Knowing the various types of car damage can help you analyze them. Through vehicle inspection automation, users can quickly gain insights about the scope and extent of the damages.
There are a variety of damages that can occur to a car. The scope and extent of the damages vary with how the car's body comes into contact with external objects. How the vehicle sustained the damage will also impact its severity and extent. Some damages will be repairable, while some can harm the car part to such an extent that it will have to be replaced by a new part.
Car damages can be classified into three primary categories, based on which component they impact. These categories are metal damage, glass damage, and miscellaneous damage.
Metal damages occur on the metallic parts of the car's body. These parts include the bumper, hood, doors, dickey, and other metal parts. Metal damages can be further classified into the following variations: dent, scratch, and tear.
Dent is the damage caused when the metal body is pressed inside when it experiences pressure from external objects. This pressure creates a concavity on the metal surface. A considerable number of dents can also occur when a car crashes. The following images illustrate this type of damage.
Scratches are the most common damages that occur on metal surfaces. When a metallic car part moves or rubs against another hard or sharp object's surface, scratches form. Scratches can also form if a hard or sharp object runs over the car's metallic surface. The following set of images depicts scratch damage.
When a car sustains tear damage, the metal part is split into pieces. This tear can occur either at the edge of the car part or inside it. The following images illustrate tear damage.
Like metal damage on metal parts, glass damages occur on the glass part of the car body. The components it impacts include the windshield, back glass, car windows, headlight, and taillight. Glass damage can be further categorized into the following variations: crack, chip, spider crack, and large range glass damage.
Crack damage occurs on the windshield of the car. The following images illustrate crack damage.
Large range glass damage
Damages that do not fall under either metal or glass damage are classified as miscellaneous damages. For example, the following images depict a gap between car parts. Dislocation of car parts also falls in this category.
We now have a comprehensive understanding of the different types of damages cars can sustain. Let us discuss how we can detect these damages through photos of the car. Although there are various types of damage, the strategy for damage detection is largely the same, except in a few instances. This general strategy involves using an object detection network trained on a large number of labeled images to detect damage in the given image.
The detection network can either have all the damages, or we can have different networks trained for specific groups of damages. The most effective approach is the latter one. The reason behind this is that different damages have different characteristics. So, a single model trained for all damages may decrease the performance of the model. This deterioration in performance materializes in the form of false positives or false negatives of the damage detected through an image.
The false positives are those error cases where the algorithm mistakenly identifies damage in an image where no damage exists. For example, the AI can erroneously mark poles, shadows, the reflection of objects, and the environment as damage. False negatives are those error cases where the AI is unable to detect the damage although it is present in the image. Our primary goal is to train the models for damage detection and to minimize the frequency of these two error cases.
As mentioned above, the default strategy is to train an object detection network. However, this approach doesn't work in a few cases. These cases will be discussed below separately, along with the techniques used to identify damages in such cases.
Let's discuss the common strategy of object detection. Object detection is a primary task in the field of computer vision in Machine Learning. Its aim is to detect objects in a given image. Deep learning and its advances are the primary algorithms that help achieve this task. Some examples of these networks are Faster-RCNN, Retinanet, YOLO, etc.
There are many networks that have been released in publications that we can utilize to tackle our problem statement. Identifying the suitable networks correctly will give the best results. Networks like Faster-RCNN and Retinanet are detection architectures. These networks can be modified by changing the backbone used to extract the features. These backbones can be MobileNet, VGG, Resnet, EfficientNet, etc. Each backbone has a different complexity of networks and processing time required to extract the features. The accuracy of each backbone varies based on the problem we are working on.
Damage Identification by Object Detection
Damages like a tear on the metal body and dislocation or gaps between parts are easily identifiable using detection networks. They give accurate results when trained on large and represented data. For example, consider the damage detection in the image below. All the tears and dislocations have been successfully identified here.
Similarly, glass damages can also be identified using this detection model. For example, the image below depicts the accurate identification of a spider crack.
In the image below, chip damage has been identified successfully.
In the image below, a large range of glass damage has been detected successfully.
Damage Identification by Ensemble Techniques and Segmentation Models
As we have already discussed, although some damages can be identified using object detection, using the same technique on other damages like scratches and glass cracks is insufficient. The chief concern in such cases is the false positives. As the problem gets harder, the accuracy of detecting the damages also becomes challenging. Nonetheless, there are methods and techniques which can help build a successful model that can fulfill our aim to detect such damages accurately. Let us analyze these techniques through the following cases: scratch detection, glass crack detection, and dent detection.
In the image below, the following scratch detection results were generated using only the object detection method.
As we can see, the bottom detections are accurate, but the top detection is a false positive. Here the model mistakenly identifies the reflection as a scratch. This problem is exaggerated when detecting damages on glossy and reflective cars. In some cases, the false positive detections can outnumber actual scratches, and the model may suggest that the car is damaged even though it is not.
Although false positives are a problem, there is still a viable solution to tackle it. The idea is to use an ensemble of detection as well as segmentation techniques. Let's see the segmentation of the scratch damage in the above image.
The green patch highlights the prediction generated using a segmentation model overlaid onto the original image. Despite the presence of reflections, the model has successfully localized the actual scratch damage with high precision.
Both object detection and segmentation models have their strengths and weaknesses. So, when we combine these two models, it is highly likely that the false positives they predict will be different and at different locations. This is because of how the prediction models function. Even the training is different for the models, with different loss functions. So, when we combine both models, we get an ensemble of models which combines the strength of the two models and minimizes the final prediction error. Let's see the combination of both the object detection and the segmentation model.
We can clearly see that boxes with overlapping segmentation pixels (here green) are accurate predictions while empty boxes are inaccurate.
We will segregate the boxes into two categories: one that overlaps with the segmentation pixels and the other with empty boxes. The above picture shows the overlapping category in the blue-colored boxes and the non-overlapping category in yellow-colored boxes. We will accept the accurate predictions (blue boxes) and reject the mispredictions (yellow boxes). The final predictions are depicted in the image below.
This final prediction of scratch detection using the ensemble of both object detection and segmentation models gives accurate and reliable predictions that the individual model could not produce. The following are some other tested images that illustrate this approach.
Glass Crack Detection
The technique used for detecting scratches is also useful for glass crack detection. Moreover, the detection of glass cracks is more challenging compared to scratch damage. Let us see how the detection model works for glass crack damage. Below is an original image of the car's windshield on which we need to accurately predict the glass cracks.
Upon running the object detection model, we get the following results.
From the results, we can determine that in addition to accurate crack predictions, there are several false positives generated by the detection model. The false positives are the three boxes highlighted on the right, and the fourth is the highlighted box at the very top. The model has inaccurately classified the reflection of poles as being glass crack damage. So, we will pass the image through the segmentation model to analyze its prediction.
From the prediction, we can see that the segmentation model accurately predicts the crack damage visible in the image. This prediction will be used in combination with the detection model results to generate the final ensemble prediction.
After the segregation of the two categories of boxes, we will finalize only the overlapping boxes. The final predictions are depicted in the image below.
The above final ensemble prediction gives accurate predictions with no false positives. This technique demonstrates the power of ensembling the two models and their impact on metrics and accuracy. The following two images illustrate the test results on other images.
The following two images depict the predictions on other test images with only the final ensemble results.
The same ensemble techniques are applicable for dent detection also. Here are the results.
Damage Identification by Tracking
It is often the case that we have trained the model on data, but the data is not large enough. This situation can arise either because of a lack of images or because of the deficiency of labeled data. In this case, the trained model may not be robust enough and may therefore lead to many false positives or false negatives. In many cases, we are given a video captured by the client, and the ML system is made to predict the damages accurately. In this case, despite the less robust model, we can use information from the neighboring frames to accurately predict the damage and its correct location. This problem involves video analysis and tracking objects (here damage) as a solution.
Let's test this method on a video of a cracked windshield. The video is read, and its frames are extracted. The windshield frames are selected, and the detection model is run on them. The below frames are the result of the above process.
Video frames of a moving car in the right direction:
Let's see the frames by zooming in.
The above frames show a car moving in the right direction, and a red box on the windshield identifies the prediction of the crack detection model. From frame-1 to frame-8, we have continued red boxes on crack six times except at two frames two times in frame-4 and frame-8, which are false positives. The tracking of crack boxes using different tracking algorithms can help us determine the box in future frames. This frequency difference of tracked boxes can help detect the true positives and false positives.
If we use any ML or DL model, we need a metric to determine how well our model performs on the test data. Based on the metric, we select the one which has the best performance value. These metrics can be mAP or F1-score.
For evaluating the above metrics, we need to get values of precision and recall, which in turn depends on IOU.
IOU is a metric that computes the ratio of intersection between the ground truth reference box and predicted box of the model to the union of two boxes. The value varies between 0 and 1. It essentially tells how much overlap the boxes have with each other. If they overlap completely, the value is 1. If there is no overlap at all, then its value is 0.
Based on the IOU score and threshold, where the threshold is a value between 0 and 1 with minimum overlap, the boxes must make, if :
● iou>=threshold : detection is correct; it is true positive (TP)
● iou<threshold: detection is incorrect; it is false positive (FP)
● If no box is predicted, it is a false negative (FN)
Based on the above conditions,
Based on the precision and recall values, we will plot the precision-recall curve.
The orange line above plots the actual values which we got after computing TP, FP, and FN for different confidence values. After using the 11-point interpolation technique to smooth the curve, we will get the red curve above. The area under the red curve will give us the Average Precision (AP). If there are multiple classes, then the mean of AP of all classes gives us the Mean Average Precision (mAP).
There is another metric for the evaluation and comparison of models. It's the F1 score. It is the harmonic mean of precision and recall. It takes account of both precision and recall. The formula to compute this score is:
In this way, we can evaluate and compare models and finally pick the one that performs the best.
Automated car damage detection reduces human error considerably. Moreover, vehicle inspection technology improves over time as more data helps train the algorithms. As we have discussed, ensemble techniques further enhance the accuracy and reliability of vehicle inspection automation.