Challenges of Background Removal and how Deep Learning overcomes them | Inspektlabs

Why Remove the Background of a Car’s Image

When designing something with a photo, the background of the car’s image may not be appealing because of aesthetic reasons. This is where photo editing comes in. Taking the background out from a car’s image also adds a touch of originality to the car’s photo.

A unique image is crucial to boosting the car business and there is no substitute for background removal. Another reason to remove the background of a car’s photo is to make the photo look just how one wants. It is also possible to remove the existing background and add a background of your choice.

Removing the background from a car’s photo is beneficial in several different ways. The features of the car look enhanced to make the car image look more suitable.

Benefits of Removing the Background from Car Images

1. Selling the Car

Most marketplaces today require white or neutral backgrounds behind the cars. This is because on neutral backgrounds, high-quality images make the features of the cars pop. this makes it convenient for the buyers to assess the elements and colors of the car. Hence, removing the background from car images and replacing the background with white or neutral tones can make the car compliant without the need of another photoshoot.

2. Coherence of the Theme

Setting up an eCommerce seller profile or designing a catalog may earn consistency in theme photography when the background is removed. Getting rid of the background helps in maintaining the consistent imagery that can be used in the website or designing catalog.

3. Reducing File Size

If the website's file size is smaller, it will load faster, which will improve the SERP, or the Search Engine Ranking Position. Thus, removing the background will not only reduce the number of extra details but will also reduce the size of the image. This also contributes to a better customer experience.

The Need for a Background Remover

1. Getting Rid of Unnecessary Objects from the Car Image

Most professional photographers are bound to make mistakes even when they aren't aware of them. Imperfections become more evident during post-processing when these mistakes become more apparent. A majority of the time, the car picture will have unnecessary items on it that cause a distraction, which has to be removed. Hence, a background remover comes in handy to remove the unwanted objects from the image of the car.

2. For the Purpose of Adding Special Edits

For adding special effects to the photo of the car, like reflections and shadows, a blank canvas is required. The possibilities for improving the picture are endless. If the background has already been edited to remove the obvious imperfections, it will be easier to add these improvements back to the photo.

3. Optimizing Photos for other purposes

Photos are an important part of everyday life in the digital age. Online shopping has now become the norm, with the only means of closely inspecting items being product photos. Many platforms require the car photos to have a neutral or transparent background or to be free of any kind of background. To ensure that car photos are accepted for upload, entrepreneurs must adhere to this requirement.

4. Turning the Background Transparent or Changing it to a Suitable Color

For aesthetic reasons, some photographers change the background of the car photos often.  They may also choose transparent backgrounds for these photos. These modifications can be made easily with background removers.

5. Making Car Photos More Visually Appealing

In promoting something, it is very important to convey a positive impression, especially in the case of cars. Regardless of whether they are printed or digital, car images have more impact and can capture the viewer's attention if they communicate the right message. Focusing on the focal point of a photograph always works, and this can be accomplished by simply removing the background from the picture.

6. Adding More Objects to the Car Photo

How about adding special effects to the car’s photos? There is also an option of adding more elements to the car’s picture, in addition to the effects that enhance it. If it is necessary to add any object, removing the background from the car image will facilitate a natural-looking edit.

7. Changing the Background of the Car’s Picture to Improve Visuals

To make it look like the car is an exotic country, the first step is to remove the original background. Transparent photos can be swapped out for a more appropriate or appealing background once they have a transparent background.

Because of all these reasons, it comes as no surprise that a background remover can have an impact on how car photos are perceived.

Master The Most Popular Techniques for Removing Backgrounds

Many beginners often find it confusing to remove the background of a car’s image, given that there are so many different options to choose from. Here are the common techniques to master to remove background that give great results.

1. Image Clipping Path

The image clipping path technique is recommended to use if the car has straight or sharp edges. The Pen Tool in Photoshop enables drawing the clipping path around the subject. Zoom in the picture to get a closer look at the car. Next, highlight the edges. Make sure that the car is entirely inside the path. The elements that are outside this path would be eliminated.

2. Image Cut-out

The technique of image cut-out is pretty self-explanatory. The car’s image is cut, and the background is removed. To give the image a professional look, a white background can also be inserted. Appealing images can also be used as backgrounds for branding purposes.

3. Image Masking

If the car’s image has fine edges, it will be difficult to clip them. In these situations, the image masking technique is useful. These techniques can be complex layer masking, alpha channel masking, translucent image masking, transparency masking, or collage masking. Photoshop has several tools to achieve this such as Magic Eraser Tool, Pen Tablet, and Background Eraser Tool for making the process easier.

4. Erasing the Background

Erasing the background from the image of a car is a simple technique, and it is suitable for beginners. The background pixels from the image can be removed using the Background Eraser Tool in Photoshop. When there are complicated backgrounds, however, this tool may not be a good option. Also, with this tool, stray pixels can be left in the image of your car.

Using these tools one can easily remove the background from their car’s pictures and make them look more appealing and aesthetic.

Challenges of Background Removal

Many car auction clients and similar service providers frequently find the need to superimpose an image of a car onto a virtual background or showroom, so that they can display it on their website. To do this, they need to first create a cut-out of the car in question.

Generating this cut-out comes with a lot of challenges. The cut-out has to be perfect, at the pixel level, and exactly follow the edges of the car. A straightforward method (which would be familiar to graphic designers) is to use Adobe Photoshop and its Object Selection Tool to create this cut-out. However, the result is rarely perfect and requires further tweaking using finer tools such as Refine Edge Brush Tool and Brush Tool. As a result, it requires constant manual intervention and has to be done individually for every car.

Enter Deep Learning. Removing background from images of cars is basically an image semantic segmentation problem. In semantic segmentation, a deep learning model labels every pixel of the image by a unique value, denoting the class to which the pixel belongs.

In background removal, we have basically two labels - First, the object of interest (in this case, the car) and second, the background.

At first, the problem seems trivial. Train a model that predicts 1 for every pixel belonging to the car and 0 otherwise.

However, upon close inspection, there are quite a lot of challenges.

1) In dense areas, i.e. areas where there are multiple cars such as an auction or parking, it is not possible / feasible to take a picture of the car in such a way that there are no other cars in the image. Take the above image as an example, apart from the blue car, there are multiple cars in the background. While generating a cutout, we need to focus on the blue car and not the other cars. In other words, we need to keep in mind the ‘saliency’ of the image.

In the above example, we can see the model’s output does not completely mask out the black car at the back. It is partially visible, indicating the model is not totally confident whether it should remove it or not.

To tackle this, we used thresholding and then used contour refinement logics to select the most appropriate mask, giving us this result:

2) Cars and Vehicles come in various shapes, sizes and colors. Training a model that generalizes well on all sorts of vehicles is quite difficult.

As you can see, the mask in this case is inappropriate. It missed out areas near the rear wheel of the car. Perhaps adding more images of trucks, vans (and other vehicles where the model fails to give good results) to the dataset might give a boost.

3) Lack of contrast between the car and the surrounding area tends to give bad results. In the image below, both the background and car are black. As a result, it’s quite difficult to accurately identify which pixels belong to the car. We can see, this results in the fact that the generated mask does not include the spoiler and the open door.

4) Extremely zoomed-in images are difficult to work with because with the numerous reflections on the car, it is hard to distinguish between the background and the reflection on the car. We need to keep the reflections but remove the background. The image below demonstrates the case:

We were able to largely tackle this issue by adding more such ‘zoomed-in’ images of reflective cars into our dataset.

5) Removing the background from images taken from the interior of a car. This in particular is extremely challenging due to the fact there are multiple occlusions and small areas that have to be masked. We tackled this by using another deep learning model that processes the output of the first model, further refining the result.

As you can see, the background is visible through multiple exclusive regions - window glass, windshield, between the steering wheel and dashboard. Masking out each of these regions while keeping the interior car parts is difficult, and as a result, the model fails to mask the small region between the steering wheel and dashboard.

6) Windows are transparent. So, while generating a cutout of the car, we need to make sure that the background seen through window glasses and windshield are not visible. Similar to problem 5, we used another deep learning model in the pipeline that masks out the background visible through transparent surfaces.

7) The final and probably the most important problem that we currently face is refining the edges of the cutout/mask. In almost all the above examples, you can see the generated output isn’t perfect, i.e., it isn’t pixel perfect. Either some minute portion of the background slips through or the mask cuts out some parts of the car. Multiple methods have been tried to tackle this problem - used novel edge detection methods such as BDCN, contour refinement and others. However, none of these seem to give consistently good results. Perhaps the key to solve this is using a dice_score + cross entropy based loss + custom boundary weighted loss. Or maybe, using higher resolution (1024x1024) input images instead of low-res (320x320) images will do the trick.