Fetching Car Details using Computer Vision and OCR.

Task Description:

🎯Create a model that will detect a car in a live stream or video and recognize characters on the number plate of the car.

🎯Secondly, it will use the characters and fetch the owner’s information using RTO APIs.

🎯Create a Web portal where all this information will be displayed (using HTML, CSS, and JS)

Pre-Requisite:

a. Flask Installed on the server
b. “pytesseract” install on the system
c. Video in which we want to detect the car’s plate and retrieve info.
d. Basic knowledge of Image Processing
e. CV2 libraries and many more for this please click here.

And Finally, A cup of coffee….

As a part of this Project, you

1. Will detect a car
2. If a car is detected, will extract the number plate
3. If number plate detected extract the Data
4. Pass the extracted data to car-registraionAPI (It’s is Precreated)
5. Finally parse the data from API and show it on your OWN SITE.

Step1: Car Detection for Detecting a Car

We will detect our car using the LBPH algorithm.

Local Binary Patterns Histogram algorithm was proposed in 2006. It is based on local binary operator. It is widely used in facial recognition due to its computational simplicity and discriminative power.
The steps involved to achieve this are:

• creating dataset
• face acquisition
• feature extraction
• classification

Working of LBPH (Easy Explanation):

The LBP feature vector, in its simplest form, is created in the following manner:

• Divide the examined window into cells (e.g. 16x16 pixels for each cell).
• For each pixel in a cell, compare the pixel to each of its 8 neighbors (on its left-top, left-middle, left-bottom, right-top, etc.). Follow the pixels along a circle, i.e. clockwise or counterclockwise.
• Where the center pixel’s value is greater than the neighbor’s value, write “0”. Otherwise, write “1”. This gives an 8-digit binary number (which is usually converted to decimal for convenience).
• Compute the histogram, over the cell, of the frequency of each “number” occurring (i.e., each combination of which pixels are smaller and which are greater than the center). This histogram can be seen as a 256-dimensional feature vector.
• Optionally normalize the histogram.
• Concatenate (normalized) histograms of all cells. This gives a feature vector for the entire window.

But here is a Twist, We are using the LBPH for Car detection as it mostly used for Facial Recognition but according to some studies and research as well as self tested it is working Quite Good.

We can get the data set of various cars and train our data via LBPH. The more the data would be better would be our prediction of a car.

Our Data for LBPH

Similarly, We have used the LBPH to Detect the Car in a Video and here’s how it worked out for us!!

As we can See above our LBPH model is Doing good Work, Now let’s move to next step.

Step2: Number Plate Detection

Our step two contains two parts:

PART-1: First we have to use the Image Processing , Contours and Masking to Extract a Rectangular Plate and Crop it.

✔Resize the image to the required size and then grayscale it:
Resizing we help us to avoid any problems with bigger resolution images, make sure the number plate still remains in the frame after resizing. Gray scaling is common in all image processing steps. This speeds up other following process sine we no longer have to deal with the color details when processing an image.

✔ Using a bilateral filter (Blurring) will remove the unwanted details from an image:
In our case only the license plate is the useful information the rest are pretty much useless for our program. This useless information is called noise and it will be reduce this noise for us.

✔Canny edge detection:
Next, we are perform edge detection. There are many ways to do it here we are using Canny edge Detection provided by openCV
Syntax for Canny Edged Detection:
“python.destination_image = cv2.Canny(source_image, thresholdValue 1, thresholdValue 2)”
The Threshold Value 1 and Threshold Value 2 are the minimum and maximum threshold values. Only the “edges that have an intensity gradient more than the minimum threshold value and less than the maximum threshold value will be displayed”
Output will be Like:

✔Now we can start looking for contours on our image:

contours=cv2.findContours(edged.copy(,
                          cv2.RETR_TREE,
                          cv2.CHAIN_APPROX_SIMPLE)
contours = imutils.grab_contours(contours)
contours = sorted(contours,key=cv2.contourArea, reverse = True)[:10]
screenCnt = None

Once the counters have been detected we sort them from big to small and consider only the first 10 results ignoring the others. In our image the counter could be anything that has a closed surface but of all the obtained results the license plate number will also be there since it is also a closed surface.
To filter the license plate image among the obtained results, we will loop though all the results and check which has a rectangle shape contour with four sides and closed figure. Since a license plate would definitely be a rectangle four sided figure hence we are checking len(approx)==4 i.e. if having 4 Coordinates only.

for c in cnts:
    # approximate the contour
    peri = cv2.arcLength(c, True)
    approx = cv2.approxPolyDP(c, 0.018 * peri, True)
    # if our approximated contour has four Coordinates, then
    # we can assume that we have found our screen
    if len(approx) == 4:
        screenCnt = approx
        break

Once we have found the right counter we save it in a variable called screenCnt and then draw a rectangle box around it to make sure we have detected the license plate correctly, Finally this is How it looks after detecting contour and creating a Rectangle over it.

✔Masking the entire picture except for the place where the number plate is Detected:
Now that we know where the number plate is, the remaining information is pretty much useless for us. So we can proceed with masking the entire picture except for the place where the number plate is.

# Masking the part other than the number plate
mask = np.zeros(gray.shape,np.uint8)
new_image = cv2.drawContours(mask,[screenCnt],0,255,-1,)
new_image = cv2.bitwise_and(img,img,mask=mask)

PART-2: Finally, we have to pass the Cropped License Plate to the Pytesseract OCR and it will give the Text to us/(Character Recognition).

To detect the text. We can will be using “pytesseract” for character Recognition.

What is Pytesseract?( Very Brief)

Pytesseract or Python-tesseract is an Optical Character Recognition (OCR) tool for python. It will read and recognize the text in images, license plates, etc. Here, we will use the tesseract package to read the text from the given image.

Mainly, 3 simple steps are involved here as shown below:-

• Loading an Image saved from the computer or download it using a browser and then loading the same. (Any Image with Text).
• Binarizing the Image (Converting Image to Binary).
• We will then Pass the Image through the OCR system.

Please Follow this link to know more about “pytesseract” will help you a lot in understanding the core.

✔Segmenting the license plate:
We are now segmenting the plate out of the whole Masked image by cropping it and saving it as a new image. We can then use this image to detect the character in it with the help of pytesseract.

# Now cropping the Plate which is Detected above.
(x, y) = np.where(mask == 255)
(topx, topy) = (np.min(x), np.min(y))
(bottomx, bottomy) = (np.max(x), np.max(y))
Cropped = gray[topx:bottomx+1, topy:bottomy+1]

✔ Number Plate Recognition with the use of Pytesseract:
It is to actually read the number plate information from the segmented image. We will use the pytesseract package to read characters from image and to retrieve the text.

import re
table = str.maketrans('', '', string.ascii_lowercase)
# Using the Pytesseract for Character Recognition
string = pytesseract.image_to_string(Cropped, config='--psm 11')
text = re.sub('[^A-Za-z0-9]+', '', string)
 
#Removing all Small and Unwanted character character (Data Filtering)
s = text
result = s.translate(table)
final_text =result[-10:]
print(final_text)

Full code: To detect the car number plate and Print the Text recognized.

Running the Above code and Checking the Output:

As expected, our code has detected the number plate in the car image, cropped and converted it into text via OCR.

Step3: Using a Pre-created API for retrieving Vehicle Info.

API is the acronym for Application Programming Interface, which is a software intermediary that allows two applications to talk to each other. Each time you use an app like Facebook, send an instant message or check the weather on your phone, you’re using an API.

This time we need an API that will go to the website to retrieve car information when the car number is passed. On further research, we found a website that provides this service for free for up to 9 uses and then uses paid subscriptions.

How to use this API in General?

⚙ Go to http://www.carregistrationapi.in/ and Create a Account.

⚙ Now Go to the Highlighted Link.

⚙ Now we have to select the India as we are Detecting Indian Car License Plate.

⚙ Finally Enter your Username Which you registered while we created the Account and Enter the Number which we wan to Search.

⚙ This is how we will get the Data in XML Format.

**Note**: Above API only Gives 10 tries to Retrieve the Number so we may have to create different different account.

How to use this API to Automatically give us the Output (As soon as OCR retrieves the data)?

So do as what we have titled above, we need to create a python code that will go and retrieve data from this site a return back data in string type.

⚙ Go to http://www.carregistrationapi.in/ and Create a Account.

>>Create an ID on this site and save its username.
>>Rest all will be taken care by the code given below.

✔We are requesting to the “Api-url” directly passing the Username and Register Number.

✔As we saw earlier the data published by the API is in XML format, So we need to convert it into JSON.

✔Now Accessing the Data through as JSON as we normally do

Output by Above code will be as Follows:

Step-4: Creating a Web Site with Front-end and Back-end Functionality.

Below part was done by my team mate “Shivansh Srivastava” , Please check his blog to understand more about this website Creation.

As we have already created all the parts of our code(Till step-3)

🎯 Car Detection
🎯 License Plate Detection And Data Retrieval
🎯 Connecting to the API to Retrieve the Vehicle Info

Now we need to design a site that will provide these services for public use integrating All the steps.

We will be Using Flask Framework, HTML, CSS, and Javascript.

You can get the source code from the GitHub link. You guys have to make few changes if cloning the GITHUB Repo
✔ First, you guys will have to change the IP’s in ./templates/*
Above means, you have to change IP in all the files present in Template Directory.

✔ Secondly, you have you replace the all the places from “C:/Users/Rishabh/Desktop/task ARTH/sm task-8/numberplate_web-main/* ” to your own Location where you have cloned this Repo.

✔In prediction.py, You have to give own location where you have installed pytesseract.pytesseract.tesseract_cmd = r' Your OWN Location'.

✔In prediction.py, you have to go to a Function named “Car_info(text)” there you will be replacing “root2” with your own “API-USERNAME”(Which you created in Step-3).

Step-5: Finally We will be Accessing our Website using any Web Browser and our server IP.

I would like to thank my mentor “Mr.Vimal Daga” for giving us this Innovative task , Teaching us all about new-technologies used in the market with each and every Concept from the Basic to the core.

This task helped me as well as my Team members to create an interactive website that can be used by anyone to upload their car video and get the result.
It helped me a lot to clear my concepts and know more about computer vision, machine learning, and full-stack development.

Link for GitHub repository:

Rishbah-76/IndianCarInfo_Fetcher

Link of Me and My Team Members LinkedIN:

Rishabh(Me) — https://www.linkedin.com/in/rishabh-465a85185/
Shivansh Srivastava — https://www.linkedin.com/in/srivastava-shivansh/
Shreya Garg — https://www.linkedin.com/in/shreya-garg-1577241b7/
Shubhyansh rai — https://www.linkedin.com/in/shubhyansh-rai-749911136/