Xinyue Zhou (xz677), Jingyi Hu (jh2677)
Our project aims to design a parking barrier modeling system with automatic number plate recognition.In our project, we used two Raspberry Pi. One for controlling the car stuck with printed number plate to move linearly toward the barrier and stop if it is not allowed to enter. The other is for controlling two Infrared IR sensors to trigger camera and servo respectively, and for image processing and recognition. The whole parking system runs as shown in the figure below: when the car moves close to the barrier within a certain distance, a sensor will react and transfer a signal to the RPi. Then, RPi will control the camera to take a photo, and processing the image get from Rpi camera using openCV and transfer individual number or letter images to a sequence of characters using OCR so that the input number plate can be checked whether matched the stored number plates string in “database”. After processing with OCR, the number plate will be recognized by RPi, then displayed on the PiTFT. If it matches with the number plates in the if-statement we set before, the barrier will lift up to let the car in, or the car will be barriered out. After the car passes through the barrier and reaches a certain distance, another sensor behind the barrier will react and transfer a signal to RPi. And RPi will control the barrier to lift down.
The general idea is that several car number plates are prestored, meaning car with these number plates are allowed to enter the park (barrier will open) while other number plates not matched the system “database” will not be allowed entering (barrier keeps closed). Additionally, the car moves automatically with changeable number plates to demonstrate the whole function of the system.
The hardware design of our parking system consists of two Raspberry Pi, a station, a barrier, two IR sensors , a modeling car and a camera. We first started with building the fundamental subsystem composed of a RPi, a camera, two IR sensors and a servo which is used to control the barrier.
For the camera, we connected it with RPi using a cable with proper length.
For IR sensors, We used GP2Y0D810Z0F, which is a distance measuring sensor to detects objects between 2 cm and 10 cm (0.8″ and 4″) away, composed of an integrated combination of PD (photo diode), IRED (infrared emitting diode) and signal processing circuit.For proper installation and use, we first soldered the pin portion and main sensor portion together, then, connected them with RPi respectively. Among three pins of each sensor, the left one needs to be grounded. The middle pin should be connected to the power supply (3V3 output on RPi), while the right one should be connected to a 1k resistance, then to GPIO to provide control signal. We chose GPIO19 and GPIO26 respectively as input pin since they are free available. When the sensor not detecting any object it input a high signal (“1”) into Pi, while once it detect an object a low signal (“0”) will be imported.
As For the servo, we used Parallax Standard Servo (#900-00005) because it is controlled by position rather than impulse time while encoding, and it can hold any position between 0 and 180 degrees.In our project, we needed the barrier lifted 90-degree up and lifted down to the initial position. Thus, the standard servo satisfied our expectation, and the quick-start circuit shows as below (using GPIO13 as output to control the servo) :
Besides the above subsystem, we also made a barrier using Cast Acrylic material and laser cut technology. Cast Acrylic is a thermosetting plastic with a superior surface finish, flatness and optical properties. It is also a hard and brittle material which is sensitive to stress concentrations. This is most evident if parts are small or thin, thus is proper to be made into a barrier. And we chose laser cutting mainly because of its easier workholding and reduced contamination of workpiece (since there is no cutting edge which can become contaminated by the material or contaminate the material). Precision may be better, since the laser beam does not wear during the process. There is also a reduced chance of warping the material that is being cut, as laser systems have a small heat-affected zone.Based on these, we designed a barrier model in CAD, which is 25cm long, drilling holes in the middle to fix camera, and also drilling holes on the one side to be connected with servo, as shown in the following figure:
Then, we designed a station to integrate the above parts into a whole. We used a paper box to accommodate battery with carrier, power bank and breadboard, and stuck the PiTFT on the top of the box to make the display clear to be seen. In addition, we used a plank to fix the position of two IR sensors and the paper box. Since the speed of the car is almost fixed, and the time cost of image processing and recognition will not vary significantly, we can determine the distance from the first sensor to the paper box to ensure enough time for RPi to analyze and control servo. In the same way, we can also determine the distance from the paper box to the second sensor to guarantee the whole car to pass through before the barrier lifts down.
As for the car, we used the one we assembled in previous lab. And we printed four different number plates with bold font and bold border for easier localization and recognition as follows:
To make the car move linearly, we attached a second Raspberry Pi on it and connected them with two Parallax Continuous Rotation Servos (#900-00008). The quick-start circuit is same as standard servo (we chose GPIO12 and GPIO13 as output of control signal), while the difference is that continuous rotation servo is controlled by impulse time.We used hardware PWM method to control two servos since they are not exactly same, it is inconvenient to set frequency or duty cycle to make them rotate with same speed. Therefore, we just gave same calibrate signals to these two servos, and used a screwdriver to adjust the rotation speeds as consistent as possible.
Image capture use Raspberry Pi Camera
from picamera import PiCamera camera = PiCamera()
camera.start_preview() sleep(2) camera.capture('image.jpg') camera.stop_preview()
sudo apt-get install libopencv-dev python-opencv
import cv2to import opencv module into python code.
Read in captured image; Change RGB to grayscale; Noise removal with iterative bilateral filter(removes noise while preserving edges); Histogram equalisation for better results; Morphological opening with a rectangular structure element; Image subtraction(Subtracting the Morphed image from the histogram equalised Image); Thresholding the image; Applying Canny Edge detection; Dilation to strengthen the edges; Finding Contours in the image based on edges; Creating the kernel for dilation; Finding Contours in the image based on edges; Sort the contours based on area; Loop over our contours: Approximate the contour with 6% error; Select the contour with 4 corners as the number plate’s counter; Obtain the 4 corner position, achieve number plate localization; Masking the background as black background Segment the number plate region from the whole image according to 4 corner coordinates;
Thresholding the segmented grayscale image again to black and white; Created a structuring elements as kernel; Morphological opening(erosion followed by dilation) with a rectangular structure element to remove noise;
Flip the foreground and background to ensure foreground is now "white" and the background is "black"; Grab the (x, y) coordinates of all pixel values that are greater than zero; Compute a rotated bounding box that contains all coordinates; Obtained the rectangle rotating angle (in range [-90,0)) using cv2.minAreaRect(); If rectangle rotates clockwise add 90 degrees to the angle; else just take the inverse of the angle to make it positive; Rotate the image matrix according to the angle obtained above
After above image processing steps, the result image is like following figure shown and is clear enough to be passed for recognition. Therefore, the aim of image processing achieved.
Number plate recognition
In this project, Optical Character Recognition(OCR) engine was used for recognition, that is transfer the characters in image to text. In this way we can check whether the character on number plate is same as the one stored in database to decide whether it is an legal number plate allowing entrance.
In order to use the OCR engine, we first installed the open source tesseract-ocr package which contains OCR engine
-libtesseract and command line program
-tesseract by running:
sudo apt-get install tesseract-ocr
After that, we can use command line in format below to run tesseract for image recognition:
tesseract imagename outputbase [-l lang] [--oem ocrenginemode] [--psm pagesegmode] [configfiles...]
In this project we used command line below to recognize image processed result “output2.jpg” and store the recognition result into “file.txt” with specifying the language as english and page segmentation modes as treat the image as a single text line:
tesseract output2.jpg file -l eng -psm 7
Then, result in txt file can be read out for following logical control.
Logical control and display using pygame
The whole system logic is that when the front end IR sensor detect the car it will trigger the pi camera to take a photo and then Raspberry pi process the image and user OCR engine to recognize the number plate and then check the recognition result with those stored in database, if the number plate matched with one in database, this is a legal number plate, servo will be controlled to lift up parking barrier to allow car entrance. At the same time, the recognition result will be displayed using pygame on PiTFT as well as the image that captured by pi camera and a message said “Allow pass”. And then, when car leaving and detected by the second IR sensor, servo will be triggered to lay down the barrier and now the system welcomes the next car coming for check with PiTFT displaying a welcome screen. However, if the recognized result was found that this is an illegal number plate, then the servo will be controlled to not lift up the barrier, and on PiTFT, instead an allowance message but a message said “Not allow” will be displayed so that “driver” need to stop the car and leave. After wait for a few seconds, the whole system welcomes the next car coming for check.
Python to achieve above whole system logic can be demonstrated by the two logic flow chart below:
The first chart is the logic flow chart for whole system outer while loop
The second one is logic flow chart for servo control function inner while loop
Hardware PWM for servo
sudo pigpiodbefore running python codes in order to launch the pigpio library as a daemon and use it correctly.
In this project, in order to have more accurate recognition result, we set the resolution of pi camera to a relatively large value 2592*1944, but this will cause the image processing time become long (13s). Our solution to this problem is to segment the image into smaller one containing the number plate after image capture but before image processing. This method is feasible because position of numblate on car is fixed, position of camera on barrier is also fixed, hence the position of number plate on the captured image is also relatively settled. And this method can reduce the image processing time to around 5s.
sudo pip install pytesseract
from PIL import Image import pytesseract im=image.open("image.jpg") text=pytesseract.image_to_string(im, lang='eng')
import subprocess cmd="tesseract output2.jpg file -l eng -psm 7" print subprocess.check_output(cmd, shell=True)
In order to run the car linear, we at first apply symmetric duty cycle to GPIO12 and GPIO13 to get PWM. However, even though the hardware PWM signal is stable, two servo rotating with different speed due to these two servo is not identical. And because we want to run the car at low speed, the servo speed is highly affected by hardware condition of servo, hence, if we use software to adjust the duty cycle to achieve approximate same speed of two servo, this point can easily changed when restart the servos. Therefore, we finally decided to use hardware PWM to provide stable calibration signal (high pause 1.5ms), and use screwdriver to calibrate the two servo to approximately same speed. This method is more efficient than calibrating using software.
We were successfully able to meet all our intended goals for the Automatic Number Plate Recognition system. After running car-control python code and adjustment by screwdriver, the car can move almost linearly towards the barrier with proper speed. The first IR sensor will trigger the camera to take a photo which will be processed by RPi through following steps: number plate localization, background remove, character enhancement, noise remove and skew correction. Then, the result image will be recognized and changed into a text. We prepared four different printed number plates for testing. Two of them are prestored (FBR 1449 and KAU 3881), while the other two are illegal (HBV9320 and 611 BUC). If the text content is consistent with a number plate we prestored before, the barrier will be lifted up, which means that the car with this number plate is allowed to pass. In the meanwhile, the text content after recognition, the initial photo taken by camera, and indication words (‘Allow Pass’) will be displayed on the PiTFT. After that, the car keeps moving and reaches the second IR sensor, triggering the barrier to be lifted down. If the text after recognition is not consistent with any in the prestored ‘database’, the barrier keeps closed. The text content, the initial photo and indication words will also be displayed, while the words will change to be ‘Not Allow’. Above description is one loop of two possible situations that can be happen. After each loop, the whole system will be automatically set back to the initial statement, which means that it is ready for waiting a next car to come.
If we had more time to continue working on this Automatic Number Plate Recognition project, one of the first features we would expand upon would be the exact linear movement of a car. Since the two continuous rotation servos are not exactly same, it is almost impossible to make the car move linearly without any error. Thus it is better and much more convenient to use a line reader to achieve it, which is also benefit to localize two IR sensors, the paper box, and fix a more precise position of taking photos, reducing the time cost of image processing consequently.
In addition, we can also build a connection between two Raspberry Pi to realize following goals: when the car with legal number plate passes through the barrier and reaches the second IR sensor, besides controlling the barrier to be lifted down, the main RPi should also send a signal to the car-control RPi to stop the car. In the other case, when the car with illegal number plate moving towards the barrier, after recognition, the main RPi should send a signal to the car-control RPi to stop the car as well. After adding these functions, there is no need for users to press the stop button.
Designed the overall software architecture. Tested and debugged of all.
Designed the overall hardware architecture. Tested and debugged of all.
|Raspberry Pi Model 3 B+||$35||2||$70|
|Servos, Resistors and Wires||Provided in lab|