Jingyi Shen(js2825) and Yunhe Shao(ys566)
ECE 5725 Final Project
Fall 2021
We designed a smart guest management system that is intended to help public buildings such as apartment complexes and office buildings to improve management efficiency.
Unlike modern smart locks for homes, our systems provide administrators with more flexibility and with a wide range of functionality for various daily needs, especially in the covid era, such as the acquisition of express and takeaway information, management of personnel entry and exit, activation of building alarms, etc..
Keywords: YOLOv3 tiny, motion sensor, hall effect sensor, telebot
Part 1: Identification of delivery package
We all know that an office building will receive a large number of express delivery every day. If the door management system can automatically identify the express brand and save relevant information, it can improve management efficiency.
Step 1: Use labelimg for image annotation, generate express image data set and corresponding annotation file
Step 2: Use YOLOv3 for model training based on the Darknet framework, save configuration files and the best weights
Step 3: Use the motion sensor, when an object is detected to move, call the camera to take a picture, and transmit it to the background. Use the trained model for real-time express type detection
The following examples show some of the types of express that we can detect:
Part 2: Access for Maintenance personnel
Maintenance personnel, such as plumbers, weeders, cleaners, etc., need to enter the building regularly, so we set the gate password. If the password is entered correctly, you can enter, and three incorrect entries will not be able to enter, and will be reported to the administrator.
Enter the 4-digit password:
If correct, the door will be open and the administrator will be notified
If you enter the error three times, it cannot be entered, and the administrator will receive the relevant information
Part 3: Storage and Retrieval of Food Delivery
Food delivery package is totally different from express package. The storage time of food delivery is relatively short. When the delivery arrives, administrator need to know its location and delivery time immediately, so that we can deal with it if there is a problem later.
Therefore, we use hall effect sensor, which can detect the presence of a magnetic field and is often used for positioning objects or measuring rotation. We use two magnets. When the food is placed on the shelf, gravity causes the two magnets to approach each other, and the sensor returns to a high voltage, indicating that it has detected that there is a takeaway delivery.
Step 1: Food Delivery Package Recognition (have one in Part 1)
Step 2: the food delivery staff will place the food delivery in the designated location
Step 3: Administrator can check in real time whether there is food delivery on the delivery shelf and the delivery time
Part 4: Alarm System
When an emergency occurs, the administrator can trigger the alarm to urge the people in the building to evacuate as soon as possible. This feature will be shown in the next section
Our complete design is based on telebot and telegram. Telebot is a bot framework for Telegram Bot API. This package provides the best of its kind API for command routing, inline query requests and keyboards, as well as callbacks. We set up an administrator group chat on telegram and set the group chat main menu. The administrator can send commands on the menu to implement all the functions we mentioned in the previous part.
/start: The administrator will receive a message: Welcome to the ultimate control panel
/history: The administrator will receive a message about all the express delivery and its classification information (Fedex, UPS, DHL)
/latest: The administrator will get the information of the most recently received package, which is convenient for the administrator to check
/door: The password function will be turned on, and maintenance workers can enter the building by entering the password. When someone successfully enters the building, the administrator will receive: "the mechanics have entered; If the password is entered incorrectly three times, the door cannot be closed and the administrator will receive a message: Potential risk of unknown person trying to enter
/alarm: When the administrator sent this command, the building’s alarm was turned on to remind people to evacuate quickly
/alarm_stop: When the danger has been eliminated or all personnel have been evacuated, the administrator sends this command to disarm the alarm
/food: The administrator will receive the current status of the takeaway shelf, whether the takeaway has been delivered and the delivery time
The picture on the right is an example of the functions we have implemented. The administrator receives different information in real time by sending different commands
The building administrator can remotely control through telegram, and can receive real-time delivery information, takeaway information, and personnel entry and exit information. At the same time, it can remind building personnel to evacuate in an emergency. After testing, our system has strong stability, so we basically realized all the functions in the proposal, and finally realized a relatively complete smart guest management system.
Problem 1: The camera cannot detect the express in real time. At first, we tried to directly call yolov3 to perform target detection on real-time video to determine whether there is an express delivery, but the real-time and accuracy did not meet our expectations, so we changed to using a motion sensor to take pictures when an object was detected in motion. Object recognition on the picture. In the future, we plan to replace yolov3 with other methods, such as feature detection to identify the type of express. In contrast, there should be a smaller delay
Problem 2: When the administrator enters the history command, it should return the type and quantity of all the current express. However, when the same type of express appears multiple times, the history will not be repeated. In the follow-up, we will modify the content of the reply to ensure that it will be returned every time there is a new package, and the order will be sorted according to the arrival order of the package
Problem 3: The password update module still needs to be improved. It is currently in the program. It is forbidden to enter when the same user's password is detected three times incorrectly. In the future, we will realize that the administrator can manually update the password through commands to ensure the controllability of the door password
js2825@cornell.edu
Responsible for Delivery Identification, Access for Maintenance Personnel, and alarm system
ys566@cornell.edu
Design Storage and Retrieval of Food Delivery module, our complete system design and system testing
import telebot import RPi.GPIO as GPIO import time from PIL import Image import os from collections import defaultdict from pygame import mixer import threading from picamera import PiCamera from pad4pi import rpi_gpio import datetime bot = telebot.TeleBot("2022892271:AAFua__pKYcbUQIhlOuwMKUZs-bWewbwDrY", parse_mode=None) # You can set parse_mode by default. HTML or MARKDOWN result_list = defaultdict(list) carrier_list = ["Fedex", "DHL", "Food"] list_pointer = 0 mixer.init() play_flag = True #-----------------------------lablel init label_stamp = 100 #-----paths-------------------------------- result_storage_path = "/home/pi/PiImage/" image_name = str(label_stamp) + ".jpg" output_path = "/home/pi/PiImage/OutTxt/" output_directory = "/home/pi/PiImage/OutTxt/" #----------------------------gpio part #----------------------------movement sensor GPIO.setmode(GPIO.BCM) GPIO.setup(4, GPIO.IN) #----------------------------hall GPIO.setwarnings(False) GPIO.setup( 20, GPIO.IN) #----------------------------camera init camera = PiCamera() camera.resolution = (3280, 2464) #---------------------------pygame initialization mixer.init() mixer.music.load("/home/pi/good_alarm.mp3") #TODO: setup pass code def get_date_taken(path): return Image.open(path)._getexif()[36867] #---------------------------gpio init operation GPIO.setup(26, GPIO.OUT) led_pin = GPIO.PWM(26, 1) led_pin.start(50) #------------------------pin pad setup # Setup Keypad KEYPAD =[["1","2","3","A"], ["4","5","6","B"], ["7","8","9","C"], ["0","F","E","D"]] password = "5725" res = 1 # same as calling: factory.create_4_by_4_keypad, still we put here fyi: ROW_PINS = [22,27,17,21] # BCM numbering; Board numbering is: 7,8,10,11 (see pinout.xyz/) COL_PINS = [26,19,6,5] # BCM numbering; Board numbering is: 12,13,15,16 (see pinout.xyz/) factory = rpi_gpio.KeypadFactory() # Try keypad = factory.create_4_by_3_keypad() or # Try keypad = factory.create_4_by_4_keypad() #for reasonable defaults # or define your own: keypad = factory.create_keypad(keypad=KEYPAD, row_pins=ROW_PINS, col_pins=COL_PINS) class KeyStore: def __init__(self, message): #list to store them self.pressed_keys ='' self.message = message #function to clear string def clear_keys(self): self.pressed_keys = self.pressed_keys.replace(self.pressed_keys,'') def store_key(self,key): global res global password if key=='D': #printing the sequence of keys. if self.pressed_keys == password: print("right password, the door is opened, please enter!") bot.reply_to(self.message, "The mechanics have entered") return 100 else: if res>=3: bot.reply_to(self.message, "Potential risk of unknown personel trying to enter") print("wrong password for 3 times, please contact the administrator") password = "sdsg" return 101 else: print("wrong password, please try again! (it is the "+ str(res)+" try)") res += 1 self.clear_keys() else: self.pressed_keys += key def pin_pad_input(): pass #-----load delivery handlers-------------------------------- def log_request(path, complete_flag): global list_pointer result_list.clear() box = [] for filename in os.listdir(path): with open(os.path.join(path, filename), 'r') as f: # open in readonly mode i = 0 j = 0 local_result_list = [] # if os.path.join(path, filename).getsize == 0: # break searchlines = f.readlines() if len(searchlines) is 0: continue last_line = searchlines[-1] image_path = searchlines[0].split(':')[0] print(image_path) if searchlines[1] != 0: predictions_num = int(searchlines[1]) - 1 for i in range(predictions_num): box = searchlines[1 + 1 + i].split(',') i += 1 for j in range(predictions_num): local_result_list.append(str(searchlines[2 + i + 1 + j].split(':')[0])) if complete_flag: if local_result_list[j] == 'fedex': result_list[image_path].append("fedex") elif local_result_list[j] == 'dhl': result_list[image_path].append("dhl") elif local_result_list[j] == 'ups': result_list[image_path].append("ups") elif local_result_list[j] == 'food delivery': result_list[image_path].append("food") list_pointer += 1 else: return local_result_list if searchlines[2 + i + 1 + j] is last_line: break j += 1 list_pointer += 1 print(result_list) if complete_flag: return result_list f.close() # /home/pi/Downloads/3.jpeg: Predicted in 0.970428 seconds. # 2 # Box 0, 0.686100, 0.433947, 0.482435 ,0.397016 # Box 1, 0.705939, 0.436732, 0.403356 ,0.375360 # ups: 89% def check_bounding_boxes(box_list): #TODO: check bounding boxes functions pass def emergency_alarm(): # Remember the current and previous button states current_state = False prev_state = True global play_flag # Load the sounds # If button is pushed, light up LED while play_flag: if (current_state == False) and (prev_state == True): mixer.music.play() current_state = True # time.sleep(10) # play_flag = False def open_door(message): #key oeprations keys = KeyStore(message) # store_key will be called each time a keypad button is pressed status_code = keypad.registerKeyPressHandler(keys.store_key) def retrieve_images(time_list): #TODO: send back lastest image pass def detection(): global label_stamp global output_path global image_name #TODO: trigger sensor and detections while True: if GPIO.input(4): image_name = str(label_stamp) + ".jpg" camera.capture(result_storage_path + image_name) output_path_full = output_path + str(label_stamp) + ".txt" os.system('cd /home/pi/darknet-nnpack/ && ./darknet detector test weight/t.data cfg/yolov3tiny_custom_trained.cfg weight/t.weights {0}{1} > {2}'.format(result_storage_path, image_name, output_path_full)) label_stamp += 1 time.sleep(5) @bot.message_handler(commands=['start', 'help']) def send_welcome(message): bot.reply_to(message, "Welcome to the ultimate control panel.") @bot.message_handler(commands=['苟利国家生死以', '膜蛤']) def send_welcome(message): bot.reply_to(message, "岂因祸福避趋之") @bot.message_handler(commands=['alarm']) def send_welcome(message): bot.reply_to(message, "door open process will be triggered") status_code = open_door(message) # while(play_flag): # usr_in = 1 # usr_in = 100 # led_pin.ChangeFrequency(int(usr_in)) # time.sleep(0.5) @bot.message_handler(commands=['latest']) def get_delivery_status(message): result_list_reply = log_request(output_directory, 0) if result_list_reply is None: bot.reply_to(message, "There are no logs currently") return 0 bot.reply_to(message, result_list_reply) @bot.message_handler(commands=['history']) def get_history(message): empty_message = 0 result_list_reply = (log_request(output_directory, 1)) if result_list_reply is None: empty_message = 1 bot.reply_to(message, "There are no logs currently") return 0 # reply_list = ''.join(str(e) for e in str(list(result_list_reply.values()))) # print(result_list) # bot.reply_to(message, reply_list) result_list_reply = "Here are the delivery carriers that have been to the lobby: \n" if 'fedex' in result_list.values(): result_list_reply += "Fedex\n" if 'ups' in result_list.values(): result_list_reply += "UPS\n" if 'dhl' in result_list.values(): result_list_reply += "DHL\n" bot.reply_to(message, result_list_reply) @bot.message_handler(commands=['door']) def get_door(message): door_staus = sensorCallback(sensor_channel) if door_staus: bot.reply_to(message, 'The door is closed!!!') else: bot.reply_to(message, 'The door is open!!!') @bot.message_handler(commands=['food']) def get_food(message): timestamp = time.time() stamp = datetime.datetime.fromtimestamp(timestamp).strftime('%H:%M:%S') if(GPIO.input(20) == False): bot.reply_to(message, "There are food delivery on shelf" + str(stamp)) else: bot.reply_to(message, "Food Lot is Empty as of " + str(stamp)) @bot.message_handler(func=lambda message: True) def echo_all(message): bot.reply_to(message, message.text) class DetectionThread(threading.Thread): def __init__(self): threading.Thread.__init__(self) def run(self): while True: detection() class ReplyThread(threading.Thread): def __init__(self): threading.Thread.__init__(self) def run(self): bot.infinity_polling() thread1 = DetectionThread() thread2 = ReplyThread() thread1.start() thread2.start()