Smart Guest Management System

Jingyi Shen(js2825) and Yunhe Shao(ys566)
ECE 5725 Final Project
Fall 2021


Introduction

We designed a smart guest management system that is intended to help public buildings such as apartment complexes and office buildings to improve management efficiency.

Unlike modern smart locks for homes, our systems provide administrators with more flexibility and with a wide range of functionality for various daily needs, especially in the covid era, such as the acquisition of express and takeaway information, management of personnel entry and exit, activation of building alarms, etc..

Keywords: YOLOv3 tiny, motion sensor, hall effect sensor, telebot


Generic placeholder image

Project Objective:

  • A smart guest management system using camera and keypad to enable building administrators to remotely obtain real-time personnel access at the gate, express delivery and package storage
  • Remote information acquisition based on telegram
  • Modular design that embraces future improvements
  • Imitate the actual door management system, including special situation alarms, storage of express delivery and other practical functions

Design

Part 1: Identification of delivery package

We all know that an office building will receive a large number of express delivery every day. If the door management system can automatically identify the express brand and save relevant information, it can improve management efficiency.

Generic placeholder image PiCamera
Generic placeholder image Motion Sensor PIR

Step 1: Use labelimg for image annotation, generate express image data set and corresponding annotation file

Step 2: Use YOLOv3 for model training based on the Darknet framework, save configuration files and the best weights

Step 3: Use the motion sensor, when an object is detected to move, call the camera to take a picture, and transmit it to the background. Use the trained model for real-time express type detection



The following examples show some of the types of express that we can detect:

Generic placeholder image UPS Package
Generic placeholder image Fedex Package
Generic placeholder image Food Delivery Package

Part 2: Access for Maintenance personnel

Maintenance personnel, such as plumbers, weeders, cleaners, etc., need to enter the building regularly, so we set the gate password. If the password is entered correctly, you can enter, and three incorrect entries will not be able to enter, and will be reported to the administrator.

Generic placeholder image Keypad Connection

Enter the 4-digit password:

If correct, the door will be open and the administrator will be notified

If you enter the error three times, it cannot be entered, and the administrator will receive the relevant information







Part 3: Storage and Retrieval of Food Delivery

Food delivery package is totally different from express package. The storage time of food delivery is relatively short. When the delivery arrives, administrator need to know its location and delivery time immediately, so that we can deal with it if there is a problem later.

Therefore, we use hall effect sensor, which can detect the presence of a magnetic field and is often used for positioning objects or measuring rotation. We use two magnets. When the food is placed on the shelf, gravity causes the two magnets to approach each other, and the sensor returns to a high voltage, indicating that it has detected that there is a takeaway delivery.

Generic placeholder image
Generic placeholder image Hall effect Sensor

Step 1: Food Delivery Package Recognition (have one in Part 1)

Step 2: the food delivery staff will place the food delivery in the designated location

Step 3: Administrator can check in real time whether there is food delivery on the delivery shelf and the delivery time







Part 4: Alarm System

When an emergency occurs, the administrator can trigger the alarm to urge the people in the building to evacuate as soon as possible. This feature will be shown in the next section




Complete front-end and back-end system design

Our complete design is based on telebot and telegram. Telebot is a bot framework for Telegram Bot API. This package provides the best of its kind API for command routing, inline query requests and keyboards, as well as callbacks. We set up an administrator group chat on telegram and set the group chat main menu. The administrator can send commands on the menu to implement all the functions we mentioned in the previous part.

Generic placeholder image Generic placeholder image Generic placeholder image Generic placeholder image

/start: The administrator will receive a message: Welcome to the ultimate control panel

/history: The administrator will receive a message about all the express delivery and its classification information (Fedex, UPS, DHL)

/latest: The administrator will get the information of the most recently received package, which is convenient for the administrator to check

/door: The password function will be turned on, and maintenance workers can enter the building by entering the password. When someone successfully enters the building, the administrator will receive: "the mechanics have entered; If the password is entered incorrectly three times, the door cannot be closed and the administrator will receive a message: Potential risk of unknown person trying to enter

/alarm: When the administrator sent this command, the building’s alarm was turned on to remind people to evacuate quickly

/alarm_stop: When the danger has been eliminated or all personnel have been evacuated, the administrator sends this command to disarm the alarm

/food: The administrator will receive the current status of the takeaway shelf, whether the takeaway has been delivered and the delivery time

The picture on the right is an example of the functions we have implemented. The administrator receives different information in real time by sending different commands






Result

The building administrator can remotely control through telegram, and can receive real-time delivery information, takeaway information, and personnel entry and exit information. At the same time, it can remind building personnel to evacuate in an emergency. After testing, our system has strong stability, so we basically realized all the functions in the proposal, and finally realized a relatively complete smart guest management system.


Problem & Future works

Problem 1: The camera cannot detect the express in real time. At first, we tried to directly call yolov3 to perform target detection on real-time video to determine whether there is an express delivery, but the real-time and accuracy did not meet our expectations, so we changed to using a motion sensor to take pictures when an object was detected in motion. Object recognition on the picture. In the future, we plan to replace yolov3 with other methods, such as feature detection to identify the type of express. In contrast, there should be a smaller delay

Problem 2: When the administrator enters the history command, it should return the type and quantity of all the current express. However, when the same type of express appears multiple times, the history will not be repeated. In the follow-up, we will modify the content of the reply to ensure that it will be returned every time there is a new package, and the order will be sorted according to the arrival order of the package

Problem 3: The password update module still needs to be improved. It is currently in the program. It is forbidden to enter when the same user's password is detected three times incorrectly. In the future, we will realize that the administrator can manually update the password through commands to ensure the controllability of the door password


Work Distribution

Generic placeholder image

Project group picture

Generic placeholder image

Jingyi Shen

js2825@cornell.edu

Responsible for Delivery Identification, Access for Maintenance Personnel, and alarm system

Generic placeholder image

Yunhe Shao

ys566@cornell.edu

Design Storage and Retrieval of Food Delivery module, our complete system design and system testing


Parts List

Total: $70


References

PiCamera Document
Keypad
Motion Sensor PIR Document
R-Pi GPIO Document
Hall effect Sensor Document

Code Appendix

import telebot
import RPi.GPIO as GPIO
import time
from PIL import Image
import os
from collections import defaultdict
from pygame import mixer
import threading
from picamera import PiCamera
from pad4pi import rpi_gpio
import datetime



bot = telebot.TeleBot("2022892271:AAFua__pKYcbUQIhlOuwMKUZs-bWewbwDrY", parse_mode=None) # You can set parse_mode by default. HTML or MARKDOWN

result_list = defaultdict(list)
carrier_list = ["Fedex", "DHL", "Food"]
list_pointer = 0

mixer.init()
play_flag = True

#-----------------------------lablel init
label_stamp = 100

#-----paths--------------------------------
result_storage_path = "/home/pi/PiImage/"
image_name = str(label_stamp) + ".jpg"
output_path = "/home/pi/PiImage/OutTxt/"
output_directory = "/home/pi/PiImage/OutTxt/"

#----------------------------gpio part

#----------------------------movement sensor
GPIO.setmode(GPIO.BCM)
GPIO.setup(4, GPIO.IN)

#----------------------------hall
GPIO.setwarnings(False)
GPIO.setup( 20, GPIO.IN)

#----------------------------camera init
camera = PiCamera()
camera.resolution = (3280, 2464)

#---------------------------pygame initialization
mixer.init()
mixer.music.load("/home/pi/good_alarm.mp3")

#TODO: setup pass code
def get_date_taken(path):
    return Image.open(path)._getexif()[36867]

#---------------------------gpio init operation
GPIO.setup(26, GPIO.OUT)
led_pin = GPIO.PWM(26, 1)
led_pin.start(50)


#------------------------pin pad setup
# Setup Keypad
KEYPAD =[["1","2","3","A"],
        ["4","5","6","B"],
        ["7","8","9","C"],
        ["0","F","E","D"]]
password = "5725"
res = 1
# same as calling: factory.create_4_by_4_keypad, still we put here fyi:
ROW_PINS = [22,27,17,21] # BCM numbering; Board numbering is: 7,8,10,11 (see pinout.xyz/)
COL_PINS = [26,19,6,5] # BCM numbering; Board numbering is: 12,13,15,16 (see pinout.xyz/)

factory = rpi_gpio.KeypadFactory()

# Try keypad = factory.create_4_by_3_keypad() or 
# Try keypad = factory.create_4_by_4_keypad() #for reasonable defaults
# or define your own:
keypad = factory.create_keypad(keypad=KEYPAD, row_pins=ROW_PINS, col_pins=COL_PINS)

class KeyStore:
    def __init__(self, message):
        #list to store them
        self.pressed_keys =''
        self.message = message

    #function to clear string
    def clear_keys(self):
        self.pressed_keys = self.pressed_keys.replace(self.pressed_keys,'')

    def store_key(self,key):
        global res
        global password
        if key=='D':
            #printing the sequence of keys.
            if self.pressed_keys == password:
              print("right password, the door is opened, please enter!")
              bot.reply_to(self.message, "The mechanics have entered")
              return 100
            else:
              if res>=3:
                bot.reply_to(self.message, "Potential risk of unknown personel trying to enter")
                print("wrong password for 3 times, please contact the administrator")
                password = "sdsg"
                return 101
              else:
                print("wrong password, please try again! (it is the "+ str(res)+" try)")
              res += 1
            self.clear_keys()
        else:
            self.pressed_keys += key


def pin_pad_input():
    pass


#-----load delivery handlers--------------------------------
def log_request(path, complete_flag):
    global list_pointer
    result_list.clear()
    box = []
    for filename in os.listdir(path):
        with open(os.path.join(path, filename), 'r') as f:  # open in readonly mode
            i = 0
            j = 0
            local_result_list = []
            # if os.path.join(path, filename).getsize == 0:
            #     break
            searchlines = f.readlines()
            if len(searchlines) is 0:
                continue
            last_line = searchlines[-1]
            image_path = searchlines[0].split(':')[0]
            print(image_path)

            if searchlines[1] != 0:
                predictions_num = int(searchlines[1]) - 1
                for i in range(predictions_num):
                    box = searchlines[1 + 1 + i].split(',')
                    i += 1
                for j in range(predictions_num):
                    local_result_list.append(str(searchlines[2 + i + 1 + j].split(':')[0]))
                    if complete_flag:
                        if local_result_list[j] == 'fedex':
                            result_list[image_path].append("fedex")
                        elif local_result_list[j] == 'dhl':
                            result_list[image_path].append("dhl")
                        elif local_result_list[j] == 'ups':
                            result_list[image_path].append("ups")
                        elif local_result_list[j] == 'food delivery':
                            result_list[image_path].append("food")
                        list_pointer += 1
                    else:
                        return local_result_list
                    if searchlines[2 + i + 1 + j] is last_line:
                        break
                    j += 1

        list_pointer += 1
        print(result_list)

    if complete_flag:
        return result_list
    f.close()

# /home/pi/Downloads/3.jpeg: Predicted in 0.970428 seconds.
# 2
# Box 0, 0.686100, 0.433947, 0.482435 ,0.397016
# Box 1, 0.705939, 0.436732, 0.403356 ,0.375360
# ups: 89%

def check_bounding_boxes(box_list):
	#TODO: check bounding boxes functions
	pass

def emergency_alarm():
	# Remember the current and previous button states
	current_state = False
	prev_state = True
	global play_flag
	# Load the sounds

	# If button is pushed, light up LED
	while play_flag:
            if (current_state == False) and (prev_state == True):
                mixer.music.play()
                current_state = True


		# time.sleep(10)
		# play_flag = False

def open_door(message):
    #key oeprations
    keys = KeyStore(message)

    # store_key will be called each time a keypad button is pressed
    status_code = keypad.registerKeyPressHandler(keys.store_key)

def retrieve_images(time_list):
	#TODO: send back lastest image
	pass

def detection():
    global label_stamp
    global output_path
    global image_name
    #TODO: trigger sensor and detections
    while True:
        if GPIO.input(4):
            image_name = str(label_stamp) + ".jpg"
            camera.capture(result_storage_path + image_name)
            output_path_full = output_path + str(label_stamp) + ".txt"
            os.system('cd /home/pi/darknet-nnpack/ && ./darknet detector test weight/t.data cfg/yolov3tiny_custom_trained.cfg weight/t.weights {0}{1} > {2}'.format(result_storage_path, image_name, output_path_full))
            label_stamp += 1
            time.sleep(5)



@bot.message_handler(commands=['start', 'help'])
def send_welcome(message):
	bot.reply_to(message, "Welcome to the ultimate control panel.")

@bot.message_handler(commands=['苟利国家生死以', '膜蛤'])
def send_welcome(message):
	bot.reply_to(message, "岂因祸福避趋之")

@bot.message_handler(commands=['alarm'])
def send_welcome(message):
        bot.reply_to(message, "door open process will be triggered")
        status_code = open_door(message)

        # while(play_flag):
        #     usr_in = 1
        #     usr_in = 100
        #     led_pin.ChangeFrequency(int(usr_in))
        #     time.sleep(0.5)


@bot.message_handler(commands=['latest'])
def get_delivery_status(message):
        result_list_reply = log_request(output_directory, 0)
        if result_list_reply is None:
            bot.reply_to(message, "There are no logs currently")
            return 0
        bot.reply_to(message, result_list_reply)

@bot.message_handler(commands=['history'])
def get_history(message):
        empty_message = 0
        result_list_reply = (log_request(output_directory, 1))
        if result_list_reply is None:
            empty_message = 1
            bot.reply_to(message, "There are no logs currently")
            return 0
        # reply_list = ''.join(str(e) for e in str(list(result_list_reply.values())))
        # print(result_list)
        # bot.reply_to(message, reply_list)
        result_list_reply = "Here are the delivery carriers that have been to the lobby: \n"
        if 'fedex' in result_list.values():
            result_list_reply += "Fedex\n"
        if 'ups' in result_list.values():
            result_list_reply += "UPS\n"
        if 'dhl' in result_list.values():
            result_list_reply += "DHL\n"
        bot.reply_to(message, result_list_reply)

@bot.message_handler(commands=['door'])
def get_door(message):
        door_staus = sensorCallback(sensor_channel)
        if door_staus:
            bot.reply_to(message, 'The  door is closed!!!')
        else:
            bot.reply_to(message, 'The door is open!!!')

@bot.message_handler(commands=['food'])
def get_food(message):
        timestamp = time.time()
        stamp = datetime.datetime.fromtimestamp(timestamp).strftime('%H:%M:%S')
        if(GPIO.input(20) == False):
            bot.reply_to(message, "There are food delivery on shelf" + str(stamp))
        else:
            bot.reply_to(message, "Food Lot is Empty as of " + str(stamp))


@bot.message_handler(func=lambda message: True)
def echo_all(message):
    bot.reply_to(message, message.text)

class DetectionThread(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)

    def run(self):
        while True:
            detection()

class ReplyThread(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)

    def run(self):
        bot.infinity_polling()

thread1 = DetectionThread()
thread2 = ReplyThread()

thread1.start()
thread2.start()