Intelligent Home Control System

Han Yu (hy532)
Yingjie Li (yl2988)

Demonstration Video

Introduction

In this project, we designed a system to control the electrical appliances and arm the house with a tracking camera. There are two parts in the system. In the ‘Controller’ part, users can turn on the lights and the fan by touching the screen. The temperature and humidity of the house show on the ‘Controller’ page. In the ‘Security’ part, users are required to set the PIN for their house. With the correct PIN, users can arm or disarm the house. When the house is in ‘Arm’ status, magnetic door sensor turns on. Once the door is open, the camera will start to track the face of intruder, taking pictures and sending an alert email to the user.

Generic placeholder image

Project Objective:

  • Design the interface on piTFT.
  • Arm the house with a tracking camera in the security part.
  • Control the electrical appliance like lights and fans in our house in the controller part.

Design and Testing

Hardware Design

In our design, we separate the power supply for RPi and the tracking camera to make sure the power supply we use is stable. We use GPIO.6, GPIO.23, GPIO.16 to control three lights, GPIO.19 to control the fan and GPIO.12 and GPIO.13 to control the servos we use for tracking camera. GPIO.5 and GPIO.26 is used as input GPIO. GPIO.26 is used to sense the status of the door and GPIO.5 is used to read in the data for temperature and humidity sensor.

The schematic

Generic placeholder image

The hardware PWM (to control the servos)

To get a more stable control of our servos using PWM, we use the hardware PWM in RPi. The RPi does support the hardware PWM but only several pins are configurable. We use GPIO.12 and GPIO.13 to control the servos by hardware PWM. The hardware PWM is more stable than the software PWM as the software PWM will result in significant wiggle and shiver on the pulse as the RPi is not a real-time system and the RPi will deal with many other processes at the same time. So, in our design, we use the hardware PWM to control the servos.

The power supply

The power supply for the fan and the servos should be separated from the RPi. In our first attempt to combine the control and the security function, we use the 5V on RPi to power the servos and the fan, the servo broke down. The TA told us that the fan needs the current to power it and so does the servos. However, the current provided by the RPi is small and fixed, if you want to power the servos and the fan at the same time, the current will be unstable and may cook something unexpected. So, in our design, we separated the power supply.

Software Design

1. Interface Design

We created three interfaces in our project including the homepage, the controller page and the security page. On the home page, we will show the date and two choices ‘Security’ and ‘Controller’ for the user to choose from.

Generic placeholder image

Fig.1 Home page

If we choose the ‘Security’, the page will turn to the security page and prompt the user to enter the PIN to arm or disarm your house.

Generic placeholder image

Fig.2 Security page

You can navigate to home page by pressing the ‘HOME’ button.

When you choose ‘Controller’ button, you can navigate to the controller page on which you can send your commands to the electrical appliance in your house. The temperature and the humidity of the house will refresh every time you click on it.

Generic placeholder image

Fig.3 Controller page

2. Electrical Appliance Controlling

In this function, you will know the temperature and the humidity in the house by clicking the space where shows the temperature and humidity. The data will be read from GPIO.5, the temperature and humidity sensor. What’s more, you can turn on and turn off the light in different room by press the corresponding buttons shown on the screen. Additionally, you can turn on and turn off the air conditioner (the fan) via the interface.

3. House Security

There are two statuses of the house, one is ‘Arm’ and the other one is ‘Disarm’. When used for the first time, you can press ‘*’ to set your own PIN number and end with ‘#’ to confirm your PIN. Then, every time you want to change the status of the house, there will be a prompt prompting you to enter your password and end your entering with ‘#’, you are required to type in the right PIN to change the status. When you choose the ‘Arm’ status, the door sensor is ready. Once the door is open, the tracking camera will work and take pictures of the intruder. When you choose ‘Disarm’, you are required to enter the authentic PIN. Once the PIN is verified authentic, the house is in the status of ‘Disarm’ and when the door is open, the tracking camera will not work.

4. Intruder Tracking

The camera we use is hold by a rotating platform of two servos. One is used to rotate horizontally and the other one is used to rotate vertically. For the first time we use the servos, we should calibrate them. Since the rotation angle is controlled by the duty cycle of the PWM sent to it, we can change the duty cycle of the PWM signal and find the angles it rotates corresponding to the duty cycle. One thing to mention here is the duty cycle corresponding to an angle will change some time.

Then, after we calibrate the servo, we can start to compose the Python code for the tracking camera. We use OpenCV to realize the recognition of human's face and track the face. First, the camera will capture a face and calculate the center coordinate of the face. Then it will compare the face's center coordinate with the center coordinate of the frame. Then the difference will be used to adjust the duty cycle of the PWM signal sent to the servo so that the servo can rotate in the corresponding angles. The recognized human’s face will always be located in the center of the frame.

5. Email Sending

After we captured the pictures we save them locally. When we save five images, we will send all of them to the user and clear them all locally. The camera captures one picture every 50 times tracking the face. We use SMTP in Python code to send email. We use Gmail to send and receive emails in our code. The images will be sent to the users as attachments of the email.

Testing nodes for the performance

1. Control of lights and the fan

After building the hardware part, we test the control function first. We set the GPIO for the device at HIGH to turn them on and at LOW to turn them off. Every time the command is executed successfully, we will get a feedback message in our terminal.

2. The tracking camera and picture captured

First, we tested the servo we use for the camera. We tested them to calibrate them. The direction of the servo is controlled by the duty cycle of the signal sent to it. So, for the first thing, we wrote a Python code to generate the signal with different duty cycles and recorded the corresponding performance of the servo to find the relation between the duty cycle and the rotation angle of the servo.

Then, we composed the Python code for the camera. First, the camera should capture the picture and recognize the human face in the picture and mark it out.

Combine the code for camera with the servo control code. when the object marked in the picture moves, the camera will move with it, marking it continuously to capture the picture. The pictures will be captured every ten times marked.

3. Sending the email

After the camera can capture and save the pictures, we tested the code for sending an email with pictures attached and an ‘Intruder’ message in the body of the email. We use Gmail to send and receive the email.

Results

Our system can display most of the functions expected at the beginning. There are mainly three parts in our system.
First, for the homepage, the date and a ‘WELCOME HOME!’ will show on the screen. There are two buttons on the screen, one is ‘Security’ and the other one is ‘Controller’. When you press ‘Security’, the page will turn to the ‘Security’ page. When you press ‘Controller’, the page will jump to the ‘Controller’ page.

Generic placeholder image

Fig.4 the model house

Second, for the ‘Security’ page, you can arm or disarm your house by easily touching on this page. To change the status of your house, you should first enter the authentic PIN to identify yourself as the master of the house. In ‘Arm’ mode, the tracking camera is ready and once the door is open, the camera will start to capture and track the face. The user will receive email to alert him of the intruder. In ‘Disarm’ mode, the camera will not work even when the door is open.
Generic placeholder image Generic placeholder image Generic placeholder image

Fig.5 the security page with prompt prompting to enter the PIN                   Fig.6 the camera tracking the face                                           Fig.7the alert email received by the user

Third, for the ‘Controller’ page, you can easily control the electrical appliance in the house. You can turn on or turn off your lights and the fan in your house by touching the screen.
Generic placeholder image

Fig.8 the controller page to control your lights

Conclusion

We finished our project successfully after 6 weeks. The results we get basically completed our expected goals. we encountered and solved many problems during the 6 weeks.

For the first thing, in hardware design, we cooked one of our servos because of the wrong power supply. We powered the servos and the fan with 5V on RPi at the same time. Since the current provided by the pin is very small while the fan and the servo need more current to start, the results of the connection may be unexpected. We changed the power supply to an external power supply after being suggested by the TA and it worked as expected.

Additionally, the servo we use may change its corresponding duty cycle some time. In many cases, every time we rebooted our system, the servo always acted strangely in an unexpected way. We thought it was cooked again but when after we get help from TA, we found that the servo is good, but the corresponding duty cycle of the angles may change for some time. So, every time we reboot our system, we should first check the servo to make sure it works as expected.

At our first attempt to run our system on piTFT, we found the piTFT just did not respond to our touch. We checked our code and hardware connection step by step and finally found that that’s because we occupied the GPIO.18 which is used to acquire the touch to information from the piTFT. We solved this problem by changing the GPIO we used to connect our hardware components.

We also found our system had a great latency at the first attempt, which is uncommon for our system. We tried to comment out the function of the GPIO we use to test which GPIO caused the latency. Finally, we found that the GPIO which reads data from Temperature and Humidity Sensor has the greatest latency as the speed for data transferring is very slow. We set every click will allow the data to refresh once. However, when the data is refreshing, the entire process will be blocked, thus there will be a lot of latency.

Future Work

We will try to make the temperature and the humidity refresh automatically without causing latency in our system. To do this, we can try to run the data reading process in the background and communicate with the main process.

We can design a Web interface or mobile phone APP to make it possible to control the system remotely.

We can add more sensors and functions to our system like the PIR motion sensor and a buzzer to alarm the intruder.

With machine learning, we can add a face recognition function to our system which can recognize the user’s face and the stranger’s face. We can build a library and train the recognition model, then use the model to recognize.

Work Distribution

The project cannot be completed successfully without anyone’s help. We have a very enjoyable cooperation with each other.

Generic placeholder image

Project group picture

Generic placeholder image

Han Yu

hy532@cornell.edu

The software design, including piTFT interface design, camera tracking design and security page design. Assembled the model house.

Generic placeholder image

Yingjie Li

yl2988@cornell.edu

The hardware design and connection. Composed the Python code for sending emails. Wrote the weekly reports and final report. Assembled the model house.

Acknowledgement

During the project, Professor Joseph Skovira helped a lot with our project. He provided us with components and gave us a lot of useful suggestions. TAs in ECE 5725 including Xitang Zhao, Mei Yang and Joao Pedro Carvao also helped us a lot. They cared much about our programs and did solve us lots of problems.

Parts List

Total: $32.28

References

PiCamera Document
SG90 Micro Servo Datasheet
Bootstrap
Pigpio Library
R-Pi GPIO Document
Temperature and Humidity Sensor (AM2302)
Magnetic Door Switch Wiring
Python Multiprocessing
OpenCV: Tracking

Code Appendix

# Final Project: homepageupdate.py
# Updated date: 12/05/2018
# Authors: Han Yu (hy532), Yingjie Li (yl2988)
# Description: Display welcome home page, controller page, and security page
#              Controller page is used for controlling household eletrical appliances, including
#              lights, air conditioner and temperature and humidity sensor.
#              Security page is used for home security. Owner can set the pin for home security
#              system, and arm or disarm their house by entering the pin number. When house is
#              armed, magnetic door sensor will work. Once it detetcts someone break in the house,
#              camera will start to work, tracking the person and sending email to the owner of the house.


import pygame
from pygame.locals import *
import os
import time
from time import sleep
import math
import RPi.GPIO as GPIO
import thread
import threading
import Adafruit_DHT
import smtplib
from email.MIMEMultipart import MIMEMultipart
from email.MIMEText import MIMEText
from email.MIMEBase import MIMEBase
from email import encoders
import cv2
import numpy as np
import pigpio


# colors
black= 0, 0, 0
white= 255, 255, 255

# size
size= width, height= 320, 240

os.putenv('SDL_VIDEODRIVER', 'fbcon') # Display on piTFT
os.putenv('SDL_FBDEV', '/dev/fb1')
os.putenv('SDL_MOUSEDRV', 'TSLIB') #Track mouse clicks on piTFT
os.putenv('SDL_MOUSEDEV', '/dev/input/touchscreen')

pygame.init()
pygame.mouse.set_visible(False)

# set the screen size and color
screen = pygame.display.set_mode(size)
screen.fill(white)
pygame.display.update()

global flag_home
global flag_controller
global flag_security
global pin
global pin_enter
global status
global isArm
global isEnter
global isSetting
global isCorrect
global isOn1
global isOn2
global isOn3
global currentposv
global currentposh
global pi
global flag_door
global temp
global hu

flag_home= True
flag_controller= False
flag_security= False
pin= []
pin_enter= []
status= 'Disarm'
isArm= False
isEnter= False
isSetting= False
isCorrect= False
isOn1= False
isOn2= False
isOn3= False
currentposv= 85000
currentposh= 75000
flag_door= False
temp= 24.5
hu= 22.3

GPIO.setmode(GPIO.BCM)

# fan
GPIO.setup(19, GPIO.OUT)
GPIO.output(19, GPIO.HIGH)

# lights
GPIO.setup(16, GPIO.OUT)
GPIO.output(16, GPIO.LOW)
GPIO.setup(6, GPIO.OUT)
GPIO.output(6, GPIO.LOW)
GPIO.setup(23, GPIO.OUT)
GPIO.output(23, GPIO.LOW)

# door switch
GPIO.setup(26, GPIO.IN, pull_up_down=GPIO.PUD_UP)

# physical bail-out button
GPIO.setup(27, GPIO.IN, pull_up_down=GPIO.PUD_UP)


# create welcome home page
def homepage():
	screen.fill(white)

	# create 'welcome home!' text
	font= pygame.font.SysFont('Times New Roman', 30, True)
	wel_surface= font.render('Welcome Home!', True, black)
	wel_rect= wel_surface.get_rect(center=(160, 120))
	screen.blit(wel_surface, wel_rect)

	# create 'security' and 'controller' buttons
	font= pygame.font.SysFont('Times New Roman', 20)
	two_buttons = {'Security':(80,200), 'Controller':(240,200)}
	for my_text, text_pos in two_buttons.items():
		button_surface = font.render('%s'%my_text, True, black)
		button_rect = button_surface.get_rect(center=text_pos)
		screen.blit(button_surface, button_rect)

	# create date
	font= pygame.font.SysFont('Times New Roman', 25)
	date= time.strftime("%B %d, %Y") 
	date_surface= font.render(date, True,black)
	date_rect= date_surface.get_rect(center=(160, 40))
	screen.blit(date_surface, date_rect)
	
	# quit button
	font= pygame.font.SysFont('Times New Roman', 20, True)
	quit_surface= font.render('X', True, black)
	quit_rect= quit_surface.get_rect(center=(300, 20))
	screen.blit(quit_surface, quit_rect)
	
	#update display
	pygame.display.flip()

# create controller page
def controllerpage():
	global temp
	global hu
	
	screen.fill(white)

	# create 'Controller' title
	font= pygame.font.SysFont('Times New Roman', 25, True)
	contr_surface= font.render('Controller', True, black)
	contr_rect= contr_surface.get_rect(center=(160, 20))
	screen.blit(contr_surface, contr_rect)

	# create display of temperature and humidity
	font= pygame.font.SysFont('Times New Roman', 15)
	Temp_Humi= {'Temperature: ':(50, 50), '*C':(140, 50), 'Humidity: ':(230, 50), '%':(300, 50)}
	for mytext, textpos in Temp_Humi.items():
		TempHumi_surface= font.render('%s'%mytext, True, black)
		TempHumi_rect= TempHumi_surface.get_rect(center=textpos)
		screen.blit(TempHumi_surface, TempHumi_rect)
	temp_surf= font.render(str(temp), True, black)
	temp_rect= temp_surf.get_rect(center=(110, 50))
	hu_surf= font.render(str(hu), True, black)
	hu_rect= hu_surf.get_rect(center=(280, 50))
	screen.blit(temp_surf, temp_rect)
	screen.blit(hu_surf, hu_rect)

	# create 'Light' text
	font= pygame.font.SysFont('Times New Roman', 20, True)
	light_surface= font.render('Light', True, black)
	light_rect= light_surface.get_rect(center=(80, 80))
	screen.blit(light_surface, light_rect)

	# create 'Air Conditioner' text
	font= pygame.font.SysFont('Times New Roman', 20, True)
	air_surface= font.render('Air Conditioner', True, black)
	air_rect= air_surface.get_rect(center=(240, 80))
	screen.blit(air_surface, air_rect)

	# create light buttons
	font= pygame.font.SysFont('Times New Roman', 15)
	lightbuttons= {'Dining Room':(80, 120), 'Bedroom':(80, 150), 'Living Room':(80, 180)}
	for mytext, textpos in lightbuttons.items():
		lb_surface= font.render('%s'%mytext, True, black)
		lb_rect= lb_surface.get_rect(center=textpos)
		screen.blit(lb_surface, lb_rect)

	# create air conditioner buttons
	font= pygame.font.SysFont('Times New Roman', 15)
	fanbuttons= {'ON':(240, 140), 'OFF':(240, 180)}
	for mytext, textpos in fanbuttons.items():
		fan_surface= font.render('%s'%mytext, True, black)
		fan_rect= fan_surface.get_rect(center=textpos)
		screen.blit(fan_surface, fan_rect)

	# create 'home' button
	font= pygame.font.SysFont('Times New Roman', 15, True)
	home_surface= font.render('HOME', True, black)
	home_rect= home_surface.get_rect(center=(280, 220))
	screen.blit(home_surface, home_rect)

	pygame.display.flip()

# create security page
def securitypage(sta, text, length):
	screen.fill(white)

	# create 'Security' title
	font= pygame.font.SysFont('Times New Roman', 25, True)
	sec_surface= font.render('Security', True, black)
	sec_rect= sec_surface.get_rect(center=(240, 20))
	screen.blit(sec_surface, sec_rect)

	# create 'home' button
	font= pygame.font.SysFont('Times New Roman', 15, True)
	home_surface= font.render('HOME', True, black)
	home_rect= home_surface.get_rect(center=(280, 220))
	screen.blit(home_surface, home_rect)

	# create status
	font= pygame.font.SysFont('Times New Roman', 15)
	status= {'Status: ': (220, 60), sta: (270, 60) }
	for mytext, textpos in status.items():
		sta_surface= font.render('%s'%mytext, True, black)
		sta_rect= sta_surface.get_rect(center=textpos)
		screen.blit(sta_surface, sta_rect)

	# create 'ARM' and 'DISARM' buttons
	font= pygame.font.SysFont('Times New Roman', 20)
	buttons= {'ARM': (240, 120), 'DISARM': (240, 170) }
	for mytext, textpos in buttons.items():
		button_surface= font.render('%s'%mytext, True, black)
		button_rect= button_surface.get_rect(center=textpos)
		screen.blit(button_surface, button_rect)

	# create number buttons
	font= pygame.font.SysFont('Times New Roman', 20)
	numpad= {'1': (40, 70), '2': (80, 70), '3': (120, 70),
	       '4': (40, 110), '5': (80, 110), '6': (120, 110),
	       '7': (40, 150), '8': (80, 150), '9': (120, 150),
	       '*': (40, 190), '0': (80, 190), '#': (120, 190)}
	for mytext, textpos in numpad.items():
		numpad_surface= font.render('%s'%mytext, True, black)
		numpad_rect= numpad_surface.get_rect(center=textpos)
		screen.blit(numpad_surface, numpad_rect)

	# create display bar
	pygame.draw.rect(screen, black, (20, 10, 120, 40), 1)

	# create text in display bar
	font= pygame.font.SysFont('Times New Roman', 10)
	if len(text) != 0:
		for index in range(len(text)):
			bar_surface= font.render('%s'%text[index], True, black)
			bar_rect= bar_surface.get_rect(topleft=(20, 15+index*15))
			screen.blit(bar_surface, bar_rect)

	# display the entering pin number in a security way
	font= pygame.font.SysFont('Times New Roman', 15)
	if length != 0:
		for i in range(length):
			pin_surface= font.render('*', True, black)
			pin_rect= pin_surface.get_rect(topleft=(20+i*5, 20))
			screen.blit(pin_surface, pin_rect)

	pygame.display.flip()

# response to entering pin number
def pin_res(num):
	global pin
	global pin_enter
	global status
	global isArm
	global isEnter
	global isSetting
	global isCorrect

	# set the pin number
	if (num == '*'):
		isEnter= True
		if (len(pin) != 0):
			text= ['Please enter the PIN number', 'ended with pound sign']
			securitypage(status, text, 0)
			if (num == '#'):
				isEnter= False
				if (pin != pin_enter):
					text= ['PIN number is wrong' 'please enter again']
					securitypage(status, text, 0)
					pin_enter= []
				else:
					text= ['Please enter the', 'new PIN number']
					securitypage(status, text, 0)
					isCorrect= True
					pin_enter= []
			else:
				if (num != '*' and num != '#' and isEnter):
					pin_enter.append(num)
					securitypage(status, [], len(pin_enter))

		if (isCorrect or len(pin) == 0):
			text= ['Please set the PIN number', 'ended with pound sign']
			securitypage(status, text, 0)
			pin= []
			isSetting= True # user is setting the pin number

	# enter pin number
	if (not isSetting):
		if (num == '#'):
			if (pin != pin_enter):
				text= ['PIN number is wrong', 'please enter again']
				securitypage(status, text, 0)
				pin_enter= []
			else:
				isEnter= False
				isCorrect= True
				text= ['Success!']
				securitypage(status, text, 0)
				pin_enter= []
		else:
			if (num != '*' and num != '#' and isEnter):
				pin_enter.append(num)
				securitypage(status, [], len(pin_enter))

	# set pin number
	else:
		if (num == '#'):
			isEnter= False
			isSetting= False
			text= ['PIN number has been set!']
			securitypage(status, text, 0)
		else:
			if (num != '*' and num != '#' and isEnter):
				pin.append(num)
				securitypage(status, [], len(pin))

def sendemail(filepath, n):
	msg= MIMEMultipart()

	# subject
	msg['Subject']= 'Alert'

	# body
	body= 'INTRUDER!'
	msg.attach(MIMEText(body, 'plain'))

	#attachment
	if (filepath != ''):
		while n != 0:
			filename= filepath + "/img" + str(n) + ".jpg"
			attachment= open(filename, 'rb')
			part = MIMEBase('application', 'octest-stream')
			part.set_payload((attachment).read())
			encoders.encode_base64(part)
			part.add_header('Content-Disposition', 'attachment', filename=filename)
			msg.attach(part)
			n -= 1

	# set SMTP
	server = smtplib.SMTP('smtp.gmail.com', 587)
	server.starttls()
	server.set_debuglevel(1)
	server.login("yingjieli1201@gmail.com","lyj961201")
	text = msg.as_string()
	server.sendmail("yingjieli1201@gmail.com","yhan96967@gmail.com", text)
	server.quit()
	time.sleep(1)
	print ("text sent")

# door open
def GPIO26_callback(channel):
    global flag_door
    if isArm:
        flag_door= True

# num_step is the distance that servo supposed to move
# direction: -1 for moving right, 1 for moving left
def move_horizontal(num_step, direction):
	global currentposh
	fullRight= 120000 # duty cycle is 12%, servo turns full right
	fullLeft= 30000 # duty cycle is 3%, servo turns full left
	step= (fullRight- fullLeft) / 480
	currentposh= currentposh + step * num_step * direction
	if currentposh > fullRight:
		currentposh= fullRight
	if currentposh < fullLeft:
		currentposh= fullLeft
	pi.hardware_PWM(13, 50, currentposh)

# num_step is the distance that servo supposed to move
# direction: -1 for moving up, 1 for moving down
def move_vertical(num_step, direction):
	global currentposv
	fullUp= 60000 # duty cycle is 6%, servo turns full up
	fullDown= 110000 # duty cycle is 11%, servo turns full down
	step= (fullDown- fullUp) / 360
	currentposv= currentposv + step * num_step * direction
	if currentposv < fullUp:
		currentposv= fullUp
	if currentposv > fullDown:
		currentposv= fullDown
	pi.hardware_PWM(12, 50, currentposv)

# tracking face
def tracking(frameShow):
	# create a VideoCapture calss object to stream video from pi camera
	vcap= cv2.VideoCapture(0)

	# create classifier to recognize human face
	hf= cv2.CascadeClassifier('haarcascade_frontalface_alt.xml')

	# check if VideoCapture class object is created successfully
	if (not vcap.isOpened()):
		print ("Error: Can't find Pi Camera")
		quit()
	else:
		print("Success: Pi Camera is open")

	videoWidth= vcap.get(cv2.cv.CV_CAP_PROP_FRAME_WIDTH)
	videoHeight= vcap.get(cv2.cv.CV_CAP_PROP_FRAME_HEIGHT)

	print("Default vedio resolution: " + str(int(videoWidth)) + "*" + str(int(videoHeight)))
	
	global pi
	global currentposh
	global currentposv
	pi= pigpio.pi()
	pi.hardware_PWM(12, 50, 85000)
	pi.hardware_PWM(13, 50, 75000)
	currentposh= 75000
	currentposv= 85000

	flag_quit= False

	try:
		time= 1
		n= 1
		while (vcap.isOpened() and not flag_quit):
			# read video by frame
			# isRead is boolean, return true when frame is read correctly
			# frame is the image of every frame
			isRead, frame= vcap.read()

			# if the video cannot be read, we break the while loop
			if (not isRead):
				break;

			# resize the frame
			frame= cv2.resize(frame, (160, 120), interpolation= cv2.INTER_CUBIC)

			# convert image to grayscale
			gray= cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

			# detect human face
			faces= hf.detectMultiScale(gray, 1.3, 5)

			filepath= "/home/pi/final_project/monitor"

			# the center coordinate of frame
			frameCenter= [80, 60]
			

			for x, y, w, h in faces:
				cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 255, 0), 2)

				# record the picture of face every 10 times detecting faces
				if time%50 == 0 and n<6:
					img= frame[y-h:y+2*h, x-w:x+2*w]
					cv2.imwrite(filepath + "/img" + str(n) + ".jpg", img)
					n += 1
					print("n= %d" %n)
				if n == 6:
					flag_quit= True
					sendemail(filepath, n-1)
					time= 1
					while n != 1:
						n -= 1
						os.remove(filepath + "/img" + str(n) + ".jpg")
					break
				time += 1

				# the center of the face detected
				faceCenter= [x+w/2, y+h/2]

				moveRL= threading.Thread()
				moveUD= threading.Thread()

				# move right or left
				# face is on the right side of the screen, camera moves right
				if faceCenter[0] > frameCenter[0]:
					direction= -1
				# face is on the left side of the screen, camera moves left
				else:
					direction= 1
				num_step= abs((int)(faceCenter[0] - frameCenter[0])/10)
				if num_step > 0:
					moveRL= threading.Thread(target= move_horizontal, args= (num_step, direction))

				# move up or down
				# face is on the lower side of the screen, camera moves down
				if faceCenter[1] > frameCenter[1]:
					direction= 1
				# face is on the upper side of the screen, camera moves up
				else:
					direction= -1
				num_step= abs((int)(faceCenter[1] - frameCenter[1])/10)
				if num_step > 0:
					moveUD= threading.Thread(target= move_vertical, args= (num_step, direction))

				moveRL.start()
				moveUD.start()

				moveRL.join()
				moveUD.join()


			if frameShow:
				cv2.imshow('Security', frame)

			key= cv2.waitKey(1) & 0xFF

			# if 'q' on the keyboard pressed or if touch 'DISARM', quit
			if (key == ord("q")):
				break
		
		# cleanup and close all the windows
		vcap.release()
		cv2.destroyAllWindows()		

	except KeyboardInterrupt:
		# cleanup and close all the windows
		vcap.release()
		cv2.destroyAllWindows()

	# cleanup and close all the windows
	vcap.release()
	cv2.destroyAllWindows()

try:
	homepage()
	while flag_home:
		# physical button to bail out
		if not GPIO.input(27):
			break
                
		for event in pygame.event.get():
			if (event.type == MOUSEBUTTONUP):
				pos= pygame.mouse.get_pos()
				x, y= pos
				print pos
				
				# quit button
				if y<40 and y>0:
					if x<320 and x>280:
						flag_home= False

				if y<220 and y>180:
					# when touch 'Controller'
					if x<280 and x>200:
						controllerpage()
						flag_controller= True
						while flag_controller:
							# if we need to update temp and humidity every loop, uncomment this
							#controllerpage()
							for event in pygame.event.get():
								if (event.type == MOUSEBUTTONUP):
									pos= pygame.mouse.get_pos()
									x, y= pos

									# when touch light buttons
									if x<120 and x>40:
										# Dining room
										if y<130 and y>110:
											if (not isOn1):
												GPIO.output(16, GPIO.HIGH)
												print("Light in dining room turns on")
												isOn1= True
											else:
												GPIO.output(16, GPIO.LOW)
												print("Light in dining room turns off")
												isOn1= False
										# Bedroom
										if y<160 and y>140:
											if (not isOn2):
												GPIO.output(6, GPIO.HIGH)
												print("Light in bedroom turns on")
												isOn2= True
											else:
												GPIO.output(6, GPIO.LOW)
												print("Light in bedroom turns off")
												isOn2= False
										# Living room
										if y<190 and y>170:
											if (not isOn3):
												GPIO.output(23, GPIO.HIGH)
												print("Light in living room turns on")
												isOn3= True
											else:
												GPIO.output(23, GPIO.LOW)
												print("Light in living room turns off")
												isOn3= False

									# when touch air conditioner button
									if x<260 and x>220:
										# when touch 'ON'
										if y<160 and y>120:
											GPIO.output(19, GPIO.LOW)
											print("Fan turns on")
										# when touch 'OFF'
										if y<200 and y>160:
											GPIO.output(19, GPIO.HIGH)
											print("Fan turns off")

									# when touch Temp & humidity part
									if y<60 and y>40:
										if x<320 and x>30:
											# temperature and humidity sensor connects GPIO5
											hu, temp= Adafruit_DHT.read_retry(Adafruit_DHT.AM2302, 5) 
											temp= round(temp, 2)
											hu= round(hu, 2)
											controllerpage()
                                                                                
									# when touch 'HOME' button, return to home page
									if y<240 and y>200:
										if x<300 and x>260:
											flag_controller= False
											homepage()

					# when touch 'Security'
					if x<120 and x>40:

						flag_security= True
						securitypage(status, [], 0)

						# when door sensor detects someone breaks in, start tracking
						while flag_security:
							if flag_door:
								tracking(False)
								flag_door= False
                                                            
							for event in pygame.event.get():
								if (event.type == MOUSEBUTTONUP):
									pos= pygame.mouse.get_pos()
									x, y= pos
									securitypage(status, [], 0)

									# when touch number pad
									if y<80 and y>60:
										if x<50 and x>30:
											pin_res('1')
										elif x<90 and x>70:
											pin_res('2')
										elif x<130 and x>110:
											pin_res('3')
									if y<120 and y>100:
										if x<50 and x>30:
											pin_res('4')
										elif x<90 and x>70:
											pin_res('5')
										elif x<130 and x>110:
											pin_res('6')
									if y<160 and y>140:
										if x<50 and x>30:
											pin_res('7')
										elif x<90 and x>70:
											pin_res('8')
										elif x<130 and x>110:
											pin_res('9')
									if y<200 and y>180:
										if x<50 and x>30:
											pin_res('*')
										elif x<90 and x>70:
											pin_res('0')
										elif x<130 and x>110:
											pin_res('#')

									# when touch 'ARM' or 'DISARM' button
									# 'ARM'
									if y<140 and y>100:
										if x<280 and x>220:
											if (isCorrect):
												status= 'Arm'
												isCorrect= False
												isArm= True
												securitypage(status, [], 0)
												GPIO.add_event_detect(26, GPIO.FALLING, callback=GPIO26_callback, bouncetime=300)

											else:
												isEnter= True
												text= ['Please enter thr PIN number!', 'Then press the button again']
												securitypage(status, text, 0)

									# 'DISARM'
									if y<190 and y>150:
										if x<280 and x>220:
											if (isCorrect):
												status= 'Disarm'
												isCorrect= False
												isArm= False
												securitypage(status, [], 0)
											else:
												isEnter= True
												text= ['Please enter thr PIN number!', 'Then press the button again']
												securitypage(status, text, 0)

									# when touch 'HOME' button
									if y<240 and y>200:
										if x<300 and x>260:
											flag_security= False
											homepage()

except KeyboardInterrupt:
	GPIO.cleanup()

GPIO.cleanup()