Magic Potter

A magic game based on Raspberry Pi
A Project By Huan Yang(hy468) and Hao Chen(hc795)


Demonstration Video


Introduction

Magic Potter is a player VS player game based on Raspberry Pi, each player will have a magic wand and wave it in front of the camera. The system will recognize the trajectory of the wand and tansform into the corresponding spell. There were a total of three rounds in which each player waved his wand in front of the camera in turn. To show the game more clearly, the battle process and the winner will display on the Pi TFT. And for the relationship between different spells, it needs players to explore. For each round, the stronger spell wins.


Project Objective

  • Achieve the recognition of the wand trajectory effciently
  • Show the game process on pi TFT in details
  • Corresponding led will shine when recognize the spells

  • Design

    At first, we planned to make a water gun, when the system recognize the corresponding trajectory, it will shoot out water to the user. However, after consideration, we afraid that the water may lead the system to short circuit, which is dangerous. And the first edition of freehand sketch is shown below. At last, we decided to make an adversarial game.


    Freehand sketch of the water gun

    This is a picture

    As for the whole system, it mainly has four parts and they are Raspberry Pi, the Raspberry camera, the Pi TFT and the LED part. The raspberry is responsable for processing the whonle pragram and issue instructions to various modules of the system. the Raspberry camera is to capture the video stream to the Raspberry pi. In addition, the Led part is to show the magic spell in a more clear way. Pi TFT is used to show the game process.

    And for the running process, at first the camera will capture video stream and pass it to Raspberry Pi. Then the Pi change the frames to gray scales and then we call a function called "cv2.HoughCircles" to detect circle (the reason is that the front end of wand is circle). And next it will compare the coordinates of circle in current frame with that in last frame. Finally the result of this comparison will give us an idea of the direction in which the wand is moving and print the corresponding spells. In addition the pygame module will show the game process on PiTFF.


    Overall schematic circuit diagram of Magic Potter

    This is a picture


    Drawings

    As for the overall structure, we use a box to cover the system parts, and the final demo is shown below.


    Overall structure of Magic Potter

    This is a picture


    Detailed assembling parts


    Pi Camera

    This is a picture


    PiTFT

    This is a picture


    LEDs

    This is a picture


    Testing

    After we finished the first edition demo. we have several steps to test the functions of the system. First of all, we used different speed to wave the magic wand to test the the recognition accuracy of the camera, then we found at low speed the trajectory could be correctly recognized. In addition, we tested that if the leds could separately light up corresponding to various trajectory. The resuls are satisfactory, which means we can add more action to the system, such as controlling a motor to water a plant. Thirdly we tested the the legibility of the content displayed by PiTFT, and tested if the physical button is working smoothly (we use physical button to restart the game). At last, we ran the whole system to see if the game is running correctly.


    Result

    As for the project result, we successfully achieve the functions we had envisioned. The system can implement the recognition of different trajectory of magic wand, and can successfully run the game. When the player move the wand in a slow and stable speed, the spells will be recognized precisely, and the game process shown on Pi TFT are detailed. Overall, the project results are satisfactory, and the deficiencies of the project can be further upgraded and adjusted in the later period.



    Future Work

    As a summary, we finally can smoothly run the game with two players. And the camera can successfully recognize the trajectory of wand in most cases. However, the recognition accuracy of the camera is still not enough, which we believe is that the performance of the currently used algorithm cannot meet our expected requirements. After exploration, we find if we capture more feature points in each frame and compare the current frame with the last one. And then calculate the coordinates' movement of similar feature points to obtain the direction of wand. In this method, it can get better results than that of current algorithm. So that in the future, we plan to update the algorithm, use a better camera and some brighter leds. All in all, we achieve the preconceived goals.

    Work Distribution

    Generic placeholder image

    Project group picture

    Generic placeholder image

    Huan Yang(hy468)

    hy468@cornell.edu

    Designed parts of the software architecture. Tested the whole system. Made parts of website, project report and demonstration video.

    Generic placeholder image

    Hao Chen(hc798)

    hc795@cornell.edu

    Designed parts of software architecture. Tested the whole system. Made parts of website, project report and demonstration video.


    Parts List

    Total: $60


    References

    PiCamera Document
    OpenCV
    Pygame
    Pigpio Library
    R-Pi GPIO Document

    Code Appendix

        import io
        from io import BytesIO
        import sys
        import pygame
        import os
        from pygame.locals import *
        from picamera import PiCamera
        import numpy as np
        import cv2
        import threading
        import math
        import time
        import pigpio
        import RPi.GPIO as GPIO
        
        GPIO.setmode(GPIO.BCM)
        GPIO.setup(5, GPIO.OUT)
        GPIO.setmode(GPIO.BCM)
        GPIO.setup(13, GPIO.OUT)
        GPIO.setmode(GPIO.BCM)
        GPIO.setup(19, GPIO.OUT)
        GPIO.setup(26, GPIO.OUT)
        GPIO.setup(23, GPIO.IN, pull_up_down=GPIO.PUD_UP)
        GPIO.setup(27, GPIO.IN, pull_up_down=GPIO.PUD_UP)
        
        os.putenv('SDL_VIDEODRIVER', 'fbcon')
        os.putenv('SDL_FBDEV', '/dev/fb0')
        os.putenv('SDL_MOUSEDRV', 'TSLIB')
        os.putenv('SDL_MOUSEDEV', '/dev/input/touchscreen')
        pygame.init()
        pygame.mouse.set_visible(False)
        
        WHITE = 255, 255, 255
        RED = 255, 0, 0
        GREEN = 0, 255, 0
        BLACK = 0, 0, 0
        
        my_buttons = {'Wizard1':(80,40), 'Wizard2':(240,40), 'Winner':(160,100)}
        winner = {(160,140):'None'}
        left_buttons = {(70,90):'None', (70, 130):'None', (70,170):'None', (70,210):'0'}
        right_buttons = {(250,90):'None', (250, 130):'None', (250,170):'None', (250,210):'0'}
        my_buttons_rect = {}
        my_font = pygame.font.Font(None, 35)
        my_font1 = pygame.font.Font(None, 20)
        my_font2 = pygame.font.Font(None, 30)
        screen = pygame.display.set_mode((320,240))
        screen.fill(BLACK)
        flag = True
        flag1 = False
        flag2 = False
        
        def GPIO5():#the defination of GPIO
            start = time.time()
            end = time.time() + 3
            while end - start > 0:
                GPIO.output(5, GPIO.HIGH)
                time.sleep(0.1)
                GPIO.output(5, GPIO.LOW)
                time.sleep(0.1)
                start = time.time()
            
            GPIO.output(5, GPIO.LOW)
            
        def GPIO13():
            start = time.time()
            end = time.time() + 3
            while end - start > 0:
                GPIO.output(13, GPIO.HIGH)
                time.sleep(0.1)
                GPIO.output(13, GPIO.LOW)
                time.sleep(0.1)
                start = time.time()
            
            GPIO.output(13, GPIO.LOW)
            
        def GPIO19():
            start = time.time()
            end = time.time() + 3
            while end - start > 0:
                GPIO.output(19, GPIO.HIGH)
                time.sleep(0.1)
                GPIO.output(19, GPIO.LOW)
                time.sleep(0.1)
                start = time.time()
            
            GPIO.output(19, GPIO.LOW)
        
        def GPIO26():
            GPIO.output(26, GPIO.HIGH)
            
        
        def Scan():#call Scan will run find wand
            global flag
            while flag:
                FindNewPoints()
        
        
        def FindNewPoints():#find the circle similar to the front end of wand shown in the video stream
            global old_frame,old_gray,p0,mask,color,ig,img,frame
            ig = [[0] for x in range(20)]
            TrackWand()
        
        def TrackWand():
            global old_frame,old_gray,p0,mask,color,ig,img,frame,flag,flag1
            color = (0,0,255)
            cam.capture(stream, 'jpeg')#capture a photo file to check the camera view
            data = np.fromstring(stream.getvalue(), dtype=np.uint8)
            old_frame = cv2.imdecode(data, 1)
            cv2.flip(old_frame,1,old_frame)
            old_gray = cv2.cvtColor(old_frame, cv2.COLOR_BGR2GRAY)
        
            # find the first frame and find circles in it
            p0 = cv2.HoughCircles(old_gray,cv2.HOUGH_GRADIENT,3,100,param1=100,param2=30,minRadius=4,maxRadius=15)
            
            p0.shape = (p0.shape[1], 1, p0.shape[2])
            p0 = p0[:,:,0:2]
         
            while flag:
                my_stream = BytesIO()
                cam.capture(my_stream, 'jpeg')
                data2 = np.fromstring(my_stream.getvalue(), dtype=np.uint8)
                frame = cv2.imdecode(data2, 1)
                cv2.flip(frame,1,frame)
                frame_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
        
                p1, st, err = cv2.calcOpticalFlowPyrLK(old_gray, frame_gray, p0, None, **lk_params)#calculate the video optical flow
                # Select good points
                good_new = p1[st==1]
                good_old = p0[st==1]
                # draw the tracks
                for i,(new,old) in enumerate(zip(good_new,good_old)):
                    a,b = new.ravel()  # a,c-the x-axis of the center´╝î b,d-the y-axis of the center
                    c,d = old.ravel()
                    
                    if (i<10):# the maximum center of circle can be detected is 10
                        IsGesture(a,b,c,d,i)
                        
                        if flag1:
                            break
                
                flag1 = False#set flags to run the while
                cam.capture(my_stream, 'jpeg')
                data3 = np.fromstring(my_stream.getvalue(), dtype=np.uint8)
                frame = cv2.imdecode(data3, 1)
                #update the frames and points
                old_gray = frame_gray.copy()#copy the gray frame
                p0 = good_new.reshape(-1,1,2)
        
        
        def Spell(spell):
            global num,flag,lupdate,rupdate
            
            
            ig = [[0] for x in range(8)]
            if (spell=="Colovaria"):
                if (num % 2) == 1:
                    lupdate = spell
                    left()
                elif (num % 2) == 0:
                    rupdate = spell
                    right()
                num += 1     
                flag = False
                print('flag is False')
                print('Colovaria','leftdown')
                print(astr)
                GPIO5()
        
            elif (spell=="Lumos"):
                if (num % 2) == 1:
                    lupdate = spell
                    left()
                elif (num % 2) == 0:
                    rupdate = spell
                    right()
                num += 1     
                flag = False
                print('flag is False')
                print('Lumos','rightdown')
                print(astr)
                GPIO13()
        
            elif (spell=="Nox"):
                if (num % 2) == 1:
                    lupdate = spell
                    left()
                elif (num % 2) == 0:
                    rupdate = spell
                    right()
                num += 1     
                flag = False
                print('flag is False')
                print('Nox','rightup')
                print(astr)
                GPIO19()
        
            Scan()
        
        
        def IsGesture(a,b,c,d,i):
            global flag1, astr
            print("point: %s" % i)# add the movement we recognized into a list
           
            if ((a<(c-4)) and (abs(b-d)<2)): 
                ig[i].append("left")
            elif ((c<(a-4)) and (abs(b-d)<2)):
                ig[i].append("right")
            elif (b<(d-6)):
                ig[i].append("up")
            elif (d<(b-6)):
                ig[i].append("down")
                
            astr = ''.join(map(str, ig[i]))
            if "rightdown" in astr:
                Spell("Lumos")
                flag1 = True
            elif "rightup" in astr:
                Spell("Nox")
                flag1 = True
            elif "leftdown" in astr:
                Spell("Colovaria")
                flag1 = True
            print(astr)
        
        
            
        def left():
            left_buttons[(70,90)] = left_buttons[(70,130)]
            left_buttons[(70,130)] = left_buttons[(70,170)]
            left_buttons[(70,170)] = lupdate
            
        def right():
            right_buttons[(250,90)] = right_buttons[(250,130)]
            right_buttons[(250,130)] = right_buttons[(250,170)]
            right_buttons[(250,170)] = rupdate
        
        def compare():
            lvalue = left_buttons[(70,170)]
            rvalue = right_buttons[(250,170)]
            
            if lvalue == 'Nox' and rvalue == 'Lumos':
                left_buttons[(70,210)] = str(int(left_buttons[(70,210)]) + 1)
            elif lvalue == 'Lumos' and rvalue == 'Nox':
                right_buttons[(250,210)] = str(int(right_buttons[(250,210)]) + 1)
                
            elif lvalue == 'Lumos' and rvalue == 'Colovaria':
                left_buttons[(70,210)] = str(int(left_buttons[(70,210)]) + 1)
            elif lvalue == 'Colovaria' and rvalue == 'Lumos':
                right_buttons[(250,210)] = str(int(right_buttons[(250,210)]) + 1)
                
            elif lvalue == 'Colovaria' and rvalue == 'Nox':
                left_buttons[(70,210)] = str(int(left_buttons[(70,210)]) + 1)
            elif lvalue == 'Nox' and rvalue == 'Colovaria':
                right_buttons[(250,210)] = str(int(right_buttons[(250,210)]) + 1)
                
        def getwinner():
            lvalue = int(left_buttons[(70,210)])
            rvalue = int(right_buttons[(250,210)])
            
            if lvalue == rvalue:
                winner[(160,140)] = 'doffall'
            
            elif lvalue > rvalue:
                winner[(160,140)] = 'Wizard1'
                
            elif lvalue < rvalue:
                winner[(160,140)] = 'Wizard2'
            
        def restart():
            global winner, left_buttons, right_buttons, num
            winner = {(160,140):'None'}
            left_buttons = {(70,90):'None', (70, 130):'None', (70,170):'None', (70,210):'0'}
            right_buttons = {(250,90):'None', (250, 130):'None', (250,170):'None', (250,210):'0'}
            num = 1
        
        lk_params = dict( winSize  = (15,15),
                        maxLevel = 2,                                                          # cv2.TERM_CRITERIA_COUNT iterate 10 times
                        criteria = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 0.03)) # cv2.TERM_CRITERIA_EPS
        dilation_params = (5, 5)
        movment_threshold = 80
        num = 1
        lupdate = ''
        rupdate = ''
        stream = BytesIO()   # Create an in-memory stream
        cam = PiCamera()    # initialize
        cam.framerate = 24    # set frequency
        
        def End():
            cam.close()
        
        while True:
            flag = True
            
            screen.fill(BLACK)
            
            if num == 3 or num == 5 or num == 7:
                compare()
                if num == 7:
                    num += 1
                
            if num == 8:
                getwinner()
                flag2 = True
            
            for my_text, text_pos in my_buttons.items():
                text_surface = my_font.render(my_text, True, GREEN)
                rect = text_surface.get_rect(center=text_pos)
                screen.blit(text_surface, rect)
            
            for text_pos, my_text in left_buttons.items():
                text_surface = my_font1.render(my_text, True, WHITE)
                rect = text_surface.get_rect(center=text_pos)
                screen.blit(text_surface, rect)
                
            for text_pos, my_text in right_buttons.items():
                text_surface = my_font1.render(my_text, True, WHITE)
                rect = text_surface.get_rect(center=text_pos)
                screen.blit(text_surface, rect)
                
            for text_pos, my_text in winner.items():
                text_surface = my_font1.render(my_text, True, RED)
                rect = text_surface.get_rect(center=text_pos)
                screen.blit(text_surface, rect)
            
            if (not GPIO.input(27)):
                sys.exit()
            
            pygame.display.flip()
            
            if num <= 6:
                Scan()
                
            while flag2:
                if (not GPIO.input(27)):
                    sys.exit()
                    
                if (not GPIO.input(23)):
                    restart()
                    flag2 = False