Projected Virtual Board

Write on any surface with an LED pen
A Project By Jiachao Chen(jc3345), Yuqing Shou(ys895).

Demonstration Video


Nowadays, many people need boards and pens to write down important notes on the board to share the essential information to everyone during a meeting or lecture. However, the physical boards and pens are not convenient to be carried, because they are either too heavy or too large. As a result, this project provide a solution to solve this problem by utilizing Texas Instrument DLP Light Crafter Display board, Raspberry Pi and a camera to build to a small, light and portable virtual board device that can project a virtual board on the wall and users can use the LED pen to write down anything on the board.

Project Objective

There are several goals in the project:

  • Computed the perspective transformation matrix by OpenCV and then apply it to the Raspberry Pi.
  • Implemented Power Law Transformation to enhance the display of the projected virtual board.
  • Recognized the red color of the LED pen light by image thresholding.
  • Utilized the LED pen to control the mouse by finding the red light’s contour area and moments.

Hardware Components

There are four basic hardware components in the project

  • Raspberry Pi.
  • DLPDLCR2000EVM DLP LightCrafter.
  • Logitech C920 HD Pro Webcam.
  • LED Pen

Raspberry Pi will be the main component in this project, because it will carry the main program to implement all the functions of the virtual board. Moreover, the virtual board will be projected by the DLPDLCR2000EVM DLP LightCrafter on a wall, and DLPDLCR2000EVM DLP LightCrafter is connected with Raspberry Pi.

In addition, the Camera will be utilized by the Raspberry Pi, and it will capture and recognize the projected virtual board to send the parameters including length and width of the virtual board and also will the distance between DLPDLCR2000EVM DLP LightCrafter and board.

Furthermore, the Digital Pen will have an LED light (or some light is brighter than the LED) on the head of the pen, and it will also have a switch that turns on and turns off the LED. In addition, the Digital Pen will also have a battery that is attached in the tail of the pen to provide the power of the Digital Pen. In a further step, the digital may have more switches to provide additional functions such as select and hold. However, at the first stage of the project, it will only provide the function of writing for the project.

The basic installation is shown below:

Generic placeholder image


Although with the limitation of computation, our algorithm of LED light detection runs on the Raspberry Pi with reduced resolution in order to reduce the computation. We set the OpenCV window size to 640x640 and the process image size to 370x640, which is a reasonable resolution that we successfully tested on the Raspberry Pi. The window size is the first window that appears for us to find first four points as input coordinates and the process image size is the size of the image that the following computer vision algorithm runs with. We set the lower bound and upper bound of the color to (160, 100, 200) and (179, 255, 255) in the HSV. It is able to detect most of red light from LED pen. During the test, we solve several problems: 1. The LED light should be strong and big enough to be captured by the camera, which means that the LED pen's light influence our project performance. 2. The computation of our algorithm and the real-time performance of our project is a trade-off, higher resolution means more accurate result but more lagging while writing or drawing. All the image processing steps are implemented through opencv. And mouse movements and operations are imitated through pyautogui. The workflow of our algorithm is in the workflow chart.

Generic placeholder image
Figure. Image Processing
Generic placeholder image
Figure. Overall workflow


We test our software and hardware components in the class lab, adjust light of the room in order to imitate the real situations in the classroom.

Generic placeholder image
Figure. Result

Possible Improvement

There are several aspects of improvements in the software and hardware that we thought about after the project is over.

  • As in this project, we are using LED pen with a little difficulty of writing and painting on a projected surface, we can instead use a laser pen as most classrooms have them and in order to achieve this, a better algorithm is needed to capture the small laser dot that shows on the projected surface.
  • The algorithm can be optimized by reducing the computation or we can move the computation to the cloud, every time the cloud finished the computation it sends the result to the Raspberry Pi.
  • The software part can be optimized to limit the space in the screen that can be used for drawing due to that not full screen is needed for writing and painting.

Work Distribution

Generic placeholder image

Project group picture

Generic placeholder image

Jiachao Chen

Generic placeholder image

Yuqing Shou

Parts List

Total: $125


OpenCV Document
DLP LightCrafter Display 2000 EVM on Raspberry Pi
R-Pi GPIO Document

Code Appendix

# 11/30/2019
# ECE5725 Final Project
# Yuqing Shou(ys895), Jiachao Chen(jc3345)
import cv2
import numpy as np
import pyautogui
from pynput.mouse import Button, Controller
import time

class Writer:
    def __init__(self, mouse, alpha = 2, gamma = 0.5, points = [(0,0),(0,0),(0,0),(0,0)], flag = False, prc_img_size = (740/2,1280/2), lower = (160, 100, 200), upper = (179, 255, 255), points_cnt = 0):
        Writer object
        :param mouse: mouse object
        :param alpha: value of alpha
        :param gamma:  value of gamma
        :param points: coordinates of the image on the opencv windows
        :param flag: initial flag
        :param prc_img_size:  process image size
        :param lower: lower bound of the color
        :param upper: upper bound of the color
        :param points_cnt: counter of points
        self.cap = cv2.VideoCapture(0)
        _,self.img =
        self.alpha = alpha
        self.mouse = mouse
        self.points = points
        self.gamma = gamma
        self.flag = flag
        self.prc_img_size = prc_img_size
        self.upper = upper
        self.lower = lower
        self.out_points = np.float32([[0,0],[prc_img_size[1],0],[0,prc_img_size[0]],[prc_img_size[1],prc_img_size[0]]])
        self.points_cnt = points_cnt

    # gamma transformation
    def gamma_adjustment(self, image, gamma):
        adjust the gamma value of an image
        :param image: input image
        :param gamma: value of gamma
        :return: new image
        invGamma = 1.0 / gamma
        table = np.array([((i / 255.0) ** invGamma) * 255
            for i in np.arange(0, 256)]).astype("uint8")
        # LUT transform
        return cv2.LUT(image, table)
    # record the coordinates of the points in the window
    def draw_points(self,event,x,y,flags,param):
        call-back function to draw points on the opencv window
        :param event: event
        :param x: x
        :param y: y
        :param flags: flags
        :param param: param
        :return: none
        if event == cv2.EVENT_LBUTTONDOWN:
                self.points[self.points_cnt] = (x,y)
                self.points_cnt = self.points_cnt + 1

    def window_points(self):
        find four points on the opencv window
        :return: none
        while True:
                cv2.imshow('img', self.img)
                if(self.points_cnt == 4):
                if (cv2.waitKey(20) & 0xFF == 27) :
    # perspective transformation
    def perspetive_transform(self, image,points):
        standard opencv perspective transformation: same object's position in different coordinates
        :param image: input image
        :param points: input coordinates of an image
        :return: new image
        in_points = np.float32(points)
        Map = cv2.getPerspectiveTransform(in_points,self.out_points)
        warped = cv2.warpPerspective(image, Map, (self.prc_img_size[1], self.prc_img_size[0]))
        return warped
    def write(self):
        main function used for writer
        :return: none
        # mouse callback functions
        print('Please points at: Top Left, Top Right, Bottom Left, Bottom Right')

        # constantly show the image img unless there are four points detected

        while True:
            # capture the frame by the camera
            _, frame =
            warped = self.perspetive_transform(frame, self.points)

            blurred = cv2.GaussianBlur(warped, (9, 9), 0)
            adjusted = self.gamma_adjustment(blurred, self.gamma)
            hsv = cv2.cvtColor(adjusted,cv2.COLOR_BGR2HSV)
            mask = cv2.inRange(hsv, self.lower, self.upper)

            # ret & otsu
            ret, otsu = cv2.threshold(mask,0,255,cv2.THRESH_BINARY+cv2.THRESH_OTSU)

            # find contours from the previous image
            contour_num = cv2.findContours(otsu.copy(), cv2.RETR_EXTERNAL,
            center = None

            a = 0
            b = 0
            # only proceed if at least one contour was found
            if len(contour_num) > 0:
                # find the largest contour in the mask, then use
                # it to compute the minimum enclosing circle and
                # centroid
                c = max(contour_num, key=cv2.contourArea)
                ((x, y), radius) = cv2.minEnclosingCircle(c)
                a = x
                b = y
                print("a:" + str(b) + ", b:" + str(b))
                M = cv2.moments(c)
                if M["m00"] != 0:
                    center = (int(M["m10"] / M["m00"]), int(M["m01"] / M["m00"]))
                else :
                    center = (0,0)

                # only proceed if the radius meets a minimum size
                if (radius>1):
                    self.flag = True
                    # draw the circle and centroid on the frame,
                    # then update the list of tracked points
          , (int(x), int(y)), int(radius),
                        (0, 255, 255), 2)
          , center, 5, (0, 0, 255), -1)

            width, height = pyautogui.size()

            m = (a/1280)*100
            n = (b/740)*100 

            # coordinates in the screen
            k = (width*m)/100
            c = (height*n)/100

            #pyautogui.FAILSAFE = False
            if self.flag == True :
                self.mouse.position = (int(k), int(c))

            self.flag = False   

            cv2.imshow('mask', mask)
            #cv2.imshow('blurred', blurred)
            #cv2.imshow('hsv', hsv)

            k=cv2.waitKey(5) & 0xFF
            if k == 27:


# main function
if __name__ == "__main__":
    mouse = Controller()
    writer = Writer(mouse)


Raspberry Pi: /boot/config.txt configuration

                # Add support for software i2c on gpio pins

                # DPI Video Setup

                hdmi_timings=854 0 14 4 12 480 0 2 3 9 0 0 0 60 0 32000000 3

Raspberry Pi Wiring

Generic placeholder image