Remote Control Robot

Qiukai Lin, Weier Mi.


We built a robot car using the components given to us in Lab 3. After constructing the robot car from Lab 3, we installed a camera and a robot arm to interface with the robot car. We assembled the robot arm in the front of the robot car, and connected the camera module to the Raspberry Pi. We then wrote codes to serve as the client and server on both the Raspberry Pi and a Windows PC laptop to enable real-time communication between the laptop and the robot car. Control signals would be sent from an Xbox 360 controller to the PC, then the PC would forward the signals via the server-client interface to the Raspberry Pi so that we could control the robot car's movement and the robot arm. We also developed interface in order to observe video stream from the camera on our laptop in real time, and we could use the camera feed as guidance to control the robot's movement from a remote position.

Block diagram

Figure 1: Block Diagram of the Project

Robot image

Figure 2: Picture of the Robot Car

Project Objective:

  • We would build a robot car with a camera so that we could control the robot remotely from the camera feed.
  • The robot car would have a robot arm that is able to rotate and grab objects.
  • The robot would be controlled with an Xbox 360 controller which is connected to a laptop.


Robot car and arm

We first assembled the robot arm that we bought from Amazon as shown in the following diagram:

Block diagram

Figure 3: Picture of the Robot Arm

We then built the robot car by reusing the robot from Lab 3. It serves as the basic platform of our final project. The car is equipped with the two servos, and each of the servos controls one wheel. We control the speed and direction of the robot car by modifying PWM signals sending to the two servos. The only significant change of the hardware is that we equipped the robot car with a robot arm, and we controlled the robot arm by sending another two PWM signals to its two servos, so it can rotate clockwise/counter-clockwise and close/open its gripper. We also connected the camera module to Raspberry Pi. The servos are powered by a battery pack consisting of 4 AA batteries (6V in total). The Raspberry Pi, on the other hand, is powered by a power bank provided by the lab. We tested the whole hardware connection by using voltmeter to see if we have all wires with expected connection.

To achieve control of the robot's movement, we defined a few control schemes, including going straight forward/backward, turning left/right while going straight forward/backward, and spin around to the left/right. To move forward, we need the left wheel to rotate counter-clockwise and the right wheel to rotate clockwise, and vice versa for moving backward. When we want the robot to turn left/right while going forward or backward, we reduce the duty cycle sent to the left/right wheel. When no acceleration signal is sent to the robot while a steering signal is present, we let the robot spin around by making the two wheels going opposite directions.

For the arm control, we would control the gripper by sending a PWM signal to it when we want it to open/close. By default the servo that controls the gripper would not rotate. We can also control the angle of the entire arm by applying another PWM signal to the servo that controls the rotation of the arm.

We tested the robot controling by using the Xbox controller to send signals, and we observed expected behaviors from the robot car and robot arm.

Xbox controller reading and encoding

In order for our laptop to interface with the Xbox controller and for the robot to read inputs in real time, we found a library called GameControlPlus from Quark Place. It has a convenient input reading library developed for several hardware devices, including joystick and gamepad. We looked through their example codes and developed our own codes. We first configured our Xbox controller to specify functions for each button, joystick and trigger, and we saved our configuration file for the use of input reading control. Our raw readings from the controller are all floats between -1 and 1, and we modified the value into speed levels (integers from -10 to 10) before sending to the robot based on our needs. Therefore, our codes read inputs from the Xbox controller, and are modified further to support server and client communication.

We tested this part by having codes run with GameControlPlus and print out the outputs to the console, and we observed expected outputs coming out in real time with our Xbox controller.

The reason we selected the Xbox controller to control the robot is that it is very common and it provides us with a lot of buttons so that we could achieve a great control of the robot. Since the controller also supports analog inputs (triggers), we can map the analog inputs to the acceleration signals and they function just like the throttle and the brake on a car in the real life. We mapped the throttle to the right trigger (RT) and the brake to the left trigger (LT). We also mapped steering of the robot to the left joystick (LS), rotation of the robot to the right stick (RS), and gripper control to button A and B on the controller.

WiFi Communication

For this part of the project, we developed server and client codes to enable real time communication between laptop and Raspberry Pi over WiFi. We set the laptop as the server and the Raspberry Pi as the client. The codes are developed using socket programming, and they are attached in Code Appendix in the end of this report.

Our server codes on the laptop side are developed in Processing IDE, which is an IDE that looks very similar to the Arduino IDE and provides a wide range of hardware processing and interfacing support. The codes are written in Java. Our server codes in Processing read inputs from the Xbox controller, including signals to indicate x-axis direction movement for the robot car (steering), y-axis direction movement for the robot car (acceleration), rotation servo on the robot arm, and switch servo for the robot arm gripper. Our server code encodes information for all the movements into a single string and sends to client code in Raspberry Pi side.

Our client codes in Raspberry Pi side are developed using Python. It has similar fashion of socket programming, and it reads inputs from server side iteratively, parses the inputs into integers, and decides what signals to send to the servos to perform desired movements.

We tested this part by running both server and client codes, and we were able to see the client code displaying inputs from the server code as expected.

Camera feed

For the camera feed, at the beginning we found a few approaches. The first would be to reuse server/client features that we used to communication Xbox controller signals, which involved with buffer images and showing in the server side. The second would be to use http connection with VLC (an open-source video player) in the server side to receive frames from Raspberry Pi camera module. However, both of those have significant delay that can’t be used for the purpose of real-time robot control.

Finally we found the RPi Cam Web Interface, which is an open source camera module interface developed for Raspberry Pi. By using this interface, we are able to use the web browser to access the camera on Raspberry Pi from our laptop and to see what the camera sees in real time. We are also able to do some other things, such as taking a picture and save to local repository, zoom in, zoom out, change contrast and sensitivity, etc.

We tested this part by start camera on Raspberry Pi, and open http in web browser of our Windows laptop. We were able to see video streaming in real time.

Block diagram

Figure 4: Screenshot of the Webpage Camera Interface


We are able to finish implementing this robot car, and it meets our expectation. It worked very well with the Xbox controller, and the camera feed has a high quality so we can use it to determine how we want to control the robot car easily.


We finished designing and implementing a robot car that can be controlled remotely over WiFi in real time. We can also see video streaming on the control center side. We can control the robot car by our Xbox controller easily and determine where we wanted the robot car and arm to perform functions based on what we see from the camera mounted on the robot.

Future Work

There are a few things that we can improve for this project.

One of them is to implement automatic driving. The robot car may be used to search for a specific item in a building of large volume, and therefore it will be beneficial if it can search for items automatically in an optimized way.

Another improvement is to use more sensors that enable the robot car to construct information network, such as drawing map of certain areas. In that case we may use IR sensors and accelerometer to draw topographic map.

The last improvement is to connect the robot car to the cloud with other robot car. In this way we enable communication among many robot cars in real time, and they can work together to do some large scale tasks, such as delivering items in a large warehouse.

Work Distribution

Qiukai Lin

Worked on server code, Xbox controller interfacing, camera feed, and robot components assembly.

Weier Mi

Worked on client code, robot car and arm control, and robot components assembly.

Parts List

Total: $64.98


RPi Camera Document
Processing IDE
Xbox Controller Inputs to Laptop

Code Appendix


The server codes reads inputs from Xbox controller, encode signals, and send
signals to clients via WiFi by using socket programming

author: Qiukai Lin
import org.gamecontrolplus.gui.*;
import org.gamecontrolplus.*;

ControlIO control;
Configuration config;
ControlDevice gpad;

int px, py, rotation;
int open, close;
int port = 8888; //Sever TCP-Port number
Server myServer;

public void setup() {
 size(400, 240);
 myServer = new Server(this, port);
 // Initialise the ControlIO
 control = ControlIO.getInstance(this);
 // Find a device that matches the configuration file
 gpad = control.getMatchedDevice("gamepad_eyes");
 if (gpad == null) {
   println("No suitable device configured");
   System.exit(-1); // End the program NOW!

public void draw() {
 background(255, 200, 255);
 String output = "";
 open = gpad.getButton("PUPILSIZE1").pressed() ? 1 : 0;
 close = gpad.getButton("PUPILSIZE2").pressed() ? 1 : 0;
 px =  (int)(gpad.getSlider("XPOS").getValue()*10);
 py =  (int)(gpad.getSlider("YPOS").getValue()*10);
 rotation = (int)(gpad.getSlider("EYELID").getValue()*10);
 output = px + " " + py + " " + rotation + " " + open + " " + close + ";";
 println("px: " + px + "   py: " + py + "   rotation: " + rotation + "   Open: " + open + "   Close: " + close);


import socket   #for sockets
import sys  #for exit
import RPi.GPIO as GPIO


GPIO.setup(13, GPIO.OUT)
GPIO.setup(19, GPIO.OUT)
GPIO.setup(5, GPIO.OUT)
GPIO.setup(6, GPIO.OUT)

servo0 = GPIO.PWM(13, 50)
servo1 = GPIO.PWM(19, 50)
servo2 = GPIO.PWM(5, 50)
servo3 = GPIO.PWM(6, 50)

servo0.start(7.5) # left
servo1.start(7.5) # right
servo2.start(7.5) # arm rotation
servo3.start(7.5) # arm gripper
    #create an AF_INET, STREAM socket (TCP)
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error, msg:
    print 'Failed to create socket. Error code: ' + str(msg[0]) + ' , Error message : ' + msg[1]
print 'Socket Created'
host = '' # subject to change in lab
port = 8888
#Connect to remote server
s.connect((host , port))
print 'Socket Connected to ' + host

step = (10 - 7.5) / 10

while True:

    data = s.recv(20) # not sure about the size
    data = data.split(';', 1)
        acc, steer, arm_dir, arm_open, arm_grab = [(int i) for i in data.split(' ')]
    except ValueError:
    if (acc == 0):
        if (steer == 0): # stops
        else: # rotates
            servo0.ChangeDutyCycle(7.5 - step * steer)
            servo1.ChangeDutyCycle(7.5 - step * steer)
    else if (acc > 0):
        if (steer == 0): # straight
            servo0.ChangeDutyCycle(7.5 + step * acc)
            servo1.ChangeDutyCycle(7.5 - step * acc)
        else if (steer > 0): # turn right
            servo0.ChangeDutyCycle(7.5 + step * acc)
            servo1.ChangeDutyCycle(7.5 - step * acc * (10-steer)/10)
        else: # turn left
            servo0.ChangeDutyCycle(7.5 + step * acc * (10-steer)/10)
            servo1.ChangeDutyCycle(7.5 - step * acc)
        if (steer == 0): # straight
            servo0.ChangeDutyCycle(7.5 - step * acc)
            servo1.ChangeDutyCycle(7.5 + step * acc)
        else if (steer > 0): # turn right
            servo0.ChangeDutyCycle(7.5 - step * acc)
            servo1.ChangeDutyCycle(7.5 + step * acc * (10-steer)/10)
        else: # turn left
            servo0.ChangeDutyCycle(7.5 - step * acc * (10-steer)/10)
            servo1.ChangeDutyCycle(7.5 + step * acc)
    servo2.ChangeDutyCycle(7.5 + step * arm_dir)
    if (arm_grab):