Demonstration Video


The Color-Tube game is a captivating puzzle that challenges players to sort the tube using strategy and logic. At the begining of the game, we have multiple tubes containing colored blocks and two empty tubes. Each tube can hold up to four blocks, and there are four blocks for each color in the game. The objective of the game is to strategically use the empty tubes to rearrange the blocks of different colors that are currently mixed in various tubes, placing blocks of the same color into the same tube.

Generic placeholder image

Our parents and friends all love this game. They enjoy spending their leisure time playing it. However, as the game progresses, the number of blocks and tubes increase, and some levels have trickly obstacles that make it difficult to come up with a solution. To keep the fun going, we need to create an automatic solving device that can take photos, analyze the position and colors of the blocks in the image, and use an efficient algorithm to generate the optimal solving strategy. The device can use an iPencil to implement this strategy in the game and complete the level.

Generic placeholder image

Project Objective:

  • Researching an efficient algorithm to solve the game.
  • Using a camera to accurately identify block color and positions.
  • Building a structrue to securely hold an iPad and an iPencil.
  • Incorporating motors and sliders to enable the movemet of the iPencil.
  • Ensure that all components are functioning properly to execute the strategy successfully.


The design process for this project includes writing the core algorithm, designing the structural framework, and implementing the computer vision (CV) routine.Throught this process, we encountered serval challenges, with two main issues standing out.
Firstly, the selection of materials and fixing elements ofr the structrue posed a challenge. Initially, we planned to use acrylic sheets as the structural material for the device, connected using hot glue. Acrylic sheets were chosen for their transparency, as they allow for better visualizatio of the algorithm execution from various angles. To support this plan, we even created a strucural diagram using Fusion 360. However, during discussions, we realized that the acylic sheets were too thin, making it impossibel to drill holes on the sides. Additionally, fixing the slide rail on the acrylic sheets between problematic and was diffcult to accommodate in the design adjustments.As a result, we had to urgently revise our plan and opt for wooden boards and screws as the structural materials.

Generic placeholder image

Secondly, concerning the CV routine, we encountered diffculties in accurately identifying the contours of the tubes and blocks. Sometimes, the blocks were not entirely detected, and at the other times, there were interfering contours present. We later discovered that proper adjustment of gray-scale levels was crucial for contour recognition. Simple gray-scale adjustment often resulted in local blurriness, causing some contours to be unrecognizable or unable to be eliminted. Using black-and-white image proved to be more suitable for identifying the countours of blocks in this game.

Generic placeholder image

Next up, was detecting the tubes. After simple threshold and the indentification of contours, we can easily use a warp transform on the padding out image from the Ipad. This will give us the four corners of the tube. We can then use the corners to find the center of the tube and map it out for the motors.

Generic placeholder image Generic placeholder image

Similarly, color recognition posed a challenge. The color range recognition based on RGB values did bot provide helpful results. This was mainly due to the presence of multiple colors in some blocks, with RGB values that were relatively close. During our learning process, we discovered the K-Means algorithm, which calculates the average RGB values within a specific region. This algorithm proved to be helpful in identifying colors with a sepcific block area.

Motor control was relatively simple as we just had to have a tight gear belt system to move linear blocks along a rail. Controlling it was also simple as we did a simple calculation to estimate how many steps for a 8 phase motor to move a certain distance.


The testing process involved several steps, including testing the functionality of the motors and camera, constructing the structural framework and slide rail, testing the functionality of the slide rail, evaluating the positioning efffectiveness of the motors and slide rail, testing the individual compoents' programs, and testing the overall operation. Throughout this process, we encountered numerous minor issues. These included problems such as the loosening of motor belts, excessive friction between the slide rail and slider component, the breaking of the fixing bolts, iPad positioning , and camera installation. We continuously tried various methods and drilled multiple positioning holes in the wooden boards. At this point, we appreciate the decision to use wooden boards, as they allowed us more flexibility for adjustments and modifications.
There are two anecdotes I can share. Firstly, regarding the motor, during the initial perdormance testing, we found that our small 5-12V stepper motor was unable to move the belt out of our expectation. This unexpected turn of events left us axious for an entire day, to the point where we started considering larger motors as an alternative. Hoever, due to time cinstriants, we couldn't change our strategy. As a last resort, we conducted further tests and discovered that the small stepper motor was indeed capable of functioning properly. Yhis incident taught us the importance of undering the connection methods for stepper motors. It was an enlightening experience that reinforced the advice our professor had given us in the first lab class: that experimental issues often arise from small details, so it's crucial to remain attentive.

During the evening testing, the lab lights were too light, causing reflections on the iPad and leading to incorrect recognitio by the CV routine. We even added a light shield (wooden block) specifically to address this issue. However, in the morning, the lighting was too dim, and the contours couldn't be recognized. These incidents made us realize the critivcal role of the camera's function. We later discover that the PiCamera allowed for more detailed settings, enabling us to adapt to different environmental conditions.
Despite encountering numerous challenges, our project proceeded according to plan and was successfully completed

Result and Conclusion

The different components of our device performed quite well, and we were initially concered that the dim lighting environment might hinder the proper function of the CV routine. However, thankfully, it worked successfully. We demostrated the device autonomously completing two levels of the game in front of everyone without any error, and I still remember the praise form the TA, "this project is truly impressive." In my mind, our project is a success.

Future Work

Based on the advice from our professor, we could indeed make better use of the camera's functions. Now the device takes a photo before solving the game and uses this image fro position and color detection, enabling the formulation of a solving strategy. However, this approach presents a challenge: the device can only operate based on a predetermined strategy, and any issues that arise cannot be modified on the spot. The professor suggested that we ultilize the video feature, allowing us to detect the completion of steps during the game-solving process and adjust the strategy accordingly. This is an excellent suggestion and will be our focus for future work.

Work Distribution

Generic placeholder image

Michael Liang

Writing the game-solving and CV algorithm, designing and constructing the structure and testing the device function.

Erhe Zheng

Parparing the components, helping to write the CV routine program, to build the structure, and testing the motors and camera functionality.

Parts List

  • Raspberry Pi Provided in lab
  • Raspberry Pi Camera V2 Provided in lab
  • The wooden boards - $35.4
  • 5-12V stepper motorss - $15.23
  • The wooden screws - $8.63
  • Timing pulley wheel - $9.71
  • Aluminum Rails and support guide set - $22.67

Total: $91.64


PiCamera Document
Pigpio Library
R-Pi GPIO Document
Stepper motor datasheet
Switch datasheet
PiCamera usage
Previous project1
Previous project2