Introduction

While most of this project is implemented in software, the camera hardware setup is critical to CloudCam’s success (naturally). In order to properly configure a camera with the Raspberry Pi, we followed [this] setup tutorial. The physical camera connection looks like this:

Once we ensured that physical connection was secure, we opened up the raspi-config tool, and clicked on the interfaces tab. In order to connect to the camera, the ‘camera’ and ‘I2C’ options needed to be enabled. We made these changes, rebooted the Pi, and used the raspistill -o test.jpg test command to make sure the camera worked. The images initially came out blurry, so we used a focus ring to adjust the camera lens. After successfully creating the test.jpg file, we were ready to get started on the actual CloudCam code!

As described in the HLD section, CloudCam operates through a state machine of four specific states, as controlled by the user’s button and touchscreen presses. For the menu handler state specifically, we divide our code into a variety of different functions, which have one of four categories: image processing functions that actually perform computations, “blit” functions that manage display, “handler” functions that monitor and respond to user input, and “miscellaneous” functions that handle other CloudCam tasks, such as Given this, a typical flow of a user’s inputs would look something like this.

Essentially, a user can take as many photos as they want, until the program is exited. For each photo they take, they can customize that photo with over a dozen different digital effects that can be compounded to create stimulating, artistic photos. Or, they could choose to use their image as an input to a Machine Learning service, and view those results any number of times. Once they are satisfied with the end result of their customizations, a user is easily able to open up a Save/Upload menu, where they can choose to discard an image, save it locally, or upload it to the Cloud.

Once the image effects menu is opened, a “level 1” menu is displayed, which divides up the different effects into four categories.

Clicking on any of these category options will open one of the following four “level 2” menus, which will allow a user to select the specific end effects they would like to apply to the image. Examples of the level 1 to level 2 menu transition are displayed below.

The menu handlers for the image effects are handled entirely by touchscreen input. The quadrant of the screen press is recorded, and mapped into the next menu transition. This also applies to the Save/Upload menu transitions, which occur when transitioning back to the free view mode, and display the following screens:

The next few sections will deep dive into how the different image effects were implemented, and how the cloud connection component of this project actually works!

For this project, we implemented a wide variety of fun image effects that a user can choose from when using the CloudCam. Many of these effects were created using the highly-optimized image processing Python library, OpenCV. The next few sections will go in-depth into how each of the grouped image applications are actually implemented.

Filters

The filtering effects are grouped as such because the effects in this category mimic the iPhone or Instagram style filters. Much of the following work was adapted from [this]. These typically involve an application of a 2D filter on an image, or involve changing the values in the different RGB channels of the image.

Noir

The simplest of the filtering-type applications, this effect uses the cv2.cvtColor(im, cv2.BGR2GRAY) to convert an image to grayscale while preserving lumosity. As the image loses two channels (going from BGR to gray), we have to account for the transition from 3D numpy arrays to 2D when displaying our image to the screen.

Sepia

Sepia is a sample image filter that gives an image an “aged photograph” look. It is computed by applying the following filter to an input image.

Warm and Cool Images

“Warm” and “cool” are offered as two separate effect options, but they simply involve adjusting the distributions of the red and blue RGB channels respectively. Warmer images appear “redder” while cooler images appear “bluer.” The channels were adjusted through a spreadLookupTable() function that adjusted the values according to a Univariate Spline transform.

Adjustments

The adjustment group of image effects include four effects that can be dynamically incremented or decremented in an image. Common effects of this type include contrast and brightness adjustment, blur and sharpness, and saturation.
Choosing an effect in the adjustment menu will open up the current status of the edited image along with an adjustment bar at the bottom, as shown below.

That bar serves as essentially a “level 2.5” menu option, as the system isn’t officially at the “view edited image state” until the “done” option is pressed. The “+” and “-” options will respectively trigger the increase and decrease modes on whichever adjustment function (e.g. increase contrast) is selected from the level 2 menu, and apply them to the edited image. The resulting image is displayed on the screen to be previewed, until the edits are finalized by selecting “done.”

Contrast and Brightness

Although these are implemented as two separate effects in the menu, they both are similar in that they involve linear operations of the form aX + b where X is the input image. The a parameter adjusts the contrast value multiplicatively, while the b parameter adjusts the brightness parammer additively. Brightness refers to the overall lightness or darkness of the image, so increasing the brightness every pixel in the frame gets lighter because of the constant bias. Contrast is the difference in brightness between objects in the image. Increasing the contrast of an image makes light areas lighter and dark areas darker. High and Low contrast images are shown below:

Contrast and brightness adjustment are respectively implemented by adjusting the alpha and beta parameters in the OpenCV function cv2.convertScaleAbs(alpha=a, beta=b), which performs the linear transform. For contrast adjustment on the user end, pressing the + button sets the contrast to 0.9, and the - button sets it to 1.1, and the images are multiplied by these constants. When the + button is selected for brightness adjustment, beta is set to +1, incrementing the overall image brightness by skewing it to the max brightness level. Similarly, if the - is pressed, beta is set to -1 to shift the image pixels down by one constant. High and Low brightness images are shown below:

Blur and Sharpness

The blur and sharpness adjustment effects are condensed into just one “blur” option, which involves the repeated application of either a blur or sharpness kernel, depending on a user’s input. If the + button is pressed, the image is filtered with a blur filter, like the one below.

This is the equivalent of an averaging, or box filter. By contrast, if “-” is pressed, a sharpness kernel is applied, such as the one below:

The result of blurring is exactly what it sounds like; the image looks fuzzier with details obscured. The result of the sharpening kernel is an image with enhanced edges and vibrant colors - almost like a colored-pencil sketch.

Saturation

The final adjustment parameter is the saturation adjustment. As stated earlier, OpenCV stores images in BGR format, and we are used to seeing images in RGB format as well. Adjusting the saturation required converting the image to an HSV type - Hue, Saturation, Value. Hue represents the color portion of a pixel as a number from 0 to 360 degrees: Saturation describes the amount of gray in a particular color, from 0 to 100 percent. Reducing this component toward zero introduces more gray into the image and produces a faded effect. By contrast, saturated images seem vibrant, with fuller colors. Sometimes, saturation appears as a range from 0 to 1, where 0 is gray, and 1 is a primary color. Value works in conjunction with saturation and describes the brightness or intensity of the color, from 0 to 100 percent, where 0 is completely black, and 100 is the brightest and reveals the most color. To adjust saturation, we converted the image to HSV through the cv2.cvtColor(img, cv2.COLOR_BGR2HSV) and split the resulting channels into h, s, and v components. We then added a constant value of 5 to all values in the image’s s channel, if the + button was pressed, or subtracted 5 if the - button was pressed.

Special Effects and Restoring

Special image effects that we included are ones that did not full under the filtering or adjusting umbrellas, but can be added to an image for other cool effects. In particular, we have support for 8-bit style images through pixelation, poster-like images through color clustering, and image outlines through Canny edge detection. As mentioned previously, these image effects can be stacked up on top of each other and used in conjunction. If a user wants to “clear out” these image effects and restore to defaults, that option is provided in this menu as well by pressing the “restore” button.

Pixelation

By “pixelating” an image, we can cool images in the style of an 8-bit video game. The pixelation process is a relatively simple matter of rescaling, using different interpolation methods. Interpolation is essentially the process of “guessing” what a neighboring pixel would be. To implement pixelation, we first downscale the image using cv2.resize(img, cv2.INTERP_LINEAR). The linear interpolator will use a combination of linear functions in order to determine neighboring pixels accurately. We then rescale the image back up using cv2.resize(img, cv2.INTERP_NEAREST). Nearest-neighbors interpolation does not adjust values of the pixels; it determines the neighboring pixels based on the exact values of pixels already present. This creates “blocks” in the rescaled image, mimicking pixels of an 8-bit game!

Color Clustering

Another special effect we added was color clustering through K means. Color clustering, or color quantization is the process of reducing the number of colors in an image. One reason to do so is to reduce the memory, but another reason, as in our case, is for the cool comic-book style poster effect. We use the OpenCV implementation of the K-means clustering algorithm for color quantization. K-means is an algorithm that attempts to classify an unknown set of data, in this case a bunch of RGB pixels, into a set number of groups or clusters. In an image, there are 3 features, R,G,B. To run this, we need to reshape the image to an array of Mx3 size, where M is the number of pixels in an image. After the clustering, we apply centroid values to all pixels, such that the resulting image will have the specified number of colors.

Canny Edge Images

The final special effect CloudCam has is the ability to generate edge images, resembling outlines. We accomplish this by using cv2.Canny(img), which performs the Canny edge detection algorithm. That algorithm utilizes a series of gradient calculations, as well as a hysteresis double-threshold, in order to compute “strong” edges of an image. Although these thresholds need to be fine-tuned on a per-image basis in order to have the optimal results, for a typical image the standard lower/upper thresholds of 100 and 200 produce meaningful results!

Stacking image effects

A feature of the CloudCam is that these image effects can stack up on top of each other to create new, unique image effects. These can be from any non-ML category. For example, a user could apply a warm filter, adjust the saturation, adjust the contrast, run a color clustering, and then pixelate an image - or any sort of combination like that, with no limit. The results of that particular sequence is shown below:

Machine Learning

The image effects described above are examples of classical, or non-learning-based Computer Vision applications. Most of these require relatively simple calculations, and run nearly instantly on the RPi. However, the newer applications of Computer Vision are typically done in conjunction with Machine Learning (ML) applications. Hundreds of different ML models exist that can generate or extract meaningful data from an input image, for a variety of different purposes. A camera that can generate inference results for trained ML models on the fly will certainly be a useful tool. To demonstrate our CloudCam’s ability to do so, we utilize two robust, pre-trained ML model frameworks, mini-Xception for Face and Emotion Recognition and MaskRCNN for common object detection and instance segmentation, as connections to the CloudCam.

To clarify, we don’t take credit for any of the ML models themselves - we borrowed pretrained versions of them. You can find the MaskRCNN model here on [gluon] and the face and emotion classification network here on [github]. These resources provide more in-depth information on the implementation and usage of these model architectures.

Examples of Emotion Recognition and MaskRCNN predictions are shown below:

Emotion Detection Results

MaskRCNN Results

AWS EC2/S3 Infrastructure

We wanted to run larger, compute-heavy machine learning models on the images, but the hardware that comes in with the Pi has CPU, RAM, and memory limitations. It is also not realistic to scale the project on the Pi if we wanted to add more classification options and models. Thus, we turned to an AWS (Amazon Web Services) infrastructure to support this computation using their cloud technologies.

Specifically, we used Amazon EC2 (Elastic Compute Cloud) and Amazon S3 (Simple Storage Service). EC2 provides virtual compute environments (instances )that you can customize and configure, where adding more resources to the instance will cost more. S3 provides cloud storage containers (called buckets) where you can store data (in our case, images).

We used EC2 to load a t2.medium instance with Ubuntu 18.04. The name T2 refers to the type, while the size indicates the amount of RAM. T2 is one of the lowest tier types of instances with minimum CPU compute power.

The general pipelines to access the server is explained above. Essentially, we use Nginx, a web server, Gunicorn, as WSGI (Web Server Gateway Interface) server and Flask, a web framework (for python). Nginx handles where requests from the internet arrive first. Gunicorn translates requests into a format the web application can handle. You can learn more about what they do [here], and [here] are instructions for building your own web server.

Start by creating an EC2 instance, ssh in, and follow the instructions! Flask controls the server logic, such as what to do when certain endpoints are called. An endpoint is the URL that the server is waiting to receive and can act on, in our case we have 2 endpoints for each of the models: url/mask/ and url/emotion/ . We have Python functions that are executed when either of these endpoints are hit (we pass the image through model inference). In addition, Flask is a RESTful web framework, which means it only takes requests such as GET and POST. Thus, you must make GET requests to our endpoints. The server is setup with AWS credentials (private and public key) so that it can make requests to S3 to upload and download images using the boto3 api.

Failed Architectures

We had to make major design changes as we approached a cloud solution. We will make note of failed architectures here for future readers.

AWS IoT Greengrass

First, we tried to use AWS IoT Greengrass. The name sounds useful right? However, it turns how this service is meant for controlling edge devices from a central server… not the other way around. AWS provides something called Greengrass Connectors, which help configure an edge device (the Pi) to do computing ON the device itself, and then send it to the server. This is meaningful when you have multiple edge devices, and want to control all of them through a Greengrass Group. However, the way even Greengrass sends information back and forth between an edge device and the central server is no different than the way we used boto3. Our goal is to not have to run on the Pi, so we looked at other solutions.

AWS Lambda

Second, we tried to use AWS Lambda. Lambda is AWS’s notion of serverless architecture. A Lambda function will wait in the cloud until it is specifically called, execute its contents and exit. There is no server allocated for the function. This allows AWS to manage all resource allocation for all Lambda functions themselves. We figured we would set up Lambda functions that would run the model through an image when specifically called on. This is a valid architecture; however, there is one major limitation. When you deploy a Lambda, you create what is called a deployment package that contains all the files and dependencies for the Lambda. The maximum unzipped file size of a Lambda is 250 MB (that is, stored on S3 or Lambda Layers, the maximum size to upload directly to Lambda is only 50 MB). The deployment package of MaskRCNN is 400 MB, which was too much, mostly because of SciPy being too large. As such, Lambda could not handle our needs, and we had to settle on a server-based architecture. If your deployment package is small enough for Lambda to handle, we highly recommend using it as it erases the need to configure an EC2 instance. A regular TensorFlow model can usually stay within the constraints.

EC2 Issues

Finally, we decided to use an EC2 server with Nginx and Flask. Configuring the environment is tedious but doable, just make sure to use a virtual environment. You can find instructions [here].

One issue we came across was that the MaskRCNN model kept crashing on the instance due to a memory error. We originally started with a t2.micro instance, part of the free tier, which only has 1 GB of RAM! However, once we upgraded it to a paid instance, a t2.medium with 4 GB of RAM, the model ran fine.

MaskRCNN takes about 100 seconds to run on a t2.medium, which has the bare minimum of resources. EC2 has many options for high compute power instances, such as those with GPU’s, but of course these will cost more. However, it keeps the application flexible to by allowing us to run as much computation as we need to simply by throwing money at it. Note that pricing is by the hour, so it is very feasible to downgrade your instance for development and upgrade it for fast performance when needed. This allows for a very scalable architecture depending on how your requirements change! For pricing reference, it costs 4 cents/hr to run the t2.medium, 16 cents/hr to run a t2.xlarge, and 53 cents/hr to run a g4dn.xlarge. You can view pricing [here].