Introduction

For our final project, we developed a graphical user interface (GUI) using PyGame that allows the user to type notes on the PiTFT touch screen. These notes are saved locally on the Raspberry Pi as plaintext files and can be revisited by the user at any time. Since we wanted to make it so that users do not have to sacrifice the convenience of being able to send the note to other people, we also added the ability to send the note through email.

For this project, we used both hardware and software to accomplish our goals. For the hardware, we used the Raspberry Pi 4 and the PiTFT touch-screen device provided to us for the previous labs in the class. The PiTFT allowed us to display the GUI and obtain touch-screen input from the user. For the software, we used PyGame to create the GUI, and Python logic to handle user inputs and application control. Using these, we were able to create an embedded system.

Design and Testing

Chris's original idea for a final project was a note-taking app that can sync up with multiple different cloud-based storage services.

Diagram showing the original note-taking idea
Figure 1. Diagram showing the original note-taking idea.

Sugam's original idea was sending emails via the terminal with the ability to encode and send images and other attachments as well.

Diagram showing the original note-taking idea
Figure 2. Command line sequence demonstrating Sugam's original idea of sending an email from the command line.

We ended up settling on combining both of these ideas together to make a program that could both allow the user to type a note and send that note by email.

Diagram showing the original note-taking idea
Figure 3. Diagram showing the functionality of the combined idea of a note-taking app with both email and Google Drive capabilities.

The first big step was to develop a working keyboard so that the user could type. We did this by first testing out rectangular on-screen buttons to see how feasible it would be to use these to type. Given the small size of the PiTFT, we were not sure if creating a smartphone style keyboard would be large enough for the user to press with little error. Hence, as a potential alternative, we thought of a small input area where users can draw the letter that they want to type, which would be converted to text using optical character recognition (OCR). However, once we found that the small keyboard buttons were functional -- or in other words, we could detect when they were pressed and differentiate which ones were actually pressed -- and we could see that it is possible to type with them despite the small PiTFT screen, we then made the entire keyboard.

The next step was actually having the text come up on the screen in a readable format. This involved not only just printing the text, but also scrolling through the note and wrapping the text onto the next line when there were too many characters in one line. Using PyGame, we were able to create an area above the keyboard where the user can see the note that they are typing, and be able to scroll through the note, as mentioned above.

After this, adding more functionality to the keyboard, such as numbers and special characters, as well as lowercase letters was the next step. This proved to be relatively easy, as we were able to reuse the code mapping the uppercase characters, and replace the text with lowercase characters, and numbers and special characters in a similar layout to the iPhone keyboard.

The next major step after that was adding the ability to go back to saved notes and load them to the screen. This was a bit more challenging, as it first involved creating a whole new GUI "screen" per se using PyGame. The first screen was the notes screen, which displayed the note on the top, and the keyboard on the bottom. We started by adding a new button on top of that screen, clicking which would prompt the Python logic to switch the screen to display all of the notes. Also, we added some new file loading and saving functionalities, where pressing any button on the keyboard would save the note to a text file. We employed a smart naming scheme, where the program would search the "./" directory for "note*.txt", where "*" represents the number, and name a newly created note with the highest number that was found in the directory plus one. This way, the first note would be "note0.txt", and new notes would be saved as increments "note1.txt", "note2.txt", etc. After this, we created the aforementioned home page menu that would have buttons to create a new note, and load in previously created notes. For the user to know which note is which, we displayed the first few characters from each note on the button. Using this menu, the user can create a new note, or load in old notes to add more text onto.

The last major part of the application was integrating the functionality of sending a note as an email. We first prototyped a simple application, which would send some simple hardcoded text as an email. This required writing email backend code, where Python would use secure socket layer (SSL) to establish a connection to Google's GMail SMTP server, and send the email through that server. Additionally, we also needed an email address to send from. For this, we created a new GMail account, and signed up for the App Passwords, using which would allow GMail to accept connections from a Python program. Then, after being able to send a simple string as an email, we added the functionality of sending a text file as an email. For the user, they should also be able to choose a subject for the email, as well as the recipient of the email. Hence, we parameterized the subject field and recipient field, making sure to check that the recipient email is valid using RegEx.

Finally, we moved on to integrating this into our notes application. This required the creation of two new PyGame screens: one as a menu to select "Subject", "Recipient", and "Send", and another to actually input those fields. Having experience from creating the "All Notes" screen previously, as well as reusing code from that section, we created these screens with relative ease. And thus, we had a fully functional GUI and logic where users could create new notes, load old notes, and send these notes as emails.

Results

Not everything performed as planned. The text wrapping will sometimes fail and a line of text won't print on the screen, even though it is still there in the file. The email will only work with an ethernet cable connected and it will crash with certain exceptions. We mostly met the goals that we had originally outlined. The only thing we did not get implemented was the Google Drive synching. Outside of the occasional bugs, we got the results we had hoped for with this project.

Conclusion

We successfully implemented a GUI interface on the PiTFT for the Raspberry Pi 4 that could type notes in the form of text files and email them to a recipient of the users choice. We honestly did not discover anything that definitely did not work. With enough effort, it really seemed that the things we hoped for for this project would be able to be implemented.

Future Work

If we had more time to work on the project, we would have spent more time fixing text wrapping and making sure that it worked in all cases. We would have made it so the keyboard could be hidden so that the note could be viewed with more screen space. We would have made it so that the user could move the cursor somehow, and we would have made the cursor actually visible. We would have created a cancel button for sending the email (you must send an email once you press the send button). We would have fixed some of the crashes involved with trying to send an email and it not working. We would have tried to get Google Drive syncing working. We would have tried to get emailing from the user's email account instead of a specific PiNotes email. We would have tried to make a more specialized interface for email sending. We would have tried to allow the user to change font size and interface color for better usability and customizability. We would have tried to allow the user to delete notes from within the PiNotes program. A search feature would have also been pretty cool to implement as well. Something like text-to-speech or speech-to-text would also be something that could have been explored more for better accessibility. Another feature such as different language keyboards would also be beneficial. Better code organization would have also been something worth pursuing, as, by the end of it, the almost one-thousand lines of code were getting hard to navigate. There are a plethora of different directions we could have taken the project if we had more time.

Budget

Our project required the use of no extra materials outside of the Raspberry Pi 4, the PiTFT, the SD card, and the power cord provided to us by the ECE 5725 course staff.

References