The goal of this project is to design and implement a Raspberry-Pi based data-logger, capable of tracking the migration pattern of birds as they pass through the base station's range of detection. In order to accurately track migration patterns, the base station will employ the use of GPS and radio communication technology. This project seeks to improve upon the current generation of the base station, providing a more cost-effective and customizable solution by replacing proprietary hardware with an open-source platform.
The Cornell University Department of Ecology and Evolutionary Biology utilizes the attachment of small radio transmitters to birds, in order to identify them and keep track of their migration patterns. Multiple base stations are deployed over a wide range of areas to scan these radio tags and keep track of bird locations. The current base station design is rugged and portable, but it is expensive and not very customizable, due to the proprietary logging device being utilized.
To this end, we set out to use a Raspberry Pi to replace the logging device, reducing cost, and increasing simplicity and customizability. By using a Raspberry Pi, we are able to augment the base station with additional functionality that was not possible with the previous iteration of the device. For instance, if the base station is in an area with an internet connection, the Raspberry Pi could upload its logs to a central database, eliminating the need for one to be physically at a base station to retrieve the data, and providing an easy way to consolidate data across base stations.
By leveraging the extensive documentation available for the Raspberry Pi, as well as the active development community, we were able to produce a small, low-power base station with fully embedded functionality. Our device is not susceptible to data loss, and is capable of storing data locally, as well as uploading it to a server when internet connectivity is present. Due to the inexpensive hardware, the device can be produced in greater numbers and distributed to a wide variety of locations, resulting in an aggregation of tracking data that would have previously been too cost-prohibitive.
Design and Testing
The primary platform for the base station is a Raspberry Pi Model B+. In addition to the Pi, there are two primary hardware components to the system: the radio receiver, and the GPS module. Through the use of device drivers and poweful libraries, the Raspberry Pi provides us with with a layer of abstraction that makes interfacing with these modules as straightforward as possible. All hardware components are described in detail below.
For this base station, we chose to use the Raspberry Pi B+. The single ARM core clocked at 700MHz on the Raspberry Pi is more than enough for the simple tasks executed in this use-case, and brings down power consumption quite a bit compared to multicore architectures. The Raspberry Pi B+ tends to use about 30% less power compared to the quad-core Raspberry Pi 2, at about 0.5 to 1W power consumption during regular usage. In the remote locations where this base station might be, power comes at a premium, so this is important. Additionally, the Pi is made to be a low power device when idle, and is powered by a micro USB cable, which is the standard charging cable for most modern phones. This means that in places where battery or solar power is needed, we can simply buy an off-the-shelf part made for a phone, as the power usage is very similar. The Pi B+ also has four USB ports and an ethernet port, making for a very versatile setup. This enables a wide variety of use cases. The external radio reader and GPS use 2 USB ports by themselves, so having the other USB ports and the Ethernet port enables the user to use WiFi if desired, or connect a keyboard with which to view the data and check the system's operation. This would also enable the use of extra sensors in the future should they be deemed necessary. The Raspberry Pi A+ only has two USB ports and no Ethernet port. Thus, the Pi B+ provides the best balance of versatility and power usage.
The tag reader consists of the antenna assembly and an MSP430 microcontroller, which receives the radio tag and sends it to the computer through a USB-UART connection. This assembly allows for a large range of detection, and for very fast delivery of data to the Pi. There is a small amount of buffering in the MSP430, which allows the system to start collecting data as the Pi is starting up. While this data would not be accurately timestamped, it still provides some level of compensation for the slow startup time of the Pi.
The GPS assembly consists of 3 simple parts: an external antenna, a SparkFun Venus GPS, and a USB-UART connector. Once powered on and connected, the GPS attempts to get a satellite fix, while relaying its current data through the UART connection. Some sample data is shown below. From this data, we only care about the GPGGA data, as it tells us the GPS time, our latitude and longitude, as well as our GPS quality.
$GPGSV,1,1,01,05,00,000,00*4D $GPRMC,120025.000,V,2400.0000,N,12100.0000,E,000.0,000.0,280606,,,N*7C $GPVTG,000.0,T,,M,000.0,N,000.0,K,N*02 $GPGGA,120026.000,2400.0000,N,12100.0000,E,0,00,0.0,0.0,M,0.0,M,,0000*6E $GPGSA,A,1,,,,,,,,,,,,,0.0,0.0,0.0*30
We can use the GPS data to not only log where our base station is, but also to get a time base for the bird scans that is universal. The GPS module sends data through the UART once per second, so we can consistently keep accurate time for the scans.
The software provides the main functionality for this project. All code is implemented in Python, and available for download in the Appendix. We made use of modularity to make very sensical, organized code, which we explain in pieces in this section.
The first piece of the program is UART interface. This is very important, as we must first make sure that our connection to our hardware is functional before any of the other analysis can happen. We made use of Python's serial library for this. In Linux, USB UART connections are mapped as filesystem objects in the /dev/ folder. After creating two connections and setting the baud rate for each to 9600, we created a round-robin schedule of communication functions, consisting of the GPS communication and the tag reader communication. We set up both connections to have a read timeout of 0.1 seconds, such that neither function would block the other for very long, and would yield if it had not received data. We only take input data that is properly terminated with a terminating character, \n, thereby eliminating cases where the string is malformed, and ensuring that we get the full data each time. Once we have data, though, we must parse it to ensure that it is the string we are looking for, and take action as a result.
One important factor of this part of the software was ensuring that the system could survive either the tag reader or the GPS being disconnected. A GPS disconnection is not fatal, as the system can simply use last the last GPS data that it received. However, if the tag reader goes missing, the system should not continue. To this end, the system will go into a loop of attempting to reconnect with the tag reader until it succeeds.
GPS Data Parsing
As mentioned before, the GPS UART connection sends in a lot of data that is not of use to us, as we only care about the GPGGA sentence. In this case, we use regular expressions (regex) to check if the input string is a match for the GPGGA sentence structure. If it is not, we can safely skip over this data and move on. If it is however, we can update our system time, latitude, and longitude with the data, which we do. To signal the start of a logging session, we also print this data to the log.
Tag Data Parsing and Logging
When we get new data from an radio tag, we must take the necessary steps to make sure that it is a valid tag (12 hexadecimal digits). This is easy to do with regex, as mentioned in the prior section. We can parse out the 4-byte tag ID, as well as the 2 bytes at the end, which give us information about signal strength and link quality. Once we do this, we need to take the necessary steps to properly log the data.
The system has two methods of logging: internet-based, and locally. The local log contains all data for the logging session, including the location, all tag scans, and indications as to whether the data has been backed up to the internet. The internet log sends a JSON packet to an offsite SQL database when there is internet, and stores it in a buffer locally if there is no internet. When we begin to log the data, we assemble a JSON message containing the scanned tag, as well as a similar string for storage in the local data log. As mentioned before, regardless of whether there is internet or not, we store the data to the local log. We then check the status of our internet connection by pinging the server we intend to send to. Here, we distinguish among four cases:
- Internet was connected before, and is still connected
- Internet was disconnected before, and is still disconnected
- Internet was connected before, but is now disconnected
- Internet was disconnected before, but is now connected
The first three cases are relatively simple. In case 1, we simply send the JSON packet to the server and write to the local log, including an indication in the local log that the data was sent. In case 2, we store the packet in a local buffer to be sent later, and write to the local log. In case 3, we close our socket connections if need be, and then store to the buffer and the local log. The 4th case is the most complicated, since we must first send all of our backlog to the server, and then send our most recent data. After looping through the buffer and sending out all the old data, we put a statement in the local log confirming that these values were logged. We then write the new data to the log and to the server, and clear the buffer for the next time we lose internet.
Sending the data to the server has potential to slow down the program, which we would like to avoid as much as possible. To this end, we made use of Python's inbuilt threading to spawn a thread to send the data. This way, the program could move on, and the spawned thread could handle making sure that the data was sent successfully by itself without hanging up the program. This will not lead to problems in the program with too many threads, since tag scans do not come in very quickly, and with an internet connection, they finish relatively quickly. If internet is lost, the program will stop spawning threads, and the threads that might still be in progress can simply wait, as there will not be many of them.
The testing for this project was very rigorous, as each section of the project was tested independently, and was incrementally integrated into the final product. The first tests involved the UART communication, in which we used a computer USB-UART interface with PuTTY to test that we could take correct inputs from the UART connection and parse them. After this, we tested the GPS and tag regex parsing on a variety of data we received from the rest of the team to ensure that we would not be missing any signals.
The next step was to find a suitable method of logging the data to the local log, and ensuring that program crashes would not cause any issues for writing to the log. Using a variety of Python string tricks, we were able to find a clean, suitable, and readable format for both the GPS and tag data in the log file. After ensuring that we could write to the log, we tested the program's resilience to crashes, ensuring that the program would not hang on to a lock on the file and disallow a future execution from accessing it.
The next unit test was ensuring the system could send out data over the internet. This was not easy to test, as the database that the base station will be writing to was not set up as of the time of this writing. However, we made use of Python's smtplib to instead send us this data via email. By unplugging the ethernet jack, we were able to use these email tests to ensure that our program was making the correct decisions about logging to the internet or not based on the state of the internet. More importantly, we were able to test if the station would send all of the backlog after it regained internet connection, ensuring that all data gets logged to the server at some point or another.
The final test of this project was putting it all together with the tag reader and GPS from the team, so we could actually check real-world functionality as opposed to just simulated strings through PuTTY, which was successful. Through all of these methods, we were able to fully validate our base station, and test the resilience of the station in an unreliable or variable internet connection.
The system worked as expected during our real-world tests. While the parameters of the project changed occasionally, we were able to adapt and meet the vast majority of the most recent criteria for the team. Although our time was limited, we were able to work quickly to implement two UART connections, as well as implement the internet functionality almost entirely, only remaining unable to test the SQL database connection. The goals we had initially outlined were met, as we were able to create a low-power base station that could handle all of the logging required by the system. The system is far more cost-effective than the logger currently in use, and can write to standard format that anyone can use.
The UART connections were found to be the weakest point of this project. The connections on the MSP430 in their current form are not extremely secure or protected, leading to some finnicky connections if we are not careful. However, since we have handlers for this, our system did not fail.
The rest of the project worked as we expected. In steady state, the speed of the project was plenty fast enough for our purposes. We are able to adjust the timeouts for the GPS and Tag UART streams, as well as the check for internet, so that we can ensure that our GPS time does not fall behind due to delays in other parts of the program. Additionally, we can make sure that a failed ping doesn't force the program to hang for an excessive amount of time.
Throughout our development, we tried various different implementations. This included trying both WiFi and Ethernet, both of which work on a plug-and-play basis, making this system very versatile, as we had desired. Additionally, our experimentation with the Raspberry Pi's UART pins offers potential in the way of adding devices, as we now know that the base station can also use these pins for another peripheral. The code would not be difficult to extend to handling this. Our last experiment was attempting to use the onboard SPI capabilities of the Pi to write to an SD card, instead of using a USB-based drive. This would serve to make the device slightly lower-power, but would also cause a much more complicated software module, and was a vast undertaking in terms of coding, so we decided against pursuing this route.
The major drawback of this new base station as opposed to the old one is startup time. While the old base station starts up in seconds, the Raspberry-Pi base station takes over a minute to start up, even longer if we are using the email-based implementation instead of the database implementation. However, the features of the new base station make this worth it, since startup time is not a particularly important feature of the system. Once the system is running, it can go forever without interruption.
This project achieved almost all of the results we desired. The Raspberry Pi-based logger is able to bring down the cost of the overall base station while extending its functionality. The major advantage here, other than cost, is the fact that we can use any method of mass computer storage now, instead of being tied to the method used by the proprietary logger. The new Raspberry Pi logger also opens the system for extensibility, as a new peripheral or change in the data format just requires a small code modification to change the entire system. This moves the base station to more open formats, such as Python code, that anyone can learn quickly and modify. Our software is very modular, and is easy to verify as a result. The project was a little bit rocky as far as which parts of the requirements we wanted to attack. We eventually declared power management out of scope for our project, since we figured that using off-the-shelf power management made for cellular phones would be sufficient, but it's certainly something that could be researched further.
One major lesson from this project was to harness the strengths of the Raspberry Pi instead of trying to avoid them. This was mostly exemplified by our attempts to use an SPI-based SD card instead of simply using USB mass storage. After hours of research, it seemed like the effort required to make use of a standard SD card filesystem via SPI would just be more effort than it was worth. Since our primary aim here was using common formats, it just did not make sense to implement the SPI SD card. Additionally, our original attempts to use the physical UART pins instead of USB UART was another lesson in using the strengths of the Raspberry Pi, as we found ourselves sometimes struggling with the physical pins and ensuring our computers could connect correctly. This would have required a change in the design of either the GPS receiver or the tag reader. While this would cut down on cost, since we would need one less UART USB converter, it was simply not worth the effort to attempt this.
This project makes a compelling case in the bigger picture for embedded logging based on the Raspberry Pi, due to all of the aforementioned advantages, including open standards, cost effectiveness, and ease of programmability. While programming on bare metal and bit-banging might introduce some speed advantages, the time to develop them is far longer, and is not as easy to spread to others. With embedded Linux devices such as the Raspberry Pi now having variants that cost as little as $5, there is increasingly more reason to adopt these devices over something closer to a microcontroller, especially if a wide variety of functionality is needed. Attempting to create this project, which includes UART, internet, and local logging, could take weeks with a microcontroller, but thanks to embedded Linux, we can drive this time down to a matter of days.
There are a few ways that this base station can be improved. In the future, when the logging server is set up be accessible, the functionality of the JSON sending function will have to be tested, as it has been implemented but not at all validated. Additionally, a future developer on this project could try implementing a method of automatically getting the current time from a variety of sources, whether this is internet, GPS, or a real-time clock (RTC) attached to the Raspberry Pi.
One major area for improvement in this system is power consumption. While the Pi B+ offers some improvement over the Pi 2, it is still not extremely good for a remote environment where power might not be as readily available. To this end, a future developer could create a battery-based system that communicates with the Pi to ensure it shuts down if battery power gets too low, and starts back up if the battery is high enough to handle it. This would be a natural addition to a solar-powered system, which would help this system spread much further. Additionally, a future developer might attempt to port this system to the Raspberry Pi Zero, which is a recent new offering from the Raspberry Pi Foundation. The Pi Zero is smaller, costs 20% of what the Pi B+ does, and could easily fulfill most of this project's needs. The power consumption on the Pi Zero is the lowest of all Raspberry Pi products, consuming roughly 1/3 of what the Pi B+ uses.
Shiva and Taylor worked together on most of this project, dividing the work equally.
Taylor's contribution primarily focused on implementing data parsing, and local logging. For data parsing, Taylor used code from the GUI team on this project to learn the different parts of the incoming data and how to parse them. The GUI was written in Java, so porting this to Python was part of Taylor's duties. Additionally, Taylor implemented the necessary code to log to a local data file, and making this easy to modify depending on what kind of storage will be used.
Shiva's contribution was primarily UART communication, internet communication, and integration. At first, we planned to implement at least one UART connection with a wire interface, since the Raspberry Pi has one and this would save us a USB port. Shiva implemented this, but decided against using it in the end, since everything that already exists on this project used USB-UART connections. After setting up this interface, Shiva used the aforementioned GUI code to learn how data was sent to the offsite server, and provided an interface for this to occur through the Python code. Finally, Shiva integrated all of the code together, including the internet test, and making decisions on logging data based on that.
Bill of Materials
|1x Raspberry Pi B+||$20|
|1x 4GB USB Stick for Logging||$5|
|Total for Logger||$25|
|2x USB-UART Cables||$20|
|1x SparkFun Venus GPS||$50|
|Total for Base Station||$105|
CodeBase Station Code and Readme
ReferencesVenus GPS Datasheet
Using the UART (for Raspberry Pi)
Python smtplib Documentation
Python datetime Documentation
GPS - NMEA Sentence Information
RegExr: Learn, Build, and Test RegEx
We would like to thank Professor Joe Skovira and our TA Gautham Ponnu for helping us through this course and project, and being extremely helpful throughout! A massive thank you to David Winkler and Rich Gabrielson, as well as the rest of the team, who welcomed us to the team with open arms and were always prompt about answering our incessant questions. Also a special thanks to Bruce Land for use of his resources at various points throughout this lab, whether this was loaning us a lamp or providing suggestions for how to approach different issues.