Speech Recognition using Raspberry Pi 2

Design and Testing    top

As mentioned above,the final project can be broadly divided into two parts pertaining to speech recognition: one dealing with the open source voice assistant, Jasper's, implementation, and the other involves implementing an offline, embedded speech recognition system.

The hardware used for the project includes a passive USB microphone to input the voice and a self-powered set of speakers. The passive USB microphone was chosen for two main reasons. First of all, the USB Microphone would work without an additional sound card attached to the Raspberry Pi 2 because the sound card it built-in. As a result, the USB microphone would serve the purpose of plug and play. In addition to this, a passive microphone was chosen over one that is controlled by a button for input in order to allow for ease of use on the user's end. This would allow the user to simply provide Jasper with a command by speaking out loud to it rather than pressing a button to have the program operate. The speakers selected are self-powered so that its power supply is independent of the Raspberry Pi 2. This was mainly chosen since a USB-powered speaker would consume an additional USB port and it also acted as a precautionary measure to protect the Raspberry Pi 2. Besides this, the audio port on the Raspberry Pi 2 is an ‘audio out’ port, so the microphone cannot be used through the audio port. In this project, we use the eBerry Plug and Play Desktop USB Microphone and Logitech Multimedia speakers. The cost of these are specified in the Bill of Materials (BOM) section.

Jasper

Jasper is an open source platform similar to voice assistants such as Siri or Cortana. Jasper's implementation that is currently specified on its official website [1] is for the Raspberry Pi 1, and not the Raspberry Pi 2. However, a manual compilation would allow the system to successfully work on the Raspberry Pi 2. The compilation that was used is courtesy of [2], since the aim of the project is testing and comparing the speech recognition aspect with other speech recognition platforms, and extending and developing additional applications that utilize the Jasper framework.

Our implementation involved Jasper using Google as the Speech To Text (STT) Engine and Espeak as the Text To Speech (TTS) Engine. Google's STT is based off the Google Speech API, where developer keys are required, where each key provides 50 requests per day. With this being said, utilizing this engine requires an active internet connection to work. To create an embedded device devoid of the internet connection, we would need a speech recognition system independent of the internet. Pocketsphinx, which was developed by CMU, does not require internet connection; however, on testing its SST engine on Jasper, we determined that the recognition was quite inaccurate and led to the execution of incorrect commands.

Jasper, on compilation, had built-in features such as telling the time, weather, news, and email. For the email functionality, Jasper would provide the number of unread emails and would specify the name of the sender of there are only a few unread emails in the inbox. These features worked accurately and gathered information from external online resources. Further additional standard modules were implemented based off the Developer API instructions provided by [1].

Using a developer token, Jasper was interfaced with Evernote. This enabled the user to instruct Jasper to take notes that were then stored on your Evernote account. The voice command used here is “Note”. After providing the note, Jasper will write the note on your Evernote account accurately and with minimum delay.

Jasper was also introduced a module to state its current kernel and CPU utilization. This is evoked using the command “Status”. Also, another module evoked by “Movies” would enable Jasper to give details on different movies, depending on the user's choice.

Finally, a standard module allowing Jasper to play music stored on the Raspberry Pi’s memory was implemented. This module was evoked by saying the keyword “Music”, currently it has been tested for just one song, but its functionalities are easily expandable. In this module, the music was played using Omxplayer.

When testing the SST engines for accuracy and functionality, everything worked smoothly. As stated above, it was determined that the offline SST, Pocketsphynx, was not reliable and resulted in several false-positive results. This conclusion was able to be made because the text output of the speach to text conversion was printed back to the user. Looking at the return values from the actual speech, the hit rate of correct words with the expected was very low. In addition to this, the built-in modules and additional modules created displayed expected results during the testing phase of the project.

In terms of speech recognition, Jasper, combined with Google's SST, has a high recognition rate for the inputs given by the user. It not only considers the word said but also takes into account similar sounding words to predict the input given by the users. On Jasper, while developing the modules, the developer can specify the words required to evoke the command as well as related words to recognize the module evoked.

However, like all recognition systems, it has certain drawbacks. Firstly, for an accurate recognition on the Jasper, internet connection is a requirement and an active internet connection for speech recognition is not always accessible or desired. Secondly, using the Google Speech recognition engine for free has a cap of 50 requests per day and forcing the user to generate multiple keys and switching them is quite tedious and not durable for an embedded system. Finally, a major downside of Google Speech is the fact that it is sensitive to the user's accent. For instance, American accents work quite well whereas, for a user with a different accent, Jasper’s recognition is quite terrible.

These are just a few reasons which arise the need for an internet devoid, accent independent, and durable speech recognition system. The second part of this project deals with implementing such a system resolving the drawbacks of Jasper.

Offline Speech Recognition - "Deluminator"

The Offline Speech Recognition portion of this project consisted of three different parts: collecting audio samples, training, and the actual voice recognition portion.

Voice Recognition with Mel Frequency Cepstral Coefficient (MFCC):

The Mel-Frequency Cepstral Coefficients (MFCC) are coefficients that are derived from taking an input audio clip. On a high level, the MFCC is derived by first taking the Fourier transforms of the signal, mapping the powers of the spectrum onto the mel scale, taking the logs of the powers at each mel frequency, and finally taking the discrete cosine transform of the log of the powers. This will result in coefficients that represent the amplitudes of the input signal. A more detailed implementation of how the coefficients are found and extracted can be found in the References section [4].

Using Python, these values were generated and then stored into a database that allowed us to create a simple learning system, which will be explained in more detail later. The program to provide the MFCC values was written incrementally, allowing to ensure that not only did the coefficients come out properly, but to ensure that they were consistent.

Audio Samples:

Before finding the MFCC values that were used to create a fingerprint for different words that were pronounced, the audio samples had to be obtained. The first step of achieving this was to simply have the Raspberry Pi record a sample of our voice, input the recording into the Python program that was written, and extract the MFCC coefficients. The first program that was used to record commands was ‘arecord’. Below are two samples of the .wav files that were recorded using ‘arecord’.

Figure 2: Audio Recordings Using 'arecord'

The problem with using 'arecord' as the form of recording was that the user was to manually start and stop the recording. The main downside of this is the inconsistency of having the user stop the recording precisely at the same time each time a recording was made. Having recordings that are roughly the same length is very important because the power spectrums that are retrieved from taking the mel transform will lead to drastically different output coefficients, making the system difficult to train. In addition to this, using ‘arecord’ lead to a lot of undesired background noise that was also mixed into the signal, leading to an imprecise recording of the desired keyword.

Because of the challenges that were faced when using 'arecord', a couple of ideas came to mind. One idea that came up was to take an input sample, filter out the background noise, and then clip the audio so that only the desired spectrum is retained before taking the mel transform of the signal. The other idea was to set a threshold value that would record the signal only if the amplitude of the signal was greater than that value. With these two ideas on mind, 'pyAudio' was discovered, which allows the user to edit audio inputs in Python. Specifically, as desired, it provides the user with the ability to manipulate an input audio file and also to set a threshold which only records when the input amplitude is larger than that threshold value.

With this library at hand, we were able to test to see if the samples were only recorded when desired and if the background noise was cancelled out compared to the recordings that were retrieved when using 'arecord'. Initially, besides retrieving the signal, the input signal was zero-padded on both ends of the voice spectrum by half a second in hopes of improving the variability in mel coefficients after the transform has been taken, as there is a greater difference between nothing being heard and the voice itself.

Figure 3: Zero Padded Signal

As can be seen above, using 'pyAudio' helped clear up the signal a lot and that the desired spectrum is prominent relative to the noiseless zero-pads and filtered-out background noise (which was accomplished by setting the threshold value). While this is the case, however, when taking the MFCC to find the spectrum coefficients, the coefficients found all came out to be 0. The reason that this was the result was due to the amount of zero padding that was added to the input signal. When the zero-padding was taken out, the coefficients were of a reasonable value. Solely looking at the output coefficients of the MFCC, it can be seen that different recordings of the same word looked different, but were much more similar than the coefficients of different words. With this, we were able to implement the offline speech recognition portion of the project. The final non-zero-padded signal can be seen below.

Figure 4: Trimmed Auio Signal

Finally, after the coefficients were extracted from the input signal, it can be seen that even though the values vary slightly, each indexed value in the array is very close to the value in the same indexed location of a different input signal for the same keyword. This can be seen in the plots below:

Figure 5: MFCC Output

Besides finding the best way to record an input sample to allow for accurate recognition, this portion of the project worked out smoothly. In terms of testing to ensure that the word that was being recorded was not only the correct word, but also audible and not covered by static or background noise, before initial training was completed, each recording was played back. This allowed us to make sure that the training set would be as accurate as possible.

Training:

In order to ensure that the system is able to recognize keywords, it must be trained. This was completed by writing a program 'training.py' that is solely used for training the system. A high-level block diagram of how the program works can be seen in the Drawings section.

Similar to what was explained above, the program will utilize the 'pyAudio' library to take an input sample .wav file and generate the MFCC values. The program will then search through the database of values based on previous training values. If there is a hit, then the program will ask “Did you say XYZ,” where "XYZ" is a keyword that has already been stored to the database. If the keyword matches what the user said, the user will type “y”, and the new coefficients will be put into the database. If the word "XYZ" that the program prints out is incorrect, the user can input “n”, and the program will ask “what did you mean?” The user will then have the ability to type in the correct word that corresponds to the recording that was just made. These new coefficients will then be added to the database of coefficients.

This type of training is very useful because it not only allows the system to be robust as more and more samples are written into the system, but also because it allows the program to behave on a more personal level. When using Jasper and relying on the recognition of Google’s speech API, while this speech recognition is one of the best out available to users today, the ability to train the speech recognition directly on the Raspberry Pi allows the system to be configured and customized to the user’s tendencies. For example, it can be seen in the video that the two group members are pronouncing the same word differently: because of the training that has been done on the group members’ voices, the system is able to identify the keyword correctly.

Specific to the system that has been created, the system has been trained to recognize the following three words: “Lumos”, “Nox” and “Close”. When “lumos” is recognized, the LEDs on a breadboard will light up. These will stay on until “nox” is said by the user, where the lights will turn off. If the user says “close” to the system, the system will quit the program.

This portion of the project went by smoothly as well. In order to test this training system, several recordings were already made to both train and test the system for accuracy and consistency.

Comparison

With the two implementations explained above, it can be concluded that speech recognition using MFCC certainly has its advantages and disadvantages. Here, there is no need for an internet connection since the Raspberry Pi 2 itself has the fingerprints of the speech stored and recognizes based off these values. This thus allows for a completely independent embedded device to operate without additional external online dependencies. Besides this, regardless of the language or accent, any word or phrase can be trained or pre-defined using this method, as proved by the demonstration of our “Deluminator”. This provides an ocean of opportunities for practical applications.

The flaw with regards to the offline speech recognition implemented is fact that words have to be trained to be recognized. Of course, the more the training values provided, the better would the results. For a dictionary of words, storing on the Raspberry Pi 2’s memory alone would not be viable and traversing through the entire memory for a particular word is quite tedious. This could be implemented over the cloud or elsewhere, but in terms of making the device purely embedded, this is not viable. The greater the training sets, the greater would be the delay in recognition. While the system can be optimized to decrease this delay, this would futher complicate the overall implementation. With this being said, one way to improve the speed of searching through a large database is to utilize all four cores of the Raspberry Pi 2.

Software

Jasper:

Developing modules for Jasper using the Developer API is user friendly and provides a wide range of applications that can be implemented. Considering a standard module implemented such as “Music,” the first step involves defining single-word strings known as “WORDS”, this is used to determine whether the module evoked is a Jasper module or not. For example,

WORDS = ["Jason", "Derulo", "Music", "Play"]

The ‘def handle(text, mic, profile)’ function is used to specify the operations to be executed when the appropriate module is evoked. Further, ‘mic.say’ is used to specify what Jasper must tell the user while ‘mic.activeListen’ emits a beep sound and waits for user input. Here, mic.say is used to tell the user the artist of the song being played. ‘subprocess.call’ executes subprocesses as well under the ‘def handle’ function. The ‘def isValid(text)’ function is used to define the input required by Jasper from the user to evoke the particular module. Here, “music” is the keyword to evoke the module.

These commands can be further expanded to suit the requirements of the developer as explained in the example for the Music module.

Offline Speech Recognition - "Deluminator"

Once the program is up and running, the microphone will passively stay on in the background, listening until a sound that is greater than the threshold value (coded in) is detected. Once this signal is detected, then the following function will be called, which will record using the microphone and save to the path provided:

record_to_file(‘recording.wav’)

The record_to_file() function above will call the record() function, which does the actual recording. This function is in charge of making sure that the amplitude of the input signal is greater than the threshold value that has been set. Once the amplitude drops before this threshold value, then the input signal will stop recording. After this is completed, the recorded signal will be normalized with respect to the maximum volume level set by the user. The purpose of doing this is to help smooth out the audio signal. Finally, the trim() function is called.

The purpose of the trim() function is to clip the recording. As stated above, the microphone will continue to listen in the background until it hears a signal which is greater than some coded threshold value. The trim() function will take the recorded signal and ensure that everything that is being saved is greater than the programmed threshold value.

In this project, the Mel Frequency Cepstral Coefficients (MFCC) for offline speech recognition are stored in numpy arrays. These arrays are two dimensional (2D) arrays and they are compared with one another using numpy functions.

The numpy boolean function used for comparison in the numpy.allclose(a, b, rtol, atol). This function will return TRUE if the following equation is satisfied,

absolute(a - b) <= (atol + rtol * absolute(b))

where ‘a’ and ‘b’ are the input numpy arrays to be compared and 'rtol' and 'atol' stand for the relative and absolute tolerance parameters.

The input values could be adjusted as per the requirements of the condition to be satisfied. In our project, we set ‘rtol’ to 0, allowing us to simply check the absolute difference of the two numpy array for a tolerance level specified by 'atol'. This allowed us to set a threshold of difference between the input array and stored arrays in the database, allowing a certain level of variation.

Training

In terms of the software written for training the system to recognize words, the recording and comparing coefficients remains the same as explained above. The main difference in this program is that it will call the training() function. The training function will write to a .wav file, obtain the MFCC values, and then check it against the most recent database. The program will then check or verify with the user on whether the recognition was completed correctly or not. If the program was able to successfully recognize the word (there is a hit against the dataset), then the new values will be added to the database of values, allowing the system to continue to learn. If, however, the program is not able to successfully recognize the word (either the word was recognized to be a different word - false positive - or there is no match whatsoever), the user is able to input the correct word that corresponded with the recording. After this is completed, the MFCC coefficients will be added to the database.