Design and testing    top

Hardware

The main hardware setup consited of a Raspberry pi B, a USB microphone from eBerry, Logitech S120 speakers, a 4 channel relay module from Songle, LEDs for demonstration and current limiting resistors.

The first task of the project was to develop a resilient and easy to install hardware module. The GPIO pins of the Raspberry Pi can output 3.3V or 5V and a maximum of 16mA per pin. The total current it can output is about 50mA. This was an important parameter to be considered while designing the external circuit. To prevent overload, we used current limiting resistors of 1K on each output pin.

To drive the main 110V line, we used a LC-201 4 channel relay module which has current specifications similar to what our Pi can drive. It requires voltage signals of 5V but we discovered that it can also be driven by 3.3V pins. Hence we did not use a separate level shifter circuit. The relay has an inverting output.

For testing purposes, we used 12V as the main line (instead of 110V) for safety, and drove two 12V, 15mA LEDs through the relay. To demonstrate how fan speed could be controlled, we implemented PWM on an LED. The LED brightness would be varied as per the users commands. Two leds would also be randomly switched on and off when a command to play music is recieved to give the effect of party lights.

Software:

To design this project we used Jasper framework to control the processing and handling of actions. And for Speech to Text conversion we used wit.ai STT and for Text to speech conversion we used Mary-TTS . We can basically use any of the speech to text converter and any text to speech converter as per the requirement but in our case since we wanted a good success rate and also did not want any daily limit on our number of STT conversions. So, we decided to go for wit.ai . The steps performed during software installation and configuration are mentioned below.

1) Setting up Audio devices :

Very first step in this project is to configure the audio devices such as mic and speaker and to ensure that they are working properly before moving on to STT and TTS. For this we connected our USB mic to the usb port of the raspberry pi board and mic to the 3.5mm audio jack and moved on to following steps.

Firstly, we confirmed if alsa (Advanced Linux Sound Architecture) is installed in our system or not by running alsa-mixer command, in our case the alsa screen showed up as shown in Fig 2 but if it is not then you may need to install it using below command

apt-get install alsamixer

Once on this screen you can check the devices you have connected using F6 key and set the audio record/play levels for each device using F3/F4 keys.

If your devices are not found in alsamixer screen then you can confirm if the devices are detected by your system by checking on a slightly lower level using below commands.

lsusb
pi@srs383_ps882:~ $ cat /proc/asound/modules
0 snd_bcm2835
1 snd_usb_audio
pi@srs383_ps882:~ $
pi@srs383_ps882:~ $ cat /proc/asound/cards
0 [ALSA ]: bcm2835 - bcm2835 ALSA
bcm2835 ALSA
1 [Device ]: USB-Audio - USB Audio Device
USB Audio Device at usb-3f980000.usb-1.5, full speed
pi@srs383_ps882:~ $

Then, to record you should be able to run the arecord command and to play you can use either speaker-test or aplay command. Please note below commands will work only if your default devices are correctly set.

Recording WAVE 'test.wav' : Signed 16 bit Little Endian, Rate 8000 Hz, Mono
^CAborted by signal Interrupt...

pi@srs383_ps882:~ $ aplay test.wav
Playing WAVE 'test.wav' : Signed 16 bit Little Endian, Rate 8000 Hz, Mono

In our case they threw error, since the default values mic and speaker were set to be the same ie. Hw:. So to resolve this we modified the commands to take hardware devices from command line itself. Commands shown below:

pi@srs383_ps882:~ $ arecord -f S16_LE -r 60000 -c 1 -D hw:1,0 test.wav
Recording WAVE 'test.wav' : Signed 16 bit Little Endian, Rate 8000 Hz, Mono
^CAborted by signal Interrupt...
pi@srs383_ps882:~ $ aplay -D hw:0,0 test.wav
Playing WAVE 'test.wav' : Signed 16 bit Little Endian, Rate 8000 Hz, Mono

Here, the -D option allows us to configure our correct audio device hardware. For this you need the sound card number and module number which you can get from below commands for arecord and aplay.

pi@srs383_ps882:~ $ arecord --list-device
**** List of CAPTURE Hardware Devices ****
card 1: Device [USB PnP Sound Device], device 0: USB Audio [USB Audio]
Subdevices: 1/1
Subdevice #0: subdevice #0
pi@srs383_ps882:~ $

Alternatively, you can also modify the values in alsa-base-config.conf file which in wheezy is stored in sudo vi /etc/modprobe.d/alsa-base.conf and in Raspberry Jessie is stored in sudo vi /usr/share/alsa/alsa-base.conf file

2) Installing Jasper:

Once we are sure that audio devices and their drivers are working fine we can move on to the Jasper installation. For this in the home directory of your Pi, clone the Jasper source code, depending on your internet speed it might take somewhere around 5-10 minutes:

git clone https://github.com/jasperproject/jasper-client.git jasper

To run properly there are some python libraries that jasper would need and to install those we can run below two commands.


sudo pip install --upgrade setuptools
sudo pip install -r jasper/client/requirements.txt

and finally to ensure that when you put the jasper script in crontab it runs properly you should make an executable.

chmod +x jasper/jasper.py

3) Installing dependencies :

Next step is to install the dependencies to support Jasper i.e. Jasper only provides the frame work to process the speech and configure the handling of actions on voice commands but there still needs to be a speech to text converter and a text to speech converter. As mentioned above we are using wit.ai in our case so we don’t need to install any extra software for that as the conversion is done over the internet. So, we only need to register for STT conversion on the wit.ai page. To do that, we first create a profile on wit.ai and create a dummy app. Then we need to select the dummy app, click settings and find Server Access token, you need to hold on to this token as it will be needed during configuration phase. Please note that in the settings you can also choose another language for wit.ai.

Note: If you choose to go for Pocketsphinx STT then extra software installation is needed.

And for text to speech conversion we are using Mary-TTS which again does not need to install any extra software. Other Options such as festival or espeak could be used if needed but they will need a package installed in local system. Also, please note that official site of mary-tts is not running the server anymore so you can either host it on your own server or use someone else’s server. In our case, we used a third party server for mary-tts

4) Cofiguring Jasper:

After the installation of jasper and its dependencies is done we can go ahead with the configuration part which basically involves setting up a profile.yml file in ~/.profile folder. This can be done either by putting in the values manually in the file or we could use is by running populate.py script which guides us through the information and stores the responses automatically in the profile.yml file.
Important part of profile.yml are mentioned here.
You will need to mention your access_token in the profile under the stt_engine setting.

stt_engine: witai
witai-stt:
access_token: YOUR_ACCESS_TOKEN

Additionally, the TTS engine needs to be configured using below statements. Please note that the server address and port are to be mentioned below the section where you have specified the tts_engine. Also, please note the “tab” in front of the sub sections

tts_engine: mary-tts
mary-tts:
server: 'mary.dfki.de'
port: '59125'
language: 'en_GB'
voice: 'dfki-spike'

5) Home automation configuration:

At this point we are done the setup of the jasper framework and next few steps will consist of utilizing that framework for our objective ie. configuring commands and handlers for home assistance
Locate the folder ~/jasper-client/client/modules/ and inside there are some preconfigured modules which are ready to use once the basic configuration for them is done. All the new modules that need to be setup will be created here.

For each of the moduels there are mainly 3 parts First, at the top there is word section:

WORDS = [“YOUR”, “WORDS”, “HERE”]

Where different words that you want to be taken as a trigger for this module can be mentioned ie. Whenever, these words are encountered by brain script of jasper this module will be invoked and executed.
Second, there is a isValid section at the bottom which basically validates the keywords received by mic script of jasper. Ie. Once the module is triggered the entire text will be matched with your preset matching pattern and if the match is found then execution will move to next step else it will be stopped.

def isValid(text):
return bool(re.search(r'\byour words here\b', text, re.IGNORECASE))

This is a normal regex match for your pattern, which means that if the subtext “your words here” is found in any text statement forwarded by mic.py then the function will return true.

Third, once the pattern match is found to be success then the control is given to the handle function which takes “text” (the actual text received by mic) , object mic(which allows you to pass your text to your TTS engine) and profile (the information specified by you in your profile.yml file which includes your gmail username, password, facebook username etc.) as the input and performs the actions as mentioned in the handle function.

def handle(text, mic, profile):
mic.say(‘Hello’)

For the purpose of our project we have written our own module LightsOn.py in which we are using this handler to write the code streams to a fifo file and the value of the code streams depend on the text being given to our module by mic.py. These code streams are then read by a polling script which performs necessary action of switching on/off a GPIO pin etc.

In LightsOn.py we have currently configured below mentioned code streams:

BEDROOM – 1
LIVING ROOM – 2
JUMP – 3
FAN – 4
FAST – 41
SLOW – 42
SLEEP – 9

Ie. Whenever the value of “text” is BEDROOM then LightsOn.py module will be invoked by brain.py and inside the LigthsOn.py module first the validation of text will be done which will come out to be true if the text contains word “BEDROOM” which may or may not be surrounded by other words. On getting “true” in return of validation the handler script will then write value 1 in the FIFO ~/home_auto and waits to receive the response. The further heavyweight work is done by the polling script, which will then read the FIFO and match the read value through its conditional statements with configured values. Upon finding that it is indeed for bedroom, it checks the current status of bedroom lights which, if ON will be turned OFF or else it will turn them on. After turning them on/off it will write the response that is to be spoken by jasper to the same FIFO , which in this case is “Bedroom lights turned on/off” . LightsOn.py module waiting to receive a response will read this value and pass it to the mic object and then exit. At this point jasper has done its part and goes on to passive listening again. An outline of software design for this can be shown as below.

The polling script that we have mentioned above is an infinitely looped python program which reads the FIFO file and if the file is found to be empty the program would sit there waiting for some data to be written there. This is achieved in our script by using os.open call with BLOCK mode. Please note by default os.open call gives you Blocking mode only. Then once the output is read from the FIFO it will check the conditional statements and after performing GPIO action will write the output to the same FIFO file and it will sit there until and unless the Module LightsOn.py will read that and then the polling script will have its first loop completed and will again go to the FIFO file and wait until some content is written into it.

On its execution below output is captured from the polling script

The final cog in the wheel is the Jasper.py script which is what runs the entire jasper framework. Other scripts such as main.py and brain.py and the module calling etc is called by Japer.py itself depending on what action needs to be taken. Below is the output from the execution of Jasper.py script.

For ease, both the polling script and Jasper.py script can be put in the crontab with @reboot option so that at every reboot both scripts start running automatically and are always listening without having to need any manual intervention.

@reboot python ~/Jasper/jasper-client/Jasper.py;
@reboot python ~/Jasper/temp_scripts/bedroom_lights.py;

---