Objective

Plants are a way that many people use to spruce up their homes and quarantine has only increased plant sales. However, plant owners rarely know the proper way to take care of their plants and get frustrated when their plants get sick or die. With our plant care system, users will no longer have to worry about watering their plants or whether or not the plants’ environment is suitable. Our system automatically cares for indoor plants through watering, monitoring, and notifying users when suboptimal conditions are detected.

Project Video

Introduction

For our project we created a system that performs automatic plant care and monitoring. Currently it is able to support the care of two plants. The system uses various sensors to monitor the plant and its environment and configure its care accordingly. The soil moisture sensor is used to check each plant’s soil water content and water using a DC water pump if water is determined to be lacking. The temperature, humidity, and sunlight sensors are used to notify the user if any of the parameters are sub-optimal for each plant’s needs. The system utilizes web scraping of the website www.mygarden.org/plants to personalize its care to each plant. A camera and OpenCV are used to track plant height and health over time. There is also a web application connected to the system that can be used to view a live feed of the plants as well as track the plants’ statuses and environmental factors over time. The web application also allows users to water the plants even while they’re away from home. The combined system allows users to rely on the system for plant watering and monitoring.

Design and Testing

System Design Overview

Diagram

Figure 1. System Schematic

Hooked Up

Figure 2. Photo of system hooked up with plants, camera, and box

Outside Box

Figure 2. Photo of system hooked up with plants, camera, and box

The system is able to care for 2 plants at any given time. At a high level, the system utilizes the sensors and web scraped data to configure the watering intervals and notify the user of any suboptimal environmental factors for their plants (sunlight, temperature, and humidity). The user can first go to the web app to enter their plant types after which the system will make configurations based on the types.

The website mygarden.org is web scraped for data regarding the user’s plant types. The website contains information about how moist the soil of each plant should be and based on that data, the watering interval is set accordingly. The 2 types of soil moistures that the website contains are: well-drained and moist but well-drained. If the soil type is identified as ‘well-drained’, the watering interval is set to 5 days. If the soil type is identified as ‘moist but well-drained’, the watering interval is set to 3.75 days. If the soil type is undefined, the watering interval is set to 3 days. Threading was used to achieve function calls at specified intervals. When a plant’s watering time is reached, the soil moisture sensors will be checked first to ensure that the soil actually needs watering. If the moisture level is determined to be already suitable, the plant will not be watered in order to avoid over-watering. Also to avoid over-watering, we made sure to turn on the water pump for only 1 second and allow that water to seep into the soil before continuing to water.

The web-scraped data is also used to set the sunlight and temperature thresholds for each plant. The website defines two sunlight conditions: full sun and partial shade. If the plant requires full sun, the sunlight threshold for that plant is set to 0.85 which (through testing) was determined to be direct sunlight conditions. If the plant requires partial shade, its sunlight threshold is set to 0.5 which was determined to be indirect sunlight conditions through testing. If the plant’s sunlight requirement is not specified, its threshold is set to 0.65 which is partial sunlight and suitable for most indoor plants.

To set the temperature thresholds, the website’s information about plant hardiness and standard indoor plant temperature ranges were used. The standard temperature range for all indoor plants is 65℉-85℉. The hardier a plant is, the better it is able to survive in low temperatures. If a plant is identified as hardy, it can handle temperatures as low as 45℉. If a plant is identified as needing protection, it is unable to tolerate temperatures lower than 60℉. Thus, the minimum temperature threshold is set for each plant using the web scraped data and the above logic. The humidity range of each plant is set as the standard range for all indoor plants (~30% - 65%). If a plant falls below or under their optimal temperature or humidity range, the piTFT display and the web application are updated to notify the user. The system box has holes to allow the DHT22 sensor to get the correct temperature and humidity values of the environment.

The camera is embedded in the system box and points towards the plants. It is set to take a picture of the plants once each day and use the image to measure height and ratio of green to brown. The color ratio is used to determine the health of the plant. If the ratio of green to brown falls under 0.8, the plant is determined to be unhealthy and the user is notified. The height and color information is stored on the web application and made available to the user. The charts on the web application allow the user to track the height and color of the plants over time.

The web application also allows the user to access the live stream of the plants via the camera as well as water the plants remotely. The web application is also where the user can go and enter the type of plants so that the system can configure its care after scraping the web.

Sunlight Sensor

A photoresistor, also known as a light dependent resistor (LDR), was used as a sunlight detection sensor. With a photoresistor, the resistance decreases with increasing light intensity and vice versa, allowing us to measure the amount of incident light in the plant’s environment. The photoresistor was connected according to the schematic below:

Alt

Figure 4. Photoresistor Schematic

To read the value outputted by the photoresistor, we utilized the gpiozero library. The gpiozero library contains a class called LightSensor, which calculates the resistance of a LDR by repeatedly discharging the connected 1µF capacitor and then measures the duration it takes to charge the capacitor back up. To get the actual value, the ldr.value() function is called, which returns a value between 1 (light) and 0 (dark). To test the functionality of the photoresistor, we experimented with values in different sunlight conditions and verified the legitimacy of the readings.

DHT22 Sensor

To measure the temperature and humidity of the plants’ environment, we used a DHT22 sensor. To use the DHT22 sensor, the CircuitPython-DHT library needed to be installed. This library is only compatible with python3, which meant that any scripts that contain readings of the DHT22 sensor needed to be explicitly called using python3. To read the DHT22 sensor, we called the dhtDevice.temperature and dhtDevice.humidity calls which returned the temperature (in Celsius) and humidity (in percentage) values respectively. We converted the temperature to Fahrenheit using the equation: degrees_celsius * (9/5) + 32. The DHT22 sensors often have invalid readings, outputting ‘None’ as the values for the temperature and humidity. This is not a major issue as the consecutive readings will return to being valid. When the readings are invalid, we output the temperature and humidity as currently ‘loading’ to the user. Below is a schematic of how the DHT22 sensor was connected to the RaspberryPi.

Alt

Figure 5. DHT22 Schematic

Soil Moisture Sensors The soil moisture sensor is used to detect the humidity in the soil. The sensor is made up of 2 components: the probe and the electronic board.

Alt

Figure 6. Soil Sensor and Electronic Board

The electronic board is made up of a potentiometer, power LED, digital output LED, and multiple pin connections. The soil moisture sensors were connected as shown in the figure below:

Alt

Figure 7. Soil Sensor Schematic

As the Raspberry Pi is unable to take in analog inputs, we utilized the digital output pin on the soil sensor. This meant that the soil moisture was detected using a set threshold. The water content in the soil was measured via the voltage that the sensor outputs, which is dependent on the amount of water. The more moisture is contained in the soil, the lower the output voltage is and vice versa. The digital signal will either be LOW (not detected) or HIGH (detected), depending on if the soil humidity exceeds the preset threshold value. To read the value, we utilized the gpiozero library’s DigitalInputDevice class and called the function digital_device.value to get the HIGH or LOW value of the sensor.

Water Pumps and Relay

The water pumps were connected to the system using a 5V 4-channel relay which is shown in the figure below:

Alt

Figure 8. 4-Channel Relay

Because the relay is powered by 5V, it needed its own power supply, hence the 5V battery pack. In the figure above, there is a jumper shorting the VCC and JD-VCC pins on the board. However, since the Raspberry Pi GPIO pins are 3.3V and the relay is 5V, we needed to remove this jumper. The VCC was then connected to the RaspberryPi’s 3.3V and the JD-VCC was connected to the 5V battery pack. This was necessary to separate the 5V relay board’s supply from the RaspberryPi’s 3.3V supply. The relay connected to the RaspberryPi in the following manner: GND to GND, VCC to 3V3, and the IN pins to corresponding GPIOs. On the other side of the relay board, the individual relays each have 3 connections: NO (normally open), C (common), and NC (normally closed). Depending on whether the IN pin is HIGH or LOW, the relay switch will flip between the NC and NO connections respectively. The DC water pumps were then connected in the following manner: water pump power to relay NC, water pump GND to RaspberryPi’s GND, relay C to RaspberryPi’s 3.3V (since the DC water pumps can be powered by any voltage between 3.3V and 5V, relay C could have also been connected to the 5V battery pack). The figure below shows how the relay and DC water pump connect to the RaspberryPi:

Alt

Figure 9. Relay and Water Pump Schematic

With this configuration, the water pumps are turned ON when their respective IN pins are set LOW and vice versa.

Camera and OpenCV

Alt

Figure 10. Pi Camera

We use the camera to check plant height and health. To measure the plant height, we use the library OpenCV to do image processing. This process mainly needs four steps. First, take a picture from the PiCamera, and we store the data into a numpy array which is more convenient to operate. Second, we find the edges of the image by changing it to a grayscale image and use the Canny Edge detection. At this point, we use the image process techniques dilation and erosion before the edge detection to make sure the edges are accurate. Also, we use the Gaussian Blur function after the edge detection to make sure that each edge from the same object has no gap. Third, once we have all the contours of the plants, we sort the contours from left to right by their x-coordinate, and since we put a reference object on the left of the plants, we can calculate mm per pixel from it. Finally, there may be not only one contour found by the program due to some noise. We find the one with the largest area as our plant and calculate its height.

The program described above is for only one plant. We also do some modification so that it also works for two plants or even more plants. For example, if we want to measure two plants, once we find the reference object, we can divide the rest of the image into two areas. Each area is for one plant. Then, we can find the contours we want in each area and calculate each plant’s height as mentioned above.

Alt

Figure 11. Greyscale and Edge Detection

Alt

Figure 12. Detecting Plants

Alt

Figure 13. OpenCV Plant Detection Result

To measure the plant height, we calculate the ratio of the number of green pixels and brown pixels. In order to make the value in the range of 0 to 100 percent, the actual formula is green_pixels / (green_pixels + brown_pixels) . To implement this, we create two different masks and each of them specify a HSV color range. Then, we can call the function cv2.inRange() to get the image that only shows some specific color. Finally, we calculate the number of pixels from each of the mask images, and use the formula to get the result.

Alt

Figure 14. Detecting Green Pixels

The last function from the camera is video streaming. The library http provides a very simple way to create a web server. Once anyone opens the URL we create, the function do_GET() will be called automatically. In the body of this function, we pass some information and the image frame from the PiCamera. Thus, a user can access the live video from a browser.

Alt

Figure 15. Video Streaming

piTFT

The piTFT serves as a local interface for the user to interact with the system and get information about the status of their plants and the environment. The main menu is made up of 4 buttons: temperature, humidity, plant 1, and plant 2. Clicking on the temperature and humidity buttons will show information about whether or not the temperature and humidity of the plants’ environment is optimal for each plant respectively. Clicking on the plant buttons will show the plant name, adequacy of soil moisture, and adequacy of sunlight for that plant. Each plant screen also has a ‘water’ button to allow the user to manually water the plants if desired. Below are examples of the piTFT’s display options.

Alt

Alt

Figure 16. piTFT Interface Examples

Web Scraping

We web-scraped the website www.mygarden.org/plants utilizing python’s BeautifulSoup library in order to configure system care to the user’s specific plant types. To do this, we take the user’s inputs in the web app for their plant types and search www.mygarden.org/plants for their inputs. Then we take the first plant result link from the search and scrape that link for information. This method of querying allows the system to handle user misspellings as the top result will still be the best matching plant for the user’s input and still find a plant on the site. We utilize the following pieces of information from the site for each plant: hardiness, sunlight requirement, and soil moisture. These pieces of information are used to configure the sunlight threshold, temperature threshold, and watering interval for each plant.

Database Design

To synchronize the display on the piTFT and Web App, we need to store the history of our sensor readings in a persistent format. To do this, we create a SQL database using SQLite3*.

Since our camera program reads data at separate time intervals than our sensors, we create 2 tables. The rows of the database schema are as follows:

(id NUMERIC, timestamp DATETIME, temp NUMERIC, hum NUMERIC, light NUMERIC)

(id NUMERIC, timestamp DATETIME, height NUMERIC, greenness NUMERIC)

The first table stores sensor data while the second table stores camera data. Then, both the piTFT program and the Web App may query the database using standard SQL to get the latest readings for each plant as well as a history (by sorting by timestamp). Further, we need to store the current system configuration. We do this in a standard JSON file. An example of a possible system state is shown below.

{
  "1": {
    "pin": 5,
    "water": "On",
    "mode": "Automatic",
    "type": "Tomato"
  },
  "2": {
    "pin": 7,
    "water": "On",
    "mode": "Automatic",
    "type": "Mint"
  }
}

The config stores the type of the plant, the mode, for each plant, whether the plant is being watered, and the pin of each plant. When an application seeks to update the config, it must read into its memory the JSON file, update it, then write it back to the file.

Before integrating our database with our sensors, we tested it by putting in dummy values and seeing that the result is what we expect. The test script for that can be seen in the Github repo under database/db_utils.py.

Web App

The Web App mirrors the functionality of the piTFT display, as well as provides other features.

The Web App functionality is developed using the Flask framework. We use Apache to handle automatic launching the application at boot.

Testing of the Web App was done through print statements to ensure our buttons were triggering actions as well as just simple playtesting. Screenshots of the Web App are shown below.

Alt Alt Alt

Figure 17. Web App UI

Results and Conclusion

Our team was able to meet all the goals we outlined in our initial description. The system is able to achieve its overall goal of caring for plants automatically. Some aspects that did not work were that the soil sensors had options for digital and analog readings but due to the nature of the RaspberryPi’s GPIOs and our lack of a DAC, we were only able to read HIGH or LOW values from those sensors. This meant that the thresholds for the soil moisture had to be set manually using the on-board potentiometers. Another aspect that did not work was that for the OpenCV to work on our plants, they had to be in front of a purely white background, which realistically is not likely. This issue may be solved by using background subtraction or detecting the plants by a Machine Learning Model. The user interfaces (piTFT and web app) of the system seem to work very well and the live feed was a nice extra component to help make the system more robust. Overall, we are very pleased with how the system turned out.

Future Work

If we had more time to work on the project, we would want to turn the web app into a mobile application for users with push notifications. We would expand camera/openCV capabilities to function well in less ideal enviromental conditions.

If we had a lot of time, we would expand our project into a large scale automatic carer for gardens and greenhouses. This would involve more complex pumps, water sources, and sensors, as well as many microcontrollers/RPIs. Such a large-scale system would be a very useful system for both professional and home gardeners.

Parts & Budget

Part Quantity Cost
Raspberry Pi 1 No cost
piTFT display 1 No cost
Photoresistor 1 No cost
1µF capacitor 1 No cost
DHT22 sensor 1 No cost
5V battery pack 1 No cost
Water pumps and 1m tubing 2 $13.99
Water container 1 No cost
4 channel relay 1 $7.99
Soil Moisture Sensors 2 No cost
Raspberry Pi camera 1 No cost
Plants 2 No cost

Resources

Scraped Site

Plant Research

DHT22 Sensor

GPIOZERO Library

4 Channel Relay Information

4 Channel Relay How-To

Monitor plant growth with OpenCV

Video Streaming with Raspberry Pi Camera

OpenCV Tutorial

Color Detection

Code Appendix

Selected code snippets are shown below. For brevity, we don't display all of our code, since that would be 2000+ lines of code.

The entire codebase for this project can be found at: Code

app.py (The Web App)

import picamera     # Importing the library for camera module
from picamera.array import PiRGBArray
from time import sleep  # Importing sleep from time library to add delay in program
import RPi.GPIO as GPIO
import sqlite3 as lite
from flask import Flask, render_template, request, Response, redirect, url_for
from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
from matplotlib.figure import Figure
import matplotlib.dates as dates
import base64
from io import BytesIO
import json
import time
import picamera
import cv2
import socket
import io

app = Flask(__name__)
app.config['SEND_FILE_MAX_AGE_DEFAULT'] = 0

# GPIO Setup
GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)

plant1_pin = 5
plant2_pin = 7

GPIO.setup(plant1_pin, GPIO.OUT)
GPIO.output(plant1_pin, GPIO.LOW)
GPIO.setup(plant2_pin, GPIO.OUT)
GPIO.output(plant2_pin, GPIO.LOW)

# Database Constants
DB_NAME = './database/sensorData.db'
TABLE_NAME = 'PlantTable'

# Relay Setup
pump_pin_1 = 21
GPIO.setup(pump_pin_1, GPIO.OUT)
GPIO.output(pump_pin_1, GPIO.HIGH)

pump_pin_2 = 20
GPIO.setup(pump_pin_2, GPIO.OUT)
GPIO.output(pump_pin_2, GPIO.HIGH)


def man_pump_water(id):

    if id == 1:
        GPIO.output(pump_pin_1, GPIO.LOW)
        time.sleep(1)
        GPIO.output(pump_pin_1, GPIO.HIGH)
    elif id == 2:
        GPIO.output(pump_pin_2, GPIO.LOW)
        time.sleep(1)
        GPIO.output(pump_pin_2, GPIO.HIGH)


def read_config():
    with open('config.json') as json_file:
        data = json.load(json_file)
        return data


def write_config(data):
    with open('config.json', 'w') as f:
        json.dump(data, f)


def getLatest(id):
    conn = lite.connect(DB_NAME)
    curs = conn.cursor()
    time, temp, hum, light, camera_time, height, greenness = 0, 0, 0, 0, 0, 0, 0
    for row in curs.execute("SELECT * FROM {} WHERE id={} ORDER BY timestamp DESC LIMIT 1".format(TABLE_NAME, id)):
        time, temp, hum, light = str(row[1]), str(
            row[2]), str(row[3]), str(row[4])
    for row in curs.execute("SELECT * FROM {} WHERE id={} ORDER BY timestamp DESC LIMIT 1".format("CameraTable", id)):
        camera_time, height, greenness = str(row[1]), str(
            row[2]), str(row[3])

    conn.close()
    return time, temp, hum, light, camera_time, height, greenness


def getHist(id):
    conn = lite.connect(DB_NAME)
    curs = conn.cursor()
    time_hist = []
    temp_hist = []
    hum_hist = []
    light_hist = []
    for row in curs.execute("SELECT * FROM {} WHERE id={} ORDER BY timestamp LIMIT 10".format(TABLE_NAME, id)):
        time_hist.append(row[1])
        temp_hist.append(row[2])
        hum_hist.append(row[3])
        light_hist.append(row[4])

    camera_time_hist = []
    height_hist = []
    greenness_hist = []
    for row in curs.execute("SELECT * FROM {} WHERE id={} ORDER BY timestamp LIMIT 10".format("CameraTable", id)):
        camera_time_hist.append(row[1])
        height_hist.append(row[2])
        greenness_hist.append(row[3])

    return time_hist, temp_hist, hum_hist, light_hist, camera_time_hist, height_hist, greenness_hist


def getCameraPlots(id):
    fig = Figure()
    FigureCanvas(fig)

    _, _, _, _, camera_time_hist, height_hist, greenness_hist = getHist(id)

    ax1 = fig.add_subplot(3, 1, 1)
    ax2 = fig.add_subplot(2, 1, 2)

    # ax = fig.subplots()
    ax1.plot(camera_time_hist, height_hist)
    ax1.set_ylabel("Height")
    ax2.plot(camera_time_hist, greenness_hist)
    ax2.set_ylabel("Greenness")

    fig.autofmt_xdate()

    ax1.set_title("Plant {} Camera Data".format(id))

    # Save it to a temporary buffer.
    buf = BytesIO()
    fig.savefig(buf, format="png")
    # Embed the result in the html output.
    data = base64.b64encode(buf.getbuffer()).decode("ascii")
    return data


def getPlots(id):
    fig = Figure()
    FigureCanvas(fig)

    time_hist, temp_hist, hum_hist, light_hist, _, _, _ = getHist(id)

    ax1 = fig.add_subplot(3, 1, 1)
    ax2 = fig.add_subplot(3, 1, 2)
    ax3 = fig.add_subplot(3, 1, 3)

    # ax = fig.subplots()
    ax1.plot(time_hist, temp_hist)
    ax1.set_ylabel("Temperature")
    ax2.plot(time_hist, hum_hist)
    ax2.set_ylabel("Humidity")

    ax3.plot(time_hist, light_hist)
    ax3.set_ylabel("Light")

    fig.autofmt_xdate()

    ax1.set_title("Plant {} Data".format(id))

    # Save it to a temporary buffer.
    buf = BytesIO()
    fig.savefig(buf, format="png")
    # Embed the result in the html output.
    data = base64.b64encode(buf.getbuffer()).decode("ascii")
    return data


@app.route("/", methods=["GET", "POST"])
def index():
    config = read_config()

    # Get form
    if request.method == "POST":
        plant1 = request.form.get("Plant1")
        plant2 = request.form.get("Plant2")
        if plant1 != "":
            config["1"]["type"] = plant1
            write_config(config)
        if plant2 != "":
            config["2"]["type"] = plant2
            write_config(config)

    time1, temp1, hum1, light1, camera_time1, height1, greenness1 = getLatest(
        1)
    time2, temp2, hum2, light2, camera_time2, height2, greenness2 = getLatest(
        2)
    plot1 = getPlots(1)
    plot2 = getPlots(2)
    camera_plot1 = getCameraPlots(1)
    camera_plot2 = getCameraPlots(2)

    # Alerts
    green_alert1 = "Plant 1 is healthy color"
    green_alert2 = "Plant 2 is healthy color"
    if float(greenness1) < 0.8:
        green_alert1 = "Plant 1 is too brown"
    if float(greenness2) < 0.8:
        green_alert2 = "Plant 2 is too brown"

    templateData = {
        'time1': time1,
        'temp1': temp1,
        'hum1': hum1,
        'light1': light1,
        'time2': time2,
        'temp2': temp2,
        'hum2': hum2,
        'light2': light2,
        'camera_time1': camera_time1,
        'height1': height1,
        'greenness1': greenness1,
        'camera_time2': camera_time2,
        'height2': height2,
        'greenness2': greenness2,
        'plot1': plot1,
        'plot2': plot2,
        'camera_plot1': camera_plot1,
        'camera_plot2': camera_plot2,
        'mode1': config["1"]['mode'],
        'mode2': config["2"]['mode'],
        'water1': config["1"]['water'],
        'water2': config["2"]['water'],
        'type1': config["1"]["type"],
        'type2': config["2"]["type"],
        'green_alert1' : green_alert1,
        'green_alert2' : green_alert2
    }
    print(read_config())
    return render_template('index_gpio.html', **templateData)


# Action is water or setting
# Both of which toggle

@app.route("/<plant>/<action>", methods=["GET", "POST"])
def toggle(plant, action):
    config = read_config()

    # Update Config
    # Get form
    if request.method == "POST":
        plant1 = request.form.get("Plant1")
        plant2 = request.form.get("Plant2")
        print(plant1, plant2)
        if plant1 != "":
            config["1"]["type"] = plant1
        if plant2 != "":
            config["2"]["type"] = plant2

    if action == 'mode':
        if config[plant][action] == 'Manual':
            config[plant][action] = 'Automatic'
        else:
            config[plant][action] = 'Manual'
    else:
        print(action)
        print(config[plant][action])
        if config[plant][action] == 'On':
            config[plant][action] = 'Off'
        else:
            config[plant][action] = 'On'

    # Control GPIO
    print("Action, plant: ", action, plant)
    if config[plant][action] == "On":
        man_pump_water(int(plant))
        print("Pumping: ", plant)

    write_config(config)

    return redirect(url_for('index'))

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=70, debug=True)

piTFT_display.py

#Code for plant monitoring display on piTFT

import RPi.GPIO as GPIO
import os
import sys
import time
import pygame
from pygame.locals import *                         #for event mouse variables
from gpiozero import LightSensor, DigitalInputDevice
import board
import json
import adafruit_dht
import threading
from bs4 import BeautifulSoup
import requests
from nltk.tokenize import TreebankWordTokenizer, RegexpTokenizer
try:
    import urllib.request as urllib2
except ImportError:
    import urllib2

import sqlite3 as lite
import sys

#set pin mode to BCM
GPIO.setmode(GPIO.BCM)

#GPIO IN (piTFT buttons)
GPIO.setup(27, GPIO.IN, pull_up_down=GPIO.PUD_UP)
GPIO.setup(17, GPIO.IN, pull_up_down=GPIO.PUD_UP)
GPIO.setup(22, GPIO.IN, pull_up_down=GPIO.PUD_UP)
GPIO.setup(23, GPIO.IN, pull_up_down=GPIO.PUD_UP)

#Tokenizer
tokenizer = RegexpTokenizer(r'\w+')

#Light Sensor Setup
light_pin = 5
ldr = LightSensor(light_pin)

#DHT Sensor Setup
dhtDevice = adafruit_dht.DHT22(board.D4, use_pulseio=False)

#Soil Sensor Setup
soil_pin_1 = 19
soil_sensor_1 = DigitalInputDevice(soil_pin_1)

soil_pin_2 = 16
soil_sensor_2 = DigitalInputDevice(soil_pin_2)

#Relay Setup
pump_pin_1 = 21
GPIO.setup(pump_pin_1, GPIO.OUT)
GPIO.output(pump_pin_1, GPIO.HIGH)

pump_pin_2 = 20
GPIO.setup(pump_pin_2, GPIO.OUT)
GPIO.output(pump_pin_2, GPIO.HIGH)

os.putenv('SDL_VIDEODRIVER', 'fbcon')              #play on piTFT
os.putenv('SDL_FBDEV', '/dev/fb1')
os.putenv('SDL_MOUSEDRV', 'TSLIB')                  #track mouse clicks on piTFT
os.putenv('SDL_MOUSEDEV', '/dev/input/touchscreen')

# Database Constants
DB_NAME = './database/sensorData.db'
TABLE_NAME = 'PlantTable'

# Config Helpers
def read_config():
    with open('config.json') as json_file:
        data = json.load(json_file)
        return data


def write_config(data):
    with open('config.json', 'w') as f:
        json.dump(data, f)


#Set up button 27 (on piTFT) for quitting later
GPIO.setup(27, GPIO.IN, pull_up_down=GPIO.PUD_UP)

#Script quit button function
def gpio27_callback(channel):
    sys.exit()

#make new button
def new_button(text, x_pos, y_pos, width, height, color):
    left = x_pos - width/2
    top = y_pos - height/2
    button_label = font.render(text, True, color)
    button_rect = button_label.get_rect(center = (x_pos, y_pos))
    pygame.draw.rect(screen, color, (left, top, width, height), 5)
    screen.blit(button_label, button_rect)

#new text label on screen
def new_label(text, x_pos, y_pos, fontsize, color):
    font = pygame.font.Font(None, fontsize)
    text_label = font.render(text, True, color)
    text_rect = text_label.get_rect(center = (x_pos, y_pos))
    screen.blit(text_label, text_rect)

#Check for touch location, return which button was touched
def button_touch(pos):
    global level
    x = pos[0]
    y = pos[1]
    if level == 0:
        if 30 <= x <= 150 and 80 <= y <= 140:
            return 1
        elif 30 <= x <= 150 and 160 <= y <= 220:
            return 3
        elif 190 <= x <= 290 and 80 <= y <= 140:
            return 2
        elif 190 <= x <= 290 and 160 <= y <= 220:
            return 4
        else:
            return 0

#back button
def back_button(pos):
    x = pos[0]
    y = pos[1]
    if 40 <= x <= 140 and 190 <= y <= 230:
        return True
    else:
        return False

def water_button(pos):
    x = pos[0]
    y = pos[1]
    if 180 <= x <= 280 and 190 <= y <= 230:
        return True
    else:
        return False

#photoresistor reading
def light_reading():
    count = 0
    GPIO.setup(light_pin, GPIO.OUT)
    GPIO.output(light_pin, GPIO.LOW)

    time.sleep(0.1)

    while(GPIO.input(light_pin) == GPIO.LOW):
        print(count)
        count += 1
    
    return count 

#DHT reading
def DHT_reading():
    try:
        # Print the values to the serial port
        temperature_c = dhtDevice.temperature
        temperature_f = temperature_c * (9 / 5) + 32
        humidity = dhtDevice.humidity
        return (temperature_f, humidity)
    except RuntimeError as error:
        # Errors happen fairly often, DHT's are hard to read, just keep going        
        return "error"
    except Exception as error:
        #dhtDevice.exit()
        return "error"

#detect moisture in soil and turn on water pumps
def p1_auto_pump_water():
    #check soil moisture every [interval] min and water if necessary
    global soil_moisture_1
    global p1_interval

    threading.Timer(p1_interval, p1_auto_pump_water).start()
    
    if soil_sensor_1.value:
        soil_moisture_1 = "Inadequate"
        GPIO.output(pump_pin_1, GPIO.LOW)
        time.sleep(1)
        GPIO.output(pump_pin_1, GPIO.HIGH)
    else:
        soil_moisture_1 = "Adequate"
    
#detect moisture in soil and turn on water pumps
def p2_auto_pump_water():
    #check soil moisture every [interval] min and water if necessary
    global soil_moisture_2
    global p2_interval

    threading.Timer(p2_interval, p2_auto_pump_water).start()
        
    if soil_sensor_2.value:
        soil_moisture_2 = "Inadequate"
        GPIO.output(pump_pin_2, GPIO.LOW)
        time.sleep(1)
        GPIO.output(pump_pin_2, GPIO.HIGH)
    else:
        soil_moisture_2 = "Adequate"

#if user presses water button
def man_pump_water():
    global level

    if level == 3:
        GPIO.output(pump_pin_1, GPIO.LOW)
        time.sleep(1)
        GPIO.output(pump_pin_1, GPIO.HIGH)
    elif level == 4:
        GPIO.output(pump_pin_2, GPIO.LOW)
        time.sleep(1)
        GPIO.output(pump_pin_2, GPIO.HIGH)

#display for each level
def display_level():
    global level
    global temp
    global humid
    global p1_sun_threshold 
    global p2_sun_threshold
    global sunlight

    # Get Status
    time1, temp1, hum1, light1, camera_time1, height1, greenness1 = getLatest(
        1)
    time2, temp2, hum2, light2, camera_time2, height2, greenness2 = getLatest(
        2)
    
    green_alert1 = "Healthy color!"
    green_alert2 = "Healthy color!"
    # Alerts
    if float(greenness1) < 0.8:
        green_alert1 = "Too brown!"
    if float(greenness2) < 0.8:
        green_alert2 = "Too brown!"

    screen.fill(BLACK)
    background_rect = background.get_rect()
    screen.blit(background, background_rect)

    if level == 0:
        new_button("Plant Care", 160, 40, 260, 40, WHITE)
        new_button("Temp", 80, 110, 100, 60, WHITE)
        new_button("Humidity", 240, 110, 100, 60, WHITE)
        new_button("Plant 1", 80, 190, 100, 60, WHITE)
        new_button("Plant 2", 240, 190, 100, 60, WHITE)
    elif level == 1:
        if temp == 0 or isinstance(temp, str):
            temp = "loading"
            new_label("Temperature: " + temp, 160, 110, 30, WHITE)
        else:
            if temp > max_temp:
                new_label(str(temp) + "F: Too hot for plants", 160, 110, 30, RED)
            elif temp < min_temp:
                new_label(str(temp) + "F: Too cold for plants", 160, 110, 30, BLUE)
            else:
                new_label("Temperature: "+ str(temp) + "F", 160, 110, 30, WHITE)
        new_button("Back", 90, 210, 100, 40, WHITE)
    elif level == 2:
        if humid == 0 or isinstance(humid, str):
            humid = "loading"
            new_label("Humidity: " + humid, 160, 130, 30, WHITE)
        else:
            if humid > max_humid:
                new_label(str(humid) + "%: Too high", 160, 110, 30, RED)
            elif humid < min_humid:
                new_label(str(humid) + "%: Too low", 160, 110, 30, BLUE)
            else:
                new_label("Humidity: " + str(humid) + "%", 160, 130, 30, WHITE)
        new_button("Back", 90, 210, 100, 40, WHITE)
    elif level == 3:
        if sunlight < p1_sun_threshold:
            text = "Not Enough Sunlight"
        else:
            text = "Adequate Sunlight"
        new_label(text, 160, 80, 30, WHITE)
        new_label(green_alert1, 160, 100, 30, WHITE)
        new_label("Soil Moisture: " + soil_moisture_1, 160, 120, 30, WHITE)
        new_button("Back", 90, 210, 100, 40, WHITE)
        new_button("Water", 230, 210, 100, 40, WHITE)
    elif level == 4:
        if sunlight < p2_sun_threshold:
            text = "Not Enough Sunlight"
        else:
            text = "Adequate Sunlight"
        new_label(text, 160, 80, 30, WHITE)
        new_label(green_alert2, 160, 100, 30, WHITE)
        new_label("Soil Moisture: " + soil_moisture_2, 160, 120, 30, WHITE)
        new_button("Back", 90, 210, 100, 40, WHITE)
        new_button("Water", 230, 210, 100, 40, WHITE)

#web scrape
def web_scrape(plant_type):
    #print("Type in the type of plant: ")
    plant_name = (plant_type).split(" ")
    url = 'https://www.mygarden.org/search?q='
    for i, tok in enumerate(plant_name):
        if (i == len(plant_name)-1):
            url += tok + '&w=plants'
        else:
            url += tok + '+'

    link_req = requests.get(url)
    data = link_req.content
    soup_1 = BeautifulSoup(data, 'html.parser')
    links = [a.get('href') for a in soup_1.find_all(href=True)]
    filt_links = [x for x in links if 'https://www.mygarden.org/plants' in x]

    first_link = filt_links[0]
    req = requests.get(first_link)
    html_page = req.content
    soup_2 = BeautifulSoup(html_page, 'html.parser')
    text = soup_2.find_all(text = True)

    exclude = ['[document]', 'noscript', 'header', 'html', 'meta', 'head', 'input', 'script']

    output = ''

    for x in text: 
        if x.parent.name not in exclude:
            output += '{} '.format(x)
    output = output.lower()

    start_ind = output.index('height')

    try:
        end_ind = output.index('see all varieties')
    except:
        end_ind = output.index('add to my exchange list')

    properties = ['height', 'color', 'soil', 'sunlight', 'ph', 'moisture', 'hardiness']
    prop_dict = dict.fromkeys(properties)

    output = output[start_ind:end_ind]
    output = tokenizer.tokenize(output.lower())

    for i,x in enumerate(output):
        if x in properties:
            min_ind = len(output)
            p_ind = properties.index(x) + 1
            for p in properties[p_ind:len(properties)]:
                try:
                    prop_ind = output.index(p)
                    if prop_ind < min_ind:
                        min_ind = prop_ind
                except:
                    pass
            prop_dict[x] = output[i + 1: min_ind]

    #print(prop_dict)
        
    try:
        ret_sun = " ".join(prop_dict['sunlight'])
    except:
        ret_sun = ""

    try:
        ret_moisture = " ".join(prop_dict['moisture'])
    except:
        ret_moisture = ""

    return ret_sun,ret_moisture

#set up parameters for specific plant types
def plant_param(p1_sun, p1_moisture, p2_sun, p2_moisture):
    global p1_interval
    global p2_interval
    global p1_sun_threshold
    global p2_sun_threshold

    #plant 1 moisture
    if "well drained" in plant1_moisture:
        p1_interval = 60
    elif "moist" in plant1_moisture:
        p1_interval = 45
    else:
        p1_interval = 30
    #plant 1 sun
    if "full" in plant1_sunlight and "partial" in plant1_sunlight:
        p1_sun_threshold = 0.70
    elif "full" in plant1_sunlight:
        p1_sun_threshold = 0.85
    elif "partial" in plant1_sunlight and "shade" in plant1_sunlight:
        p1_sun_threshold = 0.50

    #plant 2 moisture
    if "well drained" in plant2_moisture:
        p2_interval = 60
    elif "moist" in plant2_moisture:
        p2_interval = 45
    else:
        p2_interval = 30
    #plant 2 sun
    if "full" in plant2_sunlight and "partial" in plant2_sunlight:
        p2_sun_threshold = 0.70
    elif "full" in plant2_sunlight:
        p2_sun_threshold = 0.85
    elif "partial" in plant2_sunlight and "shade" in plant2_sunlight:
        p2_sun_threshold = 0.50

def getLatest(id):
    conn = lite.connect(DB_NAME)
    curs = conn.cursor()
    time, temp, hum, light, camera_time, height, greenness = 0, 0, 0, 0, 0, 0, 0
    for row in curs.execute("SELECT * FROM {} WHERE id={} ORDER BY timestamp DESC LIMIT 1".format(TABLE_NAME, id)):
        time, temp, hum, light = str(row[1]), str(
            row[2]), str(row[3]), str(row[4])
    for row in curs.execute("SELECT * FROM {} WHERE id={} ORDER BY timestamp DESC LIMIT 1".format("CameraTable", id)):
        camera_time, height, greenness = str(row[1]), str(
            row[2]), str(row[3])

    conn.close()
    return time, temp, hum, light, camera_time, height, greenness

#pygame initializations
pygame.init()
screen = pygame.display.set_mode((320, 240))
font = pygame.font.Font(None, 25)
#pygame.mouse.set_visible(False)                     #hide mouse cursor

#Colors
WHITE = (255, 255, 255)
BLACK = (0,0,0)
RED = (255,0,0)
GREEN = (65,169,76)
BLUE = (0,0,255)
YELLOW = (255,255,0)
ORANGE = (255, 165, 0)

#fill screen with BLACK
screen.fill(BLACK)
#background
background = pygame.image.load('plant_background.jpg')
background_rect = background.get_rect()
screen.blit(background, background_rect)

#menu border
pygame.draw.rect(screen, GREEN, (0,0,320,240), 10)

#display level
level = 0

#init buttons
display_level()

#init sunlight
sunlight = 0.0
p1_sun_threshold = 0.1
p2_sun_threshold = 0.9

#init soil
soil_moisture_1 = "Adequate"
soil_moisture_2 = "Adequate"

#init temp/humid
temp = 0.0
humid = 0.0
#min/max for general indoor plants
min_temp = 60
max_temp = 85
min_humid = 30
max_humid = 65

#get initial reading
DHT_reading()
DHT_reading()
DHT_reading()

#set call back for quit button, bouncetime to avoid multiple hits
GPIO.add_event_detect(27, GPIO.FALLING, callback=gpio27_callback, bouncetime=300)
GPIO.add_event_detect(23, GPIO.FALLING, callback=gpio27_callback, bouncetime=300)
GPIO.add_event_detect(22, GPIO.FALLING, callback=gpio27_callback, bouncetime=300)
GPIO.add_event_detect(17, GPIO.FALLING, callback=gpio27_callback, bouncetime=300)

#set up for specific plant1 type
#plant 1
config = read_config()
plant1 = config["1"]["type"]
print(plant1)
if plant1 != "":
    try:
        plant1_sunlight, plant1_moisture = web_scrape(plant1)
    except:
        plant1_sunlight = 0.6
        plant1_moisture = "well drained"

#plant 2
plant2 = config["2"]["type"]
print(plant2)
if plant2 != "":
    try:
        plant2_sunlight, plant2_moisture = web_scrape(plant2)
    except:
        plant2_sunlight = 0.6
        plant2_moisture = "well drained"

plant_param(plant1_sunlight, plant1_moisture, plant2_sunlight, plant2_moisture)

#thread for water pumps
p1_auto_pump_water()
p2_auto_pump_water()

timer = 0


#main loop
while True:

    t_h = DHT_reading()
    if t_h != "error":                                          #sometimes DHT will throw error
        temp = round(t_h[0], 2)
        humid = round(t_h[1], 2)
        sunlight = ldr.value
    if level == 0:
        for event in pygame.event.get():
            if event.type is MOUSEBUTTONDOWN:
                pos = (pygame.mouse.get_pos()[0], pygame.mouse.get_pos()[1])
                button_num = button_touch(pos) 
                level = button_touch(pos)
                display_level()
            if event.type == KEYDOWN:
                if event.key == pygame.K_ESCAPE:
                    sys.exit()     																		
    elif level == 1:

        for event in pygame.event.get():
            if event.type is MOUSEBUTTONDOWN:
                pos = (pygame.mouse.get_pos()[0], pygame.mouse.get_pos()[1])
                if back_button(pos):
                    level = 0
                display_level()
            if event.type == KEYDOWN:
                if event.key == pygame.K_ESCAPE:
                    sys.exit()  
    
    elif level == 2:
        for event in pygame.event.get():
            if event.type is MOUSEBUTTONDOWN:
                pos = (pygame.mouse.get_pos()[0], pygame.mouse.get_pos()[1])
                if back_button(pos):
                    level = 0
                display_level()
            if event.type == KEYDOWN:
                if event.key == pygame.K_ESCAPE:
                    sys.exit()  

    elif level == 3 or level == 4:
        for event in pygame.event.get():
            if event.type is MOUSEBUTTONDOWN:
                pos = (pygame.mouse.get_pos()[0], pygame.mouse.get_pos()[1])
                if back_button(pos):
                    level = 0
                display_level()
                if water_button(pos):
                    man_pump_water()
            if event.type == KEYDOWN:
                if event.key == pygame.K_ESCAPE:
                    sys.exit() 
    
    # Update Database
    if timer > 1000:
        DB_NAME = './database/sensorData.db'
        TABLE_NAME = 'PlantTable'
        conn = lite.connect(DB_NAME)
        curs = conn.cursor()
        query = "INSERT INTO {} values({}, datetime('now'), {}, {}, {})".format(TABLE_NAME, 1, temp, humid, sunlight)
        query2 = "INSERT INTO {} values({}, datetime('now'), {}, {}, {})".format(TABLE_NAME, 2, temp, humid, sunlight)

        if temp != None and humid != None:
            curs.execute(query)
            curs.execute(query2)
        conn.commit()
        conn.close()

        print("Reading: ", temp, humid, sunlight)
        timer = 0


    pygame.display.flip()
    timer += 1

GPIO.cleanup()

measure_two_plants.py (OpenCV)

import argparse
import imutils.contours
import cv2
import numpy as np
from picamera import PiCamera
from picamera.array import PiRGBArray
from time import sleep
import sqlite3 as lite
import sys

# command: python3 measure.py -w 16
# When executing this program, it will stop and show you the results.
# Press any key to contrinue.
# -w: the height of the reference object
parser = argparse.ArgumentParser(description='Object height measurement')

parser.add_argument("-w", "--width", type=float, required=True,
                    help="width of the left-most object in the image")
args = vars(parser.parse_args())

#Use the a photo from camera or an exist image
#True: Image / False: Camera
image_or_camera = True

if image_or_camera==False:
    #Take a photo
    camera = PiCamera()
    rawCapture = PiRGBArray(camera)
    sleep(0.1)
    camera.capture(rawCapture, format="bgr")
    image_1 = rawCapture.array
    cv2.imshow("Image", image_1)
    cv2.waitKey(0)
else:
    image = cv2.imread("resize_plants.jpg")

if image is None:
    print('Could not open or find the image:')
    exit(0)

#####   Color Detection    #####

# HSV color range
brown = [(10, 100, 20), (20, 255, 200)]
green = [(36, 25, 25), (86, 255, 255)]

# Detect green and brown color
hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
mask = cv2.inRange(hsv, green[0], green[1])
mask2 = cv2.inRange(hsv, brown[0], brown[1])

cv2.imshow("img", image)
cv2.waitKey(0)

print("Green and Brown pixels:",cv2.countNonZero(mask), cv2.countNonZero(mask2))

# count the ratio green_pixels / (green_pixels + brown_pixels)
green_color = cv2.countNonZero(mask)
brown_color = cv2.countNonZero(mask2)
ratio = green_color / (green_color + brown_color)
print('ratio: ', ratio)

#show thr result images
imask = mask>0
green_img = np.zeros_like(image, np.uint8)
green_img[imask] = image[imask]

imask = mask2>0
brown_img = np.zeros_like(image, np.uint8)
brown_img[imask] = image[imask]

res = np.hstack((image, green_img))
cv2.imshow("img", res)
cv2.waitKey(0)
res = np.hstack((image, brown_img))
cv2.imshow("img", res)
cv2.waitKey(0)

#####   Measure plant height    #####

# Cover to grayscale and blur
greyscale = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
greyscale = cv2.GaussianBlur(greyscale, (7, 7), 0)

# Detect edges and close gaps
canny_output = cv2.Canny(greyscale, 25, 30)
canny_output = cv2.dilate(canny_output, None, iterations=1)
canny_output = cv2.erode(canny_output, None, iterations=1)
canny_output = cv2.GaussianBlur(canny_output, (9, 9), 0)

res = np.hstack((greyscale, canny_output))
cv2.imshow("Image", res)
cv2.waitKey(0)

# Get the contours of the shapes, sort l-to-r and create boxes
_, contours, _ = cv2.findContours(canny_output, cv2.RETR_EXTERNAL,
                                  cv2.CHAIN_APPROX_SIMPLE)
if len(contours) < 3:
    print("Couldn't detect three or more objects")
    exit(0)

(contours, _) = imutils.contours.sort_contours(contours)
contours_poly = [None]*len(contours)  
boundRect = [None]*len(contours)
for i, c in enumerate(contours):
    contours_poly[i] = cv2.approxPolyDP(c, 3, True)
    boundRect[i] = cv2.boundingRect(contours_poly[i])

output_image = image.copy()
mmPerPixel = args["width"] / boundRect[0][3]
highestRect = 1000
lowestRect = 0

#initial plant1,2 measure values
plant1_highrect = 1000
plan1_lowrect = 0
plant2_highrect = 1000
plan2_lowrect = 0

# the x coordinate seperate two plants
image_divide_point = 300

print("number of contours", len(contours))
valid_contour = 0
for i in range(1, len(contours)):
    # print("bound coordinate: ",int(boundRect[i][0]), int(boundRect[i][1]))
    # ignore too small objects
    if boundRect[i][2] < 50 or boundRect[i][3] < 50:
        continue
    # print("The Object is at",boundRect[i][2],boundRect[i][3])
    valid_contour+=1
    # The first rectangle is our control, so set the ratio
    if highestRect > boundRect[i][1]:
        highestRect = boundRect[i][1]
    if lowestRect < (boundRect[i][1] + boundRect[i][3]):
        lowestRect = (boundRect[i][1] + boundRect[i][3])

    if int(boundRect[i][0])<image_divide_point:
        print("left contours: ",int(boundRect[i][0]), int(boundRect[i][1]))
        if plant1_highrect>boundRect[i][1]:
            plant1_highrect = boundRect[i][1]
        if plan1_lowrect < (boundRect[i][1] + boundRect[i][3]):
            plan1_lowrect = (boundRect[i][1] + boundRect[i][3])
    else:
        print("right contours: ",int(boundRect[i][0]), int(boundRect[i][1]))
        if plant2_highrect>boundRect[i][1]:
            plant2_highrect = boundRect[i][1]
        if plan2_lowrect < (boundRect[i][1] + boundRect[i][3]):
            plan2_lowrect = (boundRect[i][1] + boundRect[i][3])

    # Create a boundary box
    cv2.rectangle(output_image, (int(boundRect[i][0]), int(boundRect[i][1])),
                  (int(boundRect[i][0] + boundRect[i][2]),
                  int(boundRect[i][1] + boundRect[i][3])), (255, 0, 0), 2)
print("valid_contours:", valid_contour)
# Calculate the size of our plant
plantHeight = (lowestRect - highestRect) * mmPerPixel
#print("Plant height is {0:.0f}mm".format(plantHeight))

#First Plant
plantHeight = (plan1_lowrect - plant1_highrect) * mmPerPixel
print("First plant height is {0:.0f}mm".format(plantHeight))

#Second Plant
plantHeight2 = (plan2_lowrect - plant2_highrect) * mmPerPixel
print("Second plant height is {0:.0f}mm".format(plantHeight2))

######### Write height and greenness to database  #########
 DB_NAME = './database/sensorData.db'
 TABLE_NAME = 'CameraTable'
 conn = lite.connect(DB_NAME)
 curs = conn.cursor()
 query = "INSERT INTO {} values({}, datetime('now'), {}, {})".format(TABLE_NAME, 1, plantHeight, ratio)
 query2 = "INSERT INTO {} values({}, datetime('now'), {}, {})".format(TABLE_NAME, 2, plantHeight2, ratio)
 curs.execute(query)
 curs.execute(query2)
 conn.commit()
 conn.close()


# Resize and display the image and show the result
resized_image = cv2.resize(output_image, (360, 640))
cv2.imshow("Image", resized_image)
cv2.waitKey(0)

Work Distribution

Kevin was reponsible for developing the Web App, setting up the database/configuration files, and integrating the live feed on the Web App.

Ying-Hao was responsible for itnegrating the Camera and the OpenCV functions (measuring greenness, height, etc)

Yoon Jae was responsible for setting up all sensors and hardware, the web scraping functionality, and piTFT interface.

Team Photo

Alt