Design

Android Application

Video and Audio Stream on APP

After setting up the live stream video and audio station on RPi, we need to display the video on the Android phone by using WebView in App's MainActivity.

public class MainActivity extends AppCompatActivity implements SensorEventListener {
        private WebView webview ;
        ...
        @Override
        protected void onCreate (Bundle savedInstanceState) {
        ...
        webview =(WebView)findViewById(R.id.webView);
	// make the webview adapt to the screen
        WebSettings settings = webView.getSettings();
        settings.setUseWideViewPort(true);
        settings.setLoadWithOverviewMode(true);
	// display the video
        webview.setWebViewClient(new WebViewClient());
        webview.getSettings().setJavaScriptEnabled(true);
        webview.getSettings().setDomStorageEnabled(true);
        webview.setOverScrollMode(WebView.OVER_SCROLL_NEVER);
        webview.loadUrl("http://ip_address_of_Rpi:9000");

       }
 }

But WebView only could not play the audio, to get the real-time audio streaming, we still need to include a MediaPlayer in our Android to play the sound from the audio server.

	MediaPlayer mediaPlayer = new MediaPlayer();
        mediaPlayer.setAudioAttributes(
                new AudioAttributes.Builder()
                        .setContentType(AudioAttributes.CONTENT_TYPE_MUSIC)
                        .setUsage(AudioAttributes.USAGE_MEDIA)
                        .build()
        );
        try {
            mediaPlayer.setDataSource("http://ip_address_of_Rpi:8000/rapi.mp3");// the url for the sound data source
            mediaPlayer.prepare();
        } catch (IOException e) {
            e.printStackTrace();
        }
        //  might take long! (for buffering, etc)
        mediaPlayer.start();
 

TCP SOCKET COMMUNICATION (Application End)

In our project, we have "Admin Login" section in our interface. When entering this module, the UI would ask the user (the admins or the app holders) to login by entering password and username on the APP. After a successgul login, the UI (PiTFT) would display the username of the user who have logged in the system. And the user who has successfully logged in should have the right to open the door and watch the CCTV live. Thus, a TCP socket communication module should also be deployed on both the Android APP and RPi server end. The following code is

class Sender extends AsyncTask {
        Socket s;// socket connected to RPi
        PrintWriter pw;
        String msg;
        String type;
        BufferedReader bufferedReader;
        @SuppressLint("SetTextI18n")
        @Override
        protected Void doInBackground(Void...params){
            try {
                s = new Socket("ip address of RPi", 7000);
			// send message to RPi Server
                pw = new PrintWriter(s.getOutputStream());
                pw.write(msg);
                pw.flush();
                bufferedReader = new BufferedReader(new InputStreamReader(s.getInputStream()));
			// receive message from RPi Server
                String msg2 = bufferedReader.readLine();

                switch (type) {
                    case "login":
                        if (msg2!=null && msg2.equals("Success")) {
                            success = true;
						// login successfully
                            statusTextView.setText("Login Success");
                           showToast("Login Success");
                        }else{
						// login fail
                            statusTextView.setText("Not Login In");
                            showToast("Login Fail");
                        }
                        break;
                    case "doorOpen":
                        if (msg2!=null && msg2.equals("Success")) {
                            doorOpen = true;
                            doorStatusTextView.setText("Door Open");
                        }
                }

             ....
        }
    }
 

User Login Activity

Before allowing the user to see the CCTV live stream and control our system, a login activity must be passed to verify the user identity. Thus, a user login activity was first launched on our App. The login activity would allow users to fill in the username, password, IP address of RPi, and the designated port number.

public class StartActivity extends AppCompatActivity {

	private EditText textViewUserName;
    private EditText textViewPassword;
	...

    @Override
    protected void onCreate(Bundle savedInstanceState) {
	....
	// login button
        Button buttonEnter = (Button) findViewById(R.id.enter);
        buttonEnter.setOnClickListener(

                (x) -> {
                    uName = textViewUserName.getText().toString();
                    pwd = textViewPassword.getText().toString();
                    host = textViewIP.getText().toString();
                    TCPport = Integer.valueOf(textViewPort.getText().toString());

                    if (uName == null || uName.length() == 0 || pwd == null || pwd.length() == 0) {
                        showToast("please fill all blanks");
                        return;
                    }else{
                        StartActivity.Sender sender = new  StartActivity.Sender();
                        sender.host = host;
                        sender.port = TCPport;
                        sender.msg = "Check:"+uName+":"+pwd;
                        sender.type = "login";
                        sender.execute();
                        long timeInit = System.currentTimeMillis();

			// wait for the login result from server
                        while(System.currentTimeMillis() - timeInit<3000){
                            if (success){
                                Intent startMainActivityIntent = new Intent(StartActivity.this, MainActivity.class);
                                startActivity(startMainActivityIntent);
                                finish();
                            }
                        }
                    }
                });

 

FSM Control User Interface

In our project, apart from multiprocessing, the user interface and the system need to process several complex tasks at the same time. A finite state machine is designed to control the logic of multi thread activities and all functional modules. The user interface would behave accordingly given user inputs and performs state transitions.

1. When system starts, the system would first assgin cores to each modules, and initialize the message queue for process communication. Then the state machine would enter the MAIN state where the main interface is displayed. When user input is given (For example, the user chooses "Admin Login", "Face Login" or "Face Registration"), the state machine would transit to the corresponding state.
2. When state machine in "Admin Login" function, the state machine would start a new thread trying to get user login results from TCP module modules via message queue. When the state machine receives positive results, the state machine would transit to "User Welcome" and will then return to "MAIN" state.
3. When state machine in "Face Login" function, the state machine would start a new thread trying to get results from Face Recognition module modules via message queue. When user press "Confirm", the state machine would shift to "Check" and will decide if open the door based on recognition results. If it is the intended users, a new thread would be started to open the door asynchronously.
4. When state machine in "Face Registration" function, the state machine would start a new thread to call a bash shell script taking photos of the faces of users and save them to a designated location. When this process finish, the state machine would return to "FINISH" and then "MAIN" automatically.