Understanding Arduino software

Aims

By the end of this session we will dynamically control elements on a locally hosted web page driven by data from an API and from inputs attached to an Arduino.

• Understand what an API is and how to use one
• Retrieve and process API data
• Display API data in a web interface
• Access Arduino sensor data and display in a web interface
• Access Arduino sensor data and publish/send to an API

Why use a web interface?

For creating screen-based interfaces web technologies (HTML, CSS, JavaScript) have a lot of advantages. Compared to other programming languages learning HTML, CSS & JavaScript is relatively quick and easy. HTML provides a wealth of interface elements for content and interaction, CSS is a simple and powerful rule-based system for visual presentation and JavaScript is a widely used and forgiving programming language to enhance behaviour and make HTML dynamic.

Once the browser has received the response the communication with the server is ended and no more data can be transmitted without another HTTP request. This is a limitation of HTTP that we will overcome in later workshops.

The problem of static web pages

The problem with using web pages is that we often want to update our our interfaces with constantly changing (dynamic) data, e.g. from an API or an Arduino. As mentioned in a previous workshops web pages use HTTP, which works on the principle of making a request (to a server), receiving a response and then ending communication. To update that page would require a new HTTP request, i.e. a page refresh.

Read more about request/response model and HTTP in the Web Architecture section of our previous workshop on Web Technologies.

For example if we want to change text using data from an API or change a colour using a photoresistor light sensor sensor on an Arduino we don’t want to keep refreshing the web page. There are ways of overcoming this using JavaScript/AJAX to make new HTTP requests after the page has loaded. However we can also solve this problem using NodeJS and Websockets, which offers other advantages.

Node.js

Node.js is a server-side environment that executes JavaScript code. It is commonly used as a web server but also to make command line tools which interact with our operating systems and the hardware attached to it.

Node.js has the following benefits:

1. Runs on macOS, Windows & Linux
2. Can be used as an HTTP web server
3. Access the operating system at a low-level
4. Including access to serial ports and therefore a connected Arduino
5. Making HTTP requests to access data from API
6. Can communicate dynamic data to a web page using Websockets (see below)

WebSockets

Similar to HTTP, WebSocket is a communication protocol; an agreement between a server and client on how to transfer data. WebSockets however do not use the request/response model like HTTP. Instead it opens a consistent channel for communicating back and forth between server and client:

This is how we overcome the restrictions of HTTP allowing for a web page to change based on data from a server without making a new HTTP request for every change. Imagine the comparison between leaving someone a voice message on the phone and having a continuous conversation. This is a similar difference between making an HTTP request and opening a WebSocket.

Socket.io - TODO

Download code examples

Download the latest code and unzip it on your desktop or somewhere convenient to access.

You will be using the command line during the following exercises so it is useful to know the full file path to your code.

Exercise 1: From server (Node.js) to client (browser) using a WebSocket

Conventionally a server and client do not exist on the same computer however there is nothing to stop us from doing this and it is in fact a useful arrangement for working with APIs, Arduino and web interfaces.

The goal here is to take data that has been generated using our server-side code and pass that to our client code running in the browser. To illustrate this we will generate a regularly changing random number on the server and display it in the browser. To achieve this we will use a WebSocket that creates a consistent communication channel between server and client.

Files

You should have at least these files in your download.

├── ex1-socketio-rand
     ├── client
     │   ├── random-data.html
     │   └── random-ball.html
     └── server
         ├── package.json
         └── app.js

Open this folder in a Text Editor like Atom or your text editor of choice so you can edit the files within.

Client code

Running the code

Client-side code is code running in the browser. In our case we are interested in the JavaScript that is written in the HTML files inside the client folder. To run the code we need to open our HTML files in a browser. This is achieved by dragging the HTML from your file explorer to the browser window. It will open using the file:// protocol.

Go ahead and do this with random-data.html.

When you open the HTML file in the browser you will not see any results *yet*. The JavaScript in the HTML file is waiting for an event from your WebSocket server. We will start the WebSocket server in the next section titled **Server code**.

Looking at the code

This HTML file contains JavaScript that connects to our WebSocket server and updates an HTML element with the latest event/data. Things to note about the client-side JavaScript:

1. We are including an externally hosted JavaScript file for the Socket.io client library. <script src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/1.7.3/socket.io.js"></script>

2. We listen using the on() function for an event called random. The second parameter of the on() function accepts a function that is executed when the event is received:

// Listen for an event called 'random' from the WebSocket
socket.on('random', function (data) {
  // The code in this function is called
  // when the 'random' event has been received
});

3. Within the function that is triggered when the event is received the code selects an HTML element by it's class attribute (indicated by the dot at the beginning of .random) and updates it with the incoming data:

socket.on('random', function (data) {
  // Get the HTML element that we want to update.
  var randElement = document.querySelector(".random");
  // Change it to the random number.
  randElement.textContent = data.random;
});

Server code

The server-side code is written in JavaScript and the code will be executed in the Node.js runtime by typing node script.js on the command line.

Running the server

This script contains JavaScript that will create a WebSocket server, which emits (i.e. sends) an event to any clients that are listening. Here are the steps you need to take to run the server:

1. Change into server directory:
   The server will need to be started using the command line interface so firstly open your CLI and navigate to the appropriate directory replacing ~/Desktop/ with the location you downloaded and unzipped your source code to: cd ~/Desktop/workshop-arduino-amplified-1617/ex1-socketio-rand/server/

Be sure to replace `~/Desktop/workshop-arduino-amplified-1617/` with the location in your filesystem that you downloaded your code to.

2. Install libraries
   To run a WebSocket server our code uses a package/library called Socket.io. So first we install this using Node Package Manager (npm).

npm install socket.io --save

Using `--save` will add this library to `packages.json` so we can distribute and/or reinstall all the libraries at once using `npm install`. (TODO/WIP)

3. Run script in node.js
   Running the JavaScript code using the node [filename] command will now start the WebSocket server and start emitting/sending events. Ensuring you are in the server directory, type the following into the command line:

node app.js

Looking at the code

The server code we are running is in app.js. The purpose of this JavaScript is to setup a WebSocket using the socket.io library, then wait for connections on a specific port.

The server is created and we tell it to listen on a specific port in the first four lines of code:

var app = require('http').createServer()
var io = require('socket.io')(app);

// Listen to connections on port
app.listen(4503);

We then tell our WebSocket to listen for an event using the following line. When some client code connects to the WebSocket the function emitRandomData will called:

// Add 'connection' event listener to WebSocket, which is triggered
// when a client/browser connects.
io.on('connection', emitRandomData);

When the client connects the code inside emitRandomData function is executed. This sets an interval timer to emit (i.e. send) an event called random every 1000 milliseconds:

function emitRandomData( socket ){
  // WebSocket has connected to the browser.
  setInterval(function(){
    socket.emit('random', Math.random());
  }, 1000);
}

**Callbacks** The `emitRandomData` function above is a **callback** function. What this means is that we create a function, give it a name and then pass this name as an argument to *another* function (in this case `io.connect()`) so that it can be *called* (i.e. the code within it is executed) as the result of another event or condition occurring in the code. In our case we give `emitRandomData` to another function so that it can be called when a WebSocket connects.

Exercise 2: View API data in command line interface (CLI)

Within this exercise we intend to use Node.js to retrieve data from an API using a HTTP Request. Firstly we only want to access the data and look at the structure. Since we are running Node.js scripts on the command line we will first of all just output the results of the API call to the command line.

Firstly we need to understand what an API is.

What is an API?

Software API

An application program interface (API) is a set of clearly defined methods for interacting with a software system. An API describes the methods by which one piece of software can interact and use the data or methods from another piece of software.

As an example of this the p5.js JavaScript library uses the HTML Canvas API to draw bitmap graphics. In order to draw a rectangle using the Canvas there is a lot of code executed to draw a 4 sided shape, however all we really need to know is how to use this public function:

fillRect(x, y, width, height);

fillRect is part of the Canvas API. Everything else related to drawing a rectangle using the Canvas can be considered within a metaphorical black box as far as we're concerned.

To understand what the public functions are you would need to consult the API documentation for the canvas:

Web API

The API you are more likely to be familiar with is a web API, which operates on the same principle as the above. The only difference is that the functions of a web API are accessed over the Internet (or other network) usually using HTTP requests.

By using HTTP, web APIs are platform agnostic. Since all browsers and web servers, and most platforms and SDKs support HTTP, the developer of a web service does not need to worry about the specifics of how various 3rd party developers will access their service. The HTTP protocol encourages robust, asynchronous development to ensure that client software remains responsive whilst awaiting a response from the web service.

Servers make resources and services available at particular URL endpoints.

https://api.twitter.com/1.1/search/tweets.json (GET)  
https://api.twitter.com/1.1/friendships/destroy.json (POST)

The responses are typically formatted in JSON or XML for parsing and consumption by the client developer's software. Often, web services will permit the client developer to choose between JSON and XML, though JSON is emerging as the preferred standard. The incoming serialised data is then parsed into a usable objects or structures, with any relevant data injected into the DOM as HTML for presentation to the user.

Access to the web API is often controlled using authentication or keys, in order to limit or monetise resources, or prevent misuse by accessing, modifying or deleting user data.

There are several standards for web services and APIs, with the once-dominant SOAP (simple object access protocol) now being replaced by REST (representational state transfer). RESTful APIs are based around loose principles and rely on the underlying technologies (HTTP, URI, JSON and XML) to enforce standards.

Look at API in the browser

Let's first access an API in the most simple way we can. Many APIs are open (no authentication) and simply provide all their available data with a simple HTTP GET request.

Enter the following URL into your browser URL bar replacing the LCC postcode with your own:

http://api.postcodes.io/postcodes/se16sb

You will see in the browser information about the longitude, latitude, parliamentary constituency, etc. Viewing this in the browser isn't much use to us. The data is in a format (JSON) not designed for human eyes. It is clearly meant to be consumed by other software or code.

Because the postcode API uses the HTTP GET method we can use the browser to inspect it. If the API required that you use an HTTP POST method we would need to use other tools to make that request. Learn how to use **cURL** and **REST Client** to do this in the workshop on [Web Architecture](https://lab.arts.ac.uk/books/prototyping-lab/page/web-architecture#bkmrk-exercise%3A-a-closer-l).

Access an API using Node.js

For this exercise we are interested in only the following file. Notice there is no client code. This is because we are not sending anything to the browser yet.

├── ex2-api-to-cli
    └── server
        └── postcodes.js

The goal of this exercise is to:

1. Get some data from an API using our server-side code
2. Parse/process that data
3. Output this data to the command line interface

We will now do this using Node.js to make a request and push this data out to our command line screen. To do this we will use the postcodes.js.

Running the script in Node.js

1. Edit postcode.js and add your own postcode
   Open postcode.js with your preferred code editor and take a look at the following lines. Edit the file adding your own postcode and save it.

var myPostcode = "SE16SB"; // Add your postcode here.
var apiUrl = "http://api.postcodes.io/postcodes/" + myPostcode;

2. Change into correct directory
   Next we will run the postcode.js file with node: cd ~/Desktop/workshop-arduino-amplified-1617/ex2-api-to-cli/server/

Make sure you change `~/Desktop/` to the location where you downloaded your code

3. Install library
   Within the server-side code we are using a library/package called Request to make HTTP requests to the postcodes API. It is not yet installed so first we need to do that using npm:

npm install request

4. Run script using node.js
   Executing this script using Node.js will make a request to the API, passing the results into a function where we can parse the data and output to the CLI:

node postcodes.js

You should see an output in your terminal that gives you various postcode information.

Looking at the code

The code in this script is quite a straightforward. It can be broken down into two main parts:

1. Making an HTTP GET request to an API:

// Make an HTTP GET request to a URL.
request.get(apiUrl, function (error, response, body) {
  // This code is executed when the request is complete.
  // `body` contains the resulting data
  var data = JSON.parse(body);
  console.log(data);
});

2. Outputting that code to the CLI:

  console.log(data);
  // We can now use the . (dot) syntax to access the properties of the data object.
  // console.log(data.result.parliamentary_constituency);
  // console.log(data.result.codes.admin_district);

You can see how the examples that are commented out (preceded with a double slash //) are using a dot syntax to access properties of the data object. Uncomment those lines and try changing the values after data.result to output other values within the object to the terminal.

Exercise 3: Send API data to browser through WebSocket

Now we have sent dummy/random data from our Node.js script to a browser in Exercise 1 and accessed data from an API in Exercise 2 we can put these two exercises together to get API data and push that to a browser.

The goal of this exercise is to:

1. Get some data from an API using our server-side code
2. Parse/process that data
3. Send this data to our browser through a WebSocket.

For this part of the exercise we are interested in the following file:

├── ex2-socketio-api
    ├── client
    │   ├── postcode.html
    └── server
        └── postcodes-websockets.js

Client code

Running the code

Open the postcode.html file in your browser by dragging it onto a browser. Then open your Developer Tools and click on the Console tab. Don't get too excited yet. You will see something like this:

These are errors indicating that our client code is trying to communicate with a WebSocket using the address 127.0.0.1:4503 but it is failing with the error: ERR_CONNECTION_REFUSED. This is because we haven't yet started our WebSocket server using Node.js.

Looking at the code

The client code for this exercise has barely changed since the last example. The primary function of this code is to establish a connection to our locally hosted WebSocket server and then wait for messages to be received.

var socket = io.connect('http://127.0.0.1:4503');

socket.on('api-data-received', function (data) {
  var postcodeData = document.querySelector(".postcode-data");
  postcodeData.textContent = data;
  console.log(data);
});

1. The code connects to our socket.io server.
2. Then the code sets a listener to listen for messages with the name api-data-received.
3. The data from this message is then displayed in any HTML element with the class postcode-data and output to the console.

Server code

Again much like before we will be running a WebSocket server using Node.js. The server-side JavaScript code is in postcodes-websocket.js.

Running the server

1. Change into correct directory
   cd ex2-socketio-api/server/

2. Install library
   We are now going to use the Socket.io library again to create a WebSocket. npm install socket.io

We are installing the same library again because the last time we did installed it inside a different directory

3. Run script using node.js
   Executing this script using Node.js will make a request to the API, process the data and send the results through a WebSocket:

node postcodes-websocket.js

You will see the CLI output of API data and if you look in the browser you will see the web page and console has updated.

Looking at the code

The server code we are running is in postcodes-websocket.js. The purpose of this JavaScript is to setup a WebSocket using socket.io, then wait for clients to connect, making a request to an API and returning the data to that client.

The server code starts by establishing the server as usual, then this line adds an event listener. The listener waits for new clients to connect and calls a function called makeApiRequest, passing it a reference to the client.

io.on('connection', makeApiRequest);

This is the makeApiRequest function, the function takes a reference to the socket.io client that just connected:

1. The function makes a request to Postcodes.io, the result is passed back to us.
2. Then it checks for and displays any errors.
3. Parses the JSON from the API into a JavaScript object.
4. Logs the object then emits it to the client.

function makeApiRequest( socket ){
  request.get(apiUrl, function (error, response, body) {
    if(error){
      console.error("There was a problem with the request! Check your API URL.");
    }
    var data = JSON.parse(body);
    console.log(data);
    socket.emit('api-data-received', data.result);
  });
}