EEC172-Lab 4 Introduction to AWS and Restful API Solved

In this lab, you will explore the RESTful API and use it to connect with Amazon Web Services (AWS). First, you will create a ‘thing’ in AWS and use the HTTP GET command to retrieve status information about your thing. You will then use your code from Lab 3 to compose messages with an IR remote, which you will send to AWS using the HTTP POST command. You will use a rule in AWS to send the text to your cellphone using

Amazon’s Simple Notification Service (SNS). Much of the code you will need for accessing AWS will be provided to you. You will need to understand it at a high-level and modify it for your specific AWS account.

 

Please view the video AWS IoT Getting Started to understand the basics of AWS IoT. 

 

New Equipment Needed
• An Amazon AWS account (You can get a 1-year free trial account) 

See https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/billing-free-tier.html for information on using the AWS free tier. 

 

Development Tool Installation Procedure
To do the lab on your own computer, you will need to install the following programs:

 

1. OpenSSL  (https://www.openssl.org/)

OpenSSL will be needed to convert the .pem formatted certificate and keys to the .der format that is required. If you have other means of doing this, that is fine as well.

 

Background on the RESTful API: 

The RESTful API, based on the Representational state transfer (REST) architecture, is a web-based communication protocol that is widely used for many services. The RESTful API is typically implemented using HTTP.  Although not as optimized for low power and low-cost applications as another IoT protocol, MQTT, RESTful API is widely used due to its simplicity and genericity.

 

The RESTful API defines the format of the messages that are sent to a service to interact with that service. The function of the RESTful APIs will vary depending on the service with which you are interacting. Some of the commonly used commands are listed below.

 

Common RESTful API Commands: 

•      GET - Often used to retrieve data or information from a service. May also just be used as a simple trigger.  For AWS IoT, GET can be used to retrieve a device’s parameters from its shadow.

•      POST - Can be used to update or push data to the cloud or service.  For AWS IoT, POST could be used to update a device’s parameters in the device shadow.

•      DELETE - Used to manage and remove data structures in the cloud.  For AWS IoT, DELETE can be used to delete a device shadow.

 

Other RESTful API commands may exist, depending on the service, and are based on HTTP commands: HEAD, PUT, TRACE, OPTIONS, CONNECT, PATCH.

 

For more information on the RESTful API please see: Beginners guide to creating a REST API: http://www.andrewhavens.com/posts/20/beginners-guide-to-creating-a-rest-api 

 

Background on AWS IoT: 

Amazon Web Services (AWS) is a collection of various web-services for your cloud-based needs and removes the need of individual businesses to develop the hardware infrastructure for these initiatives. Recently, the Internet of Things (IoT) service was released, allowing connection of many hardware products.  

 

To represent these real-world devices in the cloud and maintain a persistent/coherent state with intermittent network connections, AWS IoT has a concept of a device shadow. This shadow represents the real-world device’s state in the cloud. As the device changes states, it pushes its changes to the shadow device on AWS via MQTT or a RESTful API. If the device loses network connection, and then comes back online, the device can pull the last-known state from the shadow to update itself. Updates to this shadow can trigger actions in other services as well using Rules. In this lab, a rule is used to send text messages to your cell phone.

 

Part I: Connecting to AWS (securely) and updating your Shadow 

Part I.A: Setting up an Amazon AWS account
For this lab, you will need to setup an Amazon AWS account and get a basic understanding of AWS.  One person per team will need to create an AWS account that can be shared for completing this lab.   

 

  

Figure 1: Amazon AWS Console View
 

After setting up an account, log into the console.  You should end up seeing a page similar to Figure 1.  After you are logged in, expand the services and click on the AWS IoT Core link. If it does not appear, make sure you have selected an Amazon AWS region at which this service is active. We recommend that you use the U.S. West (Oregon) AWS server for this lab.  

 

In the AWS IoT Core service, the controls on the left will list the actions you can do such as Monitor, Onboard,

Manage, Secure and Act.  In the bottom left corner of the screen, there is a Learn button and a link to Amazon’s IoT interactive tutorial. Click on it, and go through the tutorial for a high-level explanation about how AWS IoT uses Rules to do cool things. 

 

 

 

 

Part I.B: Setting Up Your First Device Thing/Shadow with the AWS IoT Console
For the next part of the lab, you will need to generate the access keys to allow your CC3200 to connect to your Amazon AWS account securely.  For a real-world device to connect to Amazon AWS you will need to do the following:

1. Create a Device Thing/Shadow This will generate a server-side representation of your real-world device (your thing).  

2. Generate the required certificates and keys for establishing an encrypted connection 3. Attach the certificate and keys to the Device Thing/Shadow 4. Give the Device Thing/Shadow permission to use the IoT functions of AWS
 

Fortunately, the above can be accomplished using the Amazon Console interface. Click on Manage, then Things and then Register a thing as shown in Figure 2.

 

  

Figure 2: Creating Your First Thing
 

You should be presented with a display similar to Figure 3. Click on the button at the bottom to Create a single thing.

 

  

Figure 3: Creating a Single Thing
 

 

For your first device thing, it is recommended you give it a descriptive name, such as ‘CC3200_Thing’ or ‘MyIdName_CC3200Board’. Leave the Thing Type, Thing Group and Attribute Key / Value options blank. Once you have entered the device name, click Next. When successful you should see something that looks like Figure 4.  

 

  

Figure 4: Creating Certificate for Your Thing
 

 

Click the Create certificate button for the One-click certificate creation as shown in Figure 4.  

  

Figure 5: Creation of Certificate, Public and Private keys

 

 

AWS should generate the required public key, private key, and certificate for identifying and encrypting traffic from your device, and allow you to download them to your computer.  It is important to download and save these files as you will NOT BE ABLE TO RETRIEVE THE PUBLIC and PRIVATE KEYS after you close the page. We will provide you with the appropriate root CA file so don’t bother downloading it from AWS. Download and save the certificate, public and private keys to a folder. Later in this tutorial, we will convert them from the .pem format to the .der format and load them into the CC3200’s serial flash.

 

After downloading, don’t forget to Activate the certificate. To learn more about how certificates are used, see: https://realtimelogic.com/blog/2013/10. Finally, once you have activated your certificate and keys, click the

Attach a policy button. Since we don’t have any existing policies, just click on Register a Thing on the next window. You have now registered your thing with an active certificate, but we still need to create and attach a policy to the certificate.  

 

Part I.C: Making a Policy to Allow Update/Get Status from Thing Shadows
To represent your thing (i.e., your CC3200 board) in the cloud, AWS uses the concept of a device shadow to which you can get the last state or push a new state to the server. Therefore, with intermittent connection to a wireless network, your device could pull the latest state from the cloud and maintain continuity. However, appropriate security measures can be made on AWS to ensure that the devices have the privilege to perform the actions requested. These next steps will walk you through how to do this:

 

From the AWS IoT console, click on Secure and then Policies, as shown in Figure 6 

 

Click on Create a policy. This should bring up a window similar to Figure 7.

 

  

Figure 7: Creating a Policy
 

Give your policy a descriptive name, and add the actions “iot:GetThingShadow” and “iot:UpdateThingShadow”. Furthermore, update the Amazon Resource Number (ARN) to signify that these actions apply to your thing created previously. Be sure to select the Allow Effect and then click on Create. You can learn more about the actions and other access management modules in the AWS documentation (http://docs.aws.amazon.com/IAM/latest/UserGuide//access_policies.html).  

 

Now that you have created a policy, you need to attach it to your certificate. From the AWS IoT console, select Secure and Certificates and then click on you’re the certificate you recently created. From the Actions pulldown menu, click on Attach policy as shown in Figure 8.

 

  

Figure 8: Attaching Policy to Certificate 

 

Select the policy that you just created and then click on Attach. Now your policy should be successfully attached to the certificate for your thing. This will allow you to use the GET and POST commands via REST from the device associated with this certificate.  

 

Part I.D: Converting the Keys/Certificates for Use with the CC3200
In this part of the lab, you will convert the private key and certificates to another format. This is necessary since AWS uses the .pem format, while your CC3200 uses the .der format.

 

The connection security between the CC3200 board and AWS is guaranteed through the TLS protocol. There are four components involved, the public and private key, a certificate and a root certificate. While the private key is used to decipher encrypted messages, the certificate is used to validate the public key. Certificates are built on a chain of trust, and so there is a root certificate which is used to sign the certificate.  

 

The key and certificates can exist in either .pem or .der format. The former has ASCII-readable characters (unsecure, readable), while the latter is in binary and more compact (still unsecure, but harder to read). Your generated files in AWS are in the .pem format but the CC3200 requires the .der format.

 

To convert the files from .pem to .der format, we will use openSSL, an open source library for transport layer security. You need to open a command prompt and navigate to the openSSL installation directory. The commands that are needed are the following:

 

For the Root Certificate Authority (rootCA) file and the certificate file

openssl x509 -outform der -in C:\...\rootCA.pem -out C:\...\rootCA.der 

(Note: we will provide the rootCA.der file)

 

openssl x509 -outform der -in C:\...\certificate.pem.crt -out C:\...\client.der 

 

For the Private Key File:

openssl rsa -outform der -in C:\...\private.pem.key -out C:\...\private.der 

 

You will obviously need to modify your file names and file paths to where you saved your keys/certificates. If necessary, move the files that need to be converted into the same file directory in which openSSL was installed. The rootCA file will be shared among all of the CC3200s used for the project.  The client and private keys should be unique to each device. It is highly recommended to group or name the files accordingly so that the keys/certificate pairs do not get mixed up or over-written.

 

Part I.E. Using UniFlash to flash key and certificates to CC3200
To enable a secure connection from your CC3200, you need to download the private key and certificates to the serial flash on your board. You will do this using the UniFlash utility, as described in the Uniflash tutorial.

 

You should flash the .der formatted certificates, root certificate (used for authenticating the AWS server), client certificate (used by AWS to authenticate our device), and the private key (for device-side encryption) to User Files using the UniFlash utility, as shown in Figure 9. You will specify the path to those files on your PC in the Url box and select the Erase, Update, and Verify checkboxes, and click Program to flash those files to the board. You can also load the application (which you will develop in a later part) binary into /sys/mcuimg.bin or you can download it from Code Composer Studio and run it from the Debugger. Take note of the path of your files on the flash (i.e. – the ‘Name’ field below) as you will need to reference them in your code. If you forget them, you can list the files on your flash by clicking on List Files in UniFlash.  Be sure to place jumper on SOP2 for flashing.

 

  

Figure 9: Sample Setup to Flash a RootCA, Private Key and Certificate 

 

 

 

 

Part I.F. Accessing AWS using the RESTful API
Amazon’s RESTful API for AWS IoT provides access to device shadows, or thing shadows.  These are JavaScript Object Notation (JSON) formatted data that allow state information for AWS things to be stored and retrieved, even when a device might be disconnected from the service. To store or retrieve data from a device shadow, Amazon provides special MQTT feeds and RESTful API calls.  

 

Select your device thing from the AWS IoT console and click on Interact as shown in Figure 10.

 

  

Figure 10: Rest API Endpoint for Your Thing 
 

From this page, you can get the Rest API endpoint for your thing as well as MQTT topics that will allow you to update your thing’s shadow. The REST API endpoint displays the web address for the AWS Thing/Shadow, and contains the web address of your account’s AWS Endpoint. For now, copy the endpoint address into a word or text file, and save it for reference. You will need it later to connect to the AWS server.  Below is an example Restful API endpoint address.  The initial string of characters will be some randomly generated string generated for your account by Amazon.

 

Endpoint:  

identifier.iot.region.amazonaws.com 

 

Example: 

https://xxxxxxxxxxxxxx.iot.us-west-2.amazonaws.com

 

Updating State Information to your Device Shadow:

https://endpoint/things/thingName/shadow 

 

 

Example GET Method (GetThingShadow)
 

GET /things/thingName/shadow HTTP/1.1 

Host: identifier.iot.region.amazonaws.com 

Connection: Keep-Alive 

 

 

Example POST Method (UpdateThingShadow)
 

POST /things/thingName/shadow HTTP/1.1 

Host: identifier.iot.region.amazonaws.com  

Connection: Keep-Alive 

Content-Type: application/json; charset=utf-8 

Content-Length: 30 

{ 

      "state":{ 

                      "desired":{ 

                                      “color":"green" 

} 

} 

} 

 

To send a RESTful API POST message to AWS, the service must first be connected to with the SimpleLink socket library. The initial connection should be to the base address for the service or website with which you are trying to communicate.  For example, you may have an AWS device thing that is said to have the RESTful API address of the following: https://<endpoint.iot. us-west-2.amazonaws.com/things/CC3200_Thing/shadow.  The initial connection with the socket library should only be to <endpoint. iot. us-west-2.amazonaws.com. The HTTPS part of the address will be managed based on the socket credentials, and the AWS device thing subdirectory will be used in the RESTful API calls.

 

Connection Configuration for AWS
•      TLS1.2

•      Cipher: TLS ECDHE RSA with AES 256 CBC SHA

•      Port: 8443

•      Certificates needed: Certificate Authority file for AWS (rootCA.der), x509 Device Certificate file (client.der) and the matching private key file (private.der)

 

 

The RESTful API application requires Internet connectivity through an Access Point (AP). The AP details are contained in the header file common.h. Modify the common.h header file, if necessary, to match the following configuration for the AP in the lab:

 

// Values for below macros shall be modified as per access-point(AP) properties 

// SimpleLink device will connect to following AP when application is executed // 

#define SSID_NAME           "eec172"          /* AP SSID */ 

#define SECURITY_TYPE       SL_SEC_TYPE_OPEN /* Security type (OPEN or WEP or WPA*/ 

#define SECURITY_KEY        ""               /* Password of the secured AP */ 

#define SSID_LEN_MAX        32 

#define BSSID_LEN_MAX       6 

 

Once common.h has been modified, you can build and test the example program. Please note that common.h does not reside in your workspace, so if you work on the lab computers and your personal laptop you will need to modify them in both places, and with the appropriate settings for the wireless network you are trying to connect with.

 

Another important change is that the secure client application must have the “current” date and time so that the credentials can be verified as valid. In this example, we use variables in the main program to update the date and time. The date and time only need to be current within the last month or so. Having the date and time hardcoded into the program is obviously not a robust or secure programming method since the code will eventually stop working unless periodically updated. However, in the interest of reducing complexity, we will use this method for this lab. The date and time variables are stored in the following constants in main.c:

 

//NEED TO UPDATE THIS FOR IT TO WORK! 

#define DATE                26   /* Current Date */ 

#define MONTH               01     /* Month 1-12 */ 

#define YEAR                2019  /* Current year */ 

#define HOUR                17    /* Time - hours */ 

#define MINUTE              0    /* Time - minutes */ 

#define SECOND              0     /* Time - seconds */ 

 

Now you can connect your CC3200 securely to AWS. Demonstrate that you can use a POST request to post state to your AWS thing and use a GET request to obtain status information on your AWS thing. You should receive an HTTP 200 status code, showing a successful POST or GET request, as well as some basic status information about your device thing.

 

NOTE: you need to give your thing shadow some initial state before you can do an HTTP GET on it. You can give the shadow state by using the HTTP POST command or by using the AWS web interface. Also, update the time in main.c everytime you run your program. The time difference of few minutes should be ok. In common.h file, SSID_Name is the name of your WiFi. We suggest that you use portable WiFi hotspot on your device with open security. The CC3200 doesn’t require a lot of data to work with AWS. If this is not possible, then change SECURITY_TYPE to match to your WiFi and enter the WiFi password in SECURITY_KEY.

 

In this part, you can start with the working example project SSL_REST_API_AWS_W19.zip. You will need to examine it carefully to understand it and modify it appropriately. In particular, you should examine the http_post() function to see how the HTTP command is sent to AWS. The data must be formatted properly, or AWS (or any other web service) will not accept it as valid. In order to use this project on your system, you may need to modify the following files or parameters:

1.      common.h – as described above for the eec172 Access Point.

2.      main.c – you will need to use your own AWS endpoint path and thing name as well as changing the HTTP command in order to test both GET and POST. You should also change the message data that you post to your thing. However, be careful to keep the correct data format. You may also need to update the date, as described above.

3.      pinmux.c – verify that the pinmux.c file sets up the pins that you intend to use. For the default application, only the UART0 and GPIO signals driving the on-board LED are actually used. If your program uses the same pins as the on-board LEDs, you will have problems.

4.      Flash the private key, the client certificate and the root Certificate Authority into the CC3200 flash, as described.

 

 

 

             

Part II:  Using SNS to send a text to your phone 

For this project you are to compose a text message using your IR remote and send it to your phone. This will be accomplished by setting up a topic in the Simple Notification Service and registering your text-capable cell phone, and establishing a Rule in the IoT service to trigger when you update your device shadow and send messages to your phone. The interactive tutorial you went through in Part I.A gives you a high-level overview on how this is accomplished in AWS.

 

For this lab we recommend that you use the “US West (Oregon)” AWS server. You need to use a server that runs both the IoT and SNS services.

 

 

To do this you must perform the following:

1.      Create a thing for the cc3200 as you have done before.

2.      Create an SNS topic  

3.      Subscribe to the SNS topic with your phone number and the SMS option.  

4.      Create a rule to forward incoming thing messages to your SNS topic.  

5.      Use the REST API to post to your topic.

 

We will walk you through creating an SNS topic and creating a rule.

 

Part II.A. Creating an SNS Topic:

 

Amazon Web Services has a collection of different web services which can be integrated to perform a larger function. So far you have been using the IoT service to communicate the states of real-world devices or things to the cloud. The Simple Notification Service (SNS) is another module that will allow you to push messages to subscribers.  Navigate to the SNS service by expanding the services tab and finding Simple Notification System under Application Integration as shown in Figure 11.   

 

  

Figure 11: Simple Notification Service in Menu of AWS Services
 

 

From the SNS dashboard, click on Create topic as shown in Figure 12.  

 

  

Figure 12: SNS Dashboard
 

Next, choose a Topic name and a Display name of no more than 10 characters and then click Create topic, as shown in Figure 13.

 

  

Figure 13: Creating an SNS Topic
 

Afterwards, Create a Subscription, where the protocol is SMS (Simple Messaging Service) for text messages.

The endpoint is your cell-phone number to receive text messages. You can then test to make sure this module works by clicking on the ‘Publish to topic’ button and manually sending a text message from the AWS Console to your cell-phone (use RAW format for now).

 

  

Figure 14: Publish a Message to your SNS Topic
 

 

Part II.B. Creating an IoT Rule: 

 

The next step is to create an IoT Rule that will trigger when you push updates to your shadow device. Earlier in the lab, we updated our device shadow (state) via the REST API. When our device shadow is updated, the IoT module publishes messages to several reserved Message Queue Telemetry Transport (MQTT) topics, a publishsubscribe ‘lightweight’ messaging protocol. In order to trigger your rule, we will be listening for updates to the device shadow via these topics. A full list of reserved topics that can be subscribed to can be found at http://docs.aws.amazon.com/iot/latest/developerguide/thing-shadow-mqtt.html .  

 

From the AWS IoT console, click on Act and then click on Create a Rule, as shown in Figure 15.

  

Figure 15: Creating a Rule
 

 You should listen to the topic ‘$aws/things/thingName/shadow/update/accepted’ (the $ and the single quotes should be included), report on the attribute ‘state.desired’, and leave the condition blank. Then attach an action that will send an SNS push notification when the rule is triggered. Select the SNS target to be the topic you created previously, and use the ‘RAW’ message format (for now). You will also need to create an IAM role to allow the IoT to access the SNS service ‘securely’ (you can learn more about Identity Access and Management [IAM] roles here: http://docs.aws.amazon.com/iot/latest/developerguide/iam-users-groupsroles.html, but will not need to understand them for this lab). Your SQL query statement for your rule should be similar to the figure below.  

 

  

Figure 16: Setting up your IoT Rule
 

Make sure the rule is enabled and try to push an update to your device shadow using your CC3200,… in a couple of seconds you should get a text message. Once that works, you can clean up the formatting using JSON message format SNS action. To figure out how to do this, go back to the SNS topic you made, click the Publish a topic button, and experiment with the JSON message generator to find the appropriate syntax. Investigate how you will need to modify your SQL statement to ‘clean-up’ your message. Helpful link and subsections for SQL: http://docs.aws.amazon.com/iot/latest/developerguide/iot-sql-reference.html 

 

  

Figure 17: Updating Device Shadow from CC3200
 

  

Figure 18: Verifying Device Shadow State
 

 

Part II.C. Integrate your game to send a message to your phone: 

Now that you have the SNS module and IoT Rules set up appropriately, you can start integrating your Lab 3 code to send a text to your phone when the game is finished. Start with the program for connecting to AWS via the REST API that you implemented in the last part. When you are done, you will be sending a text from your CC3200 through the cloud to your cellphone. Cool, yeah?

 

The Lab 3 has the game, which ends when the ball touches the boundary of the OLED Screen. At this point you can simply text Game Ended! to your phone.

 

Note: If the SMS is not getting delivered due to some issues, you can also use email. This step has to be done while Create a Subscription.