The parse service is a server-side solution provided by Facebook, Firebase is google's offering in this space.
However, arguably, the most comprehensive set of cloud services today are provided by Amazon. We shall be using, for these lab sessions, the Amazon Web Services (AWS).
Table of Contents<< Instructor's Page
<< Student's Page
Recommended Reading for this Section:
Software and AWS Setup:
In this section, we are going to connect a temperature sensor, a buzzer, and a switch to Edison. When you complete the python program, Edison will collect data from the temperature sensor and send it to AWS Kinesis (real-time stream processing service). At the beginning, the buzzer will sound. if you press the button, the buzzer sounds again, and Edison starts sending data to DynamoDB (database) in the AWS cloud. This lab is to familiarize you with connecting Edison to AWS (the cloud.)
Please review the Recommended Reading section for clarity on the interfaces, working with the services & a basic overview of what these services are.
Recommended Reading for this Section:A>> Cognito1. Boto API reference: http://boto.cloudhackers.com/en/latest/ref/cognito-identity.html
2. Boto3 API reference: http://boto3.readthedocs.org/en/latest/reference/services/cognito-identity.html
3. Amazon Cognito Identity Documentation: https://docs.aws.amazon.com/cognito/devguide/identity/
B>> Kinesis:1. Amazon Kinesis Key Pointers: http://docs.aws.amazon.com/kinesis/latest/dev/key-concepts.html
2. Understanding an Overview: https://blogs.aws.amazon.com/bigdata/post/Tx2Z24D4T99AN35/Snakes-in-the-Stream-Feeding-and-Eating-Amazon-Kinesis-Streams-with-Python
3. Kinesis intended purpose: http://techcrunch.com/2013/11/14/amazon-kinesis-a-new-aws-service-to-process-real-data-streams/
4. Working with boto & Kinesis: http://nullege.com/codes/search/boto.connect_kinesis
5. Organised Github Example: https://github.com/awslabs/kinesis-poster-worker
6. Alternate approach: http://stackoverflow.com/questions/22100206/consuming-a-kinesis-stream-in-python
7. Tutorial (Snakes in the Stream, uses boto): https://blogs.aws.amazon.com/bigdata/post/Tx2Z24D4T99AN35/Snakes-in-the-Stream-Feeding-and-Eating-Amazon-Kinesis-Streams-with-Python
C>> DynamoDb1. Amazon DynamoDB Data Model: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DataModel.html
2. Organized Tutorial (Uses Boto3): http://boto3.readthedocs.org/en/latest/guide/dynamodb.html
3. Organized Tutorial (Uses Boto): http://boto.cloudhackers.com/en/latest/dynamodb2_tut.html
4. Scanning a Table (Different Techniques): http://stackoverflow.com/questions/12499822/how-can-i-get-the-total-number-of-items-in-a-dynamodb-table
Required for this lab:
The temp. sensor, buzzer, and switch are all in the Grove-Starter Kit Plus package.
Software and AWS Setup:
To interface with the Amazon Web Services in python, we use the boto library. If you haven't already, install it as follows:
Install boto (& in your main project use boto3, as that is the latest version.) & verify that the .boto file has been created.
pip install boto
Insert the following code into the file (if it does not already exist):
Type the following to create a folder for Lab 2.
mkdir ~/Lab2 cd ~/Lab2
We will ultimately run this python file to send data from Edison to the cloud.
2. Begin writing into edisonAWSDemo.py:
import boto import boto.dynamodb2
Now with AWS:
1. Go to aws.amazon.com and login to your console.
2. Click on your user name at the top right corner and navigate to My Account page.
3. On that page below Account Details you can find your Account ID. You will be putting this in your python file, along with other identifiers, as shown below.
Cognito is a service that offers identity management and data synchronization. What that essentially means is that services you create can be accessed using your unique Cognitoidentities for your users and these can be authenticated with identity providers. Supported identity providers include Amazon, Google, Facebook. In addition, you can have unauthenticated users or custom authentication that you define. You can create these identities and manage them easily using the Identity & Access Management (IAM) service provided by AWS.
To interface with these different AWS services, we can either choose to manually initialize them using the console or programmatically in code. When using python (our language of choice for this tutorial), we shall make use of the boto library for AWS interfacing.
We use the AWS GUI-based interface for Cognito & IAM, considering that we are working with a fairly small number of users & easily-managed identities. Working with these programmatically is left as an exercise to the reader.
Follow the steps below to get started with the Cognito setup. The steps in the GUI & the corresponding steps in the python code file have been accompanied with explanations wherever possible.
4. Go back to the console page and click on "Cognito," under Mobile Services. Click on "Create new identity pool." Type edisonDemoKinesis as the name to the identity pool. An identity pool is a store of user identity data specific to your account. Using Amazon Cognito Sync, you can retrieve the data across client platforms, devices, and operating systems, so that if a user starts using your app on a phone and later switches to a tablet, the persisted app information is still available for that user. You can read more about identity pools here.
5. Check the box that says Enable access to unauthenticated identities (we are currently not using an identity provider such as Google, Amazon or Facebook). In our case, the unauthenticated identity is our Edison application.
6. After creating cognito identity, go back to the Cognito dashboard and click on it. Click "Edit identity pool" on top right corner. In that page, you can find your Identity Pool ID.
7. Now go to the main console and navigate to Identity & Access management (IAM). In that page, under IAM Resources click on "Roles." You can see your unauthenticated role asCognito_edisonDemoKinesisUnauth_Role. Click on it. Under Summary, you can find your Role ARN.
Type the Account ID, Identity Pool ID, and Role ARN in the edisonAWSDemo.py file. You can see below an example:
ACCOUNT_ID = '364418251412' IDENTITY_POOL_ID = 'us-east-1:c64c86ba-c46b-4287-852e-034b2e2fcef7' ROLE_ARN = 'arn:aws:iam::364418251412:role/Cognito_edisonDemoKinesisUnauth_Role'
Further add the following code that sets up your cognito & STS interface using boto. Please note that, to setup these two services, these steps are essential. As we shall see in further examples below, working with these lines allows for access to a number of identity protected features on the Amazon Web Services.
# Use cognito to get an identity. cognito = boto.connect_cognito_identity() cognito_id = cognito.get_id(ACCOUNT_ID, IDENTITY_POOL_ID) oidc = cognito.get_open_id_token(cognito_id['IdentityId']) # Further setup your STS using the code below sts = boto.connect_sts() assumedRoleObject = sts.assume_role_with_web_identity(ROLE_ARN, "XX", oidc['Token'])
http://boto.cloudhackers.com/en/latest/ref/cognito-identity.html provides some quick references to the details regarding the API.
DynamoDB is the Amazon NoSQL cloud service. It is a service providing <10ms latency. The service allows for working with Key-Value Pairs, which we shall be using predominantly in this assignment, in addition to data store models.
A strongly recommended resource to understand this is
Use the above as a background for the creation of your program.
As above, let's get started with the AWS setup part first.
8. On the AWS web page, return to the main console and select "DynamoDB" under Database. Click "Get Started."
9. Under "Table name," type edisonDemoDynamo
10. Under "Primary key," type deviceid.
11. Return to the IAM section in the main console. There, head over to the Roles section as before and select the Cognito_edisonDemoKinesisUnauth_Role.
12. There, click "Attach Policy" under Permissions and check the "AmazonDynamoDBFullAccess" policy. Click "Attach Policy."
Back on your python code, to make your life a tad bit easier, define a constant as below. Fill in whatever name you select from above.
DYNAMODB_TABLE_NAME = 'edisonDemoDynamo'
Configuring the DynamoDB has two possibilities. One, as above, we interface the database that we created above in our Python code.
Add the following lines in your program.
# Prepare DynamoDB client client_dynamo = boto.dynamodb2.connect_to_region( 'us-east-1', aws_access_key_id=assumedRoleObject.credentials.access_key, aws_secret_access_key=assumedRoleObject.credentials.secret_key, security_token=assumedRoleObject.credentials.session_token) from boto.dynamodb2.table import Table table_dynamo = Table(DYNAMODB_TABLE_NAME, connection=client_dynamo)
Do review the following link to get started with the boto library and interfacing to the AWS services.
IMPORTANT NOTE: When following the tutorial, please be sure to include the "connection=client_dynamo" in your function call as above.
For example, when creating a Table, be sure to add the above.
users = Table.create('users', schema=[HashKey('username')], connection=client_dynamo)
IMPORTANT NOTE: After creating the table, be sure to put a small delay (~10-20sec) right after the Table.create(....) command, so as to allow Amazon to actual create & configure the table.
Check out the recommended reading sections for information on working with the boto3 library.
You will need to review this section for a portion of your assignment for the week.
Finally, we shift to one of Amazon's real-time processing & streaming service, Kinesis.
Amazon Kinesis is a platform for streaming data on AWS, offering powerful services to make it easy to load and analyze streaming data, and also providing the ability for you to build custom streaming data applications for specialized needs. Web applications, mobile devices, wearables, industrial sensors, and many software applications and services can generate staggering amounts of streaming data – sometimes TBs per hour– that need to be collected, stored, and processed continuously
For example, Amazon Kinesis can be used to perform tasks such as real-time analyses on streaming data and create alarms when the load performance is bad or when there are unusual user behaviors.
As before, begin by setting up the Amazon Kinesis service on your AWS Console online.
11. Return to the console and select "Kinesis" under Analytics.
12. Click "Go to Kinesis Streams," and then click "Create Stream." Type edisonDemoKinesis as the stream name. Type 1 as number of shards. Click "Create."
13. Return to the IAM section in the main console. There, head over to the Roles section as before and select the Cognito_edisonDemoKinesisUnauth_Role.
14. There, click "Attach Policy" under Permissions and check the "AmazonKinesisFullAccess" policy. Click "Attach Policy."
15. Add the following lines to interface the Kinesis stream in your python file, change the name of your stream name as per what you fill up above.
KINESIS_STREAM_NAME = 'edisonDemoKinesis'
Following this, add the relevant code needed for the Kinesis client.
# Prepare Kinesis client client_kinesis = boto.connect_kinesis( aws_access_key_id=assumedRoleObject.credentials.access_key, aws_secret_access_key=assumedRoleObject.credentials.secret_key, security_token=assumedRoleObject.credentials.session_token)
For your reference, use the following link to familiarize yourself with working with the Kinesis stream.
Based on the instructions described on the Assignment Page, complete this code. Test and run.
Bibliography:1. Lab Based on: http://hack-day.s3-website-us-east-1.amazonaws.com/Welcome.html
2. Documentation as above.