Amazon DynamoDB Demo

January 19th, 2012

Yesterday Amazon announced Amazon DynamoDB, their Internet-scale NoSQL database service. Back in November we were fortunate enough at CloudSpokes to be invited to participate in a private beta for Amazon DynamoDB. We’ve since then had some time to “work it over” a bit and I have put together a demo application to show off some functionality. The code for the HomeController is below but you can find source for the entire applications at our GitHub repo for your forking convenience.

The API is very straight forward and easy to work with. If you’ve used other NoSQL databases then you should have no problem wrapping your head around Amazon DynamoDB. It has simple storage and query methods, allowing you to store and access data items with a flexible number of attributes using simple “Put” or “Get” web services APIs. Amazon DynamoDB provides a native API for HTTP and SDKs for Java, PHP and .NET. More are reportedly in the works.

What is Amazon DynamoDB and why would I want to use it?

Amazon DynamoDB is a fast, highly scalable, highly available, cost-effective non-relational database service that scales automatically without limits or administration. This service is tightly coupled with Amazon S3 andAmazon EC2, collectively providing the ability to store, process and query data sets in the cloud.

If you have massive amounts of highly transactional data then Amazon DynamoDB might be for you:

  • Store Social Graph data for processing
  • Storing GPS data for devices
  • Data storage for Hadoop processes
  • Record user activity logs
  • NFC processes
  • Recording clicks for A/B testing

Blazing Fast – Amazon DynamoDB runs on a new solid state disk (SSD) architecture for low-latency response times. Read latencies average less than 5 milliseconds, and write latencies average less than 10 milliseconds. We found our applications to be extremely responsive.

Hands Off Administration – Amazon DynamoDB is a fully managed service – no need to worry about hardware or software provisioning, setup and configuration, software patching, or partitioning data over multiple instances as you scale. For instance, when you create a table, you need to specify the request throughput you want for your table. In the background, Amazon DynamoDB handles the provisioning of resources to meet the requested throughput rate.

Auto Scaling – To continue with the “no administration” theme, Amazon DynamoDB can automatically scale machine resources in response to increases in database traffic without the need of client-side partitioning. Alternatively, you can also proactively manage performance with a few simple commands.

Security Baked In – Amazon DynamoDB is integrated with AWS Identity and Access Management (access keys and tokens) allowing you to provide access to defined users and groups, assign granular security credentials and user access, much more.

Centralized Monitoring – As with most everything in AWS-land, you can easily view metrics for your Amazon DynamoDB table in the AWS Management Console. You can also view your request throughput and latency for each API as well as resource consumption through Amazon CloudWatch.

API Overview

From a high level, Amazon DynamoDB API provides the following functionality:

  • Create a table
  • Delete a table
  • Request the current state of a table
  • Get a list of all of the tables for the current account
  • Put an item
  • Get one or more items by primary key
  • Update the attributes in a single item
  • Delete an item
  • Scan a table and optionally filter the items returned using comparison operators
  • Query for items using a range index and comparison operators
  • Increment or decrement a numeric value

Data Model

Amazon DynamoDB stores data in tables containing items with a collection of name-value pairs (attributes). Items (anaglous to a record) are managed by assigning each item a primary key value. Unlike traditional databases, the table is schemaless and only relies on the primary key. Items can contain combination of attributes. For example:

“Name” = “Member Search with Redis”
“ChallengeId” = 1219
“Categories” = “aws”, “ruby”, “mobile”
“Ratings” = 17, 36

Primary Keys & Indexes

When creating a new table, you define the primary key and type of key to be used. Amazon DynamoDB supports a one name/value pair primary key (a hash primary key; string or number) or two name/value pair primary key (a hash-and-range primary key) for index values.

Hash key example: “ChallengeId” = 1219
Hash-and-range key example: “MemberId” = “romin”, “MemberNumber” = “976″

Note: the Query API is only available for hash-and-range primary key tables. If you are using a simple hash key, then you need to use the Scan API.

Data Types

Amazon DynamoDB supports two scalar data types (Number and String) and multi-valued types (String Set and Number Set). Everything is stored in Amazon DynamoDB as a UTF-8 string value. You designate the data as a Number, String, String Set, or Number Set in the request but there is no distinction between an int, long, float, etc. For example:

item.put(“member”, new AttributeValue().withS(“romin”));
item.put(“challenge”, new AttributeValue().withN(“1219″));

Amazon DynamoDB supports both Number Sets and String Sets:

“Challenges”:[{
 "members":{"SS":["kenji776, romin, akkishore"]},
 ”wins” : {“NS”:["14", "10", "8"]}
}]

Amazon DynamoDB uses JSON as the transport protocol. However, the JSON data is parsed and stored nativly on disk.

That’s a quick overview so make sure to check out the DynamoDB documentation for more details. The documentation is very well done and has clear instructions and code samples for Java, PHP and .Net. If you are into database performance, check out the details on provisioned throughput, data consistency, conditional operations, performance factors and more.

How to get started?

Sign up for a new AWS account (if you don’t already have one) and get your AWS Access Key ID and Secret Access Key from your account’s security section. Walk through their Getting Started Guide for samples. In your code, just add your credentials to the AWSDynamoDBClient and you are ready to start making requests. All of the API calls are pretty straightforward and work as you would expect them to.

Pricing is, again, pay as you go but Amazon DynamoDB is part of the AWS’s Free Usage Tier so check it out for more info.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
package com.cloudspokes.dynamodb;
 
@Controller
public class HomeController {
 
  static AmazonDynamoDBClient dynamoDB;
  private String tableName = "kiva-loans";
  // aws keys
  public static final String ACCESSKEY = "YOURKEY";
  public static final String SECRETKEY = "YOURSECRET";
 
  public HomeController() {
    try {
      setup();
    } catch (Exception e) {
      // TODO Auto-generated catch block
      e.printStackTrace();
    }
  }
 
  /**
   * Displays the list of loans from the table
   */
  @RequestMapping(value = "/loans", method = RequestMethod.GET)
  public String loans(Locale locale,
      @RequestParam(value = "keyword", required = false) String keyword,
      Model model) {
 
    ArrayList<Loan> loans = new ArrayList<Loan>();
    ScanRequest scanRequest = new ScanRequest(tableName);
 
    if (keyword != null) {
      HashMap<String, Condition> scanFilter = new HashMap<String, Condition>();
      Condition condition = new Condition().withComparisonOperator(
          ComparisonOperator.EQ.toString()).withAttributeValueList(
          new AttributeValue().withS(keyword));
      scanFilter.put("country", condition);
      scanRequest = new ScanRequest(tableName).withScanFilter(scanFilter);
    }
    ScanResult scanResult = dynamoDB.scan(scanRequest);
 
    for (int i = 0; i < scanResult.getCount(); i++) {
      HashMap<String, AttributeValue> item = (HashMap<String, AttributeValue>) scanResult
          .getItems().get(i);
      Loan loan = new Loan();
      loan.setActivity(item.get("activity").getS());
      loan.setCountry(item.get("country").getS());
      loan.setFunded_amount(Double.parseDouble(item.get("funded_amount")
          .getN()));
      loan.setId(Integer.parseInt(item.get("id").getN()));
      loan.setName(item.get("name").getS());
      loan.setStatus(item.get("status").getS());
      loan.setUse(item.get("use").getS());
      loans.add(loan);
    }
 
    model.addAttribute("loans", loans);
    return "loans";
  }
 
  /**
   * Displays a loan item
   */
  @RequestMapping(value = "/show/{id}", method = RequestMethod.GET)
  public String show(@PathVariable String id, Locale locale, Model model) {
    model.addAttribute("loan", getLoan(id));
    return "show";
  }
 
  /**
   * Displays a form to create a new loan item
   */
  @RequestMapping(value = "/new", method = RequestMethod.GET)
  public String newLoan(Locale locale, Model model) {
    model.addAttribute("loan", new Loan());
    return "new";
  }
 
  /**
   * Inserts a new loan item into dynamodb
   */
  @RequestMapping(value = "/new", method = RequestMethod.POST)
  public String addLoan(@ModelAttribute("loan") Loan loan,
      BindingResult result) {
 
    // populate an item with the data to put
    HashMap<String, AttributeValue> item = new HashMap<String, AttributeValue>();
    item.put("id", new AttributeValue().withN(String.valueOf(loan.getId())));
    item.put("activity", new AttributeValue().withS(loan.getActivity()));
    item.put("country", new AttributeValue().withS(loan.getCountry()));
    item.put("funded_amount", new AttributeValue().withN(String
        .valueOf(loan.getFunded_amount())));
    item.put("name", new AttributeValue().withS(loan.getName()));
    item.put("status", new AttributeValue().withS(loan.getStatus()));
    item.put("use", new AttributeValue().withS(loan.getUse()));
 
    // put the item to the table
    try {
      PutItemRequest req = new PutItemRequest(tableName, item);
      PutItemResult res = dynamoDB.putItem(req);
    } catch (AmazonServiceException ase) {
      System.err.println("Failed to create item in " + tableName);
    }
 
    return "redirect:show/" + loan.getId();
  }
 
  /**
   * Displays the item for editing
   */
  @RequestMapping(value = "/edit/{id}", method = RequestMethod.GET)
  public String editLoan(@PathVariable String id, Locale locale, Model model) {
    model.addAttribute("loan", getLoan(id));
    return "edit";
  }
 
  /**
   * Submits the updates loan data to dynamodb
   */
  @RequestMapping(value = "/edit/{id}", method = RequestMethod.POST)
  public String updateLoan(@PathVariable String id,
      @ModelAttribute("loan") Loan loan, BindingResult result) {
 
    Key key = new Key().withHashKeyElement(new AttributeValue().withN(id));
    HashMap<String, AttributeValueUpdate> updates = new HashMap<String, AttributeValueUpdate>();
 
    AttributeValueUpdate update = new AttributeValueUpdate().withValue(
        new AttributeValue(loan.getStatus())).withAction("PUT");
    updates.put("status", update);
 
    // update the item to the table
    try {
      UpdateItemRequest req = new UpdateItemRequest(tableName, key,
          updates);
      UpdateItemResult res = dynamoDB.updateItem(req);
    } catch (AmazonServiceException ase) {
      System.err.println("Failed to update item: " + ase.getMessage());
    }
 
    return "redirect:../show/" + id;
  }
 
  /**
   * Fetches loan data from Kiva an inserts it into dynamodb
   */
  @RequestMapping(value = "/loadData", method = RequestMethod.GET)
  public String loadData(Locale locale, Model model) {
 
    // delete all of the current loans
    deleteAllLoans();
 
    // make the REST call to kiva
    DefaultHttpClient httpClient = new DefaultHttpClient();
    HttpGet getRequest = new HttpGet(
        "http://api.kivaws.org/v1/loans/newest.json");
    getRequest.addHeader("accept", "application/json");
    HttpResponse response;
    String payload = "";
 
    try {
      response = httpClient.execute(getRequest);
 
      if (response.getStatusLine().getStatusCode() != 200) {
        throw new RuntimeException("Failed : HTTP error code : "
            + response.getStatusLine().getStatusCode());
      }
 
      BufferedReader br = new BufferedReader(new InputStreamReader(
          (response.getEntity().getContent())));
      payload = br.readLine();
    } catch (ClientProtocolException e) {
      // TODO Auto-generated catch block
      e.printStackTrace();
    } catch (IOException e) {
      // TODO Auto-generated catch block
      e.printStackTrace();
    }
 
    JSONObject json = (JSONObject) JSONSerializer.toJSON(payload);
    // get the array of loans
    JSONArray loans = json.getJSONArray("loans");
 
    for (int i = 0; i < loans.size(); ++i) {
      JSONObject loan = loans.getJSONObject(i);
      // populate an item with the data to put
      HashMap<String, AttributeValue> item = new HashMap<String, AttributeValue>();
      item.put("id", new AttributeValue().withN(loan.getString("id")));
      item.put("name", new AttributeValue().withS(loan.getString("name")));
      item.put("status",
          new AttributeValue().withS(loan.getString("status")));
      item.put("funded_amount",
          new AttributeValue().withN(loan.getString("funded_amount")));
      item.put("activity",
          new AttributeValue().withS(loan.getString("activity")));
      item.put("use", new AttributeValue().withS(loan.getString("use")));
      item.put("country", new AttributeValue().withS(loan.getJSONObject(
          "location").getString("country")));
 
      try {
        PutItemRequest req = new PutItemRequest(tableName, item);
        PutItemResult res = dynamoDB.putItem(req);
        System.out.println("Put result: " + res);
      } catch (AmazonServiceException ase) {
        System.err.println("Failed to create item in " + tableName);
      }
 
    }
 
    httpClient.getConnectionManager().shutdown();
 
    return "redirect:loans";
  }
 
  /**
   * Displays the home page
   */
  @RequestMapping(value = "/", method = RequestMethod.GET)
  public String home(Locale locale, Model model) {
    return "home";
  }
 
  /**
   * Fetches a specific loan item
   */
  private Loan getLoan(String id) {
    Key key = new Key().withHashKeyElement(new AttributeValue().withN(id));
    GetItemRequest req = new GetItemRequest(tableName, key);
    GetItemResult res = dynamoDB.getItem(req);
    HashMap<String, AttributeValue> item = (HashMap<String, AttributeValue>) res
        .getItem();
 
    Loan loan = new Loan();
    loan.setActivity(item.get("activity").getS());
    loan.setCountry(item.get("country").getS());
    loan.setFunded_amount(Double.parseDouble(item.get("funded_amount")
        .getN()));
    loan.setId(Integer.parseInt(item.get("id").getN()));
    loan.setName(item.get("name").getS());
    loan.setStatus(item.get("status").getS());
    loan.setUse(item.get("use").getS());
    return loan;
  }
 
  /**
   * Deletes all items from dynamodb
   */
  private void deleteAllLoans() {
    ScanRequest scanRequest = new ScanRequest(tableName);
    ScanResult scanResult = dynamoDB.scan(scanRequest);
    for (int i = 0; i < scanResult.getItems().size(); i++) {
      HashMap<String, AttributeValue> item = (HashMap<String, AttributeValue>) scanResult
          .getItems().get(i);
      try {
        Key key = new Key()
            .withHashKeyElement(new AttributeValue("id"))
            .withHashKeyElement(item.get("id"));
        DeleteItemRequest request = new DeleteItemRequest(tableName,
            key);
        DeleteItemResult result = dynamoDB.deleteItem(request);
        System.out.println("Result: " + result);
      } catch (AmazonServiceException ase) {
        System.err.println("Failed to delete item in " + tableName);
      }
    }
  }
 
  private void setup() throws Exception {
    BasicAWSCredentials creds = new BasicAWSCredentials(ACCESSKEY,
        SECRETKEY);
    dynamoDB = new AmazonDynamoDBClient(creds);
    dynamoDB.setEndpoint("http://dynamodb.us-east-1.amazonaws.com");
  }
 
}
VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

Categories: Salesforce

Leave a comment

Leave a comment

Feed

http://blog.jeffdouglas.com / Amazon DynamoDB Demo