Big Data Made Small with Heroku, DynamoDB, and Elastic Map Reduce

One million tweets per day.

An average of fifteen words per tweet.

Four (awesome) days of Dreamforce 2012…

Out of the 60 million words that scrolled across the screen on the Model Metrics Art of Code exhibit Moving the Cloud during Dreamforce 2012, which were the most frequently used? Well, “social” was #1, then “touch” and “mobile”. The word cloud above shows the rest of the top 100. But how did we calculate that? And, more importantly, how can we do so in a way that will easily scale up to working with much larger data sets?

Well, Moving the Cloud is written in Node.js, and I didn’t want to do anything that would tax the production version of the page, so the first thing I did was to create a simplified version of it by stripping out the UI/HTTP layer and adding in the Dynamo package for working with Amazon DynamoDB. DynamoDB is a highly performant, highly scaleable NoSQL database service hosted by Amazon Web Services. Amazon automatically handles scaling the storage space for you with super-fast SSD drives. Your main configuration options are to set the max number of allowed reads per second, and the max number of writes per second. Changing these values takes less than a minute, and you can set up CloudWatch alarms to let you know if you’re getting close to the limits. You pay more for higher limits, and we were seeing around 25-50 tweets per second max, so I set the write limit to 100. The read limit only really matters when you want to start reporting on the data, so I set it pretty low initially.

As you can see from the Trendy-Dynamo code in GitHub, the actual communication with DynamoDB from Node.js is pretty simple. DynamoDB stores Key/Value pairs, and has no defined schema aside from requiring a primary key. The Twitter Streaming API returns JSON documents with a lot of extra cruft, so I pulled out the relavent information and stored in in DynamoDB:

Back in the olden days of aught four, I might have set this running on an old linux box laying around my house (I still actually have a few big towers stacked in the basement, along with boxes of power supplies and old parts, but they haven’t been turned on in ages). Then my ISP would drop the connection, or the power supply would fail, and I’d be missing a bunch of data. Enter Heroku. Such an app can literally be hosted for free on the Heroku Cedar Stack with one Worker Dyno:

Okay, so that’s the initial setup — let’s move ahead a few days — #DF12 is over, and we have 60 million words to count. This is where Elastic Map Reduce (EMR) comes in. EMR is a hosted instance of Apache Hadoop, and Map-Reduce is a handy algorithm for taking huge data sets and breaking them down into smaller, manageable chunks. Think of it like this — imagine in this image that each of the three multi-colored blocks on the left side is one individual tweet…

Say the red block is the word “salesforce”, the yellow block is the word “is”, and the blue block is the word “social”. The first step of the process is to count the instances of each word in that tweet. Then, we increment the count of that word in every tweet. Simple, right? Over time, we break down 60 million words into a reduced set where each word occurs only once, but is accompanied by a number that represents the total number of occurrences. To do this with EMR, the first thing we need to do is to snapshot the data from DynamoDB into Amazon S3. To do this, I’ve used an interactive command line Hadoop tool named Apache Hive. It allows you to map external tables and to query them with SQL-like syntax.

Using Hive, I created an external table for DynamoDB:

CREATE EXTERNAL table dynamo_tweet (tweet_id string, tweet_text string)
STORED BY 'org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler'
TBLPROPERTIES ("dynamodb.table.name" = "df12tweet","dynamodb.column.mapping" = "tweet_id:Tweet ID,tweet_text:text");


And an external table for S3:

CREATE EXTERNAL TABLE s3_df12snapshot (tweet_id string, tweet_text string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
LOCATION 's3://mm-trendy-dynamo/demo_output/';

And then copy from one to the other:

INSERT OVERWRITE TABLE s3_df12snapshot
SELECT * FROM dynamo_tweet;

Snapshotting takes a little while, so go get a coffee or something… Don’t worry, I’ll wait.

…And, we’re back. Okay, so now we need to actually run the Map-Reduce job to count each word. Luckily, EMR gives us a sample application that does just that:

Select the Word Count job, walk through the rest of the wizard, and let it start processing. The amount of time it takes is basically a factor of how many EC2 instances you throw at it, and the processing power of each. When it finishes, the output of the job will be stored in S3, and you can create another external table in Hive:

CREATE EXTERNAL TABLE s3_df12mapreduce (tweet_word string, tweet_count int)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
LOCATION 's3://mm-trendy-dynamo/outputmapreduce/';

And then query it:

SELECT * FROM s3_df12mapreduce
WHERE LENGTH(tweet_word) > 4
ORDER BY tweet_count DESC
LIMIT 100;

What you do with this map/reduced data is then up to you, but if you’re interested in how I created the word cloud, I used this D3-Cloud Javascript Library

TL;DR: I made a wordcloud with some tweets.

Read More

Moving the Cloud — HTML5 and CSS3 on Node.js

Moving the Cloud is an experiment in using HTML5 and CSS3 technologies and Node.js. It was also an experiment on using Node.js on the Cedar stack in Heroku, but that didn’t quite work out as expected (more on that later). It’s also an Easter egg on the Model Metrics Homepage. Click on the animated 1s and 0s down in the bottom left corner of the screen to get to it.

And… if you want, fork the source over on GitHub. The readme.md file there gives some information on how to get it set up.

I was asked to put together a temporary (it’ll probably be around for a few months) artsy installation piece for the Art of Code section on modelmetrics.com. Going into the project, I wanted to do a social media visualization because, well, I think they’re cool. Some of my favorite examples are Twistori.com and Vizeddit — they visualize Twitter and Reddit respectively.

Moving the Cloud uses Node.js to pull tweets containing some keywords like “Cloud”, “Salesforce”, “Social”, “Mobile”, or “Model Metrics” from the Twitter Streaming API and stream them to clients using websockets. They move across the screen right to left, sort of like a cloud — GET IT? (nudge nudge wink wink)?

Twitter Streaming API

The Twitter Streaming API (well, technically APIs) allows you to open up a long-running socket connection with Twitter, and stream tweets that match a certain criteria. For this, I used the Public Stream API with the /filter endpoint so that I could receive tweets that matched my keywords. Technically, you could do this directly from the client using HTML5 websockets, but you have to authenticate with Twitter. You can generate keys for a specific app at dev.twitter.com, so you don’t have to just hard-code your Twitter username and password into the app, but still, you don’t really want to embed that in client-side code. So, in this example, I’m using Node.js as a go-between. It authenticates with Twitter, sets up a socket connection with the Streaming API, and dispatches those tweets out to any clients (browsers) that connect to it.

Node.js

If you haven’t used Node.js, you should. It’s a way to run Javascript on the server-side that takes advantage of Javascript’s inherent event-based non-blocking nature to create a server that works especially well for realtime or data-intensive apps like instant messaging apps. It’s possible to build a regular website (like a blog or something) with it using Express (which I am using here for simplicity), but I’m not sure it’s necessarily the best choice for something like that, as RoR, PHP, and Python (for instance) already do a pretty good job in that space. For a Twitter streaming API app, though, Node is great.

The Node.js server script itself is pretty simple–it uses the node-twitter library to connect to Twitter, and Socket.io to handle the setting up the websocket connections with whatever client browser happens to connect to it. It also handles fallback to XHR long polling for browsers that don’t support websockets. Aside from an array to hold active connections and some error handling, that’s about it on the server side.

HTML5/CSS3

Like I said earlier, one of my goals for this project was to demonstrate some HTML5 and CSS3 technologies. Because of that, it does not work on IE. It would probably be possible to get it sort of working in IE by falling back from HTML5/CSS3 technologies to older ones, but it wouldnt’ work as well, and sometimes you just have to leave the old crappy browsers behind. It does work perfectly fine on current versions of Chrome, Safari, Firefox, Mobile Safari (iPhone/iPad), and the Android Browser, though the transitions are kind of choppy on Firefox. Anyway, here’s some of what’s going on:

WebSockets

Websockets are awesome. If you’re familiar with client/server programming, you’re probably familiar with the concept of sockets. If you’re not, you use them all the time anyway. With a typical HTTP request, for instance, a socket connection is opened with the server, a request is made (GET, POST, etc.), a response is given and the socket is closed. In order for the server to be able to push data down to the client, a socket connection needs to stay open.

There’s some hacks like XHR Long Polling (Comet) that work on older browsers, basically by opening a dummy request socket and delaying the response until something push-worthy happens, but it’s limited by the HTTP 1.1 spec that says a browser should have no more than 2 open sockets with a server, and, well, it’s hacky. Websockets, on the other hand, let you open up a bi-directional full-duplex socket between a web browser and a server. In Moving The Cloud, tweets are sent from the Node.js server to the each client using websockets. Socket.io handily fails back to XHR long polling if websockets aren’t available.

CSS3 Transitions

CSS3 transitions allow you to do 2D and 3D transformations without using any Javascript. This is great because, for the most part, it’s faster to do things in CSS where possible. On some platforms, it’s even hardware accelerated. Webkit-based browsers even include a special mode that lets you see which elements are hardware accelerated. This is Safari’s CA_COLOR_OPAQUE=1 mode:

Show hardware acceleration in Safari CA_COLOR_OPAQUE=1

Show hardware acceleration in Chrome –show-composited-layer-borders

Firefox, best I can tell, does not have a similar mode. And CSS3 transitions are really slow in the current version of Firefox, too. So, I guess Mozilla has some work to do on that front.

In Moving the Cloud, I used the JQuery Transit plugin because it handles fallbacks to Javascript transitions when CSS3 transitions aren’t available.

CSS3 Web Fonts

Web Fonts use the @font-face rule to include font files that can be hosted on the web – they don’t have to be installed on the viewer’s computer. This has given rise to some great services like Google Web Fonts and fontsforweb.com. This is great, because the Model Metrics logo uses GothamLight, which isn’t a typical web font. In the olden-days, back when onions were worn on belts, logos that had to use a specific font would typically just be images, and the font for the rest of the site would just be a web-safe font that was close enough.

For instance, the rest of the modelmetrics.com site (except the logo) uses the Helvetica font family: “HelveticaNeue-Light”, “Helvetica Neue Light”, “Helvetica Neue”, Helvetica, Arial, “Lucida Grande”, sans-serif. My bit uses GothamLight:

@font-face{
font-family: "GothamLight";
src: url('http://fontsforweb.com/public/fonts/1151/GothamLight.eot');
src: local("Gotham-Light"), url('http://fontsforweb.com/public/fonts/1151/GothamLight.ttf') format("truetype");
}
.fontsforweb_fontid_1151 {
font-family: "GothamLight";
}

CSS3 Opacity

CSS3 adds an opacity parameter, so any element can a specified level of opacity (or what most people would call transparency). It’s pretty simple to use – just add an opacity: attribute to an HTML element, and specify a value. The footer in Moving the Cloud, for instance, has an opacity value of 0.9. Just enough so that you can see tweets floating by underneath.

HTML5 Boilerplate

When starting a new web app, it’s often a good idea to start with a reset.css to normalize your app across browsers. HTML5 Boilerplate gives you that and more. It’s is a “”professional front-end template that helps you build fast, robust, adaptable, and future-proof websites”. It’s a great starting point for an HTMl5 app that handles many of the idiosyncrasies between various browsers.

Known Issues

Moving the Cloud works quite well in most modern browsers, but there are some known issues:

Internet Explorer

Doesn’t work – right now I’m just using an [if IE] statement to show a message to the user that they should use a different browser – yes, I know this is bad form, but it’s not a critical portion of the website, and it’s meant to demonstrate new technologies not imitations of new technologies. Partially it doesn’t work because IE doesn’t support Websockets or CSS3 Transitions, but technically speaking, Socket.io should be able to fail back to XHR long polling, and JQuery Transit should be able to fail back to Javascript, so I’m not 100% sure why it isn’t working. I could spend some more time on it, but to be honest, even if I got it working, I’m assuming it will be really slow and crappy anyway.

Firefox

Speaking of which, it’s kind of slow and crappy on Firefox. I think this is just because CSS3 transitions are slow and crappy on Firefox. Mozilla really needs to step it up – it runs great on my iPhone but not Firefox on a new MacBook Pro.

Heroku

I really wanted to get this working in Heroku. Really really really did. And, technically it does work – on one dyno. I think the issue is that the Cedar stack doesn’t support websockets from Node.js, so it has to use XHR Long Polling, and even though it should work with a RedisStore as a man in the middle, it works sort of intermitently if you try to scale the app up to use more than one dyno. So, it’s running in EC2. I’ll try again when Cedar supports websockets. More information here on stackoverflow.com.

Read More