Fetching Tweets with Node.js and Heroku

∞ published on March 9, 2013 at 12:24AM UTC with comments and reactions

When I added my recent tweets to this website, I knew Twitter’s 1.0 API was deprecated and I would need to find a solution by mid-March that didn’t require embedding my Twitter OAuth credentials in client-side code. Node.js and Heroku to the rescue…

I am warning you now: this is a detailed, step-by-step tutorial. If you aren’t at least slightly interested in Node.js or Heroku, I’m doing you a favor and asking you to leave. For the rest of you, I hope this can help you in overcoming a few hurdles I ran into with running my first Node.js app on Heroku.

The Problem

This website is static, which means that any dynamic content needs to be pulled from a web service via AJAX. This also means that any authenticated web service would require me to embed my credentials in JavaScript, allowing anyone to steal them. Fortunately, both Last.fm and Twitter have authentication-free APIs… well… Twitter did. Twitter is shutting down their 1.0 API and forcing everyone to use their new 1.1 API that requires OAuth.

The Solution

I love JavaScript and Node.js. Heroku can run Node.js applications. Heroku has a free tier. With these in mind, I decided to write a basic Node.js application that just queries Twitter’s 1.1 API for recent tweets and passes them through as a JSONP response for my website to consume.

Setup

Before jumping in and writing code, there are a few things you need to do first:

Signup for a free Heroku account.
Download and install the Heroku Toolbelt. They have pre-built packages for Mac OS X, Windows, and Debian/Ubuntu. This will install git if you don’t already have it.

Login to Heroku from your development environment:

$ heroku login
Enter your Heroku credentials.
Email: jordanryanmoore@example.com
Password: 
Could not find an existing public key.
Would you like to generate one? [Yn] Y
Generating new SSH public key.
Uploading ssh public key /Users/jordanryanmoore/.ssh/id_rsa.pub

Create your application. You need to create a git repository. I wanted to host my source on GitHub, so I cloned my repository to a local directory. Alternatively, you could just create a new local repository.

$ git clone git@github.com:jordanryanmoore/api.jordanryanmoore.com.git

Next, you’ll need to create your Heroku app from within the project directory:

$ cd api.jordanryanmoore.com
$ heroku create
Creating wacky-name-123... done, stack is cedar
http://wacky-name-123.herokuapp.com/ | git@heroku.com:wacky-name-123.git
Git remote heroku added

Heroku generates random names for new apps. Personally, I would recommend renaming your app to something meaningful now instead of later.

$ heroku apps:rename api-jordanryanmoore-com
Renaming wacky-name-123 to api-jordanryanmoore-com... done
http://api-jordanryanmoore-com.herokuapp.com/ | git@heroku.com:api-jordanryanmoore-com.git
Git remote heroku updated

On to code…

Hello, World

I wanted to start the minimum amount of code to get something running. You need a package.json file to install your dependencies and declare which version of the Node.js engine you’d like to use.

{
  "name": "api.jordanryanmoore.com",
  "engines": {
    "node": ">= 0.6.0",
    "npm": ">= 1.1.0"
  },
  "dependencies": {
    "express": "3.x"
  }
}

I’m using Express for my app server, and as desired, the code footprint is tiny — only 6 lines of code in server.js:

var express = require('express');

var app = express();

app.get('/', function(request, response) {
  response.send('Hello, world.');
});

app.listen(process.env.PORT);

Your app will be accessible on port 80 in production, but Heroku assigns a unique port for your app that you need to specify using the PORT environment variable. To choose which port to run the app on locally, you can specify the port in an .env configuration file:

PORT=8080

The last you need to create is Procfile. This file informs Heroku to create a web worker by executing server.js with node.

web: node server

Try running your app locally:

$ foreman start

You should now be able to access your app by loading http://localhost:8080/. If all is well, you just need to commit and push to the remote heroku/master branch.

$ git add package.json server.js Procfile
$ git commit -m "Hello, world."
[master 038d45c] Hello, world.
 3 files changed, 20 insertions(+)
 create mode 100644 Procfile
 create mode 100644 package.json
 create mode 100644 server.js
$ git push heroku master
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 546 bytes, done.
Total 5 (delta 0), reused 0 (delta 0)

-----> Node.js app detected
-----> Resolving engine versions
       Using Node.js version: 0.8.21
       Using npm version: 1.2.12
-----> Fetching Node.js binaries
-----> Vendoring node into slug
-----> Installing dependencies with npm
       npm WARN package.json api-jordanryanmoore-com@ No README.md file found!
       npm http GET https://registry.npmjs.org/express
       npm http 200 https://registry.npmjs.org/express
       npm http GET https://registry.npmjs.org/express/-/express-3.1.0.tgz
       npm http 200 https://registry.npmjs.org/express/-/express-3.1.0.tgz
       npm http GET https://registry.npmjs.org/connect/2.7.2
       npm http GET https://registry.npmjs.org/commander/0.6.1
       npm http GET https://registry.npmjs.org/range-parser/0.0.4
       npm http GET https://registry.npmjs.org/mkdirp/0.3.3
       npm http GET https://registry.npmjs.org/cookie/0.0.5
       npm http GET https://registry.npmjs.org/buffer-crc32/0.1.1
       npm http GET https://registry.npmjs.org/fresh/0.1.0
       npm http GET https://registry.npmjs.org/methods/0.0.1
       npm http GET https://registry.npmjs.org/send/0.1.0
       npm http GET https://registry.npmjs.org/cookie-signature/0.0.1
       npm http GET https://registry.npmjs.org/debug
       npm http 200 https://registry.npmjs.org/mkdirp/0.3.3
       npm http 200 https://registry.npmjs.org/range-parser/0.0.4
       npm http GET https://registry.npmjs.org/mkdirp/-/mkdirp-0.3.3.tgz
       npm http GET https://registry.npmjs.org/range-parser/-/range-parser-0.0.4.tgz
       npm http 200 https://registry.npmjs.org/cookie/0.0.5
       npm http 200 https://registry.npmjs.org/commander/0.6.1
       npm http 200 https://registry.npmjs.org/fresh/0.1.0
       npm http 200 https://registry.npmjs.org/connect/2.7.2
       npm http GET https://registry.npmjs.org/cookie/-/cookie-0.0.5.tgz
       npm http GET https://registry.npmjs.org/commander/-/commander-0.6.1.tgz
       npm http GET https://registry.npmjs.org/fresh/-/fresh-0.1.0.tgz
       npm http GET https://registry.npmjs.org/connect/-/connect-2.7.2.tgz
       npm http 200 https://registry.npmjs.org/methods/0.0.1
       npm http 200 https://registry.npmjs.org/buffer-crc32/0.1.1
       npm http GET https://registry.npmjs.org/methods/-/methods-0.0.1.tgz
       npm http GET https://registry.npmjs.org/buffer-crc32/-/buffer-crc32-0.1.1.tgz
       npm http 200 https://registry.npmjs.org/cookie-signature/0.0.1
       npm http GET https://registry.npmjs.org/cookie-signature/-/cookie-signature-0.0.1.tgz
       npm http 200 https://registry.npmjs.org/send/0.1.0
       npm http GET https://registry.npmjs.org/send/-/send-0.1.0.tgz
       npm http 200 https://registry.npmjs.org/range-parser/-/range-parser-0.0.4.tgz
       npm http 200 https://registry.npmjs.org/mkdirp/-/mkdirp-0.3.3.tgz
       npm http 200 https://registry.npmjs.org/debug
       npm http 200 https://registry.npmjs.org/cookie/-/cookie-0.0.5.tgz
       npm http GET https://registry.npmjs.org/debug/-/debug-0.7.2.tgz
       npm http 200 https://registry.npmjs.org/commander/-/commander-0.6.1.tgz
       npm http 200 https://registry.npmjs.org/fresh/-/fresh-0.1.0.tgz
       npm http 200 https://registry.npmjs.org/buffer-crc32/-/buffer-crc32-0.1.1.tgz
       npm http 200 https://registry.npmjs.org/cookie-signature/-/cookie-signature-0.0.1.tgz
       npm http 200 https://registry.npmjs.org/methods/-/methods-0.0.1.tgz
       npm http 200 https://registry.npmjs.org/connect/-/connect-2.7.2.tgz
       npm http 200 https://registry.npmjs.org/send/-/send-0.1.0.tgz
       npm WARN package.json methods@0.0.1 No README.md file found!
       npm http 200 https://registry.npmjs.org/debug/-/debug-0.7.2.tgz
       npm http GET https://registry.npmjs.org/mime/1.2.6
       npm http GET https://registry.npmjs.org/qs/0.5.1
       npm http GET https://registry.npmjs.org/formidable/1.0.11
       npm http GET https://registry.npmjs.org/bytes/0.1.0
       npm http GET https://registry.npmjs.org/pause/0.0.1
       npm http 200 https://registry.npmjs.org/mime/1.2.6
       npm http GET https://registry.npmjs.org/mime/-/mime-1.2.6.tgz
       npm http 200 https://registry.npmjs.org/pause/0.0.1
       npm http GET https://registry.npmjs.org/pause/-/pause-0.0.1.tgz
       npm http 200 https://registry.npmjs.org/formidable/1.0.11
       npm http GET https://registry.npmjs.org/formidable/-/formidable-1.0.11.tgz
       npm http 200 https://registry.npmjs.org/qs/0.5.1
       npm http GET https://registry.npmjs.org/qs/-/qs-0.5.1.tgz
       npm http 200 https://registry.npmjs.org/bytes/0.1.0
       npm http GET https://registry.npmjs.org/bytes/-/bytes-0.1.0.tgz
       npm http 200 https://registry.npmjs.org/mime/-/mime-1.2.6.tgz
       npm http 200 https://registry.npmjs.org/pause/-/pause-0.0.1.tgz
       npm http 200 https://registry.npmjs.org/formidable/-/formidable-1.0.11.tgz
       npm http 200 https://registry.npmjs.org/qs/-/qs-0.5.1.tgz
       npm http 200 https://registry.npmjs.org/bytes/-/bytes-0.1.0.tgz
       express@3.1.0 node_modules/express
       ├── fresh@0.1.0
       ├── methods@0.0.1
       ├── range-parser@0.0.4
       ├── cookie@0.0.5
       ├── cookie-signature@0.0.1
       ├── buffer-crc32@0.1.1
       ├── debug@0.7.2
       ├── commander@0.6.1
       ├── mkdirp@0.3.3
       ├── send@0.1.0 (mime@1.2.6)
       └── connect@2.7.2 (pause@0.0.1, bytes@0.1.0, formidable@1.0.11, qs@0.5.1)
       npm WARN package.json api-jordanryanmoore-com@ No README.md file found!
       express@3.1.0 /tmp/build_3266udmwu5t95/node_modules/express
       connect@2.7.2 /tmp/build_3266udmwu5t95/node_modules/express/node_modules/connect
       qs@0.5.1 /tmp/build_3266udmwu5t95/node_modules/express/node_modules/connect/node_modules/qs
       formidable@1.0.11 /tmp/build_3266udmwu5t95/node_modules/express/node_modules/connect/node_modules/formidable
       cookie-signature@0.0.1 /tmp/build_3266udmwu5t95/node_modules/express/node_modules/cookie-signature
       buffer-crc32@0.1.1 /tmp/build_3266udmwu5t95/node_modules/express/node_modules/buffer-crc32
       cookie@0.0.5 /tmp/build_3266udmwu5t95/node_modules/express/node_modules/cookie
       bytes@0.1.0 /tmp/build_3266udmwu5t95/node_modules/express/node_modules/connect/node_modules/bytes
       send@0.1.0 /tmp/build_3266udmwu5t95/node_modules/express/node_modules/send
       debug@0.7.2 /tmp/build_3266udmwu5t95/node_modules/express/node_modules/debug
       mime@1.2.6 /tmp/build_3266udmwu5t95/node_modules/express/node_modules/send/node_modules/mime
       fresh@0.1.0 /tmp/build_3266udmwu5t95/node_modules/express/node_modules/fresh
       range-parser@0.0.4 /tmp/build_3266udmwu5t95/node_modules/express/node_modules/range-parser
       pause@0.0.1 /tmp/build_3266udmwu5t95/node_modules/express/node_modules/connect/node_modules/pause
       commander@0.6.1 /tmp/build_3266udmwu5t95/node_modules/express/node_modules/commander
       mkdirp@0.3.3 /tmp/build_3266udmwu5t95/node_modules/express/node_modules/mkdirp
       methods@0.0.1 /tmp/build_3266udmwu5t95/node_modules/express/node_modules/methods
       Dependencies installed
-----> Building runtime environment
-----> Discovering process types
       Procfile declares types -> web

-----> Compiled slug size: 4.4MB
-----> Launching... done, v2
       http://api-jordanryanmoore-com.herokuapp.com deployed to Heroku

To git@heroku.com:api-jordanryanmoore-com.git
 * [new branch]      master -> master

You can try it out by going to the URL in your web browser or running:

$ heroku open
Opening api-jordanryanmoore-com... done

Easy, right? Now for a little more complexity…

Fetching Tweets

I need to provide a JSONP endpoint to fetch my tweets. I added a new depency on node-oauth in package.json (not shown) and required two more modules in server.js to make this easier for myself:

var
  url = require('url'),
  express = require('express'),
  oauth = require('oauth');

Next, I added a new Express handler to server.js:

app.get('/tweets.json', function(request, response) {
  var query = {
    'trim_user': '1'
  };

  var count = parseInt(request.param('count'));

  if (!isNaN(count)) {
    query.count = count;
  }

  var twitter = new oauth.OAuth(
    'https://api.twitter.com/oauth/request_token',
    'https://api.twitter.com/oauth/access_token',
    process.env.TWITTER_CONSUMER_KEY,
    process.env.TWITTER_CONSUMER_SECRET,
    '1.0A',
    null,
    'HMAC-SHA1'
  );

  twitter.get(
    url.format({
      protocol: 'https:',
      hostname: 'api.twitter.com',
      pathname: '/1.1/statuses/user_timeline.json',
      query: query
    }),
    process.env.TWITTER_ACCESS_TOKEN,
    process.env.TWITTER_ACCESS_TOKEN_SECRET,
    function(err, data) {
      if (err) {
        response.jsonp(err);
      } else {
        var tweets = [];

        JSON.parse(data).forEach(function(tweet) {
          tweets.push({
            'id_str': tweet.id_str,
            'created_at': tweet.created_at,
            'text': tweet.text
          });
        });

        response.jsonp({
          'statusCode': 200,
          'data': tweets
        });
      }
    }
  );
});

Most of it should be self-explanatory, with the potential exception of the process.env.TWITTER_* variables. You can find yours at https://dev.twitter.com/apps and can define them as environment variables by running:

$ heroku config:set TWITTER_CONSUMER_KEY=your-twitter-consumer-key
Setting config vars and restarting api-jordanryanmoore-com... done, v4
TWITTER_CONSUMER_KEY: your-twitter-consumer-key

If you want to have local overrides (like we do with PORT), you can specify them in the .env file:

PORT=8080
TWITTER_CONSUMER_KEY=your-twitter-consumer-key
TWITTER_CONSUMER_SECRET=your-twitter-consumer-secret
TWITTER_ACCESS_TOKEN=your-twitter-access-token
TWITTER_ACCESS_TOKEN_SECRET=your-twitter-access-token-secret

All you need to do now is commit and push again. Ta da!