Your Own Little node.js PaaS - Part 1

PaaS (platform-as-a-service) services are blooming these days (Heroku, Nodejitsu, nodester, cloudcontrol and many more). All offer some awesome features. I set out to try and set up a very simple node.js PaaS for myself within an Amazon EC2 Ubuntu instance. It obviously lacks many of the features in commercial PaaS services but it’s a lot of fun and also gives you complete control over the runtime environment. For me, I needed some non-node Linux services running alongside my app (e.g. SMTP, squid, sshd).

This post will walk through:

1. Creating a simple “hello world” repository and pushing it into github.
2. Launching an EC2 instance and deploying this repository on it.
3. Setting up a github post-receive hook to automatically redeploy the #master branch whenever changes are committed.

In a future post I will use connect-girror to host multiple apps on a single EC2 instance.

Some background:

• Amazon EC2 and Ubuntu
• git and github
• node.js

Creating the root repository

The first step is to create a git repository that will serve as the root of your PaaS:

Create a new github repository.

Clone this repo to your local box:

Create an initial app.js and package.json:

This is the canonical node.js “Hello, world”. Notice that the port we are listening to is defined in process.env.port. This will be set later in the Ubuntu upstart script that keeps this app alive on the production server. Port 5000 is used as fallback for the development environment.

Now commit the changes and push them into github:

Launch an Amazon EC2 instance

The next step is to launch an Amazon EC2 instance. I’m using Ubuntu here but you could use any other *nix machine. Details will vary, especially around setup and the init scripts (upstart in Ubuntu).

A list of available stock Ubuntu instances is available here. I have been successfully using a 64-bit ebs-backed instance. Note that an non-ebs root store might not have enough space for the stuff we need on this box.

When launching your instance, paste the gist below in the userdata field of the instance, altering the REPO variable to point to your newly created github repository.

In case this is a private repository, you can use the https://USER:PASSWORD@github.com/account/repo scheme, or install a github private key on your server and use the github ssh endpoint (git@github.com/account/repo.git).

This script basically does the following:

1. Installs nodejs, npm, git, gnu make and gcc compiler for c++. The latter two are required for compiling native modules in case you will use any.
2. Installs girror, which is a node.js module/utility I created that facilitates in syncing clean mirrors of git repositories to a local directory.
3. Creates /etc/init/paas.conf which is an Ubuntu upstart script invoked during system initialization. When start paas is called (or the system is initialized), this script will fetch any changes from the app’s git repository, call npm install (in case there’s a package.json file in the app’s working directory and then start node app.js from the app’s directory (setting port to 80).
4. Starts the paas service we just created.

In the firewall settings, open up at least port 22 (SSH) and port 80 (HTTP).

After you hit Launch, EC2 will spin up the new instance for you and will run the userdata script you specified. Note that the initial boot might take a few minutes (about 5) since we are installing a few things.

If everything goes well (see the System Log in the AWS console for debugging), at this point your app should be running and you should be able to hit it via HTTP (replace with your own Amazon instance DNS):

How cool is this man?

Tailing logs from your server

Logs are appended into /var/log/paas.log, so tailing them should be quite easy:

I like to keep an open iTerm frame with this command running at all times so I can see what’s up with my apps.

Set up a GitHub hook

The next step is to add a hook handler to our app and set up a github post-commit hook which will send an HTTP POST to your instance whenever changes are committed into the master branch.

Edit your app.js and add the following code:

The mechanism for handling post-commit triggers is simply to exit the app using process.exit(1). This will trigger a respawn by the upstart script which will call girror $REPO $WORKDIR which will fetch any changes from $REPO. This is very preliminary and not very sophisticated but actually works… One thing you would need for a more advanced workflow is only respond to changes within the #master branch (or any other branch that you wish deployed). This way, you could control exactly which changes affect your production environment.

Security notice: since the root app contains the super secret deployment endpoint, you should probably make it a private repository and not public.

Commit and push the changes to github:

Now, you will need to redeploy your app manually by invoking upstart’s restart command. You can do this remotely using ssh:

In the logs, you should see something like this:

Starting app from https://github.com/eladb/mylittlepaas
16:26:27 - TRACE - done
My little PaaS (v2) started on port 80

Did you notice v2? Yeah!!

The next step is to setup a github post receive hook.

In your repository’s administration page, under Service Hooks (!https://github.com/account/repo/admin/hooks), add a WebHook URL that points to your secret deployment endpoint. In our case it is http://ec2xxx.compute.amazonaws.com/mysupersecretdeploymentendpoint:

Click Update Settings and then Test Hook while monitoring your tailed logs. You should see deployment activity going:

Post-receive trigger. Exiting in 1 second
Starting app from https://github.com/eladb/mylittlepaas
16:53:14 - TRACE - done
My little PaaS (v2) started on port 80

Now we are getting somewhere…

If you GET /, you will see the new version deployed:

Installing npm packages

The paas.conf upstart script executes npm install before starting your app. This means that any npm packages you specify under the dependencies field of your app’s package.json file will be installed automatically before the app starts. One disadvantage of this approach is that sometimes modules use wildcard semver versioning when specifying dependencies, which can potentially cause app breakage if some module misbehaved and did not adhere to npm’s versioning guidelines. npm provides a mechanism called shrinkwrapping which allows you to fix the versions of all the modules and submodules you are using. This is the best practice for bundling app dependencies.

Another approach is to commit all dependent modules directly into your repository’s node_modules directory. This will work only if you are not using modules with native parts which must be compiled on the target architecture.

Let’s install the asciimo module and emit some ascii art.

Edit your app’s package.json and add asciimo@0.3.1 under the dependencies field:

Now, install and shrinkwrap:

Edit app.js:

Commit and push:

Code should be now be deployed automatically (and npm installed), so GET / should yield:

Recap

So, at this point we have:

1. A running ec2 instance hosting the node.js app under a github hosted repository.
2. Any changes to the master branch of this repository will be automatically fetched and the app will be restarted.
3. Logs can be tailed via ssh using tail -f /var/log/paas.log

In the next post I will extend this app to host apps from multiple repositories using connect-girror.

Any comments/feedback/ideas/contributions/flaming are very welcome!