Background

Moving a project from an exploration and prototyping stage to a full product development stage requires re-evaluating technology decisions. One of my client projects is undergoing this process right now - moving from a small demo product to gain insight into the problem and show to investors to a going concern with dedicated staff and engineering resources.

Previously, we’d built a ruby on rails application with a ember-cli app embedded inside it using ember-cli-rails. The application was deployed to heroku, which provided some great benefits - very easy deployment (even though ember-cli-rails requires adding a second buildpack), very easy scaling, and pretty reasonable prices.

But increased engineering resources means re-evaluating the stack. We expected that at the scale we needed, heroku would be fairly expensive. The application does a serious amount of back-end processing and requires a heft database, meaning that we were looking at $200/m for Heroku Postgres tier 1, plus more than $250/month for five 2x professional dynos, before we even implemented the web tier. And if we fail in certain aspects of our engineering effort, we’re going to need to patch it with caching - looking at between $70 and $165/m for Memcachier (which we had had very good experiences with!) was not good.

Additionally, we’d like to gain flexibility on how we process some events and handle our back end technology stack. Although I love ruby and ruby on rails, ruby doesn’t offer the best tools for the heavy duty data processing and machine learning we want to work towards. Pretty much anything we want to deploy can be deployed on heroku, either using the JVM or Python build packs, but getting the auto-scaling and batch processing we think we might need would be a lot of effort.

So, we decided to re-evaluate working with AWS Elastic Beanstalk. Elastic beanstalk is ultimately a fairly thin wrapper around a whole suite of Amazon Web Service tools - anything you can do with Elastic Beanstalk you could do without it, but perhaps not as easily. It also has a command line tool set to allow heroku-esque git deployments, though they are fundamentally not the same as pushing to heroku’s embedded git repository.

Prerequisites

You’ll need an Amazon Web Services account (you can do everything here except connect the service to a domain using Route 53 using the free tier, but the Route 53 bill should be less than $1/m). Additionally, you’ll want to install the amazon web services command line toolkit.

To install the AWS CLI, you can follow the most up to date installation aws cli installation instructions. The long and the short of it is (on unix): $ sudo pip install awscli. You do need to use sudo because of some of the binaries that the installer provides.

Then install the Elastic Beanstalk CLI - $ sudo pip install awsebcli. You’ll need to set up your credentials from AWS; run aws configure and enter an AWS access key and secret key.

Unfortunately, the old version of the elastic beanstalk CLI tool gained a lot of traction, and so the documentation that relies entirely on the old tool is still frequently first in the search results. If you end up on an AWS documentation page and the last modified date is 2010, it probably isn’t accurate anymore. I suggest using Google’s convenient “in the past year” filter on your searches to avoid going down the wrong path.

For Elastic Beanstalk, there is a distinction between applications and environments. Applications include multiple environments. Environments can either be web environments, or worker environments - much like heroku’s web / worker dyno distinction. You can have extra environments for testing, staging, etc., which mirror or differ in settings from your primary production environment appropriately.

In addition, the application/environment distinction along with Route 53 (AWS’s DNS setup tool) makes a swapping-based deployment process easy. Instead of having a static ‘staging’ environment and an alternate ‘production’ environment, you can create two ‘production’ environments - say production-1 and production-2. Using Route 53, you can route your web traffic to production-1, and use production-2 as a staging environment. When it comes time to deploy, instead of re-doing the deployment process in production-1, just change your Route 53 to make your web traffic point at production-2 (which, due to the way Route 53 works, is very, very fast). Leave production-1 running - it’ll automatically scale down to minimal resources, and if you need to rollback, instead of having to roll back a deployment, just direct your traffic back at production-1!

Of course, this setup does mean that you’ll be incurring charges for a minimum of two sets of production web servers, but if you’re a reasonably large application that could be worth it.

Setting Up Rails on Elastic Beanstalk

First, create a demo Rails app, and go ahead and add some custom route that will generate a response. The default “welcome” page is not served correctly in production, so you will need something other than that. Go ahead and create a git repository and commit the code to the git repository.

You can actually create your elastic beanstalk application and environment entirely from the command line, but I do not suggest doing so - there are a lot of parameters to set and for your first setup, browsing through them is a good idea.

In the AWS console, select elastic beanstalk and “create new application”. Frustratingly, when I did so it did not give me an opportunity to create an application and instead immediately created “My First Elastic Beanstalk Application”. You can click through, delete it, and then go back to “Create a New Application”.

After you’ve created the application, go ahead and create an environment. The application will ask if you want to choose an IAM (identity access management) role; just click through and one will be created for you.

Now choose your configuration (Ruby). If this is a production environment, you’ll want to pick “Load balancing, auto scaling” as your environment type, instead of “Single Instance”. Note that creating a “Load balancing, auto scaling” environment will incur charges from creating an Elastic Load Balancer (about $15/m). Single instance environments are appropriate for testing and the occasional specialized job server.

On the next screen, go ahead and choose “Sample Application”. The default Batch Size is fine. Click next. AWS suggests that you name environments as application-environment, for example, myproject-production.

AWS will now ask if you want to create an RDS DB instance - If you don’t have one already, go ahead and make one, though it will frustratingly default to Magnetic storage and you’ll need to change it later from the RDS console if you want SSD-backed databases.

Finally, choose an instance type. For testing out the system, t2.micro is great. All of the other defaults are fine, you can enter your email address if you want to receive updates about the environment health (great for production).

Click through, configure your database if necessary (username and password are the important parts) and finally confirm everything. Creating a new Elastic Beanstalk application takes quite a while - easily ten minutes, in my experience, especially if you are creating a database. Don’t fear, though, once it’s created deployments are quick and easy.

Once it is set up and the environment says that it is “Healthy” on the elastic beanstalk dashboard, you can go ahead and upload your application. To do so, go to your application directory, and run eb init. Then run eb deploy - it’ll take the current commit, zip it up, upload the application to S3, and then deploy it to all of your instances. Once that’s done (the first time can take some time), run eb open to view your application live. Done!

Except, if you used a newly generated Rails app, you’ll probably get an error. To view the logs, run eb logs. The error message should be something about secret_key_base. If you run your rails app with -e production, you should see the same error locally. If so, go ahead and set the SECRET_KEY_BASE environment variable by editing the environment from the AWS console. Select the environment, then “Configuration” on the left hand side, then the “Software Configuration” card (should be fourth down, after ‘notifications’) and then enter the environment variable.

That’s it. To deploy again, run eb deploy.

A Word on Instance Types

After several rounds of releasing new and improved instance types at ever decreasing prices, making sense of the AWS instance options is not straightforward. Here’s my current understanding:

All of the ’t' instances (t1.micro, t2.micro, etc.) are bistable - they can’t maintain full power for extended periods of time, but can run at full power for a little bit. This should make them good for hobby projects and for test environments where overall speed isn’t necessary.

The number in the instance name is its generation - all else equal, always prefer later generations, as they are cheaper and more powerful. The later generations don’t always have all of the options, though; e.g. there is no m3.small instance type. This means that m3.medium is the smallest instance type that should be used in production for business critical web applications. The m3.medium instance costs about $35/month to run, with about a 30% discount for making a one year commitment, and has more RAM than a Heroku dyno, so it should be a competitive option.

You can only set one instance type for each Elastic Beanstalk environment, because instances are designed to be totally expendable - they’ll crash or be pulled down pretty regularly, and it’s up to the scaling manager to take care of bringing up new ones. This means that the size of your instances impacts the granularity of your scaling - if you’re running $0.25/hr instances and decide to scale, it’s going to cost an extra $0.25/hr.

However, creating environments is very cheap - there’s really no overhead to do so. So go ahead and create test, staging, etc environments to your heart’s content, and use different instance types for different environments. In theory, you could even host the same application on two different subdomains (maybe app1.example.com and app2.example.com) if you really needed heterogeneous instance types, and perform some simple in-app load balancing to spread traffic across them.

Connecting to a Database and Connecting Your Service to a Domain

I’ll cover connecting to a database and connecting the web service to a domain in a future post.