Warning: Creating default object from empty value in /homepages/39/d161420129/htdocs/p373.net/wp/wp-content/themes/p373b/admin/functions.php on line 183
Rails deployments from 4 minutes to 40 seconds

Rails deployments from 4 minutes to 40 seconds

Currently my deployment(which is pretty close to a default setup) takes about 4 minutes.  Currently we take the server offline into a “maintenance” mode for the entire deployment process.  That’s 4 minutes of downtime.  No bueno.
Here’s what we’re going to do:
  1. 1. Move the maintenance mode (think of it like a mutex) to wrap only the most critical parts of the deployment
  2. 2. Make asset compilation happen offline
  3. 3. Investigate unnecessary rake tasks and scripts that load full Rails environment
Maintenance Mode

It would be great to have zero downtime deployments, but considering I wear the hats of lead dev, devops, cto, and probably a few others, I need quick solutions that have serious bang for buck.  So, spending the time to develop a scalable zero downtime solution for a startup that is just starting to grow traffic is not feasible.  Instead, a downtime window that is under a minute is acceptable to us.  So the first thing I did was evaluate what are the critical tasks that need to be ACID compliant.  Real quick I noticed that really the only tasks I need to have an in atomic operation are database migrations and the change in symlink that moves the app to the new version.  The deployed app is idempotent to all other changes.

Offline Asset Compilation

While the asset pipeline is a glorious thing from a development perspective, its also the bain of a Rails dev’s existence when it comes to deployment.  Compiling those assets takes forever.  A number of people have published a few ways around this.  There are even gems to help out like Turbo Sprockets which will only compile assets that have changed.

However, the fact remains that if you need to compile any of your stylesheets and js, it will take time.  This is problematic because if you take your app offline during the compilation process, it will leave your site unavailable for longer than it has to.  Even if you don’t take your site offline during the compilation process, than its hosing your box’s cpu cycles while users are using it.
Asset compilation should happen offline and the assets should be pushed up outside your deployment’s downtime window.  You should be using assets with a digest that are loaded via a manifest.yml file which references said digests, so compiling and pushing them up shouldn’t affect anything.
Well, as it turns out, someone has already solved this problem(mostly).
Robert Rostamizadeh has already handled most of the heavy lifting.  However, his solution is incomplete if you fit in either or both of the following cases:
  1. 1. You use Capistrano to put up a maintenance page for all or part of the deployment
  2. 2. You use a gem like AssetSync to handle deploying assets to the cloud
In my case, I fit into both.  To summarize, my goal here is to compile my assets locally, push them to my server, then use AssetSync to push to the cloud, all without taking the site offline.  The site should only be taken offline for critical sections of the deployment, like migrations and changing up the symlinks.
So here’s the Capistrano task I came up with:
# = Capistrano Assets recipe
# Provides a couple of tasks for precompiling assets locally
# Assumes usage of AssetSync to deploy to CloudProvider or CDN 
# Inspired by http://www.rostamizadeh.net/blog/2012/04/14/precompiling-assets-locally-for-capistrano-deployment

namespace :deploy do

  namespace :assets do

    desc <<-DESC
      Precompiles assets locally.  If you are using AssetSync, it should send them
      to the cloud automatically.
    task :precompile, roles: :web do
      # i keep this in so that it triggers login at the beginning of the process
      from = source.next_revision(current_revision)
      run_locally("rake assets:clean && rake assets:precompile")
      run_locally "cd public && tar -jcf assets.tar.bz2 assets"
      top.upload "public/assets.tar.bz2", "#{shared_path}", :via => :scp
      run "cd #{shared_path} && tar -jxf assets.tar.bz2 && rm assets.tar.bz2"
      run_locally "rm public/assets.tar.bz2"
      run_locally("rake assets:clean")

    desc <<-DESC
      Sync assets to the cloud via Asset Sync.  Must be run remotely(so it pulls remote aws credentials)
      and after code is deployed so it gets proper asset manifest to sync
    task :sync, roles: :web do
      run("cd #{latest_release} && /usr/bin/env rake assets:sync RAILS_ENV=production")

    desc <<-DESC
      [internal] Updates the symlink for assets
    task :symlink, roles: :web do
      run ("rm -rf #{latest_release}/public/assets &&
            mkdir -p #{latest_release}/public &&
            mkdir -p #{shared_path}/assets &&
            ln -s #{shared_path}/assets #{latest_release}/public/assets")


You can see I follow Robert’s solution mostly, but with a couple of notable changes:
  1. 1. I removed his check to see if assets have changed.  His solution seemed a bit problematic as it relies on the git log to suss out what’s changed.  It sounds like an acceptable situation on the surface but I ran into some edge case issues which I won’t go into here.  TurboSprockets will take care of this for me anyway, and it uses the manifest.yml to compare digests.  That sounds a bit more robust to me.
  2. 2. Add an extra deploy:assets:sync task to handle syncing to the cloud with AssetSync gem
  3. 3. I rearranged the Capistrano callback stack see below
Investigate unnecessary rake tasks and scripts that load full Rails environment
I noticed that starting and stopping our background process(DelayedJob) was taking a while.  Its not such a big deal anymore as I’ve moved it outside the maintenance window, however it still bothers me.  Our app has several initializers that load in some optimizations and take some time to do so.  Do I really need the whole Rails environment to do so?  I’m just trying to restart a process.
To summarize, the best bet to reduce your deployment time, is to compile your assets offline, and suck in that maintenance window to only the most critical tasks.  Here’s our new Capistrano stack(in english)
  1. 1. Before we update code(“deploy:update_code”), precompile and push to server and cloud
  2. 2. stop our background processes
  3. 3. enable maintenance mode
  4. 4. symlink
  5. 5. migrate
  6. 6. restart web server
  7. 7. disable maintenance mode
  8. 8. start up background processes

40 Seconds is still pretty long for users to have to wait to use your site, but this time is mostly Passenger restarting itself and I’ll leave that to another article, probably one where I talk about ditching Passenger entirely…

    This entry was posted in Coding, Ruby/Rails, Technology and tagged , , , . Bookmark the permalink. Both comments and trackbacks are currently closed.