Using Capistrano with Dynamic Virtual Environments

The Problem

At Metal Toad we use Capistrano to facilitate deploying projects. It allows us to support different environments, pulling and pushing databases and files, for all sorts of products.

In a virtual environment, like AWS,  servers are no longer static. This means the list of servers that Capistrano requires, have a chance of being out-of-date. In a blog last year, I found a way to address this problem, but that solution had several shortcomings:

  • It required us to commit an AWS Security Key to the repo or for every developer to have an IAM user for the AWS Account.
  • It didn’t allow us to use AutoScaling groups because AutoScaling groups require deploys every time a new server is provisioned.

To overcome the above issues, I tried to use several different tools to weigh the pro’s and con’s and find the best solution.

First, I used Capistrano by creating a ‘local’ environment to run on boot. But compiling the SASS on boot caused too much of a delay in getting a new server in rotation.

Next, I tried making the Capistrano deploy create a tar file of the repository and compiled SASS. Capistrano would then upload the tar file to S3 to be downloaded on boot. This solution worked but resulted in lots of moving parts that were difficult to monitor.

Lastly, I tried AWS CodeDeploy.  CodeDeploy provides an easy way to deploy updated code to instances in AutoScaling groups and other EC2 Instances. CodeDeploy’s one drawback, it doesn’t allow for developers to pull files and databases.

In the end, I found the best way to work around the shortcomings of Capistrano and CodeDeploy is to use them together. I programmed Capistrano to manage our CodeDeploy. This solution enables our developers to use one tool they are already familiar with, while seamlessly scaling up and down the AWS environment.

 

How To

Amazon has great documentation for setting up CodeDeploy and AutoScaling. So I will avoid repetition and post a link to their setup instructions.  I will mention an S3 Bucket and a CodeDeploy Application setup is required.

 

Getting Capistrano to manage CodeDeploy is straightforward. First, add some environment variables by ensuring the below is in your environment.rb file:

set :deploy_to, "/var/www/sites/master/#{application}-#{stage}"
set :code_depoy_app, "Code_deploy_app"
set :code_deploy_config, "CodeDeployDefault.AllAtOnce"
set :code_deploy_group, "Prod"
set :bucket_name, "bucket_name"
 
role :web, "webserver" 

The “webserver” should be a single server with an AWS role that allows the server to run code deploy.

Replace the other variable values with ones from your environment.

Second, you need to add a new task to the deploy namespace. This task will be triggered in the :symlinks_files task.

 desc "Trigger AWS Code Deploy"
  task :code_deploy do
    set :start_date, capture('date +%s')
    set :end_date, capture('date +%s -d "5 minutes"')
    run "cp  #{deploy_to}/#{shared_dir}/cached-copy/appspec.yml #{deploy_to}/appspec.yml"
    run "cd #{deploy_to} && tar -czf ~/#{application}.tar.gz --exclude=./drupal/sites/default/private --exclude=./db  --exclude=./trevor  --exclude=./drupal/sites/default/files --exclude=.git  ./;"
    run "aws s3 cp ~/#{application}.tar.gz s3://#{bucket_name}/#{application}.tar.gz"
    run "aws deploy create-deployment --application-name #{code_depoy_app} --deployment-config-name #{code_deploy_config} --deployment-group-name #{code_deploy_group} --description test --s3-location bucket=#{bucket_name},key=#{application}.tar.gz,bundleType=tgz --region us-west-2"
    run "id=`aws deploy list-deployments --create-time-range start=#{start_date},end=#{end_date} --region us-west-2 | grep \'\"d-\' | awk -F\'\"\' \'{print$2 }\'` ; while true; do foo=`aws deploy get-deployment --deployment-id=$id --region us-west-2`; if echo $foo | grep --quiet \'\"status\": \"Failed\"\'; then  echo \"failed\"; break; elif echo $foo | grep --quiet \'\"status\": \"Succeeded\"\'; then  echo \"succeed\"; break;  fi; echo $foo | grep \'\"status\": \"\'; sleep 10; done; if echo $foo | grep --quiet \'\"status\": \"Failed\"\'; then  false; elif echo $foo | grep --quiet \'\"status\": \"Succeeded\"\'; then true; fi; "
  end

The above code performs several key functions:

  • Moving the appspec.yml file that CodeDeploy uses, to the correct path in the tar file.
  • Creates a tar file that will be deployed by CodeDeploy and places it in the S3 bucket.
  • It triggers the CodeDeploy.
  • It checks the status of the CodeDeploy and waits for a successes or a fail. If it fails, it will give an error and notify the user. If it succeeds, it proceeds with the deploy.

In addition to setting up CodeDeploy in AWS, there are some configuration files that CodeDeploy require to be present when deployed:

Create appsec.yaml file in the root of our git repository.

version: 0.0
os: linux
hooks:
  BeforeInstall:
    - location: shared/cached-copy/code_deploy_scripts/create_file_structure.sh
      runas: root
  AfterInstall:
    - location: shared/cached-copy/code_deploy_scripts/Move_Application.sh
    - location: shared/cached-copy/code_deploy_scripts/update_sym_links.sh
      runas: root
      timeout: 180
  ApplicationStart:
    - location: shared/cached-copy/code_deploy_scripts/restart_apache.sh
      runas: root
      timeout: 3600
  ValidateService:
    - location: shared/cached-copy/code_deploy_scripts/check_site.sh
      timeout: 3600
      runas: deploy

In the repository root, create a folder “code_deploy_scripts”, this folder will be updated when the code is deployed.

create_file_structure.sh

deploy_dir='/var/www/sites/virtual'
 
if [ ! -d "/var/www/sites/virtual" ]; then
  mkdir -p "/var/www/sites/virtual"
fi
 
if [ ! -d "$deploy_dir/$APPLICATION_NAME" ]; then
  mkdir -p "$deploy_dir/$APPLICATION_NAME"
fi

Move_Application.sh

#!/bin/bash
 
deploy_dir='/var/www/sites/virtual'
 
if [ ! -d "$deploy_dir/$APPLICATION_NAME/releases/$DEPLOYMENT_ID" ]; then
  mkdir -p "$deploy_dir/$APPLICATION_NAME/releases/$DEPLOYMENT_ID"
fi
 
rsync -a --delete --links  "/opt/codedeploy-agent/deployment-root/3b4185dd-80b9-49d8-828f-e9796d8309e5/$DEPLOYMENT_ID/deployment-archive/"* "$deploy_dir/$APPLICATION_NAME"

update_sym_links.sh

deploy_dir='/var/www/sites/virtual'
 
#remove old links
rm -f "$deploy_dir/$APPLICATION_NAME/current"
#new links
releaseid=`ls -1rt $deploy_dir/$APPLICATION_NAME/releases/ | tail -1`
 
ln -s "$deploy_dir/$APPLICATION_NAME/releases/$releaseid" "$deploy_dir/$APPLICATION_NAME/current"
#remove old links before recreating them.
rm -f "$deploy_dir/$APPLICATION_NAME/current/drupal/sites/default/files"
ln -s "/mnt/efs/goldenglobes/files" "$deploy_dir/$APPLICATION_NAME/current/drupal/sites/default/files"
rm -f "$deploy_dir/$APPLICATION_NAME/current/drupal/sites/default/local_settings.php"
ln -s "$deploy_dir/$APPLICATION_NAME/shared/default/local_settings.php" "$deploy_dir/$APPLICATION_NAME/current/drupal/sites/default/local_settings.php"

check_site.sh simply enter a command to verify the site is running.

restart_apache.sh just restarts the apache service.

 

Both tools together are performing well and I have not heard any complaints from the developers. In addition, numerous load tests utilizing our AutoScaling group have worked without incident, while ensuring the latest code base is in use. Utilizing these two great tools together have helped us here at Metal Toad and hopefully can now be of use to your team on your next project.

 

Add new comment

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • You can enable syntax highlighting of source code with the following tags: <code>, <blockcode>, <cpp>, <java>, <php>. The supported tag styles are: <foo>, [foo].
  • Web page addresses and email addresses turn into links automatically.
  • Lines and paragraphs break automatically.

About the Author

Nathan Wilkerson, Cloud Operations Manager

Nathan started building computers, programming and networking with a home IPX network at age 13. Since then he has had a love of all things computer; working in programming, system administration, devops, and Cloud Computing. Over the years he's enriched his knowledge of computers with hands on experience and earning his AWS Certified Solutions Architect – Professional.

Recently, Nathan has transitioned to a Cloud Operations Manager role. He helps clients and internal teams interface with the Cloud Team using the best practices of Kanban to ensure a speedy response and resolution to tickets.

Ready for transformation?