Artificial Intelligence
What is AWS EC2?
AWS EC2 is short for Amazon Web Services Elastic Compute Cloud.
Capistrano Deploy to dynamic AWS EC2
At Metal Toad we use Capistrano to deploy our projects to their respective servers.
Nathan Wilkerson, VP of Engineering
Sep 9, 2014
Filed under:
At Metal Toad we use Capistrano to deploy our projects to their respective servers. Normally this is done with a configuration file for each stage (Dev, Staging, QA, and Production) that contains a list of servers. Below is an example of one of those files.
# Set the deployment directory on the target hosts.
set :deploy_to, "/var/www/sites/virtual/<client>-stage.metaltoad-sites.com"
# The hostnames to deploy to.
role :web, "stg01-<client>.ec2.metaltoad.net", “stg02-<client>.ec2.metaltoad.net”
# Specify one of the web servers to use for database backups or updates.
# This server should also be running Drupal.
role :db, "stg01-<client>.ec2.metaltoad.net", :primary => true
# The path to drush
set :drush, "cd #{current_path}/#{app_root} ; /usr/local/bin/drush"
# The username on the target system, if different from your local username
ssh_options[:user] = 'deploy'
This process has worked great, but now we are creating Custom Clouds in AWS, we need a more dynamic solution.
The biggest feature of AWS is the ability to pay for the servers you need, and only when you need them. That means we need to constantly change the lists of servers in our Capistrano configuration. These changes lead to deploys failing when a server that was there isn’t or when a server that should have gotten a deploy, didn’t.
Below is my solution to dynamically populate the server lists:
# Set the deployment directory on the target hosts.
set :deploy_to, "/var/www/sites/virtual/#{application}"
# The hostnames to deploy to.
set :access_key, ""
set :secret_key, ""
set :ec2_servers, ""
set :ec2_gateway, ""
set :ec2_db_gateway, ""
set :client, ""
run_locally "echo `ec2-describe-instances -O #{access_key} -W #{secret_key} --filter \"instance-state-code=16\"| grep #{client} | grep Name | grep web | awk -F' ' '{ print $5\".ec2.metaltoad.net \" }'` | sed ':a;N;$!ba;s/\n/, /g' > /tmp/#{client}-servers"
File.open("/tmp/#{client}-servers", 'r') do |f1|
while line = f1.gets
puts line
ec2_servers=line.split(" ")
end
puts ec2_servers
role(:web) { ec2_servers }
end
run_locally "echo `ec2-describe-instances -O #{access_key} -W #{secret_key} --filter \"instance-state-code=16\" --filter \"tag-key=Gateway\" | grep #{client} | grep Name | awk -F' ' '{ print $5\".ec2.metaltoad.net \" }'` | sed ':a;N;$!ba;s/\n/, /g' > /tmp/#{client}-gateway"
File.open("/tmp/#{client}-gateway", 'r') do |f1|
while line = f1.gets
puts line
ec2_gateway=line.split(" ")
end
set :gateway, ec2_gateway
end
# Specify one of the web servers to use for database backups or updates.
# This server should also be running Drupal.
run_locally "echo `ec2-describe-instances -O #{access_key} -W #{secret_key} --filter \"instance-state-code=16\" --filter \"tag-key=DB-Gateway\"| grep #{client} | grep Name | awk -F' ' '{ print $5\".ec2.metaltoad.net \" }'` | sed ':a;N;$!ba;s/\n/, /g' > /tmp/#{client}-database"
File.open("/tmp/#{client}-database", 'r') do |f1|
while line = f1.gets
puts line
ec2_db_gateway=line.split(" ")
end
role(:db) { ec2_db_gateway }
end
# The username on the target system, if different from your local username
ssh_options[:user] = 'deploy'
# The path to drush
set :drush, "cd #{current_path}/#{app_root} ; /usr/local/bin/drush"
namespace :deploy do
desc "Notify New Relic"
task :newrelic do
run_locally 'curl -H "<newrelic key>" -d "deployment[application_id]=4214617" -d "deployment[user]=`whoami`" https://rpm.newrelic.com/deployments.xml'
# end
#end
after "deploy",
"deploy:newrelic"
This new script uses the AWS CLI to get a list of servers for our EC2 instances. It uses a combination of host names, and tags to identify the correct servers for the specific client. It then identifies our gateway, builds a list of web servers, and a list of servers with access to the database.
I hope other people will find this as useful as I do.