Useful ECS deploy script

ecr
aws

#1

Just passing this along since after a few iterations of the original Go project example script have found this version to be a bit more useful.

A few things to note is:

  • Takes a command line arguments so you can deploy differently between staging and production
  • Will “complete” the service name for AWS CloudFormation created services

As with anything, improvements are always welcome.

Usage:

if [[ $CIRCLE_BRANCH = staging ]]; then
  sh .circleci/ecs_deploy.sh \
    --cluster MY_CLUSTER \
    --service MY_SERVICE \
    --task MY_TASK \
    ${AWS_ACCOUNT_ID}.dkr.ecr.us-east-1.amazonaws.com/${CONTAINER_IMAGE}:${CIRCLE_SHA1}
fi

Script ecs_deploy.sh

#!/bin/bash

usage() {
    echo "Usage: $0 --cluster CLUSTER_NAME --service SERVICE_NAME --task TASK_NAME DOCKER_IMAGE"
    exit 1
}

while true ; do
    case "$1" in
        -t|--task) TASK_NAME=$2 ; shift 2 ;;
        -s|--service) SERVICE_NAME=$2 ; shift 2 ;;
        -c|--cluster) CLUSTER_NAME=$2 ; shift 2 ;;
        -h|--help) usage ;;
        --) shift ; break ;;
        *) break ;;
    esac
done

[ $# -eq 0 -o -z "$TASK_NAME" -o -z "$SERVICE_NAME" -o -z "$CLUSTER_NAME" ] && usage

DOCKER_IMAGE=$1

expr='.serviceArns[]|select(contains("/'$SERVICE_NAME'-"))|split("/")|.[1]'
SNAME=$(aws ecs list-services --output json --cluster $CLUSTER_NAME | jq -r $expr)

OLD_TASK_DEF=$(aws ecs describe-task-definition --task-definition $TASK_NAME --output json)
NEW_TASK_DEF=$(echo $OLD_TASK_DEF | jq --arg NDI $DOCKER_IMAGE '.taskDefinition.containerDefinitions[0].image=$NDI')
FINAL_TASK=$(echo $NEW_TASK_DEF | jq '.taskDefinition|{family: .family, volumes: .volumes, containerDefinitions: .containerDefinitions}')

aws ecs register-task-definition --family $TASK_NAME --cli-input-json "$(echo $FINAL_TASK)"
aws ecs update-service --service $SNAME --task-definition $TASK_NAME --cluster $CLUSTER_NAME

#3

@koblas - Thanks again for providing this improved script for ecs deployment. 3 question for you/community.

  1. Is there a reason you are retrieving the full name of the service (arn:aws:ecr:…service/$MY_SERVICE).?

aws ecs update-service seems to support --service $MY_SERVICE instead of using the full name. Didn’t know if you experienced something I haven’t yet with complex clusters.

  1. Removed this question relating to optional fields, because you take the previous task def and change the image, thus preserving all the relevant fields.

  2. How do you handle the checking if the previous task is drained properly? We use a a for loop of checking aws ecs describe-services. Do you have a more “graceful” recommendation?


#4

FYI: Here is the script I am using that is based off your script. Feedback is welcome! I do need to make one for that works cleanly with Fargate in the future.

Summary:

  1. Set AWS Creds and Push Docker Container within the script
  2. Set more environment variables in the run command on the Circle CI (not shown here, but assuming straightforward)
  3. Removed the need to evaluate the full service-name
  4. Save a few more variables for clean-up use
  5. Make sure the task def becomes the primary and all other tasks are de-registered

Nothing is changed for this section

#!/usr/bin/env bash
usage() {
    echo "Usage: $0 --cluster CLUSTER_NAME --service SERVICE_NAME --task TASK_NAME DOCKER_IMAGE"
    exit 1
}

while true ; do
    case "$1" in
        -t|--task) TASK_NAME=$2 ; shift 2 ;;
        -s|--service) SERVICE_NAME=$2 ; shift 2 ;;
        -c|--cluster) CLUSTER_NAME=$2 ; shift 2 ;;
        -h|--help) usage ;;
        --) shift ; break ;;
        *) break ;;
    esac
done

[ $# -eq 0 -o -z "$TASK_NAME" -o -z "$SERVICE_NAME" -o -z "$CLUSTER_NAME" ] && usage

DOCKER_IMAGE=$1
### Set AWS Creds
aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID
aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
aws configure set default.region $AWS_REGION
aws configure set default.output json
###
##### Push the new docker image
echo "Logging in"
eval $(aws ecr get-login --region $AWS_REGION --no-include-email)
echo "Tagging the latest image"
docker tag $CONTAINER_IMAGE:$DEPLOY_ENV $DOCKER_IMAGE
echo "Pushing the latest image"
docker -D push $DOCKER_IMAGE
######
echo "Get the previous task definition"
OLD_TASK_DEF=$(aws ecs describe-task-definition --task-definition $TASK_NAME --output json)
OLD_TASK_DEF_REVISION=$(echo $OLD_TASK_DEF | jq ".taskDefinition|.revision")

echo "dropping in the new image"
NEW_TASK_DEF=$(echo $OLD_TASK_DEF | jq --arg NDI $DOCKER_IMAGE '.taskDefinition.containerDefinitions[0].image=$NDI')

echo "create a new task template with all the required information to bring over"
FINAL_TASK=$(echo $NEW_TASK_DEF | jq '.taskDefinition|{family: .family, volumes: .volumes, containerDefinitions: .containerDefinitions, taskRoleArn: .taskRoleArn}')
#Set variables for re-use
echo "Upload the task information and register the new task definition along with optional information"
UPDATED_TASK=$(aws ecs register-task-definition --cli-input-json "$(echo $FINAL_TASK)")
echo "Storing the Revision"
UPDATED_TASK_DEF_REVISION=$(echo $UPDATED_TASK | jq ".taskDefinition|.taskDefinitionArn")
echo "Updated task def revision: $UPDATED_TASK_DEF_REVISION"
echo "switch over to the new task definition by selecting the newest revision"
SUCCESS_UPDATE=$(aws ecs update-service --service $SERVICE_NAME --task-definition $TASK_NAME --cluster $CLUSTER_NAME)
echo "Verify the new task definition attached and the old task definitions de-register aka cleanup"
for attempt in {1..8}; do
    #will return true if the updated task def is fully up and running in the service and the primary task def
    IS_ECS_READY=$(aws ecs describe-services --cluster $CLUSTER_NAME --services $SERVICE_NAME | jq '.services[0] .deployments | .[] | select(.taskDefinition == '${UPDATED_TASK_DEF_REVISION}') | (.desiredCount == .runningCount and .status == "PRIMARY")')
    echo "Is ECS updated: $IS_ECS_READY"
    if [ $IS_ECS_READY = false ]; then
        echo "Waiting for $UPDATED_TASK_DEF_REVISION"
        echo "It needs to become the primary task def and reach desired instance count"
        sleep 20
        echo "Lets find all active task definitions"
        ACTIVE_TASK_DEFS=$(aws ecs list-task-definitions --family-prefix $TASK_NAME | jq '.taskDefinitionArns')
        echo "Here are the active task definitions: $ACTIVE_TASK_DEFS"
        #will return true if there are more than 1 task definitions still active
        IS_MULTIPLE_ACTIVE_TASK_DEFS=$(echo $ACTIVE_TASK_DEFS | jq 'map(select(. != '${UPDATED_TASK_DEF_REVISION}')) | length > 1')
        echo "Are there multiple active ones: $IS_MULTIPLE_ACTIVE_TASK_DEFS"
        IS_ALL_TASKS_DRAINED=false #should default this to false, so it doesnt throw an error below
        continue
    elif [ $IS_ALL_TASKS_DRAINED = true ]; then
        echo "Successfully cleaned up old tasks and running the new task."
        PRIMARY_TASK=$(aws ecs list-task-definitions --family-prefix $TASK_NAME | jq '.taskDefinitionArns[]')
        echo "$PRIMARY_TASK is the only running task"
        break
    else
        #iterate through the active tasks and register them
        echo $ACTIVE_TASK_DEFS | jq -r '.[] | select(. != '${UPDATED_TASK_DEF_REVISION}')' | \
        while read arn; do
            deregistered_status=$(aws ecs deregister-task-definition --task-definition $arn | jq '.taskDefinition .status');
            echo "Setting $arn task definition to " + $(echo $deregistered_status)
        done
        ACTIVE_TASK_DEFS=$(aws ecs list-task-definitions --family-prefix $TASK_NAME | jq '.taskDefinitionArns')
        echo "All obsolete tasks have been moved to INACTIVE"
        echo "but we want to make sure they are drained as well"
        sleep 30
        IS_ALL_TASKS_DRAINED=$(aws ecs describe-services --cluster $CLUSTER_NAME --services $SERVICE_NAME | jq '.services[0] .deployments | length == 1')
        echo "Are all obsolete tasks drained/stopped: $IS_ALL_TASKS_DRAINED"
    fi

#5

FYI… For using this with Fargate, you have to add in a few things.
The FINAL_TASK line will look something like this:
FINAL_TASK=$(echo $NEW_TASK_DEF | jq '.taskDefinition|{networkMode: .networkMode, family: .family, volumes: .volumes, .executionRoleArn: "arn:aws:iam::your-ecs-admin-role", taskRoleArn: .taskRoleArn, containerDefinitions: .containerDefinitions}')

Namely, I had to add the networkMode param as well as the taskRoleArn and executionRoleArn. The executionRoleArn is especially annoying because it doesn’t get printed out from the aws ecs describe-task-definition, so you have to manually inject it in each time even if your last task definition has it in there.

Hope this helps.


#6

Your post helped me to do the same but in bitbucket pipelines.

family is not required if you are using cli-input-json so you can just write this:
aws ecs register-task-definition --cli-input-json "$(echo $FINAL_TASK)"


#7

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.