Using CircleCI workflows to deploy to Kubernetes

Wednesday, July 5, 2017

As I have been continuing to work with CircleCI’s new 2.0 beta I started looking at the workflows recently. These are like Pipelines in Jenkins, that allow for a series of build/test/deploy steps that can be done either sequentially or in parallel depending upon your use case. To set up a true Continuous Integration using this, and the new Keubernetes cluster, I set up a new job to deploy to Kubernetes. It took a little time digging through some documentation between both CircleCI and Kubernetes, so I wanted to share my findings.

The easiest way I found to get this working was to actually setup a Kubernetes configuration file within the build container that does the deployment. Obviously, putting my actual Kubernetes configuration file into the container, and thus into Git, is definitely not ideal. There are way too many secrets inclusive in there (including the server SSL keys which could be used for a number of nefarious purposes). Luckily there is a way to set these options via the command line using kubectl config. By using this, plus variables that are set in the CircleCI UI (and thus outside of Git) I have gotten this working.

A little context on our setup. We are running Kubernetes in AWS using Kops. Hence we are also using the Amazon ECR and need to be able to push containers to thier appropriate repositories. This has two build steps, the first builds my test application in a container that has the AWS CLI installed, and then pushes the resulting image to the ECR. Then the second build installs the Kubectl tool, sets up the configuration, and deploys the container to the Kubernetes cluster.

  version: 2
  jobs:
    build:
      working_directory: /app
      docker:
        - image: docker:17.05.0-ce-git
      steps:
        - checkout
        - setup_remote_docker
        - run:
            name: Install Dependencies
            command: |
              apk add --no-cache \
                py-pip=9.0.0-r1
              pip install \
                docker-compose==1.12.0 \
                awscli==1.11.76
        - restore_cache:
            keys:
              - v1-{{ .Branch }}
            paths:
              - /caches/app.tar
        - run:
            name: Load Docker Image Layer Cache
            command: |
              set +o pipefail
              docker load -i /caches/app.tar | true
        - run:
            name: Build Application Docker Image
            command: |
              docker build --cache-from=app -t app .
        - run:
            name: Save Docker Image Layer Cache
            command: |
              mkdir -p /caches
              docker save -o /caches/app.tar app 
        - save_cache:
            key: v1-{{ .Branch }}--{{ epoch }}
            paths:
              - /caches/app.tar
        - deploy:
            name: Push Docker Image to ECR 
            command: |
              if [ "${CIRCLE_BRANCH}" == "master" ]; then
                login="$(aws ecr get-login)"
                ${login}
                docker tag app "${ECR_ENDPOINT}/testing:${CIRCLE_SHA1}"
                docker push "${ECR_ENDPOINT}/testing:${CIRCLE_SHA1}"
              fi
    kubernetes:
      working_directory: /deploy
      docker:
        - image: golang:1.7-wheezy
      steps:
        - checkout
        - setup_remote_docker
        - run:
            name: Install Dependencies
            command: |
              curl  -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
              chmod +x ./kubectl
              mv ./kubectl /usr/local/bin/kubectl
        - run:
            name: Setup Kubectl and Deploy to Kubernetes Cluster
            command: |
              kubectl config set-credentials $KOPS_USERNAME/$KOPS_CLUSTER_NAME --username=$KOPS_USERNAME --password=$KOPS_PASSWORD
              kubectl config set-cluster $KOPS_CLUSTER_NAME --insecure-skip-tls-verify=true --server=$KOPS_SERVER
              kubectl config set-context default/$KOPS_CLUSTER_NAME/$KOPS_USERNAME --user=$KOPS_USERNAME/$KOPS_CLUSTER_NAME --namespace=default --cluster=$KOPS_CLUSTER_NAME
              kubectl config use-context default/$KOPS_CLUSTER_NAME/$KOPS_USERNAME
              kubectl create -f /deploy/kubernetes/deployment.yaml
  workflows:
    version: 2
    build_and_deploy:
      jobs:
        - build
        - kubernetes:
            requires:
              - build

There are still a few things that I need to work out. First, I need to get the SSL keys working so I don’t have to use the insecure-skip-tls-verify=true switch. I’m sure there is a way to get the SSL keys into the configuration as well. Second, the kubectl create command will only work the first time it is run, since it’s, well, a create command. For all subsequent runs a rolling-update should be applied instead. I need to look into the logic around this and see exactly how to set it up.

devopskubernetescircleci

Kubernetes, Cluster Auto-Scaling and RBAC

Infrastructure Change - Enter CircleCI