As I have been continuing to work with CircleCI’s new 2.0 beta I started looking at the workflows recently. These are like Pipelines in Jenkins, that allow for a series of build/test/deploy steps that can be done either sequentially or in parallel depending upon your use case. To set up a true Continuous Integration using this, and the new Keubernetes cluster, I set up a new job to deploy to Kubernetes. It took a little time digging through some documentation between both CircleCI and Kubernetes, so I wanted to share my findings.
The easiest way I found to get this working was to actually setup a Kubernetes configuration file within the build container that does the deployment. Obviously, putting my actual Kubernetes configuration file into the container, and thus into Git, is definitely not ideal. There are way too many secrets inclusive in there (including the server SSL keys which could be used for a number of nefarious purposes). Luckily there is a way to set these options via the command line using kubectl config. By using this, plus variables that are set in the CircleCI UI (and thus outside of Git) I have gotten this working.
A little context on our setup. We are running Kubernetes in AWS using Kops. Hence we are also using the Amazon ECR and need to be able to push containers to thier appropriate repositories. This has two build steps, the first builds my test application in a container that has the AWS CLI installed, and then pushes the resulting image to the ECR. Then the second build installs the Kubectl tool, sets up the configuration, and deploys the container to the Kubernetes cluster.
version: 2
jobs:
build:
working_directory: /app
docker:
- image: docker:17.05.0-ce-git
steps:
- checkout
- setup_remote_docker
- run:
name: Install Dependencies
command: |
apk add --no-cache \
py-pip=9.0.0-r1
pip install \
docker-compose==1.12.0 \
awscli==1.11.76
- restore_cache:
keys:
- v1-{{ .Branch }}
paths:
- /caches/app.tar
- run:
name: Load Docker Image Layer Cache
command: |
set +o pipefail
docker load -i /caches/app.tar | true
- run:
name: Build Application Docker Image
command: |
docker build --cache-from=app -t app .
- run:
name: Save Docker Image Layer Cache
command: |
mkdir -p /caches
docker save -o /caches/app.tar app
- save_cache:
key: v1-{{ .Branch }}--{{ epoch }}
paths:
- /caches/app.tar
- deploy:
name: Push Docker Image to ECR
command: |
if [ "${CIRCLE_BRANCH}" == "master" ]; then
login="$(aws ecr get-login)"
${login}
docker tag app "${ECR_ENDPOINT}/testing:${CIRCLE_SHA1}"
docker push "${ECR_ENDPOINT}/testing:${CIRCLE_SHA1}"
fi
kubernetes:
working_directory: /deploy
docker:
- image: golang:1.7-wheezy
steps:
- checkout
- setup_remote_docker
- run:
name: Install Dependencies
command: |
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod +x ./kubectl
mv ./kubectl /usr/local/bin/kubectl
- run:
name: Setup Kubectl and Deploy to Kubernetes Cluster
command: |
kubectl config set-credentials $KOPS_USERNAME/$KOPS_CLUSTER_NAME --username=$KOPS_USERNAME --password=$KOPS_PASSWORD
kubectl config set-cluster $KOPS_CLUSTER_NAME --insecure-skip-tls-verify=true --server=$KOPS_SERVER
kubectl config set-context default/$KOPS_CLUSTER_NAME/$KOPS_USERNAME --user=$KOPS_USERNAME/$KOPS_CLUSTER_NAME --namespace=default --cluster=$KOPS_CLUSTER_NAME
kubectl config use-context default/$KOPS_CLUSTER_NAME/$KOPS_USERNAME
kubectl create -f /deploy/kubernetes/deployment.yaml
workflows:
version: 2
build_and_deploy:
jobs:
- build
- kubernetes:
requires:
- build
There are still a few things that I need to work out. First, I need to get the SSL keys working so I don’t have to use the insecure-skip-tls-verify=true switch. I’m sure there is a way to get the SSL keys into the configuration as well. Second, the kubectl create command will only work the first time it is run, since it’s, well, a create command. For all subsequent runs a rolling-update should be applied instead. I need to look into the logic around this and see exactly how to set it up.