devops

gitlab, gitlab runners, k3s, and building all the things

Saturday, June 25, 2022

Although originally using Gitea for hosting, and Drone for my CI engine, this setup proved to be difficult. I continually had problems with Done, and actually had the Gitea database corrupt at one point. With all this I decided to just move back to Gitlab. They run a great service, and it’s what we use at work so I am quite familiar with them and their CI/CD setup. I installed the Gitlab Runner on my internal K3s cluster, and am now using that to do all my builds and deployments behind my firewall. I hit a few bumps along the way, so I wanted to document the final setup for posterity and anyone else who might be having issues as well. …more

nginx-ingress, cert-manager, and default wildcard certificates

Friday, January 28, 2022

I have quite a few internal only services running on my Kubernetes cluster. For all these services I wanted to use SSL, but using the default method of getting an ACME certificate from Let’s Encrypt (HTTP-01) wouldn’t work due to my setup. I also didn’t want to have to generate a certificate for every service I decided to spin up. Using cert-manager I was able to generate an ACME wildcard certificate and then set it to the default certificate on the nginx-ingress. …more

Drone and Hugo

Wednesday, August 4, 2021

This blog is deployed using a series of technologies from Kubernetes and Helm down to its core using Hugo. One of the annoyances with Hugo is that one needs to build the static content before it is deployed. This is easy enough to do, all you need to do is invoke the hugo command while in the correct directory, but I constantly forget to to this manual step. Instead of manually trying to remember to run this, I built out my Drone CI to do it for me. …more

Cloudflared and Internal Services

Tuesday, July 20, 2021

In my new position I work more with Kubernetes than I ever have in the past. This is a great thing, because I think Kubernetes is an amazingly powerful tool, and is quite enjoyable to work with once you understand the in’s and out’s of it. As such I decided to migrate just about everything I run over to a new Kubernetes cluster. Initially I ran two clusters: one internal to my firewall for non-world accessible stuff, and one in the cloud for world accessible stuff. Then I found out about Cloudflare Tunnels (formerly Argo Tunnel) and moved to a single cluster proxying through Cloudflare. …more

Quickly deploying Python code to Lambda

Thursday, December 12, 2019

Lambda is pretty awesome, and we use it for a ton of things at work. I’ve deployed around 20 different scripts at this point, that do everything from monitoring type of tasks to a full Slackbot. Although generally we use Ansible to deploy everything (including our Lambda tasks), there are times when things need to go and they need to go now. For this I wrote a simple script that zips up a package that can be manually deployed to Lambda with all the necessary libraries and things your script needs. …more

drone.io and the GCR

Wednesday, May 22, 2019

For the last week or so I’ve been evaluating drone.io as a replacement for our current CI engine (spoiler: we aren’t going to use it). Something that I ran across that wasn’t immediately obvious on how to achieve was pulling private images from Google Container Repository. Although there is a thread on the Discouse forums that touches on it, I had to do some testing on my own to actually get it working. Here is what I did. …more

Testing Ansible AWX/Tower roles

Friday, September 21, 2018

As I continue to push forward the use of AWX in our infrastructure, there is continued need to test what we are producing. Much like any other kind of development, sometimes we need a local environment to test something. This came up recently for me in AWX because I was testing a role that modified the AWS Task container in order to provide enhanced functionality. I wrote the following little shell provisioner that gets AWX up and running quickly for testing. …more

Using Nginx in front of AWX for SSL

Wednesday, August 8, 2018

I’ve got AWX deployed in production currently. Obviously for production, you need to SSL for authentication. Since AWX doesn’t offer this currently, and I’m not paying $20/month just for a ELB for this, I decided to slap Nginx in front of the containers to proxy SSL. This ended up being a larger pain than I expected because of the websockets. Finally got it figured out and wanted to post my nginx config for anyone else trying to do the same thing. …more

Migration to Gitlab and Gitlab Runners

Wednesday, July 25, 2018

The Linux community, and the open-source community in general, was up in arms last month as Microsoft announced a purchase of Github. For many reasons, which I am not going to detail here, I agree with the general concern and outrage over this. Due to this, I have migrated all my repos from Github (where I was a paying member) to Gitlab. The migration itself is dead easy, but with this migration you lose access to other tools. This mainly impacted me with the loss of CircleCI. Enter Gitlab runners! …more

docker-py deprecated in favor of docker

Tuesday, July 17, 2018

Just a quick note that the Python module docker-py has been deprecated in favor of docker. I run a few Ansible hosts that run Docker containers for various reasons and recently they started to fail during their Ansible runs. Turns out that docker-py now fails during said runs. After some searching I found that this module hasn’t been updated in a very long time, and has been deprecated in favor of the docker module instead. I wrote the following small Ansible task to fix this on my machines. …more