Cloudflared and Internal Services

Tuesday, July 20, 2021

In my new position I work more with Kubernetes than I ever have in the past. This is a great thing, because I think Kubernetes is an amazingly powerful tool, and is quite enjoyable to work with once you understand the in’s and out’s of it. As such I decided to migrate just about everything I run over to a new Kubernetes cluster. Initially I ran two clusters: one internal to my firewall for non-world accessible stuff, and one in the cloud for world accessible stuff. Then I found out about Cloudflare Tunnels (formerly Argo Tunnel) and moved to a single cluster proxying through Cloudflare.

What is Cloudflare Tunnels?

Cloudflare tunnels is a service that proxies traffic from Cloudflare’s edge points back to your services. It is nice because it allows for the routing of traffic without having to poke holes in your firewall. This means you can make internal services external, with no additional firewall configuration. It also means you have the extra protection of using Cloudflare as your endpoint. What’s the catch? Well first it costs money. There is a free option (which I will get to later), but for most use cases it’s going to run you $5/month plus metered billing of $0.10/GB. I still felt this was worth it, as it is much cheaper than the cost of running any K8s cluster in the cloud. The second catch is that you have to use Cloudfare DNS. You could probably use delegation for only certain subdomains, but I really didn’t mind moving my DNS so it wasn’t an issue for me.

I’m using this service on a bunch of my internal stuff, but I am going to walk through the Gitea setup because it proxies traffic for both the frontend and the ssh connections.

The Free Option

Before we get into the nitty gritty of all this, I want to touch on the free option. You can technically use Cloudflare Tunnels for free, but there are a few catches. First, you can only use it for HTTP connections. Second, you do not get to choose your own subdomain. You are given a random subdomain ending in trycloudflare.com. Lastly, their is no guarantee that the tunnel is persistent, or that the subdomain will remain the same. Basically this is a better option than ngrok for testing inside of K8s (IMO).

Using the free option is easy, add the sidecar container to any current deployment / statefulset or whatever with the following options (with http://127.0.0.1:8000 being where you main pod is serving traffic):

        - name: tunnel
          image: docker.io/cloudflare/cloudflared:2021.7.0-amd64
          imagePullPolicy: Always
          command: ["cloudflared", "tunnel"]
          args:
            - --url=http://127.0.0.1:8080
            - --no-autoupdate
          env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace

Then look at the logs for the tunnel for your url:

$ k logs hello-kubernetes-hello-k8s-7ccd9778d7-pb7q7 tunnel 
2021-07-20T17:03:11Z INF Cannot determine default configuration path. No file [config.yml config.yaml] in [~/.cloudflared ~/.cloudflare-warp ~/cloudflare-warp /etc/cloudflared /usr/local/etc/cloudflared]
2021-07-20T17:03:11Z INF Version 
2021-07-20T17:03:11Z INF GOOS: linux, GOVersion: go1.16.4, GoArch: amd64
2021-07-20T17:03:11Z INF Settings: map[no-autoupdate:true url:http://127.0.0.1:8080]
2021-07-20T17:03:11Z INF Initial protocol h2mux
2021-07-20T17:03:11Z INF Starting metrics server on 127.0.0.1:46537/metrics
2021-07-20T17:03:11Z INF Connection established connIndex=0 location=ATL
2021-07-20T17:03:12Z INF Each HA connection's tunnel IDs: map[0:rtz6n2d2augbnvctnp9hgslzvye0ngl890rqf8fb2z5kdhg6ne4g]
2021-07-20T17:03:12Z INF +------------------------------------------------------------+
2021-07-20T17:03:12Z INF |  Your free tunnel has started! Visit it:                   |
2021-07-20T17:03:12Z INF |    https://planets-festivals-trader-ssl.trycloudflare.com  |
2021-07-20T17:03:12Z INF +------------------------------------------------------------+
2021-07-20T17:03:12Z INF Route propagating, it may take up to 1 minute for your new route to become functional
2021-07-20T17:03:13Z INF Connection established connIndex=1 location=DFW
2021-07-20T17:03:14Z INF Connection established connIndex=2 location=ATL
2021-07-20T17:03:14Z INF Each HA connection's tunnel IDs: map[0:rtz6n2d2augbnvctnp9hgslzvye0ngl890rqf8fb2z5kdhg6ne4g 1:rtz6n2d2augbnvctnp9hgslzvye0ngl890rqf8fb2z5kdhg6ne4g]

That’s it, you have a free tunnel from the internet into your K8s pod.

Running Gitea with Cloudflare Tunnels

Setting up the Tunnel

To set up the tunnel you first have to have a Cloudflare account and sign up for Argo Tunnel (Argo is the old name, but it hasn’t been changed in the UI). Then install cloudflared command line utility on your computer. Once you have that installed you need to login using cloudflared tunnel login. Then we can create the tunnel:

$ cloudflared tunnel create test                       
Tunnel credentials written to /home/user/.cloudflared/25628b00-7922-47c0-a5b8-afe6bda7db77.json. cloudflared chose this file based on where your origin certificate was found. Keep this file secret. To revoke these credentials, delete the tunnel.

Created tunnel test with id 25628b00-7922-47c0-a5b8-afe6bda7db77

Finally use cloudflared tunnel route dns <tunnel_name> <subdomain.yourdomain.net> to set up the DNS entry for your new service. Now we can move into the Kubernetes setup.

Setting up Gitea

The Gitea setup is pretty standard. It is deployed using the Helm chart provided by them, but with two additions. The first addition is the use of a sidecar container running cloudflared. This is what proxies the connections from the containers out to the internet and back. Using this setup you do not need to run a Kubernetes service or ingress. It is all handled by the sidecar. The second is setting up Gitea to use it’s own internal SSH server on a different subdomain than the front end.

In order to add the sidecar container edit the templates/gitea/statefulset.yaml file and add a new container under the containers section:

        - name: tunnel
          image: docker.io/cloudflare/cloudflared:2021.7.0-amd64
          imagePullPolicy: Always
          args:
            - tunnel
            - --config
            - /etc/cloudflared/config/config.yaml
            - run
          volumeMounts:
          - name: tunnel-creds
            mountPath: /etc/cloudflared/creds
            readOnly: true
          - name: tunnel-config
            mountPath: /etc/cloudflared/config
            readOnly: true
          - name: tunnel-cert
            mountPath: /etc/cloudflared
            readOnly: true
          env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace

You can see from this definition that there are three important files being added, the tunnel-creds, the tunnel-config, and the tunnel-cert. Make sure to add these in the volumes definition:

      volumes:
        - name: config
          secret:
            secretName: {{ include "gitea.fullname" . }}
        - name: tunnel-creds
          secret:
            secretName: gitea-credentials
        - name: tunnel-cert
          secret:
            secretName: tunnel-cert
        - name: tunnel-config
          configMap:
            name: cloudflared-gitea
            items:
            - key: config.yaml
              path: config.yaml

Creating the Secrets and ConfigMap

Creating the secrets and the config map is pretty straight forward. To create the tunnel-creds create a new secret using the tunnel config created by the tunnel create above: k create secret generic tunnel-credentials --from-file=credentials.json=/home/user/.cloudflared/25628b00-7922-47c0-a5b8-afe6bda7db77.json

Next create the tunnel-cert using the cert.pem in your ~/.cloudflared directory: k create secret generic tunnel-cert --from-file=cert.pem=/home/user/.cloudflared/cert.pem

Finally we create the configMap. This is a simple config file that cloudflared uses to determine how to route your traffic. Here is the one used for Gitea. The last service of http_status:404 is required, and has to be last. Also notice that SSH and HTTP traffic are served on different subdomains. This is important for the next part to work.

apiVersion: v1    
kind: ConfigMap
metadata:
  name: cloudflared-gitea
  namespace: gitea
data:    
  config.yaml: |    
    # Name of the tunnel you want to run    
    tunnel: gitea
    credentials-file: /etc/cloudflared/creds/credentials.json
    # Serves the metrics server under /metrics and the readiness server under /ready    
    metrics: 0.0.0.0:2000
    # Autoupdates applied in a k8s pod will be lost when the pod is removed or restarted, so    
    # autoupdate doesn't make sense in Kubernetes. However, outside of Kubernetes, we strongly    
    # recommend using autoupdate.    
    no-autoupdate: true 
    # The `ingress` block tells cloudflared which local service to route incoming    
    # requests to. For more about ingress rules, see    
    # https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ingress    
    #    
    # Remember, these rules route traffic from cloudflared to a local service. To route traffic    
    # from the internet to cloudflared, run `cloudflared tunnel route dns <tunnel> <hostname>`.    
    # E.g. `cloudflared tunnel route dns example-tunnel tunnel.example.com`.    
    ingress: 
    - hostname: git.binaryronin.io 
      service: http://localhost:3000
    - hostname: git-ssh.binaryronin.io
      service: ssh://localhost:2222
    - service: http_status:404

Once this is all deployed the front end will be working, but SSH needs to go through something Cloudflare calls “Zero Trust”.

SSH and Zero Trust

To use Zero Trust a Cloudflare Teams account is required. This account is free, so just add it to your current Cloudflare offerings. Once inside here create a new application for the ssh connection (I called mine git-ssh) and point the application domain to the domain for the SSH connection. For “Identity Provider” just select “One-time PIN” for now. If you have a more complex setup you can attach various IDPs and use those. I think the rules section is pretty self-explanatory, for my use-case I just added my email address as the only one that could auth.

Once that is all setup, on your local machine create a new entry in your ~/.ssh/config that passes authorization off to the local cloudflared command:

Host git-ssh.binaryronin.io
        ProxyCommand /usr/bin/cloudflared access ssh --hostname %h

And that’s it! When you go to push to the server, you will be redirected to your browser which will ask you to log in.

Conclusion

I highly recommend checking out this option if you are running a homelab and want to serve things to the outside world. I have a few other services using this currently (including this blog) and it’s nice to add a sidecar container and be done with it. Cloudflare handles all the routing, all the SSL, everything. No need for ingress controllers with certificate managers, or loadbalancers, or hostPorts. And added bonus you get Cloudflare caching and DDoS mitigation as well.

devopsk8sk3scloudflared

CentOS and Newer Versions of Python