Loan a cloud IP to your minikube cluster

by Alex Ellis

I'm going to show you you can loan a public cloud IP adress to your minikube cluster from your managed cloud Kubernetes cluster.

After following this tutorial, you'll be able to run a local service on your laptop with minikube and have it appear with a public IP through your Kubernetes cluster.

Pictured: when a client connects to the public Cluster's IP, then they will be proxied to a local service running inside a minikube cluster.

This is most useful either as a freelancer when you want to share work with clients, or as part of a team or OSS community collaborating on code and services. You will also benefit from a quick dev/test loop without pushing code into a remote registry or cluster, something that's ideal when integrating with partners and services such as webhooks from Stripe, Strava, PayPal, GitHub, GitLab, you name it.

Introducing inlets

For the tutorial we'll be using a popular Open Source networking tunnel and service proxy called inlets. Inlets is listed on the CNCF landscape.

Conceptual architecture diagram showing a use-case for GitHub webhooks

The traditional usage of inlets is as a client/server pair, where the client runs behind a restrictive firewall or NAT configuration and the server runs somewhere with a public IP. Exit-servers can be automated for use with inlets through inletsctl or inlets-operator for Kubernetes services. The service in the private network then appears on the outside world through the IP of the "exit-server".

Setup a cloud Kubernetes cluster

This step is important, you must use a managed Kubernetes offering which can give you a LoadBalancer, such as an ALB or ELB in the AWS ecosystem.

I'm going to use AWS EKS because the Open Source team offered some credits to use for writing tutorials like this. Recently the cost of the EKS control plane was halved, from $0.20 USD / hour, to $0.10 / hour which makes the overall cost cheaper too.

Get eksctl, which is a CLI created by Weaveworks to setup the managed Kubernetes cluster:

curl --silent --location "https://github.com/weaveworks/eksctl/releases/download/latest_release/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp

sudo mv /tmp/eksctl /usr/local/bin

Now install the AWS CLI and configure your ~/.aws/credentials file or environment variables.

Now authenticate:

eksctl create cluster

# Or customise

eksctl create cluster \
    --nodes 1 \
    --node-type t3.large \
    --region eu-west-1

Once up and running you'll have a production-grade cluster from AWS which can issue LoadBalancers, this is how we'll be getting a public IP when required.

Run minikube, KinD, or some other local Kubernetes cluster

I'm a big fan of the speed of k3s, k3d, and KinD for local Kubernetes development, but minikube is still very popular with the community. Let's use that.

Once done, you can create a local cluster:

minikube start

Schedule a workload on your minikube cluster

We now need to schedule a workload on our local minikube cluster. This is the service which needs a public IP.

Let's use my Express.js + Node.js microservice

git clone https://github.com/alexellis/expressjs-k8s
cd expressjs-k8s

kubectl apply -f ./yaml/dep.yaml

Now we have a deployment running, but need to create a service for it.

kubectl apply -f ./yaml/svc.yaml

You should be able to access the service via curl and a port-forward:

kubectl port-forward deploy/expressjs 8080:8080 &

curl -s 127.0.0.1:8080/links | jq
[
  {
    "name": "github",
    "url": "https://github.com/alexellis"
  },
  {
    "name": "twitter",
    "url": "https://twitter.com/alexellisuk"
  },
  {
    "name": "blog",
    "url": "https://blog.alexellis.io"
  },
  {
    "name": "sponsors",
    "url": "https://github.com/users/alexellis/sponsorship"
  }
]

Note that our service exposes port 8080 and is named "expressjs".

Start an inlets-server on the cloud cluster

Now let's start an inlets-server on the cloud cluster. Normally we'd simply use the inlets-operator for this stage, and it would automate everything for us. In this tutorial we're trying to use a cloud Kubernetes cluster as our exit-node, rather than a VM.

Set your Kubernetes context to point at the remote cloud cluster.

kubectl config get-contexts
kubectl config set-context <CONTEXT>

Generate a token to be used by the inlets client to authenticate to the inlets server.

export TOKEN=$(head -c 16 /dev/urandom | shasum | cut -d" " -f1)
kubectl create secret generic inlets-token --from-literal token=$TOKEN

echo The token is: $TOKEN

echo $TOKEN > inlets-token.txt

You need to record this token and use it again in the minikube cluster.

Create the exit-server deployment and service

Now create a deployment for the inlets-server:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inlets-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: inlets-server
  template:
    metadata:
      labels:
        app: inlets-server
    spec:
      containers:
      - name: inlets-server
        image: inlets/inlets:2.6.3
        imagePullPolicy: Always
        command: ["inlets"]
        args:
        - "server"
        - "--control-port=8123"
        - "--port=8080"
        - "--token-from=/var/inlets/token"
        volumeMounts:
          - name: inlets-token-volume
            mountPath: /var/inlets/
      volumes:
        - name: inlets-token-volume
          secret:
            secretName: inlets-token

You can use kubectl apply -f - and then paste the text above.

We now need a LoadBalancer service, to get a public IP:

apiVersion: v1
kind: Service
metadata:
  name: inlets-server
  labels:
    app: inlets-server
spec:
  type: LoadBalancer
  ports:
    - name: inlets-data
      port: 8080
      protocol: TCP
      targetPort: 8080
      nodePort: 30080
    - name: inlets-control
      port: 8123
      protocol: TCP
      targetPort: 8123
      nodePort: 30023
  selector:
    app: inlets-server

If you've ever tried to use a LoadBalancer with EKS before, you'll remember that you get a DNS entry, and not an actual IP. That's OK, don't worry.

You'll see two ports: 8080 and 8123, the data port can be whatever you like, so you could have used port 80 if you wanted.

For the control port I've picked 8123, so that the two don't clash, you can also customise this value if you like.

If you want to create a DNS record, you can use a "CNAME" and paste in the output from the kubectl command.

kubectl get svc/inlets -o wide
NAME     TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT(S)
inlets   LoadBalancer   10.100.146.196   a25ab4229452511eaadd40a443f60541-488879946.eu-west-1.elb.amazonaws.com   8080:30080/TCP,8128:30081/TCP

Note the EXTERNAL_IP down now, which is actually a DNS entry.

When completed, our users will connect to the EXTERNAL_IP using port 8080 and the inlets-client will connect in via port 8128 to establish a persistent websocket.

You can see the logs and check on the server deployment:

kubectl get deploy/inlets-server

NAME            READY   UP-TO-DATE   AVAILABLE   AGE
inlets-server   1/1     1            1           46s

kubectl logs deploy/inlets-server

2020/02/01 18:59:16 Welcome to inlets.dev! Find out more at https://github.com/inlets/inlets
2020/02/01 18:59:16 Starting server - version 2.6.3
2020/02/01 18:59:16 Server token: "3de9710abadb56509eb8e634777bcbddad194331"
2020/02/01 18:59:16 Data Plane Listening on :8080
2020/02/01 18:59:16 Control Plane Listening on :8123

Start an inlets-client inside minikube

Use kubectl get-contexts and kubectl set-context to make sure that kubectl is pointing at the minikube cluster.

If you want to check that you're in the correct cluster, at any time run: kubectl cluster-info

Create a secret for the inlets-client to use, this must use the value entered in the public cloud cluster:

kubectl create secret generic inlets-token --from-literal token=$(cat inlets-token.txt)

Start the inlets-client

The inlets-client will connect to the inlets-server and establish a bi-directional websocket. For any requests that hit the inlets-server, they will be redirected top the inlets-client inside our local minikube cluster.

We just need to set three things:

  • --remote - the remote cluster's LoadBalancer IP (a DNS-entry, for EKS users), use ws:// for plaintext and wss:// for TLS.
  • --token - the authentication secret name (inlets-token) for the authentication between inlets
  • --upstream - the upstream address that inlets needs to point at i.e. http://expressjs.default:8080
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inlets-client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: inlets-client
  template:
    metadata:
      labels:
        app: inlets-client
    spec:
      containers:
      - name: inlets-client
        image: inlets/inlets:2.6.3
        imagePullPolicy: Always
        command: ["inlets"]
        args:
        - "client"
        - "--remote=ws://a25ab4229452511eaadd40a443f60541-488879946.eu-west-1.elb.amazonaws.com:8123"
        - "--upstream=http://expressjs.default:8080"
        - "--token-from=/var/inlets/token"
        volumeMounts:
          - name: inlets-token-volume
            mountPath: /var/inlets/
      volumes:
        - name: inlets-token-volume
          secret:
            secretName: inlets-token

Edit the above text, then paste it into kubectl apply -f -, or use kubectl apply -f filename.yaml.

You can see the logs and check on the client deployment:

kubectl get deploy/inlets-client
NAME            READY   UP-TO-DATE   AVAILABLE   AGE
inlets-client   1/1     1            1           46s

kubectl logs deploy/inlets-client

2020/02/01 18:56:29 Welcome to inlets.dev! Find out more at https://github.com/inlets/inlets
2020/02/01 18:56:29 Starting client - version 2.6.3
2020/02/01 18:56:29 Upstream:  => http://expressjs.default:8080
2020/02/01 18:56:29 Token: "3de9710abadb56509eb8e634777bcbddad194331"
time="2020-02-01T18:56:29Z" level=info msg="Connecting to proxy" url="ws://a25ab4229452511eaadd40a443f60541-488879946.eu-west-1.elb.amazonaws.com:8123/tunnel"

Connect to your new service via the cloud cluster

You can now connect to the expressjs service via the cloud IP.

curl -s -i http://a25ab4229452511eaadd40a443f60541-488879946.eu-west-1.elb.amazonaws.com:8080/links

You can also create a Route53 DNS record for the LoadBalancer, for instance expressjs.example.com.

You can see the curl working with the public URL

Here's the website rendered with Copyright 2019.

Let's use kubectl against our local cluster to change to a newer Docker image that presents Copyright 2020 instead and refresh the page:

Remember to use your minikube context for this:

kubectl get deploy -o wide
NAME            READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS      IMAGES                     SELECTOR
expressjs       1/1     1            1           39m     expressjs       alexellis2/service:0.3.5   app=expressjs

I pushed a new version of the image tag as 0.3.6, so let's update it:

kubectl set image deployment/expressjs expressjs=alexellis2/service:0.3.6
deployment.apps/expressjs image updated

Now check back on the public URL via our EKS cluster:

Take it further

  • Count the cost

    The OSS version of inlets is free for you to use, so the only cost you'll have to bear is for your cloud Kubernetes cluster, any LoadBalancers that you create and for bandwidth that you use. It goes without saying that you should setup your cloud cluster in a region that has low latency to where you're connecting from.

  • Pause and resume

    If you stop minikube, you can leave the inlets-server running with its ALB, the inlets-client deployment will reconnect automatically once you start up minikube again.

  • Proxy multiple services or ports

    The OSS version of inlets can use a single LB and IP to proxy to multiple different upstream services using the --upstream parameter repeated. To use multiple upstreams, you'll need a DNS entry for each one i.e. --upstream=http://prometheus.example.com=prometheus:9090 followed by --upstream=http://tekton.example.com=tekton-webhook-receiver:8080

    inlets-pro supports pure TCP proxying, so you could use it to get a public IP for an IngressController such as Traefik, Nginx-Ingresshttps://github.com/kubernetes/ingress-nginx, or Caddy. inlets-pro also supports mapping more than one port from the external host to your internal cluster, i.e. 80 and 443. You'll be able to obtain a TLS certificate from LetsEncrypt with a DNS01 or HTTP01 challenge and serve it directly from your local cluster.

  • Easy encryption with inlets-pro

    If you'd like to enable encryption out of the box, you can use inlets-pro instead, which also comes with a Docker image ready for you. Follow one of my inlets-pro tutorials.

  • Setup an Ingress record and TLS certificate

    Rather than creating a LoadBalancer for each service we want to expose, you could use your favourite IngressController and create a simple ingress record that points at the inlets-service. Just change it to a ClusterIP.

  • Take it easy on yourself and use the inlets-operator

    The inlets-operator makes all this automated, and provisions a cheap VM for each tunnel. Life is too short, enjoy it while you can with our custom-built automation. Support exists for Packet, DigitalOcean, Scaleway, AWS and more. Feel free to raise a PR or suggestion if your cloud provider is missing and you'd like to request it.

Tidy up your public Kubernetes cluster

The inlets-operator is a much cheaper way to run an exit-node than using a managed Kubernetes cluster, but if your corporate already has an account, or you get free credits, then the approach we used here could be a better option for you.

You should be able to clean up all the resources with:

# In the public cloud cluster
kubectl delete svc/inlets-server
kubectl deploy svc/inlets-server

# In the local cluster
kubectl delete svc/inlets-client
kubectl deploy svc/inlets-client

# Delete the clusters if you wish
eksctl delete cluster
minikube delete

Learn inlets

inlets on the CNCF landscape

inlets is a rich tool with its own automation, tutorials, podcasts, videos, and community.

If you'd like to explore commercial use of inlets-pro, feel free to send an email to sales@openfaas.com

For comments and suggestions, you can follow @inletsdev on Twitter