Cannot access Kubernetes endpoint within WSL2 Docker Desktop from Linux machine
I've setup an authentication service in a Kubernetes cluster which lives in a Docker Desktop + WSL2 environment on a Windows 11 Pro machine.
It can be reached, e.g. with a CURL or PostMan request from within the same Windows machine outside that WSL2 layer, but it cannot be accessed from a Linux machine within the same LAN network.
In order to verify that a connectivity to that auth service actually works, I've setup a service with type "NodePort". With the external port obtained from 'kubectl get svc' the curl requests both work fine from the Windows and Linux machine, and the service responds with the proper answer.
However, note that the NodePort service completely circumvents the TSL/SSL of the K8s ingress which comes with a fake certificate, so the request - although successfully - only works for requests over HTTP.
Because I need to work with cookies in the response from the auth service, requests over HTTPS are crucial, however the K8s Ingress does not send a response at all.
On the windows machine, I've opened relevant ports 80, 443 (and 22 just to be sure), and just to be sure, I also disabled the windows firewall temporarily to see if it's actually blocking the connection attempt. With a disabled firewall, the request still doesn't come through so to me it looks like the Ingress doesn't know how to answer the request (or silently drops it).
But before going into any assumptions, let me give you an overview of the machines and the POST requests between them:
The auth deployment and service are defined as follows:
apiVersion: apps/v1
kind: Deployment
metadata:
name: gts-auth-depl
spec:
replicas: 1
selector:
matchLabels:
app: gts-auth
template:
metadata:
labels:
app: gts-auth
spec:
containers:
- name: gts-auth
image: creativewarlock/gts-auth
imagePullPolicy: IfNotPresent
env:
- name: MONGO_URI
value: 'mongodb://gts-auth-mongo-srv:27017/auth'
- name: JWT_KEY
valueFrom:
secretKeyRef:
name: gts-jwt-secret
key: JWT_KEY
imagePullSecrets:
- name: docker-hub
---
apiVersion: v1
kind: Service
metadata:
name: gts-auth-srv
spec:
type: ClusterIP
selector:
app: gts-auth
ports:
- name: gts-auth
protocol: TCP
port: 3000
targetPort: 3000
The ingress for the auth and web client (not relevant though, but just to show):
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: gts-ingress-srv
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/use-regex: 'true'
spec:
rules:
- host: gts.dev # Entry in System32/etc/hosts: 127.0.0.1 gts.dev (alternatively: localhost)
http:
paths:
- path: /api/users/?(.*) # will match /api/users/signin
pathType: Prefix
backend:
service:
name: gts-auth-srv
port:
number: 3000
- path: /?(.*) # map all other requests to web client
pathType: Prefix
backend:
service:
name: gts-client-srv
port:
number: 8080
The NodePort service for the external / direct access to the auth service:
apiVersion: v1
kind: Service
metadata:
name: gts-auth-srv-ext
spec:
type: NodePort
selector:
app: gts-auth
ports:
- name: gts-auth-srv
protocol: TCP
port: 3000
targetPort: 3000
Here's how Kubernetes describes the Ingress:
k describe pod ingress-nginx-controller-6bf7bc7f94-krv2c -n ingress
-nginx
Name: ingress-nginx-controller-6bf7bc7f94-krv2c
Namespace: ingress-nginx
Priority: 0
Node: docker-desktop/192.168.65.4
Start Time: Thu, 11 Aug 2022 12:31:42 +0200
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/name=ingress-nginx
pod-template-hash=6bf7bc7f94
Annotations: <none>
Status: Running
IP: 10.1.28.235
IPs:
IP: 10.1.28.235
Controlled By: ReplicaSet/ingress-nginx-controller-6bf7bc7f94
Containers:
controller:
Container ID: docker://0cef9bec68d309011a32e18bf4f3a2a7ec7050b3db6cc7bafd87565e370f6695
Image: registry.k8s.io/ingress-nginx/controller:v1.3.0@sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5
Image ID: docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5
Ports: 80/TCP, 443/TCP, 8443/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Args:
/nginx-ingress-controller
--publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
--election-id=ingress-controller-leader
--controller-class=k8s.io/ingress-nginx
--ingress-class=nginx
--configmap=$(POD_NAMESPACE)/ingress-nginx-controller
--validating-webhook=:8443
--validating-webhook-certificate=/usr/local/certificates/cert
--validating-webhook-key=/usr/local/certificates/key
State: Running
Started: Sun, 19 Feb 2023 11:50:07 +0100
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Sat, 18 Feb 2023 11:44:18 +0100
Finished: Sun, 19 Feb 2023 11:49:36 +0100
Ready: True
Restart Count: 132
Requests:
cpu: 100m
memory: 90Mi
Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: ingress-nginx-controller-6bf7bc7f94-krv2c (v1:metadata.name)
POD_NAMESPACE: ingress-nginx (v1:metadata.namespace)
LD_PRELOAD: /usr/local/lib/libmimalloc.so
Mounts:
/usr/local/certificates/ from webhook-cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hpxhw (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
webhook-cert:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-admission
Optional: false
kube-api-access-hpxhw:
ConfigMapOptional: <nil> DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal RELOAD 54m (x3 over 95m) nginx-ingress-controller NGINX reload triggered due to a change in configuration
Here are the logs of the Ingress controller. The last line shows the response upon the curl request from the windows machine. The curl HTTPS request from the linux machine is not being logged here:
k logs -f ingress-nginx-controller-6bf7bc7f94-krv2c -n ingress-nginx
W0219 10:50:08.009613 7 client_config.go:617] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0219 10:50:08.010796 7 main.go:230] "Creating API client" host="https://10.96.0.1:443"
W0219 10:50:30.162337 7 main.go:271] Initial connection to the Kubernetes API server was retried 1 times.
I0219 10:50:30.162386 7 main.go:274] "Running in Kubernetes cluster" major="1" minor="24" git="v1.24.1" state="clean" commit="3ddd0f45aa91e2f30c70734b175631bec5b5825a" platform="linux/amd64"
I0219 10:50:30.278395 7 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0219 10:50:30.299891 7 ssl.go:531] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0219 10:50:30.314983 7 nginx.go:258] "Starting NGINX Ingress controller"
I0219 10:50:30.321843 7 event.go:285] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"407f816e-98b9-4ab6-a08f-95efafa21ddc", APIVersion:"v1", ResourceVersion:"612", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0219 10:50:31.516611 7 nginx.go:301] "Starting NGINX process"
I0219 10:50:31.516698 7 leaderelection.go:248] attempting to acquire leader lease ingress-nginx/ingress-controller-leader...
I0219 10:50:31.517522 7 nginx.go:321] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0219 10:50:31.517893 7 controller.go:167] "Configuration changes detected, backend reload required"
I0219 10:50:31.543621 7 leaderelection.go:258] successfully acquired lease ingress-nginx/ingress-controller-leader
I0219 10:50:31.543648 7 status.go:84] "New leader elected" identity="ingress-nginx-controller-6bf7bc7f94-krv2c"
I0219 10:50:31.610775 7 controller.go:184] "Backend successfully reloaded"
I0219 10:50:31.610910 7 controller.go:195] "Initial sync, sleeping for 1 second"
192.168.65.4 - - [19/Feb/2023:13:09:46 +0000] "POST /api/users/signin/ HTTP/1.1" 200 150 "-" "curl/7.83.1" 259 0.054 [default-gts-auth-srv-3000] [] 10.1.28.246:3000 150 0.050 200 5b8761d90f76b4c160b0a2effc980b85
So my questions are:
-
Is there something wrong with using a fake certificate in the Kubernetes setup? Since curl/postman requests work from the windows machine (outside the WSL2 layer), I would think it's just a matter of configuring the Windows firewall to make HTTPS request from other machines in my LAN work as well. However, turning off the firewall proved, that the requests land in nirvana and the ingress does not log anything.
-
What needs to be configured in the Ingress to properly handle the HTTPS request from the outside of that windows machine?
-
I'm lacking knowledge about tools to properly track the request that somewhere dies either on its way (unlikely) or inside the cluster.
Any suggestions on how to get a HTTPS request work from another machine within my LAN are very welcome.
Update: On another windows machine (Windows 10 Pro) with the same WSL2 / Docker desktop / Kubernetes cluster setup, the request from the Linux machine is being answered with a 404 HTML document from NGINX:
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
Analogously with the request from the Linux to the Win11 machine, the request here does not produce any logs in the Ingress pod, either.
Comments
Post a Comment