Cannot access Kubernetes endpoint within WSL2 Docker Desktop from Linux machine

By -

I've setup an authentication service in a Kubernetes cluster which lives in a Docker Desktop + WSL2 environment on a Windows 11 Pro machine.

It can be reached, e.g. with a CURL or PostMan request from within the same Windows machine outside that WSL2 layer, but it cannot be accessed from a Linux machine within the same LAN network.

In order to verify that a connectivity to that auth service actually works, I've setup a service with type "NodePort". With the external port obtained from 'kubectl get svc' the curl requests both work fine from the Windows and Linux machine, and the service responds with the proper answer.

However, note that the NodePort service completely circumvents the TSL/SSL of the K8s ingress which comes with a fake certificate, so the request - although successfully - only works for requests over HTTP.

Because I need to work with cookies in the response from the auth service, requests over HTTPS are crucial, however the K8s Ingress does not send a response at all.

On the windows machine, I've opened relevant ports 80, 443 (and 22 just to be sure), and just to be sure, I also disabled the windows firewall temporarily to see if it's actually blocking the connection attempt. With a disabled firewall, the request still doesn't come through so to me it looks like the Ingress doesn't know how to answer the request (or silently drops it).

But before going into any assumptions, let me give you an overview of the machines and the POST requests between them:

enter image description here

The auth deployment and service are defined as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
    name: gts-auth-depl
spec:
    replicas: 1
    selector:
        matchLabels:
            app: gts-auth
    template:
        metadata:
            labels:
                app: gts-auth
        spec:
            containers:
                - name: gts-auth
                  image: creativewarlock/gts-auth
                  imagePullPolicy: IfNotPresent
                  env:
                      - name: MONGO_URI
                        value: 'mongodb://gts-auth-mongo-srv:27017/auth'
                      - name: JWT_KEY
                        valueFrom:
                            secretKeyRef:
                                name: gts-jwt-secret
                                key: JWT_KEY
            imagePullSecrets:
                - name: docker-hub
---
apiVersion: v1
kind: Service
metadata:
    name: gts-auth-srv
spec:
    type: ClusterIP
    selector:
        app: gts-auth
    ports:
        - name: gts-auth
          protocol: TCP
          port: 3000
          targetPort: 3000

The ingress for the auth and web client (not relevant though, but just to show):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
    name: gts-ingress-srv
    annotations:
        kubernetes.io/ingress.class: nginx
        nginx.ingress.kubernetes.io/use-regex: 'true'
spec:
    rules:
        - host: gts.dev # Entry in System32/etc/hosts: 127.0.0.1 gts.dev (alternatively: localhost)
          http:
              paths:
                  - path: /api/users/?(.*)    # will match /api/users/signin
                    pathType: Prefix
                    backend:
                        service:
                            name: gts-auth-srv
                            port:
                                number: 3000
                  - path: /?(.*)                # map all other requests to web client
                    pathType: Prefix
                    backend:
                        service:
                            name: gts-client-srv
                            port:
                                number: 8080

The NodePort service for the external / direct access to the auth service:

apiVersion: v1
kind: Service
metadata:
    name: gts-auth-srv-ext
spec:
    type: NodePort
    selector:
        app: gts-auth
    ports:
        - name: gts-auth-srv
          protocol: TCP
          port: 3000
          targetPort: 3000

Here's how Kubernetes describes the Ingress:

k describe pod ingress-nginx-controller-6bf7bc7f94-krv2c -n ingress
-nginx
Name:         ingress-nginx-controller-6bf7bc7f94-krv2c
Namespace:    ingress-nginx
Priority:     0
Node:         docker-desktop/192.168.65.4
Start Time:   Thu, 11 Aug 2022 12:31:42 +0200
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/name=ingress-nginx
              pod-template-hash=6bf7bc7f94
Annotations:  <none>
Status:       Running
IP:           10.1.28.235
IPs:
  IP:           10.1.28.235
Controlled By:  ReplicaSet/ingress-nginx-controller-6bf7bc7f94       
Containers:
  controller:
    Container ID:  docker://0cef9bec68d309011a32e18bf4f3a2a7ec7050b3db6cc7bafd87565e370f6695
    Image:         registry.k8s.io/ingress-nginx/controller:v1.3.0@sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5
    Image ID:      docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller    
      --election-id=ingress-controller-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert  
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Sun, 19 Feb 2023 11:50:07 +0100
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Sat, 18 Feb 2023 11:44:18 +0100
      Finished:     Sun, 19 Feb 2023 11:49:36 +0100
    Ready:          True
    Restart Count:  132
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-6bf7bc7f94-krv2c (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hpxhw (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-hpxhw:
    ConfigMapOptional:       <nil>    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s      
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s    
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  RELOAD  54m (x3 over 95m)  nginx-ingress-controller  NGINX reload triggered due to a change in configuration

Here are the logs of the Ingress controller. The last line shows the response upon the curl request from the windows machine. The curl HTTPS request from the linux machine is not being logged here:

k logs -f ingress-nginx-controller-6bf7bc7f94-krv2c -n ingress-nginx

W0219 10:50:08.009613       7 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0219 10:50:08.010796       7 main.go:230] "Creating API client" host="https://10.96.0.1:443"
W0219 10:50:30.162337       7 main.go:271] Initial connection to the Kubernetes API server was retried 1 times.   
I0219 10:50:30.162386       7 main.go:274] "Running in Kubernetes cluster" major="1" minor="24" git="v1.24.1" state="clean" commit="3ddd0f45aa91e2f30c70734b175631bec5b5825a" platform="linux/amd64"
I0219 10:50:30.278395       7 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0219 10:50:30.299891       7 ssl.go:531] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0219 10:50:30.314983       7 nginx.go:258] "Starting NGINX Ingress controller"
I0219 10:50:30.321843       7 event.go:285] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"407f816e-98b9-4ab6-a08f-95efafa21ddc", APIVersion:"v1", ResourceVersion:"612", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0219 10:50:31.516611       7 nginx.go:301] "Starting NGINX process"
I0219 10:50:31.516698       7 leaderelection.go:248] attempting to acquire leader lease ingress-nginx/ingress-controller-leader...
I0219 10:50:31.517522       7 nginx.go:321] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"    
I0219 10:50:31.517893       7 controller.go:167] "Configuration changes detected, backend reload required"        
I0219 10:50:31.543621       7 leaderelection.go:258] successfully acquired lease ingress-nginx/ingress-controller-leader
I0219 10:50:31.543648       7 status.go:84] "New leader elected" identity="ingress-nginx-controller-6bf7bc7f94-krv2c"
I0219 10:50:31.610775       7 controller.go:184] "Backend successfully reloaded"
I0219 10:50:31.610910       7 controller.go:195] "Initial sync, sleeping for 1 second"
192.168.65.4 - - [19/Feb/2023:13:09:46 +0000] "POST /api/users/signin/ HTTP/1.1" 200 150 "-" "curl/7.83.1" 259 0.054 [default-gts-auth-srv-3000] [] 10.1.28.246:3000 150 0.050 200 5b8761d90f76b4c160b0a2effc980b85

So my questions are:

  • Is there something wrong with using a fake certificate in the Kubernetes setup? Since curl/postman requests work from the windows machine (outside the WSL2 layer), I would think it's just a matter of configuring the Windows firewall to make HTTPS request from other machines in my LAN work as well. However, turning off the firewall proved, that the requests land in nirvana and the ingress does not log anything.

  • What needs to be configured in the Ingress to properly handle the HTTPS request from the outside of that windows machine?

  • I'm lacking knowledge about tools to properly track the request that somewhere dies either on its way (unlikely) or inside the cluster.

Any suggestions on how to get a HTTPS request work from another machine within my LAN are very welcome.

Update: On another windows machine (Windows 10 Pro) with the same WSL2 / Docker desktop / Kubernetes cluster setup, the request from the Linux machine is being answered with a 404 HTML document from NGINX:

<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

Analogously with the request from the Linux to the Win11 machine, the request here does not produce any logs in the Ingress pod, either.



Comments

Post a Comment

Popular posts from this blog

Today Walkin 14th-Sept

By -
Image
14th Sept 2017 ( GO Through all List) Read Walkin details properly , Try to call company before attending if possible !!! ==================================== 1. Myntra through fresherworld (Call Based , Venue shared to shortlisted candidates) _____________________________ 2. Magna : Hiring for Client Harman : C++
Read more

Spring Elasticsearch Operations

By -
Elasticsearch Operations Spring Data Elasticsearch uses two interfaces to define the operations that can be called against an Elasticsearch index. These are ElasticsearchOperations and ReactiveElasticsearchOperations. Whereas the first is used with the classic synchronous implementations, the second one uses reactive infrastructure. The default implementations of the interfaces offer: Read/Write mapping support for domain types. A rich query and criteria api. Resource management and Exception translation. ElasticsearchTemplate The ElasticsearchTemplate is an implementation of the ElasticsearchOperations interface using the Transport Client. @Configuration public class TransportClientConfig extends ElasticsearchConfigurationSupport {   @Bean   public Client elasticsearchClient() throws UnknownHostException {                      Settings settings = Settings.builder().put("cluster.name", "elasticsearch")....
Read more

Hibernate Search - Elasticsearch with JSON manipulation

By -
Elasticsearch with JSON manipulation Elasticsearch ships with many features. It is possible that at some point, one feature you need will not be exposed by the Search DSL. To work around such limitations, Hibernate Search provides ways to: Transform the HTTP request sent to Elasticsearch for search queries. Read the raw JSON of the HTTP response received from Elasticsearch for search queries. Direct changes to the HTTP request may conflict with Hibernate Search features and be supported differently by different versions of Elasticsearch. Similarly, the content of the HTTP response may change depending on the version of Elasticsearch, depending on which Hibernate Search features are used, and even depending on how Hibernate Search features are implemented. Thus, features relying on direct access to HTTP requests or responses cannot be guaranteed to continue to work when upgrading Hibernate Search, even for micro upgrades (x.y.z to x.y.(z+1)). Use this at your own risk....
Read more