who/where actually work liveness probe in kubernetes?

2/15/2019

in my kubernetes cluster, http liveness probe always failed with this message

Liveness probe failed: Get http://10.233.90.72:8080/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

so, coredns and kubernetes-dashboard (any other using http liveness probe) pods being infinitely restart.

while pod running (between events start and restart), i check the endpoints for the pod with executing command curl http://10.233.90.72:8080/health on the busyboxplus pod. this command are working normally, i can see OK return. but liveness probe still failed. pod is restarting...

in this situation, i want to debug liveness probe, but i don't have any idea who/where actually work liveness probe in kubernetes? is this pod? or node?

how can i debug liveness probe? does anyone have same issue..?

please advice for me.

kubectl version:
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

version info:
 OS: Ubuntu 18.04
 Kubernetes: 1.13.3
 Docker: 18.09.2

thanks in advance

-- GwiYeong Jeong
kubernetes
kubernetes-cluster
ubuntu-18.04

1 Answer

2/15/2019

Did you check the DNS already ? using the busybox:1.28 try to execute dns lookup to the pod and see what you get.

nslookup pod-ip-in-dash.pod.cluster.local

Another thing you can do which you might already did, check in kube-system if coredns pod is running.

Let me know how went,

-- Angel Ramirez
Source: StackOverflow