Skip to content
Snippets Groups Projects
Berk Yavuz's avatar
Berk Yavuz authored
7ec5c53d
History
Name Last commit Last update
yaml
README.md

Helm

Index


Troubleshooting and Debugging

The first step in troubleshooting is triage. What is the problem? Is it your Pods or your Service?

Exercise: Debugging Pods

  1. Firt we need a pod for debugging, use the following commands:

yaml/pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: container-nginx
    image: nginx:stablee
    ports:
    - containerPort: 80
$ kubectl apply -f yaml/my-pod.yaml
  1. After applying wait for a second and run the commands below
$ kubectl describe pod my-pod
Name:               my-pod
Namespace:          default
Priority:           0
PriorityClassName:  <none>
+
+
+
+
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  14s   default-scheduler  Successfully assigned default/my-pod to minikube
  Normal   Pulling    11s   kubelet, minikube  Pulling image "nginx:stablee"
  Warning  Failed     9s    kubelet, minikube  Failed to pull image "nginx:stablee": rpc error: code = Unknown desc = Error response from daemon: manifest for nginx:stablee not found
  Warning  Failed     9s    kubelet, minikube  Error: ErrImagePull
  Normal   BackOff    8s    kubelet, minikube  Back-off pulling image "nginx:stablee"
  Warning  Failed     8s    kubelet, minikube  Error: ImagePullBackOff

Describing is not just for pods, it can be used many of the character.

  1. As you can see there is a problem on image so edit and apply the yaml

yaml/my-pod-r.yaml

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: container-nginx
    image: nginx:stable
    ports:
    - containerPort: 80

Command

$ kubectl apply -f yaml/my-pod-r.yaml
  1. Lets check if its running
$ kubectl port-forward pods/my-pod 8084:80
  1. and open localhost:8084

Exercise: Events

For the most part, events are easy to see when you are trying to debug issues for a specific resource. Using kubectl describe pod <podname> for example will show events at the end of the output for the pod. Another way to view events is to grab the events from the resources API directly. This can be done using kubectl get events

  1. Lets get the events
$ kubectl get events

Output

LAST SEEN   TYPE      REASON    OBJECT       MESSAGE
45m         Normal    Pulling   pod/my-pod   Pulling image "nginx:stablee"
15m         Normal    BackOff   pod/my-pod   Back-off pulling image "nginx:stablee"
10m         Warning   Failed    pod/my-pod   Error: ImagePullBackOff

These are the old events before we updated the pod.

  1. Get events with spesific fields
$ kubectl get events --field-selector type=Warning

Output

LAST SEEN   TYPE      REASON   OBJECT       MESSAGE
13m         Warning   Failed   pod/my-pod   Error: ImagePullBackOff
  1. Get events for spesific objects
$ kubectl get events --field-selector involvedObject.kind!=Service

Exercise: Debugging Services

Imagine you deployed a service but it does not providing the conatinerized app inside a pod. The first thing

  1. Update the pod for this section

yaml/my-pod-r-svc.yaml

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
  labels:
    app: nginx
spec:
  containers:
  - name: container-nginx
    image: nginx:stable
    ports:
    - containerPort: 80

Commands

$ kubectl apply -f yaml/my-pod-r-svc.yaml
  1. Create a service or just apply from the /yaml

yaml/svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: example-service
  labels:
    app: example-service
spec:
  selector:
    app: wrong-label
  ports:
  - port: 80
    nodePort: 32500
    targetPort: 80
  type: NodePort

$ kubectl apply -f yaml/svc.yaml

Actually it needs to work for pod my-pod but the labels are incorrect.

  1. Lets try to access Nginx app from the browser. Open localhost:32500.

Note: for minikube users the host will be minikube ip.

  1. The connection will be refused. For debugging our service list the pods in terms of labels for simulating service selector.
$ kubectl get pod --selector app=wrong-label

The output will be No resource found. because the selectors are incorrect in our service. This selector can be used more complex situations.

  1. Just fix the selector with app: nginx and update. Use the yaml below or just apply the yaml inside yaml/svc-fix.yaml

yaml/svc-fix.yaml

apiVersion: v1
kind: Service
metadata:
  name: example-service
  labels:
    app: example-service
spec:
  selector:
    app: nginx
  ports:
  - port: 80
    nodePort: 32500
    targetPort: 80
  type: NodePort

Command

$ kubectl apply -f yaml/svc-fix.yaml
  1. Open the browser again and refresh the page.

Docker for desktop users => localhost:32500

For Minikube users => minikubeip:32500

  1. We got the traditional nginx welcome page and fix the label-selector issue with using selector flag. Lets try to access the service with a creating temp test pod. Create the pod with image busybox
$ kubectl run debug-pod --image busybox:latest -it /bin/sh
  1. Now you are on the pods shell. Lets try to access the service with name.
/# wget example-service

Output

Connecting to example-service (10.103.180.135:80)
saving to 'index.html'
index.html           100% |***************************************************************************************************************************************************|   612  0:00:00 ETA
'index.html' saved

You succesfully get the index.html. This means the service and pod working correctly.


Exercise: Debugging Cluster

Try the following commands and examine the outputs.

  1. Node informations
$ kubectl get nodes -o wide
  1. Describe the nodes for details
$ kubectl describe node <nodename>

Logging

Exercise: Basic Logging in Kubernetes

  1. Create a pod that generates logs.

yaml/log-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  containers:
  - name: count
    image: busybox
    args: [/bin/sh, -c,
            'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done']

Command

$ kubectl apply -f yaml/log-pod.yaml
  1. Look at the logs generated by the pod
$ kubectl logs counter