Helm
Index
Troubleshooting and Debugging
The first step in troubleshooting is triage. What is the problem? Is it your Pods or your Service?
Exercise: Debugging Pods
- Firt we need a pod for debugging, use the following commands:
yaml/pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: my-pod
spec:
containers:
- name: container-nginx
image: nginx:stablee
ports:
- containerPort: 80
$ kubectl apply -f yaml/my-pod.yaml
- After applying wait for a second and run the commands below
$ kubectl describe pod my-pod
Name: my-pod
Namespace: default
Priority: 0
PriorityClassName: <none>
+
+
+
+
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 14s default-scheduler Successfully assigned default/my-pod to minikube
Normal Pulling 11s kubelet, minikube Pulling image "nginx:stablee"
Warning Failed 9s kubelet, minikube Failed to pull image "nginx:stablee": rpc error: code = Unknown desc = Error response from daemon: manifest for nginx:stablee not found
Warning Failed 9s kubelet, minikube Error: ErrImagePull
Normal BackOff 8s kubelet, minikube Back-off pulling image "nginx:stablee"
Warning Failed 8s kubelet, minikube Error: ImagePullBackOff
Describing is not just for pods, it can be used many of the character.
- As you can see there is a problem on image so edit and apply the yaml
yaml/my-pod-r.yaml
apiVersion: v1
kind: Pod
metadata:
name: my-pod
spec:
containers:
- name: container-nginx
image: nginx:stable
ports:
- containerPort: 80
Command
$ kubectl apply -f yaml/my-pod-r.yaml
- Lets check if its running
$ kubectl port-forward pods/my-pod 8084:80
- and open localhost:8084
Exercise: Events
For the most part, events are easy to see when you are trying to debug issues for a specific resource. Using kubectl describe pod <podname>
for example will show events at the end of the output for the pod.
Another way to view events is to grab the events from the resources API directly. This can be done using kubectl get events
- Lets get the events
$ kubectl get events
Output
LAST SEEN TYPE REASON OBJECT MESSAGE
45m Normal Pulling pod/my-pod Pulling image "nginx:stablee"
15m Normal BackOff pod/my-pod Back-off pulling image "nginx:stablee"
10m Warning Failed pod/my-pod Error: ImagePullBackOff
These are the old events before we updated the pod.
- Get events with spesific fields
$ kubectl get events --field-selector type=Warning
Output
LAST SEEN TYPE REASON OBJECT MESSAGE
13m Warning Failed pod/my-pod Error: ImagePullBackOff
- Get events for spesific objects
$ kubectl get events --field-selector involvedObject.kind!=Service
Exercise: Debugging Services
Imagine you deployed a service but it does not providing the conatinerized app inside a pod. The first thing
- Update the pod for this section
yaml/my-pod-r-svc.yaml
apiVersion: v1
kind: Pod
metadata:
name: my-pod
labels:
app: nginx
spec:
containers:
- name: container-nginx
image: nginx:stable
ports:
- containerPort: 80
Commands
$ kubectl apply -f yaml/my-pod-r-svc.yaml
- Create a service or just apply from the
/yaml
yaml/svc.yaml
apiVersion: v1
kind: Service
metadata:
name: example-service
labels:
app: example-service
spec:
selector:
app: wrong-label
ports:
- port: 80
nodePort: 32500
targetPort: 80
type: NodePort
$ kubectl apply -f yaml/svc.yaml
Actually it needs to work for pod
my-pod
but the labels are incorrect.
- Lets try to access Nginx app from the browser. Open
localhost:32500
.
Note: for minikube users the host will be
minikube ip
.
- The connection will be refused. For debugging our service list the pods in terms of labels for simulating service selector.
$ kubectl get pod --selector app=wrong-label
The output will be No resource found.
because the selectors are incorrect in our service. This selector can be used more complex situations.
- Just fix the selector with
app: nginx
and update. Use the yaml below or just apply the yaml insideyaml/svc-fix.yaml
yaml/svc-fix.yaml
apiVersion: v1
kind: Service
metadata:
name: example-service
labels:
app: example-service
spec:
selector:
app: nginx
ports:
- port: 80
nodePort: 32500
targetPort: 80
type: NodePort
Command
$ kubectl apply -f yaml/svc-fix.yaml
- Open the browser again and refresh the page.
Docker for desktop users => localhost:32500
For Minikube users => minikubeip:32500
- We got the traditional nginx welcome page and fix the label-selector issue with using selector flag. Lets try to access the service with a creating temp test pod. Create the pod with image busybox
$ kubectl run debug-pod --image busybox:latest -it /bin/sh
- Now you are on the pods shell. Lets try to access the service with name.
/# wget example-service
Output
Connecting to example-service (10.103.180.135:80)
saving to 'index.html'
index.html 100% |***************************************************************************************************************************************************| 612 0:00:00 ETA
'index.html' saved
You succesfully get the index.html
. This means the service and pod working correctly.
Exercise: Debugging Cluster
Try the following commands and examine the outputs.
- Node informations
$ kubectl get nodes -o wide
- Describe the nodes for details
$ kubectl describe node <nodename>
Logging
Exercise: Basic Logging in Kubernetes
- Create a pod that generates logs.
yaml/log-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: counter
spec:
containers:
- name: count
image: busybox
args: [/bin/sh, -c,
'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done']
Command
$ kubectl apply -f yaml/log-pod.yaml
- Look at the logs generated by the pod
$ kubectl logs counter