Certified Kubernetes Administrator

Kubernetes: Troubleshooting
 Learning Objectives

By the end of this lesson, you will be able to:

Check pod status, logs, and events

Check control panel failure status in pod and service

Check node status and describe a node to get the Kubernetes

environment status

Use top command to check node performance

Check the status of master and cert connectivity, service

connectivity, and nslookup

Perform kube-proxy check, CNI plugin check, and network

failure check
Application Failure
 Check Pod Status, Logs, and Events

Use the kubectl get pods command to check the state of the pod

Use the kubectl logs counter command to fetch the logs

Use the kubectl get events command to fetch the events for all resources
 Checking Pod Status, Logs, and Events

Problem Statement: You are given a project to check the pod state, logs, and events.
Control Panel Failure
 Checking Control Panel Failure Status

Problem Statement: You are given a project to check control panel failure status in pod and
service.
Worker Node Failure
 Checking Node Status

The kubectl describe node <insert-node-name-here> command is used to check the status of
a node. It displays the node status and a few other details.

The node status contains the following information:

Addresses Conditions

Capacity and Allocatable Info

 Top Command: Checking Node Performance

The top command kubectl top [options] is used to see the resource consumption of the nodes
or pods.

It is important that Heapster is configured correctly and is working on the server.

Below are a few options inherited from the parent commands:

⮚ --alsologtostderr=false ⮚ --context=""
⮚ --application-metrics-count-limit=100 ⮚ --enable-load-reader=false
⮚ --as="" ⮚ --kubeconfig=""
⮚ -as-group=[] ⮚ --log-dir=""
⮚ --azure-container-registry-config="" ⮚ -log-file=""
⮚ --boot-id-file="/proc/sys/kernel/random/boot_id" ⮚ --logtostderr=true
⮚ --cache-dir="/builddir/.kube/http-cache" ⮚ --profile="none"
⮚ --certificate-authority="" ⮚ --skip-headers=false
⮚ --client-certificate="" ⮚ --token=""
⮚ --cloud-provider-gce-lb-src- ⮚ --username=""
⮚ --cluster=“”” ⮚ --version=false
 Master and Cert Connectivity Status

The master and cert connectivity terminates at the kubelet’s HTTP endpoints.

The --kubelet-certificate-authority flag is used to provide the apiserver with a root certificate bundle.

The master and cert connectivity status is used to:

⮚ Fetch logs for pods

⮚ Attach to running pods
⮚ Provide the kubelet’s port-forwarding functionality
 Checking Node Status

Problem Statement: You are given a project to check node status and describe a node to
know the status of the Kubernetes environment.
 Top Command to Check Node Performance

Problem Statement: You are given a project to demonstrate the use of the top command to
check node performance.
 Master and Cert Connectivity Status

Problem Statement: You are given a project to check the status of master and cert
connectivity.
 Worker Node Failure Check

Problem Statement: You are given a project to perform the worker node failure check.
Network Failure
 Service Connectivity Check

Command to check the connectivity of a service:

kubectl expose deployment hostnames --port=80 --target-port=9376

service/hostnames exposed

Command to check the details of the existing services:

kubectl get svc hostnames

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hostnames ClusterIP 10.0.1.175 <none> 80/TCP 5s
 nslookup Check

nslookup can be used to check the working of the DNS.

You can execute nslookup in the environment once the pod is running. You will see the
following if the DNS is running:

kubectl exec -ti busybox -- nslookup kubernetes.default

Server: 10.0.0.10
Address 1: 10.0.0.10
Name: kubernetes.default
Address 1: 10.0.0.1
 nslookup Check

Things to check if the nslookup check fails:

⮚ Local DNS configuration

⮚ Running of DNS pod
⮚ Errors in the DNS pod
⮚ Is DNS service up?
⮚ Are DNS endpoints exposed?
⮚ Are DNS queries being processed?
 Kube-Proxy Check and CNI Plugin Check

The kube-proxy network runs on each node. The Kube-proxy [flag] is used for communicating with the
master node and routing.

The CNI is responsible to adhere to the appc/CNI specification, designed for interoperability.

The --network-plugin=cni command-line option is used to select the CNI plugin.

 Service Connectivity Check

Problem Statement: You are given a project to perform a service connectivity check.
 nslookup Check

Problem Statement: You are given a project to perform a nslookup check.

 Kube-Proxy Check and CNI Plugin Check

Problem Statement: You are given a project to perform a kube-proxy check and CNI plugin
check.
 Network Failure Check

Problem Statement: You are given a project to perform a network failure check.
 Key Takeaways

You are now able to:

Check pods, logs, and events

Check control panel failure status in pod and service

Check node status and describe a node to get the Kubernetes

environment status

Use top command to check node performance

Check the status of master and cert connectivity, service

connectivity, and nslookup

Perform kube-proxy check, CNI plugin check, and network

failure check
Knowledge Check
Knowledge
Check
_____________ command is used to check the state of the pod.
1

a. kubectl get pods

b. kubectl logs counter

c. kubectl get events

d. kubectl describe node

 Knowledge
Check
_____________ command is used to check the state of the pod.
1

a. kubectl get pods

b. kubectl logs counter

c. kubectl get events

d. kubectl describe node

The correct answer is a

kubectl get pods command is used to check the state of the pod.
Knowledge
Check
Which of the following can be used to check the working of DNS?
2

a. Top command

b. nslookup

c. kubectl get pods

d. kubectl describe node

 Knowledge
Check
Which of the following can be used to check the working of DNS?
2

a. Top command

b. nslookup

c. kubectl get pods

d. kubectl describe node

The correct answer is b

nslookup can be used to check the working of DNS.

Knowledge
Check
_______________ flag is used to provide the apiserver with a root certificate bundle.
3

a. --kubectl describe node

b. --kubelet-certificate-authority

c. --Kubelet get svc hostnames

d. --kubelet exec -ti busybox -- nslookup kubernetes.default

 Knowledge
Check
_______________ flag is used to provide the apiserver with a root certificate bundle.
3

a. --kubectl describe node

b. --kubelet-certificate-authority

c. --Kubelet get svc hostnames

d. --kubelet exec -ti busybox -- nslookup kubernetes.default

The correct answer is b

--kubelet-certificate-authority flag is used to provide the apiserver with a root certificate bundle.
Knowledge
Check
Which of the following commands is used to check the existence of the service?
4

a. kubectl expose deployment hostnames

b. kubelet-certificate-authority

c. kubectl get svc hostnames

d. Kube-proxy [flag]
 Knowledge
Check
Which of the following commands is used to check the existence of the service?
4

a. kubectl expose deployment hostnames

b. kubelet-certificate-authority

c. kubectl get svc hostnames

d. Kube-proxy [flag]

The correct answer is c

kubectl get svc hostnames command is used to check the existence of the service.
Knowledge
Check
What is the purpose of fetching the master and cert connectivity status?
5

a. To fetch logs for pods

b. To attach to running pods

c. To provide the kubelet’s port-forwarding functionality

d. All of the above

 Knowledge
Check
What is the purpose of fetching the master and cert connectivity status?
5

a. To fetch logs for pods

b. To attach to running pods

c. To provide the kubelet’s port-forwarding functionality

d. All of the above

The correct answer is d

The master and cert connectivity status is used for fetching logs for pods, attaching to running pods, and
providing the kubelet’s port-forwarding functionality.
Problem Statement: While developing a highly scalable application, real
challenges will come into picture during the deployment of that application
into production or real-time data scenario. Having the application stuck
while working in normal case scenarios in real-time production
environment is one of the biggest drawbacks for any product after
development. Kubernetes plays an important role in dubbing the tool once
your application deployment is in the production environment.

Objective: Once your application is running, you’ll inevitably need to

debug the problems with it. Use Kubernetes to debug your application.