Certified Kubernetes Administrator
Kubernetes: Troubleshooting
Learning Objectives
By the end of this lesson, you will be able to:
Check pod status, logs, and events
Check control panel failure status in pod and service
Check node status and describe a node to get the Kubernetes
environment status
Use top command to check node performance
Check the status of master and cert connectivity, service
connectivity, and nslookup
Perform kube-proxy check, CNI plugin check, and network
failure check
Application Failure
Check Pod Status, Logs, and Events
Use the kubectl get pods command to check the state of the pod
Use the kubectl logs counter command to fetch the logs
Use the kubectl get events command to fetch the events for all resources
Checking Pod Status, Logs, and Events
Problem Statement: You are given a project to check the pod state, logs, and events.
Control Panel Failure
Checking Control Panel Failure Status
Problem Statement: You are given a project to check control panel failure status in pod and
service.
Worker Node Failure
Checking Node Status
The kubectl describe node <insert-node-name-here> command is used to check the status of
a node. It displays the node status and a few other details.
The node status contains the following information:
Addresses Conditions
Capacity and Allocatable Info
Top Command: Checking Node Performance
The top command kubectl top [options] is used to see the resource consumption of the nodes
or pods.
It is important that Heapster is configured correctly and is working on the server.
Below are a few options inherited from the parent commands:
⮚ --alsologtostderr=false ⮚ --context=""
⮚ --application-metrics-count-limit=100 ⮚ --enable-load-reader=false
⮚ --as="" ⮚ --kubeconfig=""
⮚ -as-group=[] ⮚ --log-dir=""
⮚ --azure-container-registry-config="" ⮚ -log-file=""
⮚ --boot-id-file="/proc/sys/kernel/random/boot_id" ⮚ --logtostderr=true
⮚ --cache-dir="/builddir/.kube/http-cache" ⮚ --profile="none"
⮚ --certificate-authority="" ⮚ --skip-headers=false
⮚ --client-certificate="" ⮚ --token=""
⮚ --cloud-provider-gce-lb-src- ⮚ --username=""
⮚ --cluster=“”” ⮚ --version=false
Master and Cert Connectivity Status
The master and cert connectivity terminates at the kubelet’s HTTP endpoints.
The --kubelet-certificate-authority flag is used to provide the apiserver with a root certificate bundle.
The master and cert connectivity status is used to:
⮚ Fetch logs for pods
⮚ Attach to running pods
⮚ Provide the kubelet’s port-forwarding functionality
Checking Node Status
Problem Statement: You are given a project to check node status and describe a node to
know the status of the Kubernetes environment.
Top Command to Check Node Performance
Problem Statement: You are given a project to demonstrate the use of the top command to
check node performance.
Master and Cert Connectivity Status
Problem Statement: You are given a project to check the status of master and cert
connectivity.
Worker Node Failure Check
Problem Statement: You are given a project to perform the worker node failure check.
Network Failure
Service Connectivity Check
Command to check the connectivity of a service:
kubectl expose deployment hostnames --port=80 --target-port=9376
service/hostnames exposed
Command to check the details of the existing services:
kubectl get svc hostnames
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hostnames ClusterIP 10.0.1.175 <none> 80/TCP 5s
nslookup Check
nslookup can be used to check the working of the DNS.
You can execute nslookup in the environment once the pod is running. You will see the
following if the DNS is running:
kubectl exec -ti busybox -- nslookup kubernetes.default
Server: 10.0.0.10
Address 1: 10.0.0.10
Name: kubernetes.default
Address 1: 10.0.0.1
nslookup Check
Things to check if the nslookup check fails:
⮚ Local DNS configuration
⮚ Running of DNS pod
⮚ Errors in the DNS pod
⮚ Is DNS service up?
⮚ Are DNS endpoints exposed?
⮚ Are DNS queries being processed?
Kube-Proxy Check and CNI Plugin Check
The kube-proxy network runs on each node. The Kube-proxy [flag] is used for communicating with the
master node and routing.
The CNI is responsible to adhere to the appc/CNI specification, designed for interoperability.
The --network-plugin=cni command-line option is used to select the CNI plugin.
Service Connectivity Check
Problem Statement: You are given a project to perform a service connectivity check.
nslookup Check
Problem Statement: You are given a project to perform a nslookup check.
Kube-Proxy Check and CNI Plugin Check
Problem Statement: You are given a project to perform a kube-proxy check and CNI plugin
check.
Network Failure Check
Problem Statement: You are given a project to perform a network failure check.
Key Takeaways
You are now able to:
Check pods, logs, and events
Check control panel failure status in pod and service
Check node status and describe a node to get the Kubernetes
environment status
Use top command to check node performance
Check the status of master and cert connectivity, service
connectivity, and nslookup
Perform kube-proxy check, CNI plugin check, and network
failure check
Knowledge Check
Knowledge
Check
_____________ command is used to check the state of the pod.
1
a. kubectl get pods
b. kubectl logs counter
c. kubectl get events
d. kubectl describe node
Knowledge
Check
_____________ command is used to check the state of the pod.
1
a. kubectl get pods
b. kubectl logs counter
c. kubectl get events
d. kubectl describe node
The correct answer is a
kubectl get pods command is used to check the state of the pod.
Knowledge
Check
Which of the following can be used to check the working of DNS?
2
a. Top command
b. nslookup
c. kubectl get pods
d. kubectl describe node
Knowledge
Check
Which of the following can be used to check the working of DNS?
2
a. Top command
b. nslookup
c. kubectl get pods
d. kubectl describe node
The correct answer is b
nslookup can be used to check the working of DNS.
Knowledge
Check
_______________ flag is used to provide the apiserver with a root certificate bundle.
3
a. --kubectl describe node
b. --kubelet-certificate-authority
c. --Kubelet get svc hostnames
d. --kubelet exec -ti busybox -- nslookup kubernetes.default
Knowledge
Check
_______________ flag is used to provide the apiserver with a root certificate bundle.
3
a. --kubectl describe node
b. --kubelet-certificate-authority
c. --Kubelet get svc hostnames
d. --kubelet exec -ti busybox -- nslookup kubernetes.default
The correct answer is b
--kubelet-certificate-authority flag is used to provide the apiserver with a root certificate bundle.
Knowledge
Check
Which of the following commands is used to check the existence of the service?
4
a. kubectl expose deployment hostnames
b. kubelet-certificate-authority
c. kubectl get svc hostnames
d. Kube-proxy [flag]
Knowledge
Check
Which of the following commands is used to check the existence of the service?
4
a. kubectl expose deployment hostnames
b. kubelet-certificate-authority
c. kubectl get svc hostnames
d. Kube-proxy [flag]
The correct answer is c
kubectl get svc hostnames command is used to check the existence of the service.
Knowledge
Check
What is the purpose of fetching the master and cert connectivity status?
5
a. To fetch logs for pods
b. To attach to running pods
c. To provide the kubelet’s port-forwarding functionality
d. All of the above
Knowledge
Check
What is the purpose of fetching the master and cert connectivity status?
5
a. To fetch logs for pods
b. To attach to running pods
c. To provide the kubelet’s port-forwarding functionality
d. All of the above
The correct answer is d
The master and cert connectivity status is used for fetching logs for pods, attaching to running pods, and
providing the kubelet’s port-forwarding functionality.
Problem Statement: While developing a highly scalable application, real
challenges will come into picture during the deployment of that application
into production or real-time data scenario. Having the application stuck
while working in normal case scenarios in real-time production
environment is one of the biggest drawbacks for any product after
development. Kubernetes plays an important role in dubbing the tool once
your application deployment is in the production environment.
Objective: Once your application is running, you’ll inevitably need to
debug the problems with it. Use Kubernetes to debug your application.
