hello everyone my name is abishek and welcome back to my Channel today is episode

one of kubet is troubleshooting Zero to Hero series and in this video we will
learn one of the most common beginner level kubernetes error that is image pull
back off so this might sound simple but there are multiple ways in which someone
can land into image back off and in this video we will learn what are those
multiple ways how to reproduce image pull backoff on your local kubernetes
cluster how to troubleshoot image pull backoff errors and finally how to fix it
on your kubernetes clusters so it's going to be a very interesting video please
try to watch it till the end because this is first episode I want to address a
couple of things number one is this series will focus on most common kubernetes
errors that develops Engineers face on their day-to-day life we will start with
image pull back off and going ahead we will increase the complexity of the
kubernetes troubleshooting second Point all the material notes commands that I'm
going to use in in this particular series will be shared through this GitHub
repository for example in today's video we will learn about image pull backoff so
if you go to this folder you will find the complete notes related to image pull
backoff and in this MD files you will also find the commands that I'm going to
use in this video so you can start the repository folk it watch it to get the
continuous updates now let's move to the Whiteboard and start understanding
image P back off there is a reason why I want to start this kubernetes
troubleshooting series with image pull back off error because many of us when we
have started our journey with kubernetes the first thing that we have done is to
deploy a pod onto the kubernetes cluster and this error is actually related to
this particular task but what exactly is this as the name suggests this error is
related to pulling a container image onto the kubernetes cluster so when you
deploy your application onto the the kubernetes cluster it can be through a p
directly or a deployment stateful set demon set or a replica set in any of the
cases when you try to deploy your application as a container image onto the
kubernetes cluster you might run into this error called image pull back offs but
abishek what are those scenarios so there are two scenarios I can broadly
categorize into first scenario is when you provide an invalid image name in the
same category I will put a non-existent image name what are these things let's
say you want to deploy an engine application onto the kubernetes cluster and you
have taken example from the official kubernetes dogs so here we have an example
for deploying engine application we have everything so I will just copy this and
I will paste it on my notepad and using Cube CTL I will try to apply this but
while copying this now I missed this particular thing instead of X or Som of I
deleted this or it got overridden and the image name got changed to ngi n y
instead of ngi NX so this became an invalid image name why because there is no
such image called ngi NY and with the specific tag so here what you are seeing is
a repository it's not an image right so there is no such image called ngi n y in
the docker Hub so that's why you will face this error called image P backoff
because it's an invalid image similarly let's say in your organization there is
an image called f colon 1.1.1 and you are asked to deploy this image F do Fu
colon 1.1.1 which is f is the image name and 1.1.1 is the tag and unfortunately
this was deleted but you have created the deployment and you have deployed your
application which is pointing to a non-existent container image so even in this
case it will result in image pull back of error now these are very simple use
cases there is scenario two where where you are trying to deploy your application
as a container image but the container image is actually private secure what
does that mean so if you go to dockerhub all the images that you see here like
the most of the images are public so you can simply use the docker pull image and
download them but let say you go to my repositories and you will see one of the
images that are tagged as private not tagged as private but they are marked as
private because this repository that I have here image pull demo I have
intentionally marked this as private so that no one without my authorization can
download this image so even in such cases you will run into image pull back off
error but abishek what if no my organization wants to deploy this image or what
if I want to deploy the image which I have made secure in my dockerhub for that
case what you need to do is you need to use a concept called image P secret so
in your kubernetes deployment manifest file if this is my kubernetes deployment
and assume enginex 1.14.2 is a private image so what I need to do is I will just
copy this I will take my terminal just to show you people so let's say I'm
creating a file called Vim enginex deploy yaml and if this particular image is a
private or secure image what I will do is I will add this particular thing called
image P secret and I will pass the name of the secret which stores my Docker
credentials why Docker credentials because we are talking about pulling an image
from from dockerhub what if you want to pull an image from ECR that is elastic
container registry of AWS again the process is exactly the same here you will
provide the secret name which will have your AWS credentials if it is azure you
just need to pass your Azure credentials secret how we will create this secret I
will show in a couple of minutes but what what I want to make sure is you all
understand the cases where you will see the image pull back of error you can see
in this case as well and you can see in this case as well so whenever you are
seeing a image pull back off error when you do Cube CTL get pods you need to
also understand did you land into image pull backup because of this or because of
this how will you identify that for that there are a couple of ways one is you
can use the describe command which I'm going to show or you can also use the
events command both of them are Cube Cil commands one advice that I can give you
when you start troubleshooting or learning troubleshooting of kubernetes just
have this page as a quick reference this is Cube CTL quick reference chart just
search for cube CTL quick reference and here you will have all sort of cube CTL
commands so even if you forget some Cube CTL command you can just refer this page
and you can get that ccdl command now let's understand what exactly is the second
part of this error called back off so we clearly understood what is image pull
but how about back off it's very very simple you might see this term called back
off when you are learning kubernetes or while you are performing some activities
on kubernetes it is quite common what exactly this means is back off DeLay So
initially let's take this one you you have provided an invalid image name on your
kubernetes deployment. yaml file so kubernetes will not throw the image pull back
off error directly the first thing that it throws is error image pull error
image pull that means first error that you see even before image pull back off is
this error called error image pull and after kubernetes gets this error from next
time what it will do is it will wait for some time because this error can be
because of network issues as well the image name might be correct but because of
some network issues cuet which is running your pod might not be able to pull the
image or there can be some transient issues intermittent issues so what
kubernetes will do is it will wait for some time and after some time it will try
one more time and again after some time waiting for some time it will try one
more time this is a continuous process where kubernetes will incrementally
increase the weight time that is initially it might wait for 5 seconds then it
will wait for 20 seconds let's say then it will eventually move towards waiting
till 5 minutes to pull the container image so kubernetes will not just attempt to
pull your container image once it will attempt multiple times and each time it
will add a delay incrementally and try to pull the image this process is called
as back off delay that's why this error is called as image pull off right there
are two parts of it and both of them are very critical to understand so I'll just
repeat it one more time image pull back off so this error is related to pulling
a container image onto your kubernetes cluster if the image is invalid or non-
existent or the image that you are trying to pull you don't have access to it
that is permission denied in any of these cases you will run into image P backoff
error and this error is called as image pull back off because initial error that
kubernetes throws is error image pull and later what kubernetes does is it will
not leave in the first attempt it will try to pull the image after a certain time
maybe 10 seconds let's say and then it will wait incrementally let's say now for
20 seconds and give one more attempt then after 40 seconds and give one more
attempt so this incremental process of delay of pulling the image is called as
back off that's when you will notice the error pull image error is converted to
image pull back off now let's try to do this practically on our kubernetes
cluster if you don't have a kubernetes cluster the simple thing that you can do
is start using mini Cube so you can just search for mini Cube start and you will
land into this page whether you
are on Linux Mac OS or Windows let's say if your laptop is not compatible you
can also create an E2 instance with 2 CPU and 4GB Ram requirement you can select
Linux and you can simply execute this commands if you have a Mac OS just change
it and if your architecture is arm you need to select the right Fields here again
for Windows so you can simply copy paste these commands and your kubernetes
cluster will be up and running once you execute mini Cube start so I have my
kubernetes cluster up and running there is a reason why I'm not talking about the
kubernetes basics because we have already completed kubernetes 0 To Hero where we
have covered all the kubernetes fundamentals and Basics it will just be
repetitive so my mini Cube cluster if I just do mini Cube state St is up and
running so I don't have to start the cluster again now let's copy the same
example so first let me see if there is any deployment on the cluster no there is
no deployment so we will add this file enginex deploy do yaml let's first see if
we can deploy this image or not so I'll just do ql apply minus F ngx deploy yaml
you can also create some shortcuts to cube CTL by creating an alas which we will
do as we proceed with the next videos so if I do Cube CTL get pods you will see
the containers they are currently in the container creating state I'm using this
command called minus W I mean this is just a field to cube Cil and what it will
do it will continuously watch and update us about the status of the container
that you are deploying through a pod it is same as running the cube CDL get pods
multiple times instead of running it multiple times you can just execute Cube CTL
get pods minus w and it will report the status continuously to you so it said all
the parts are in the running status that means this deployment is a valid one now
what I'm going to do is I will just delete Cube CTL delete deploy engine X
deployment and intentionally let's say I will modify this to ngi NY as I've
shown you there is no container image with ngi n on dockerhub which is the
default container registry if you want to point anything else you can simply
change this let's say you want to point it to ECR you want to point it to ACR
from where you want to pull the container image you need to provide the registry
name if you don't provide the regist name you can assume it is as similar as
docker.io engine enginex followed by whatever it is if you use any other
container registries you have to provide the registry name first okay now let's
try to deploy this Cube CTL apply minus F enginex deploy yam again the deployment
is created but when you do Cube CTL get pods let's use minus W see as I've told
you the first error that you will see is error image pull so today we are
learning about both errors error image pull as well as image pull back off
eventually error image pull will be transition to image pull back off this is
what I have explained now let's try to understand image pull back off from the
point of view of a private container repo so what we can do is we will create a
container image we will push that container image onto my dockerhub repository
and we will make that repo private so we will take a simple example I will pull
the same engine X image you can use any image it doesn't matter but just for the
Simplicity I'm using enginex 1.14.2 and next thing that I'll do is I will tag
this container image with my dockerhub uh thing and I will push it to my Docker
Hub and then make the container repo private if you already have a image on your
Docker Hub you can directly make that container repo private okay okay so it's
extracting now I have this container image on my local what I'll do is I will do
Docker tag engine x 1.14.2 with abishek Fi which is my dockerhub username
followed by the repository that I want to create let me create enginex image
demo colum V1 I'm just giving the tag as V1 now I will push this onto my
dockerhub so it will create a Enix IM image demo repository on my Docker hub for
that I'll just do Docker pull when I am going to push it manifest unknown oh
sorry I need to push so when I push this by default the container repo will be
public we will go to our Docker Hub and change it to private if you're using a
free dockerhub uh user then you can only create one repo as private so I think I
already have one what I'm going to do is I will go to the Repository change the
one that I already have to public while doing the demo I actually was trying the
same thing and I made one private so now okay so I made the previous one as
public so now I can change the one that we created as private so this is the one
engine X image demo I'll just go to this settings make private perfect so let's
copy this because we are going to use it onto the deployment file enginex
deploy yaml and I will change the container image this time with this tag should
be V1 and let's delete the previous engine X deployment so let's get rid of
this perfect so now if I do Cub cail get pods I have these parts terminating
probably if I do it one more time the ports are gone now we will do Cube CTL
apply minus F engine X deploy doy again the deployment is created but if we do
cail get pods and try to watch the status of the parts again you will see the
same thing error IM image pull and image pull back off why because we don't have
access to pull that container image what is the process that takes place when you
are trying to deploy this PA so Cube CDL is the one that is sorry cubelet is the
one that is responsible for running the p and when cuet tries to pull that
container image cuet will see if it has access to it it will refer the
deployment. yaml file to see if there is a secret where it can get the username
and password to pull the image from the container registry if cuet does not find
that then it will say that okay I cannot pull the container image there is some
issue and you will see the image pull back off that's what is happening now now
this let's try to understand how to troubleshoot for it we need to create a
secret so if you go back to our kubernetes troubleshooting GitHub repository you
have this file called private image where have explain the same concept and you
can simply use this cctl command it's not a rocket signs all that it is doing
Cube CTL create secret and the secret is of type Docker registry just like we
have a generic secret TLS secret here we are using this Docker registry type
secret name can be anything you have to reference this name inside the
deployment. yaml which I'm going to show so you just have to remember what is the
name and you provide your Docker email then you provide your Docker password
you also provide your Docker username make sure you provide everything correct if
you don't provide things right then your image pull Secrets w't work and the
docker server if you go to the kubernetes documentation and just search for
kubernetes image pull Secrets pull an image from private registry so the docker
server that you have to provide is always the same so if you scroll down you will
find this is the docker registry server that you have to provide if you're using
anything other than dockerhub let's say you're using ECR I'll also tell you what
you need to use as the docker server if if you using ECR don't worry then your
Docker username in my case the username is abishek F5 then you need to provide
your Docker password as well as the email address for the purpose of security let
me just pause the recording and enter this command so I executed the command and
if you do Cube CTL get secrets you should see a secret like this in our case the
name of the secret is demo and see here it says that the secret is of type Docker
config Json perfect now what I have to do is I have to reference this secret very
simple just go back to your deployment file Vim engine deploy yaml and you can
just provide it is also provided in the documentation here this particular thing
called image P secrets and provide the name of the image P Secrets as demo
right so I'm just providing cubet through the deployment the secret which has
Docker credentials the now let's see if this works Cube CTL apply minus F engine
X deploy do yaml let's do cubec get pods minus W so again I'm seeing container
creating let's see if there is any progress if I have provided the right username
and password it should work still the status is the same because if you go back
to the engine deploy yaml the mistake that we have done is we need to provide
abishek fi followed by engine X image demo now let's see if this works Cube CPL
apply you don't have to delete the deployment if you're using Cube CTL apply
command now again let's use Cube CTL get pods so it is trying to create the
container image again so it is terminating the previous ones container is
creating I hope now yeah so the containers are in the running status if you just
cancel this everything and if I quickly show you Cube C get pods all the
containers are in in the running status so this is how you can use image pull
Secrets when you are dealing with private container images this is how you
troubleshoot image pull back off so the image pull back off issue is solved but
there is one more thing okay abishek for dockerhub you have provided the
credentials for the docker Hub and you have explained how to use the docker
username and password what if my organization is using AWS EC elastic container
registry there is nothing to worry it's exactly the same If You observe both the
commands Cube CTL create secret Docker registry followed by the name of the
secret similarly
even here you will follow the same thing you will provide the docker server to
Docker username Docker password and name space if you are interested in this
case the docker server will not be O.D doer but it will be your AWS account
followed by this thing followed by the region followed by Amazon aws.com Docker
username for AWS will always be the same capitals AWS and Docker password you can
retrieve using this particular command and finally if name space is required you
can put the name space now you might ask me how did you understand this like how
do you know that the docker server should be this particular thing it can be you
it can be me anyone has to refer the documentation just like for pulling a secret
from the docker Hub I referred this documentation and came to know that the
docker server is index. docker.io similarly you can also search for your AWS
documentation just say ECR image pull Secrets AWS talk pulling an image and here
you can find the complete details see AWS account ID followed by the region these
are the parameters that have exactly placed here AWS account AWS region and rest
everything is the same if you want for Azure you can search and you can find the
Azure Docker server replacement thank you so much for watching this video in the
next episodes we will learn more kubernetes errors we will also learn some
shortcuts aliases cheat sheets and whatnot with every day we will try to increase
the complexity of kubernetes Errors see you all in the next video take care bye-
bye