K8s and F5 BIG-IP LTM

Integration – Lab Guide

CHAPTER 4
Lab Architecture
Management (10.1.1.0/24)

10.1.1.8 10.1.1.4

External (10.1.10.0/24) Internal (10.1.20.0/24)

F5 BIG-IP
Client K8s Master
10.1.10.8 10.1.10.4 LTM VE 10.1.20.4 10.1.20.5 10.1.1.5

10.1.20.6
K8s Worker1
10.1.1.6

10.1.20.7
K8s Worker2
10.1.1.7

2 | ©2022 F5 CONFIDENTIAL
F5 BIG-IP Requirements
Basic BIG-IP configuration up to LTM and DNS/GTM provisioned.

3 | ©2022 F5 CONFIDENTIAL
F5 BIG-IP VE Requirements
Basic BIG-IP networking configured (VLAN and Self-IP).

Allow Service none (except with VXLAN for ClusterIP mode).

4 | ©2022 F5 CONFIDENTIAL
Access Preparation
At minimum, open the following access:

1) F5 GUI 2) Client CLI

5 | ©2022 F5 CONFIDENTIAL
How to Connect to the Lab – F5 GUI

Username: admin
Password: admin
For GUI access

6 | ©2022 F5 CONFIDENTIAL
Copy+Paste From PPTX to Web Browser
The commands for the lab is provided in this PPTX. One example:

su - ubuntu

7 | ©2022 F5 CONFIDENTIAL
Copy+Paste From PPTX to Web Browser
The commands for the lab is provided in this PPTX. One example:

su - ubuntu

Select and Copy the above command line and Paste the command to the web shell.

You may also type the command directly to the web shell.

Shortcut Key Tips:

• MS Edge – Paste: Ctrl + Shift + V

• Google Chrome – Paste: Ctrl + V

8 | ©2022 F5 CONFIDENTIAL
Ensure You Use "ubuntu" Account
Run the following command to login to "ubuntu" account from the web shell.

su - ubuntu

You should see "ubuntu@client:~$" prompt è

Note: It is a security practice to use normal privilege user account. Only use the privilege elevated user account
when it is really needed.

9 | ©2022 F5 CONFIDENTIAL
Download YAML Files to Local
Use "git clone" to download the YAML files to local storage inside the client VM and copy all cloned YAML files to
the home directory.

git clone https://github.com/halimfaf/F5ASEAN-K8sBootcamp.git

cd F5ASEAN-K8sBootcamp/Lab4/

10 | ©2022 F5 CONFIDENTIAL
Alternative: Command List in TXT
A command list in TXT format is also available from the cloned GitHub repo.

The command list for this lab is lab4-commands-for-copy-paste.txt

You can use "cat", "more", "less", "cat", "head" or "tail" command to read the TXT file. Then you can Copy+Paste
from and to the same web shell.

11 | ©2022 F5 CONFIDENTIAL
 Lab 4 – CIS NodePort

CONFIDENTIAL
Lab 4 – CIS NodePort
Lab Tasks

4.1) Confirm AS3 Installation in BIG-IP

4.2) F5 CIS Installation in K8s

4.3) F5 IPAM Installation in K8s

4.4) Create a deployment in K8s

4.5) Send test traffic and inspect it

13 | ©2022 F5
 Task 4.1)
Confirm AS3 Installation in BIG-IP

CONFIDENTIAL
Task 4.1) Confirm AS3 Installation in BIG-IP
Bring up the F5 GUI and follow the steps below:

15 | ©2022 F5 CONFIDENTIAL
Task 4.1) Confirm AS3 Installation in BIG-IP
If the AS3 is not yet installed, you can installed it by following the installation guide:
https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/userguide/installation.html

16 | ©2022 F5 CONFIDENTIAL
 Task 4.2)
F5 CIS Installation in K8s

CONFIDENTIAL
Task 4.2.1) F5 CIS Installation in K8s
Installing F5 CIS starts with installing the CRD to K8s cluster:

kubectl create -f customresourcedefinitions.yml

Note: You can download the original CRD from the CIS Github to the Master node:
https://raw.githubusercontent.com/F5Networks/k8s-bigip-
ctlr/master/docs/config_examples/customResourceDefinitions/customresourcedefinitions.yml
18 | ©2022 F5 CONFIDENTIAL
Task 4.2.2) F5 CIS Installation in K8s
Create the secret containing the admin's username and password of the F5 device.

kubectl create secret generic bigip-login -n kube-system --from-literal=username=admin --from-literal=password=admin

Create the cluster role, the cluster role binding, and the service account.

kubectl create -f bigip-ctlr-clusterrole.yaml

19 | ©2022 F5 CONFIDENTIAL
Task 4.2.3) F5 CIS Installation in K8s
Examine the CIS deployment setting.

nano f5-cis-deployment.yaml

Once done, exit the editor by Ctrl-X.

20 | ©2022 F5 CONFIDENTIAL
Task 4.2.4) F5 CIS Installation in K8s
Install the F5 CIS.

kubectl create -f f5-cis-deployment.yaml

Check the CIS installation result

kubectl get pods -A -o wide

21 | ©2022 F5 CONFIDENTIAL
Task 4.2.5) F5 CIS Installation in K8s
Check the F5 CIS logs.

kubectl logs -n kube-system `kubectl get pods -A | grep k8s-bigip-ctlr | awk '{print $2}'`

22 | ©2022 F5 CONFIDENTIAL
 Task 4.3)
F5 IPAM Installation in K8s

CONFIDENTIAL
Task 4.3.1) F5 IPAM Installation in K8s
Installing the F5 IPAM in Kubernetes cluster starts with creating the cluster role, the cluster role binding, and the
service account.

kubectl create -f f5-ipam-rbac.yaml

Create persistent volume to store F5 IPAM data.

kubectl create -f f5-ipam-persistentvolume.yaml

24 | ©2022 F5 CONFIDENTIAL
Task 4.3.2) F5 IPAM Installation in K8s
Confirm that the persistent volume has been correctly created.

kubectl get persistentvolume -A

kubectl get persistentvolumeclaim -A

25 | ©2022 F5 CONFIDENTIAL
Task 4.3.3) F5 IPAM Installation in K8s
Examine the F5 IPAM deployment setting.

nano f5-ipam-deployment.yaml

Once done, exit the editor by Ctrl-X.

26 | ©2022 F5 CONFIDENTIAL
Task 4.3.4) F5 IPAM Installation in K8s
Install the F5 IPAM.

kubectl create -f f5-ipam-deployment.yaml

Check the IPAM installation result

kubectl get pods -A -o wide

27 | ©2022 F5 CONFIDENTIAL
Task 4.3.5) F5 IPAM Installation in K8s – change capture
Check the F5 CIS logs.

kubectl logs -n kube-system `kubectl get pods -A | grep f5-ipam | awk '{print $2}'`

28 | ©2022 F5 CONFIDENTIAL
 Task 4.4)
Create a Deployment in K8s

CONFIDENTIAL
Task 4.4.1) Create a Deployment in K8s
Deploy a JSON-based Echo Server.

kubectl create -f echo-api-deployment.yaml

kubectl get pods -A -o wide

30 | ©2022 F5 CONFIDENTIAL
Task 4.4.2) Create a Deployment in K8s
Before creating the Ingress for the Echo Server, let's see the settings in the NGINX IC.

kubectl exec -n nginx-ingress `kubectl get pods -A | grep nginx-ingress | awk '{print $2}'` -- ls -l /etc/nginx/

kubectl exec -n nginx-ingress `kubectl get pods -A | grep nginx-ingress | awk '{print $2}'` -- ls -l
/etc/nginx/conf.d

31 | ©2022 F5 CONFIDENTIAL
Task 4.4.3) Create a Deployment in K8s
Create the Ingress for Echo Server.

kubectl create -f echo-api-ingress.yaml

Once the Ingress is created, NGINX IC automatically creates the corresponding configuration.

kubectl exec -n nginx-ingress `kubectl get pods -A | grep nginx-ingress | awk '{print $2}'` -- ls -l
/etc/nginx/conf.d

32 | ©2022 F5 CONFIDENTIAL
Task 4.4.4) Create a Deployment in K8s
The next step is to create VirtualServer CRD for the Echo Server which will trigger a new configuration creation in
F5 BIG-IP. Before that event, let's inspect the F5 configuration using F5 GUI.

Virtual Server List is Empty

Partition List contains only Common Pool List is Empty

33 | ©2022 F5 CONFIDENTIAL
Task 4.4.5) Create a Deployment in K8s
Create the VirtualServer CRD for Echo Server.

kubectl create -f echo-api-vs.yaml

34 | ©2022 F5 CONFIDENTIAL
Task 4.4.6) Modify F5 CIS from Cluster to NodePort
Once the VirtualServer CRD is created in the K8s clutser, CIS creates the virtual servers, the pools, the nodes, the
traffic policy, etc in F5 device.

35 | ©2022 F5 CONFIDENTIAL
 Task 4.5)
Send Test Traffic and Inspect It

CONFIDENTIAL
Task 4.5) Send Test Traffic and Inspect It
Send several test traffic by repeating the following command.

curl -s --resolve echo.example.com:80:10.1.10.200

http://echo.example.com/api | jq

Observe several important keys such as HOSTNAME, x-forwarded-for,

ECHO_API_SVC_SERVICE_HOST, and
ECHO_API_SVC_SERVICE_PORT.

37 | ©2022 F5 CONFIDENTIAL
Task 4.5) Send Test Traffic and Inspect It
F5 is configured to send the to NGINX pod which then only goes to the
actual apps pod.

Scale up and down NGINX pod and observe the effect

kubectl scale --replicas=5 -n nginx-ingress deployment/nginx-ingress

Monitor K8s command below and BIG-IP pool members. Compare with
earlier what you see now

FROM CLIENT CLI FROM F5 BIG-IP

kubectl get pods -A -o wide

38 | ©2022 F5 CONFIDENTIAL
 Bonus Lab 4 – CIS Cluster

CONFIDENTIAL
Bonus Lab 4 – CIS Cluster
Lab Tasks

4.1) Cleanup F5 CIS

4.2) Configure BGP in K8s Calico

4.3) Configure BGP in F5 BIG-IP

4.4) Modify F5 CIS from NodePort to Cluster mode

4.5) Send test traffic and inspect it

40 | ©2022 F5
 Bonus Task 4.1)
Cleanup F5 CIS

CONFIDENTIAL
Bonus 4.1) Cleanup F5 CIS
Delete the VirtualServer CRD for Echo Server.

kubectl delete -f echo-api-vs.yaml

Scale down the NGINX back to 1 replica.

kubectl scale --replicas=1 -n nginx-ingress deployment/nginx-ingress

42 | ©2022 F5 CONFIDENTIAL
Bonus 4.1) Cleanup F5 CIS
Remove the F5 CIS deployed earlier.

kubectl delete -f f5-cis-deployment.yaml

Check the CIS has been deleted (k8s-bigip-ctlr-* is gone from the pod list)

kubectl get pods -A -o wide

43 | ©2022 F5 CONFIDENTIAL
 Bonus Task 4.2)
Configure BGP in K8s Calico

CONFIDENTIAL
Bonus 4.2.1) Configure BGP in K8s Calico
Configuring BGP in K8s Calico starts with downloading calicoctl for kubectl and moving it to PATH:

curl -L https://github.com/projectcalico/calico/releases/download/v3.23.3/calicoctl-linux-amd64 -o kubectl-calico

chmod +x kubectl-calico

sudo mv kubectl-calico /usr/local/bin

45 | ©2022 F5 CONFIDENTIAL
Bonus 4.2.2) Configure BGP in K8s Calico
Now you can verify if the calicoctl is working fine and the config should be blank since we have not configured it
yet

kubectl calico get bgpconfig

kubectl calico get bgppeer

46 | ©2022 F5
Bonus 4.2.3) Configure BGP in K8s Calico
Create the BGP Configuration with appropriate AS number.

kubectl calico create -f calico-bgpconfig.yaml

Create the BGP Peer configuration with F5 internal IP as its peer.

kubectl calico create -f calico-bgppeer.yaml

47 | ©2022 F5 CONFIDENTIAL
Bonus 4.2.4) Configure BGP in K8s Calico
Now you can verify again if the calico config is updated correctly

kubectl calico get bgpconfig

kubectl calico get bgppeer

48 | ©2022 F5
 Bonus Task 4.3)
Configure BGP in F5 BIG-IP

CONFIDENTIAL
Bonus 4.3.1) Configure BGP in F5 BIG-IP
SSH into F5 BIG-IP:

ssh [email protected]

(password: root)

50 | ©2022 F5 CONFIDENTIAL
Bonus 4.3.2) Configure BGP in F5 BIG-IP
Configuring BGP in F5 BIG-IP starts with turning on BGP on route domain:

tmsh modify /net route-domain 0 routing-protocol add { BGP }

tmsh save /sys config

51 | ©2022 F5 CONFIDENTIAL
Bonus 4.3.3) Configure BGP in F5 BIG-IP
Verify that theres no BGP peering established:

imish

show ip route

show ip bgp neighbors | grep neighbor

exit

52 | ©2022 F5 CONFIDENTIAL
Bonus 4.3.4) Configure BGP in F5 BIG-IP
Then we can configure BGP in ZebOS with all K8s node as peer:
imish

enable

config terminal

router bgp 64567

neighbor 10.1.20.5 remote-as 64567

neighbor 10.1.20.5 activate

neighbor 10.1.20.6 remote-as 64567

neighbor 10.1.20.6 activate

neighbor 10.1.20.7 remote-as 64567

neighbor 10.1.20.7 activate

end

write

exit

53 | ©2022 F5 CONFIDENTIAL
Bonus 4.3.5) Configure BGP in F5 BIG-IP
Verify that theres BGP peering established:

imish

show ip route

show ip bgp neighbors | grep neighbor

exit

54 | ©2022 F5 CONFIDENTIAL
Bonus 4.3.6) Configure BGP in F5 BIG-IP
Exit out from SSH into F5 BIG-IP:

exit

55 | ©2022 F5 CONFIDENTIAL
 Bonus Task 4.4)
Modify F5 CIS from NodePort to
Cluster mode

CONFIDENTIAL
Bonus 4.4) Modify F5 CIS from NodePort to Cluster mode
Edit the f5-cis-deployment.yaml file using your favorite editor. Modify the below lines into to change F5 CIS mode
from NodePort to Cluster:
TO FROM

"--pool-member-type=nodeport", "--pool-member-type=cluster",

57 | ©2022 F5 CONFIDENTIAL
Bonus 4.4) Modify F5 CIS from NodePort to Cluster mode
Install the F5 CIS.

kubectl create -f f5-cis-deployment.yaml

Check the CIS installation result

kubectl get pods -A -o wide

58 | ©2022 F5 CONFIDENTIAL
Bonus 4.4) Modify F5 CIS from NodePort to Cluster mode
Verify the modified lines are reflected correctly (pool-member-type is cluster).

kubectl describe deployment -n kube-system k8s-bigip-ctlr-deployment

59 | ©2022 F5 CONFIDENTIAL
Bonus 4.4) Modify F5 CIS from NodePort to Cluster mode
Create the VirtualServer CRD for Echo Server.

kubectl create -f echo-api-vs.yaml

60 | ©2022 F5 CONFIDENTIAL
Bonus 4.4) Modify F5 CIS from NodePort to Cluster mode
Once the VirtualServer CRD is created in the K8s clutser, CIS creates the virtual servers, the pools, the nodes, the
traffic policy, etc in F5 device. Notice any difference from earlier?

61 | ©2022 F5 CONFIDENTIAL
 Bonus Task 4.5)
Send Test Traffic and Inspect It

CONFIDENTIAL
Bonus 4.5) Send Test Traffic and Inspect It
Send several test traffic by repeating the following command.

curl -s --resolve echo.example.com:80:10.1.10.200

http://echo.example.com/api | jq

Observe several important keys such as HOSTNAME, x-forwarded-for,

ECHO_API_SVC_SERVICE_HOST, and
ECHO_API_SVC_SERVICE_PORT.

63 | ©2022 F5 CONFIDENTIAL
Bonus 4.5) Send Test Traffic and Inspect It
F5 is configured to send the to NGINX pod which then only goes to the
actual apps pod.

Scale up and down NGINX pod and observe the effect

kubectl scale --replicas=5 -n nginx-ingress deployment/nginx-ingress

Monitor K8s command below and BIG-IP pool members. Compare with
earlier what you see now

FROM CLIENT CLI FROM F5 BIG-IP

kubectl get pods -A -o wide

64 | ©2022 F5 CONFIDENTIAL
 Summary

CONFIDENTIAL
What We Have Learned in This Chapter
• Connect K8s pod properly to outside K8s network using an ADC (Application Delivery Controller)
that has automation capability.

• Deploy F5 CIS to automate the F5 BIG-IP LTM according to the pod lifecycle.

• Inspect the traffic flow from client à F5 à K8s pod.

66 | ©2022 F5 CONFIDENTIAL