About this Lab
1. Overview
etcd is a key value store used by kubernetes to store all data
It provides consistency and high availability
You should plan to backup etcd regularly
More information on etcd can be found here
Agenda of this lab is to practice taking backup and restoring etcd
IP address of master should be same otherwise restore get complicated due to certificate issues
Lets Practice
2. Create kubernetes cluster
Task: Create Kubernetes cluster with 3 worker nodes.
Master: 1 node
Worker: 2 node
Hint
Solution
Create docker hub account. Docker Hub if you already have one skip this step
Open Play with Kubernetes login with your docker hub account.
Click on start
It will start a 4 hr session
create three instance
click on + ADD NEW INSTANCE three time to add three instances
on first instance enter below command, this node will be master nodekubeadm init --apiserver-advertise-address $(hostname -i) --pod-network-cidr 10.5.0.0/16enter below command on first node
kubectl apply -f https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter.yamlcapture output of kubeadm join XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
you may also use kubeadm token list to find token
use this command on second and third node kubeadm join <IP address of master/first node>:6443 –token
enter captured command in second and third node
kubeadm join XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Check node status, all 3 nodes should be in ready state
3. Create a Pod with Kubectl
Task: Create a pod with name web and image nginx
name: web
image: nginx
Use Kubectl command
verify with below command
kubectl get podsSolution
this command will create pod with image nginx and name web
kubectl run web --image=nginx4. Expose pod port outside cluster
Task: Create a pod using Kubectl and expose port, port should be accessible from outside kubernetes cluster
name: web
image: nginx
port: 80
verify with below command
kubectl get allSolution
kubectl expose pod web --port 80 --name=nginx-svc --type=NodePort --target-port=80
5. Find port used by Nodeport
Task: Find port used by nodeport service in Kubernetes to expose container
Solution
kubectl get all
6. Create a Deployment
Task: Create a deployment with following details:
name: demo
image: nginx
Use Kubectl command
verify with below command
kubectl create deployment demo --image=nginx
Solution
this command will create a deployment with name demo and image nginx
kubectl run web --image=nginx7. Check state of all objects
Task: Check state of all objects in default namespace
Solution
kubectl get all
8. Find etcd pod
Task: Find name of etcd pod in this kubernetes cluster
Solution
kubectl get pods -n kube-system
9. Extract etcdctl
Task: Copy etcdctl from etc pod to local bin directory
Solution
kubectl cp -n kube-system <name of etcd pod>:usr/local/bin/etcdctl etcdctl
ls -la
10. update permission on etcdctl
Task: Change permission on etcdctl file to executable
Solution
chmod +x etcdctl
ls -la
11. Move etcdctl to right directory
Task: Move etcdctl to any directory in PATH
Solution
cp etcdctl /usr/bin/
13. Check etcd config
Task: Check etcd config in yaml file
Solution
cat /etc/kubernetes/manifests/etcd.yaml
--advertise-client-urls
--cert-file
--peer-cert-file
--peer-key-file
--data-dir
--initial-advertise-peer-urls
--initial-cluster
14. Take etcd backup
Task: Take etcd backup
Solution
Test etcdctl command
ETCDCTL_API=3 etcdctl --endpoints=https://<use --advertise-client-urls>:2379 --cacert=<use cert-file from above command > --cert=<use peer-cert-file from above command> --key=<use peer-key-file from above command> versionFirst set of words are to set etcdctl api version to 3 as there are some incompatibilities with previous version
--etcdctl for taking backup require endpoints information to connect and backup (these could be more than one in case of HA)--cacert option is required for taking backup to verify certificates of TLS-enabled secure servers using this CA bundle
--cert option is required for taking backup to identify secure client using this TLS certificate file
--key option is required for taking backup to identify secure client using this TLS key file
Use Below command to backup
ETCDCTL_API=3 etcdctl --endpoints=https://<use --advertise-client-urls>:2379 --cacert=<use cert-file from above command > --cert=<use peer-cert-file from above command> --key=<use peer-key-file from above command> snapshot save <location of file where you want to save backup>First set of words are to set etcdctl api version to 3 as there are some incompatibilities with previous version
--etcdctl for taking backup require endpoints information to connect and backup (these could be more than one in case of HA)--cacert option is required for taking backup to verify certificates of TLS-enabled secure servers using this CA bundle
--cert option is required for taking backup to identify secure client using this TLS certificate file
--key option is required for taking backup to identify secure client using this TLS key file
15. Check etcd backup status
Task: Check etcd backup status
Solution
etcdctl snapshot status etcdbackup
16. Delete components
Task: Delete deployment demo, pod web and svc nginx-svc
Solution
kubectl delete deployment demo
delete deployment demo
Step 21 :
kubectl delete pod web
delete pod web
Step 23 :
kubectl delete service/nginx-svc
delete service we created
17. Check state of all ojects
Task: Check state of all objects in default namespace
Solution
kubectl get all
20. check directory used by etcd
Task: check directory used by etcd, as mentioned in etcd config
Solution
ls -la /var/lib/
21. Restore etcd
Task: Restore etcd
Solution
ETCDCTL_API=3 etcdctl --data-dir=/var/lib/etcd --initial-advertise-peer-urls=https://<use initial-advertise-peer-urls from step 16>:2380 --initial-cluster=default=https://<use initial-cluster from step 16>:2380 snapshot restore <use file where you save backup in step 16>
First set of words are to set etcdctl api version to 3 as there are some incompatibilities with previous version
--data-dir is required to create directory and restore etcd
--initial-advertise-peer-urls is required
--initial-cluster is required
this information could be obtained from earlier step
25. Check state of all objects
Task: Check state of all objects in default namespace
Solution
kubectl get all
26. Cleanup
Task: Delete all open nodes/instances and close session
- Select the node and click on DELETE
- Repeat same for any other open nodes
- click close session
}}