k8s setup 2

Posted on |

background

this blog try to deploy two service in k8s: redis and dashboard. the other is engineering.

manual deploy via kubectl

1
2
3
4
5
6
7
kubectl create ns test-busybox
kubectl run busybox --namespace=test-busybox \
                      --port=8280 \
                      --image=busybox \
                      -- sh -c "echo 'Hello' > /var/www/index.html && httpd -f -p 8280 -h /var/www/"
kubectl get pods -n test-busybox  #should display `Running`, but `Pending`
  • error1: pending pod
1
kubectl describe pods/busybox -n test-busybox

gives:

1
Warning  FailedScheduling  <unknown>  default-scheduler  0/2 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 1 node(s) had taint {node.kubernetes.io/unreachable: }, that the pod didn't tolerate.

a few things to check:

  • swapoff -a to close firewall on working node

  • kubectl uncordon to make node schedulable kubectl uncordon

  • error 2: failed create pod sandbox
1
Warning  FailedCreatePodSandBox  25s (x4 over 2m2s)  kubelet, ubuntu    Failed to create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.2": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

solution is to copy k8s.grc.io/pause:3.2 image to ubuntu node, and restart kubelet on working node.

  • error 3: no network plugin CNI
1
networkPlugin cni failed to set up pod "busybox_test-busybox" network: open /run/flannel/subnet.env: no such file or directory

a temp solution is to cp /run/flannel/subnet.env from master node to worker node, then restart kubelet at the worker node. as further study, the cp subnet.env to worker node is not the right solution, as every time the worker node shutdown, this subnet.env file will delete, and won’t restart when reboot the worker node the next day.

so the final solution here is to pull quay.io/coreos/flannel image to worker node, as well as k8s.gcr.io/kube-proxy. in later k8s version, kube-proxy is like a proxy, what’s really inside is the flannel daemon. so we need both kube-proxy and flannel at worker node, to guarantee the network working.

we can see the busybox service is running well: 

1
2
3
4
5
6
7
kubectl expose pod busybox --type=NodePort   --namespace=test-busybox
kubectl get pods --output=wide -n test-busybox
NAME      READY   STATUS    RESTARTS   AGE     IP         NODE     NOMINATED NODE   READINESS GATES
busybox   1/1     Running   0          7m57s   10.4.0.3   ubuntu   <none>           <none>
kubectl get service busybox -n test-busybox
NAME      TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
busybox   NodePort   10.107.117.219   <none>        8280:32431/TCP   33s

but the problem here is, we can’t access this service from host machine.

exposing an external IP to access an app in cluster

to expose service externally, define the service as eitherLoadBalancer or NodePort type. but LoaderBalancer requires external third-party: 23562 implement of load balancer, e.g. AWS.
why loadBalancer service doesn’t work: if you are using a custom Kubernetes Cluster (using minikube, kubeadm or the like). In this case, there is no LoadBalancer integrated (unlike AWS or Google Cloud). With this default setup, you can only use NodePort or an Ingress Controller.

1
2
3
4
5
6
7
8
9
10
11
kubectl apply -f /home/gwm/k8s/busybox.yaml
kubectl get deployments hello-world  	#display info of Deployment
kubectl describe deployments hello-world
kubectl get replicasets		#display info of ReplicaSet
kubectl describe replicasets
kubectl expose deployment hello-world --type=NodePort --name=my-service  # create a service object that exposes the deployment 
kubectl get services my-service 
kubectl describe services my-service
#cleanup when test done
kubectl delete services my-service
kubectl delete deployment hello-world

looks the NodePort service doesn’t work as expected: 

1
2
curl http://10.20.180.12:8280 
curl: (7) Failed to connect to 10.20.180.12 port 8280: Connection refused

if pods can’t be cleaned by kubectl delete pods xx, try kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE>.

how to access k8s service outside the cluster

kubectl config

reconfigure a node’s kubelet in a live cluster

Basic workflow overview

The basic workflow for configuring a kubelet in a live cluster is as follows:

Write a YAML or JSON configuration file containing the kubelet’s configuration.
Wrap this file in a ConfigMap and save it to the Kubernetes control plane.
Update the kubelet’s corresponding Node object to use this ConfigMap.
  • dump configure file of each node 
1
NODE_NAME="the-name-of-the-node-you-are-reconfiguring"; curl -sSL "http://localhost:8001/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"' > kubelet_configz_${NODE_NAME}

our cluster have ubuntu and meng(as leader) two nodes. with these two config files, we found two existing issues:

1) network config on two nodes doesn’ match each other

1
2
3
4
5
6
<   "clusterDomain": "xyz.abc",
---
>   "clusterDomain": "cluster.local",
<     "10.3.0.10"
---
>     "10.96.0.10"

after generrating the NODE config files above, we can edit these files, and then push the edited config file to the control plane:

1
2
NODE_NAME=meng; kubectl -n kube-system create configmap meng-node-config --from-file=kubelet=kubelet_configz_${NODE_NAME} --append-hash -o yaml
NODE_NAME=ubuntu; kubectl -n kube-system create configmap ubuntu-node-config --from-file=kubelet=kubelet_configz_${NODE_NAME} --append-hash -o yaml

after this setting up, we can check the new generated configmaps:

1
2
kubectl get configmaps -n kube-system
kubectl edit configmap meng-node-config-t442m526c5 -n kube-system

tips: configMaps is also an Object in k8s, just like namespace, pods, svc. but which is only in /tmp, need manually dump.

namely:

1
2
meng-node-config-t442m526c5          1      35m
ubuntu-node-config-ghkg27446c        1      18s
  • set node to use new configMap, by kubectl edit node ${NODE_NAME}, and add the following YAML under spec:
1
2
3
4
5
configSource:
    configMap:
        name: CONFIG_MAP_NAME # replace CONFIG_MAP_NAME with the name of the ConfigMap
        namespace: kube-system
        kubeletConfigKey: kubelet
  • observe the node begin with the new configuration
1
kubectl get node ${NODE_NAME} -o yaml

2) kubectl command doesn’t work at worker node

basically, worker node always report error: Missing or incomplete configuration info. Please point to an existing, complete config file when running kubectl command.

which needs to copy /etc/kubernetes/admin.conf from master to worker, then append cat "export KUBECONFIG=/etc/kubernetes/admin.conf" >> /etc/profile at worker node.

organizing cluster accesss using kubecnfig files

docker0 iptables transfer

when starting docker engine, docker0 VNC is created, and this vnc add its routing rules to the host’s iptables. From docker 1.13.1, the routing rules of docker0 vnc is only transfer to localhost of the host machine, namely docker0 to any other non-localhost is forbidden, which leads to the service can only be access on the host machine, where this pod/container is running. in multi-nodes k8s, we need enable iptable FORWARD.

append the following line to ExecStart line in file /lib/systemd/system/docker.service: 

1
ExecStartPost=/sbin/iptables -I FORWARD -s 0.0.0.0/0 -j ACCEPT

then restart docker engine: 

1
2
systemctl daemon-reload
systemctl restart docker.service

after enable docker0 iptable rules, the following test service can be accessed on both nodes.

deploy redis service

create a k8s-redis image

1
2
3
4
5
6
7
8
9
10
11
# use existing docker image as a base
FROM ubuntu:16.04
# Download and install dependency
RUN apt-get update && apt-get install -y --no-install-recommends redis-server
# EXPOSE the port to the Host OS
EXPOSE 6379
# Tell the image what command it has to execute as it starts as a container
CMD ["redis-server"]

build the image and push to both nodes.

deploy a redis-deployment

  • create redis-deployment.yaml: 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: load-balancer-example
  name: kredis-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: load-balancer-example
  template:
    metadata:
      labels:
        app.kubernetes.io/name: load-balancer-example
    spec:
      containers:
      - image: 10.20.181.119:5000/k8s_redis
        name: kredis
        ports:
        - containerPort: 6379
  • expose deployment as service
1
2
3
4
5
6
7
8
9
10
11
12
13
kubectl create ns test-busybox
kubectl apply -f redis-deployment.yaml
kubectl get deployments redis-deployment 	#display info of Deployment
kubectl describe deployments redis-deployment
kubectl get replicasets		#display info of ReplicaSet
kubectl describe replicasets
kubectl expose deployment redis-deployment --type=NodePort --name=my-redis  # create a service object that exposes the deployment 
kubectl get services my-redis 
kubectl describe services my-redis 
kubectl get pods --output=wide
#clean up later (afer step 3)
kubectl delete services my-redis
kubectl delete deployment redis-deployment

access as pod

1
2
3
4
5
6
7
8
9
gwm@meng:~/k8s/alpine$ kubectl get pods --output=wide 
NAME                                 READY   STATUS    RESTARTS   AGE   IP          NODE     NOMINATED NODE   READINESS GATES
kredis-deployment-7567b7f4b7-wmqgd   1/1     Running   0          16h   10.4.1.18   ubuntu   <none>           <none>
gwm@meng:~/k8s/alpine$ redis-cli -p 6379
Could not connect to Redis at 127.0.0.1:6379: Connection refused
not connected> 
gwm@ubuntu:~$ redis-cli -p 6379
Could not connect to Redis at 127.0.0.1:6379: Connection refused
not connected>

as we can see here, as redis-server as pod, won’t expose any port. and pod-IP(10.4.1.18) is only accessible inside cluster

access as service

1
2
3
4
5
6
7
8
9
10
11
kubectl get services --output=wide 
NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE   SELECTOR
kredis-deploy   NodePort    10.104.43.224   <none>        6379:31962/TCP   23h   app.kubernetes.io/name=load-balancer-example
kubernetes      ClusterIP   10.96.0.1       <none>        443/TCP          8d    <none>
root@ubuntu:~# docker container inspect 60bfd6c5ccac  | grep 31962 
root@ubuntu:~# redis-cli -p 31962 
127.0.0.1:31962> 
root@ubuntu:~# redis-cli -p 31962 -h 10.20.181.132 
10.20.181.132:31962> 
gwm@meng:~$ redis-cli -h 10.20.181.132 -p 31962 
10.20.181.132:31962>

so basically, we can access redis as service with the exposed port 31962, and the host node’s IP(10.20.181.132), (rather than the serivce cluster IP(10.104.43.224).

tips, only check service, won’t tell on which node, the pod is running. so need check the pod, and get its node’s IP.

with docker StartExec with iptable FORWARD, redis-cli on on both ubuntu node and meng node can access the service.

in summary: if we deploy service as NodePort, we suppose to access the service with its host node’s IP and the exposed port, from external/outside of k8s.

endpoints

k8s endpoints. what’s the difference from endpoints to externalIP ? 

1
2
3
kubectl get endpoints 
NAME         ENDPOINTS           AGE
kubernetes   10.20.180.12:6443   8d

it gives us the kubernetes endpoints, which is avaiable on both meng and ubuntu nodes.

1
2
3
4
gwm@meng:~$ curl http://10.20.180.12:6443 
Client sent an HTTP request to an HTTPS server.
gwm@ubuntu:~$ curl http://10.20.180.12:6443 
Client sent an HTTP request to an HTTPS server.

not every service has ENDPOINTS, which gives the way to access outside of the cluster. but NodePort type service can bind to the running pod’s host IP with the exported port.

whenever expose k8s service to either internally or externally, it goes through kube-proxy. when kube-proxy do network transfer, it has two ways: Userspace or Iptables.

clusterIP, is basically expose internnaly, with the service’s cluster IP; while nodePort type, is basically bind the service’s port to each node, so we can access the service from each node with the node’s IP and this fixed port.

apiserver

core of k8s: API Server, is the RESTful API for resource POST/GET/DELETE/UPDATE. we can access through: 

1
2
3
4
curl apiserver_ip:apiserver_port/api
curl apiserver_ip:apiserver_port/api/v1/pods
curl apiserver_ip:apiserver_port/api/v1/services
CURL apiserver_ip:apiserver_port/api/v1/proxy/nodes/{name}/pods/
  • check apiServer IP
1
2
kubectl get pods -n kube-system --output=wide
kube-apiserver-meng            1/1     Running   2          8d     10.20.180.12    meng     <none>           <none>

if check the LISTEN ports on both worker and master nodes, there are many k8s related ports, some are accessible, while some are not.

k8s dashboard

the following is from dashboard doc in cn

  • download src
1
2
3
docker search kubernetes-dashboard-amd64
docker pull k8scn/kubernetes-dashboard-amd64
docker tag k8scn/kubernetes-dashboard-amd64:latest k8s.gcr.io/kubernetes-dashboard-amd64:latest
  • clear old dashboard resources

if there are old running dashboard, can clear first. 

1
2
3
4
5
kubectl get clusterroles kubernetes-dashboard --output=wide 
kubectl get clusterrolebindings kubernetes-dashboard  --output=wide 
kubectl delete clusterroles kubernetes-dashboard 
kubectl delete clusterrolebindings kubernetes-dashboard 
kubectl delete ns kubernetes-dashboard
  • start a fresh dashboard
1
2
kubectl apply -f https://kuboard.cn/install-script/k8s-dashboard/v2.0.0-beta5.yaml 
kubectl apply -f https://kuboard.cn/install-script/k8s-dashboard/auth.yaml

or src from github/dashboard/recommended.yaml, and run:

1
2
kubectl create -f admin-user.yaml
kubectl create -f recommended.yaml

admin-user.yaml is defined wih admin authorization. if not define or applied, when login to dashboard web UI, it gives some errors like:

1
namespaces is forbidden: User "system:serviceaccount:kubernetes-dashboard:kubernetes-dashboard" cannot list resource "namespaces" in API group "" at the cluster scope

so there are two tips during creating dashboard.

  • auth/admin-user.yaml is required

  • add NodePort type service to expose dashboard. if not, can’ access dashboard on host machine.

refer from deploy dashboard && metrics-server

  • create external-http.yaml to expose NodePort service

  • create admin-user.yaml for admin manage

  • get the ServiceAccount token
1
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')
  • go to https://nodeIP.6443, tips, dashboard service is using https
  • login dashboard

there are two ways to auth to login dashboard:

– kubeconfig, the configure to access the cluster

– token, every service account has a secret with valid Bearer Token, that can used to login to Dashboard.

  • system checks
1
2
3
kubectl get secrets -n kubernetes-dashboard
kubectl get serviceaccount -n kubernetes-dashboard
kubectl describe serviceaccount kubernetes-dashboard  -n kubernetes-dashboard

metrics-server

metrics-server is a replace of Heapster. 

1
kubectl edit deploy -n kubernetes-dashboard dashboard-metrics-scraper

roles

the right way to create a role:

  • create a ServiceAccount
  • bind a role for the ServiceAccount(cluster-admin role is needed)
  • make a ClusterRoleBinding for ServiceAccount

list all container images in all ns

1
2
3
4
kubectl get pods --all-namespaces -o jsonpath="{..image}" |\
tr -s '[[:space:]]' '\n' |\
sort |\
uniq -c

refere

kubectl cheatsheet

deployments from k8s doc

deploy tiny web server to k8s

k8s production best practices

cni readme

configure network plugins:

k8s与flannel网络原理

清晰脱俗的直解K8S网络

k8s: iptables and docker0

linux docker and iptables

controlling access to k8s APIserver

understand RBAC auth