13/12/2020 - KUBERNETES
In this example we are going to use Kubernetes readiness and liveness probes to automatically detect unhealthy containers/applications in order to handle requests appropriately. It helps us providing as reliable service as possible with minimum downtime.
Indicates whether the Container is "fully" ready for accepting traffic to respond to requests.
Readiness probes are designed to let Kubernetes know when your Container is ready to serve traffic. Kubernetes makes sure the readiness probe passes before allowing a Service to send traffic to the Pod. If a readiness probe starts to fail, Kubernetes stops sending traffic to the Pod until it passes. A Pod is considered ready when all of its Containers are ready to accept connections or requests. If a Container is not in ready state, you don't want to kill it, but you don't want to send requests to it either. For example, a Pod takes time to start/restart so in such cases Kubernetes either stops sending traffic to it or just diverts traffic to already running Pod(s) if exists.
If your Container needs to work on loading large data, configuration files, waiting for external server connections or migrations during start up, specify a readiness probe.
Indicates whether the Container is running. If the liveness probe fails, the kubelet kills the Container and the Container is subjected to its "restart policy".
Liveness probes let Kubernetes know if your Container is alive or dead. If your app is alive, Kubernetes leaves it alone. If your Container is dead, Kubernetes removes the Pod and starts a new one to replace it.
An application can crash, become unresponsive etc. If you wish your Container to be restarted in such cases, specify a liveness probe. You also need to specify a restartPolicy of Always or OnFailure. However, sometimes applications are allowed to crash. In such cases you don't need to specify liveness probe. Kubernetes is clever enough to perform an appropriate action in accordance with the restartPolicy value.
Configuration for Command, HTTP and TCP liveness/readiness probe implementations are identical. You can control probes with options below.
A command is executed, e.g. cat /tmp/healthy, in the target Container. If the command succeeds, it returns 0. If the command returns a non-zero value (failure), it gets restarted.
Sending a HTTP GET request to the server that is running in the Container and listening on the specified port - e.g. /health. Any response code greater than or equal to 200 and less than 400 indicates success. Any other code indicates failure so it triggers restart.
There will be an attempt to open a socket to your Container on the specified port. If it can establish a connection, the Container is considered healthy, if it can't it is considered a failure. For a TCP probe, the kubelet makes the probe connection at the node, not in the Pod, which means that you cannot use a service name in the host parameter since the kubelet is unable to resolve it.
├── Makefile
├── deploy
│ └── k8s
│ ├── deployment.yaml
│ └── service.yaml
├── docker
│ └── dev
│ └── Dockerfile
└── main.go
## Build application binary.
.PHONY: build
build:
go build -race -ldflags "-s -w" -o bin/main main.go
## Build application binary and run it.
.PHONY: run
run: build
bin/main
## Send a dummy request to the local server.
.PHONY: local-test
local-test:
curl -I --request GET http://localhost:3000/health
## -------------------------------------------------------------
## Build, tag and push application image to registry then clean up.
.PHONY: push
push:
docker build -t you/address-finder:latest -f ./docker/dev/Dockerfile .
docker push you/address-finder:latest
docker rmi you/address-finder:latest
docker system prune --volumes --force
## Deploy application to kubernetes cluster.
.PHONY: deploy
deploy:
kubectl apply -f deploy/k8s/deployment.yaml
kubectl apply -f deploy/k8s/service.yaml
## Send a dummy request to the exposed pod on kubernetes.
.PHONY: k8s-test
k8s-test:
curl -I --request GET $(shell minikube service address-finder-service --url)/health
#
# STAGE 1: build
#
FROM golang:1.15-alpine3.12 as build
WORKDIR /source
COPY . .
RUN CGO_ENABLED=0 go build -ldflags "-s -w" -o bin/main main.go
#
# STAGE 2: run
#
FROM alpine:3.12 as run
COPY --from=build /source/bin/main /main
ENTRYPOINT ["/main"]
package main
import (
"log"
"net/http"
"time"
)
func main() {
log.Println("app will be read in 10 sec")
time.Sleep(time.Second*10)
log.Println("app is ready")
rtr := http.NewServeMux()
rtr.HandleFunc("/health", health)
if err := http.ListenAndServe(":3000", rtr); err != nil {
log.Fatalln(err)
}
}
func health(_ http.ResponseWriter, r *http.Request) {
log.Println("X-Probe:", r.Header.Get("X-Probe"))
}
apiVersion: apps/v1
kind: Deployment
metadata:
name: address-finder-deployment
labels:
app: address-finder
spec:
replicas: 1
selector:
matchLabels:
app: address-finder
template:
metadata:
labels:
app: address-finder
spec:
containers:
- name: golang
image: you/address-finder:latest
ports:
- containerPort: 3000
readinessProbe:
httpGet:
path: /health
port: 3000
httpHeaders:
- name: X-Probe
value: readiness
initialDelaySeconds: 10 # Wait x seconds before initialising probe.
periodSeconds: 10 # Probe ever x seconds.
timeoutSeconds: 5 # Time out probe after x seconds.
successThreshold: 1 # Consider probe to be successful after minimum x consecutive successes.
failureThreshold: 5 # Try x times before giving up. Pod will be marked as "Unready".
livenessProbe:
httpGet:
path: /health
port: 3000
httpHeaders:
- name: X-Probe
value: liveness
initialDelaySeconds: 10 # Wait x seconds before initialising probe.
periodSeconds: 10 # Probe ever x seconds.
timeoutSeconds: 5 # Time out probe after x seconds.
successThreshold: 1 # Consider probe to be successful after minimum x consecutive successes.
failureThreshold: 2 # Try x times before giving up. Container will be restarted.
apiVersion: v1
kind: Service
metadata:
name: address-finder-service
spec:
type: NodePort
selector:
app: address-finder
ports:
- protocol: TCP
port: 80
targetPort: 3000
Run make push and make deploy to finalise deployment. Here we verify deployment probes.
$ kubectl describe deployment.apps/address-finder-deployment
…
Liveness: http-get http://:3000/health delay=10s timeout=5s period=10s #success=1 #failure=2
Readiness: http-get http://:3000/health delay=10s timeout=5s period=10s #success=1 #failure=5
…
When we test right after deployment and before 10 sec.
$ make k8s-test
curl -I --request GET http://192.168.99.100:32560/health
curl: (7) Failed to connect to 192.168.99.100 port 32560: Connection refused
make: *** [k8s-test] Error 7
When we test 10 sec after start up.
$ make k8s-test
curl -I --request GET http://192.168.99.100:32560/health
HTTP/1.1 200 OK
Date: Sun, 13 Dec 2020 17:19:49 GMT
Content-Length: 0
These are the logs.
$ kubectl logs pod/address-finder-deployment-6f5fc66d4d-k7pb2
2020/12/13 17:19:28 app will be read in 10 sec
2020/12/13 17:19:38 app is ready
2020/12/13 17:19:40 X-Probe: liveness
2020/12/13 17:19:46 X-Probe: readiness
2020/12/13 17:19:49 X-Probe: // This is the moment when we got 200 OK while testing, compare the time
2020/12/13 17:19:50 X-Probe: liveness
2020/12/13 17:19:56 X-Probe: readiness
2020/12/13 17:20:00 X-Probe: liveness
2020/12/13 17:20:06 X-Probe: readiness
2020/12/13 17:20:10 X-Probe: liveness
2020/12/13 17:20:16 X-Probe: readiness
…