13/01/2024 - DOCKER, GO, KUBERNETES
At the time of writing, Go runtime is not aware of the CPU limits set on the container that runs in a Kubernetes pod. Hence it will use all available CPU cores in the node. Instead of this, it is better to explicitly set it manually in configurations. The usual practise is to match GOMAXPROCS with the number of cores that you have in your system. For instance, your node has 2 cores but the container in a pod is meant to be using 1 core then the GOMAXPROCS should also be 1 too. Not configuring this often comes with performance penalty. I won't get into too much details of this subject because you can check the links below and the Go team has noticed it too.
My node has 2 CPU cores and 4GB memory in total. Here is my settings.
$ minikube start --memory 4000 --cpus=2
- Using the hyperkit driver based on user configuration
- Starting "minikube" primary control-plane node in "minikube" cluster
- Creating hyperkit VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
- Preparing Kubernetes v1.30.0 on Docker 26.0.1 ...
$ kubectl describe node
...
Capacity:
cpu: 2
memory: 3912944Ki
Allocatable:
cpu: 2
memory: 3912944Ki
...
$ kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
minikube 295m 14% 1495Mi 39%
CPU(cores) - 295m means 295 millicpu. If 1000m equals to 1 CPU, thus 295m means 29.5% of 1 CPU.
CPU% - The total CPU usage percentage of the node which is 14% in this case.
MEMORY(bytes) - The total memory usage of the node which 1495MB (1.495GB) in this case.
MEMORY% - The total memory usage percentage of the node which is 39% in this case.
apiVersion: apps/v1
kind: Deployment
metadata:
name: gomax-deployment
namespace: default
labels:
app: gomax
spec:
replicas: 1
selector:
matchLabels:
app: gomax
template:
metadata:
labels:
app: gomax
spec:
containers:
- name: golang
image: you/gomax:latest
imagePullPolicy: Always
ports:
- containerPort: 8000
resources:
limits:
cpu: 1000m # My K8S node has 2 CPU cores in total so I limit this pod to 1 core.
memory: 1000Mi
requests:
cpu: 750m
memory: 750Mi
env:
- name: GOMAXPROCS
value: "1" # 1 core. Should equal to resources.limits.cpu setting.
---
apiVersion: v1
kind: Service
metadata:
name: gomax-service
namespace: default
spec:
type: NodePort
selector:
app: gomax
ports:
- protocol: TCP
port: 80
targetPort: 8000
package main
import (
"fmt"
"log"
"os"
"runtime"
"strconv"
)
func main() {
fmt.Println("CPU CORES:", runtime.NumCPU())
fmt.Println("ENV GOMAX:", os.Getenv("GOMAXPROCS"))
fmt.Println("------------")
fmt.Println("CUR GOMAX:", runtime.GOMAXPROCS(0))
// You need this in your application.
if val := os.Getenv("GOMAXPROCS"); val != "" {
max, err := strconv.Atoi(val)
if err != nil {
log.Fatalln(err)
}
runtime.GOMAXPROCS(max)
}
// End
fmt.Println("NEW GOMAX:", runtime.GOMAXPROCS(0))
}
# No configuratiosn at all.
$ kubectl describe node
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
default gomax-deployment-6cf48b6f75-92thz 0 (0%) 0 (0%) 0 (0%) 0 (0%) 61s
$ kubectl logs gomax-deployment-6cf48b6f75-92thz golang
CPU CORES: 2
ENV GOMAX:
------------
CUR GOMAX: 2
NEW GOMAX: 2
# Only K8S limits
resources:
limits:
cpu: 1000m
memory: 1000Mi
requests:
cpu: 750m
memory: 750Mi
$ kubectl describe node
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
default gomax-deployment-5564c56bf5-pv7bc 750m (37%) 1 (50%) 750Mi (19%) 1000Mi (26%) 82s
$ kubectl logs gomax-deployment-5564c56bf5-pv7bc golang
CPU CORES: 2
ENV GOMAX:
------------
CUR GOMAX: 2
NEW GOMAX: 2
# Only GOMAXPROCS variable set.
env:
- name: GOMAXPROCS
value: "1"
$ kubectl describe node
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
default gomax-deployment-7fbd97f7d9-rdkh2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 77s
$ kubectl logs gomax-deployment-7fbd97f7d9-rdkh2 golang
CPU CORES: 2
ENV GOMAX: 1
------------
CUR GOMAX: 1
NEW GOMAX: 1
# All settings (what we want).
resources:
limits:
cpu: 1000m
memory: 1000Mi
requests:
cpu: 750m
memory: 750Mi
env:
- name: GOMAXPROCS
value: "1"
$ kubectl describe node
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
default gomax-deployment-84d77fddfc-xhl2r 750m (37%) 1 (50%) 750Mi (19%) 1000Mi (26%) 73s
$ kubectl logs gomax-deployment-84d77fddfc-xhl2r golang
CPU CORES: 2
ENV GOMAX: 1
------------
CUR GOMAX: 1
NEW GOMAX: 1