Kubernetes node pools

A cluster's workers come in node pools. Every worker of a pool has the pool's size and zone, and joins the cluster with the pool's labels and taints. Use pools to give each workload the machines it needs: large workers for a database, small ones for everything else, or a pool kept for one team.

A cluster starts with the pools you set up in the Node pools step when you create it: the first is named default unless you rename it, and you can add more there, within your plan's workers per cluster. See Create a cluster.

Manage node pools

Open the cluster and go to Nodes. The Node pools table lists each pool's size, zone, how many of its workers are ready, its labels and taints, and its state.

• Add node pool: choose a name, a worker size, a zone and how many workers, and optionally labels and taints. The dialog shows the price per worker and a pay-as-you-go estimate for the pool's workers: see Kubernetes pricing and usage.
• Edit: change the number of workers, the labels or the taints. The name, size and zone stay as they are; to change them, add a new pool and delete the old one.
• Delete: drains the pool's workers and removes them, then the pool.

Owners, admins and developers can manage node pools. Viewers can see them.

The cluster shows Updating while a change applies, and keeps serving the whole time. New workers join with their pool's labels and taints, so nothing is scheduled on them before those apply. When workers leave because you lowered a count, the cluster keeps a worker in every zone that holds one of its volumes as long as the count allows it, removes workers that are not ready yet or have no volume attached before the others, and otherwise removes the newest first; deleting a pool removes all of its workers. Each is taken out of the load balancers and drained, with up to 15 minutes for your pod disruption budgets, and its volumes move to the workers that remain in their zone.

Names, sizes and zones

What a worker keeps for itself

Every worker keeps some CPU and memory back for its operating system and for the Kubernetes agent and container runtime that run it, and starts evicting pods when its free memory falls below 100 MiB, so a busy pod cannot starve the worker itself. What is left is the worker's allocatable capacity: the most your pods can request on it.

• The memory kept back includes the 100 MiB eviction threshold. The kernel takes a little more before Kubernetes counts anything, hence "about".
• The cluster's own services, such as DNS and the pod network, run on workers too and use part of what is allocatable.
• The Free worker's two vCPUs share one core's worth of CPU time.
• kubectl describe node <name> shows a worker's capacity, what is allocatable and what its pods request.

Workers of clusters created before these reservations existed keep nothing back but the 100 MiB eviction threshold.

Labels the platform sets

Every worker carries these labels, for node selectors and affinity rules:

Your labels and taints

A pool can carry up to 20 labels and 10 taints of your own. They apply to every worker of the pool, the running ones and those still to come.

• Keys follow Kubernetes label syntax: a name of up to 63 characters, optionally after a domain and a slash, such as team or example.com/tier.
• A taint has a key, an optional value and an effect:
  • NoSchedule keeps pods without a matching toleration off the pool's workers.
  • PreferNoSchedule keeps them off when there is room elsewhere.
  • NoExecute also evicts the pods already there that do not tolerate it.
• Change them on the pool, not on its nodes: whenever the cluster changes, every worker gets its pool's labels and taints again. Labels and taints you set on a node yourself are left alone, unless the pool sets the same key.

Run a workload on one pool

Give a pool named gpu the label team=ml and the taint dedicated=gpu with the effect NoSchedule. Then select the pool and tolerate its taint:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: trainer
spec:
  replicas: 2
  selector:
    matchLabels:
      app: trainer
  template:
    metadata:
      labels:
        app: trainer
    spec:
      nodeSelector:
        node.sahabti.com/pool: gpu
      tolerations:
        - key: dedicated
          operator: Equal
          value: gpu
          effect: NoSchedule
      containers:
        - name: trainer
          image: nginx:1.27-alpine

kubectl apply -f trainer.yaml
kubectl get pods -l app=trainer -o wide

The node selector brings the pods to the pool, and the toleration lets them past its taint. Without the selector they could also run on other pools; without the toleration they cannot run on the tainted pool at all.

Rules

• Plan limits: the workers of every pool count against your plan's workers per cluster, and a pool can only use your plan's worker sizes. Lowering a count and deleting a pool are always allowed.
• Quota: every worker of a paid size takes its vCPU, memory and disk from your plan's quota, like an instance, so a new pool or more workers need room in it; the dialog shows what they use. The Free worker and the control plane do not count.
• At least one pool: a cluster keeps at least one node pool. Add another before you delete the last one.
• A pool for the cluster's own services: DNS and the cluster's other services do not tolerate your taints, so at least one pool must have no NoSchedule or NoExecute taint. A change that would leave every pool with one is refused.
• Reserved keys: keys in the kubernetes.io, k8s.io and sahabti.com domains, and their subdomains, belong to Kubernetes and the platform, so a pool's labels and taints cannot use them.