Kubernetes reference

The facts to look up once a cluster is running: its version, its workers, its limits, and what to check when something does not work. To create and use a cluster, start with Kubernetes.

Versions

New clusters run Kubernetes 1.36, the version the create wizard offers and marks Recommended. The wizard lists only the versions offered for new clusters.

A cluster keeps the version it was created with: an existing cluster cannot be upgraded or moved to another version today. Its name, network and high availability are fixed at creation too. The cluster's page shows its version as Kubernetes 1.36, and its Nodes section shows each worker's exact release, such as v1.36.4.

The end of support is the day the Kubernetes project stops patching a version, security fixes included. The workspace's owners and admins are emailed 30 days and 7 days before it for every cluster on that version, and the cluster's Overview shows it: see What we tell you.

Workers

• Workers are 64-bit x86 machines, so your images must include linux/amd64.
• The Free worker's network is capped at 10 Mbit/s, and its two vCPUs share one core's worth of CPU time.
• A worker has no swap. Its disk holds its own system, the images it pulls and your pods' temporary storage, such as emptyDir volumes.
• Up to 24 volumes can be attached to one worker.
• Every cluster uses the same internal ranges: pods in 10.244.0.0/16, Services in 10.96.0.0/12, and the DNS domain cluster.local.

Kubernetes node pools lists the labels every worker carries.

What is left for your pods

Each worker keeps part of its CPU and memory back for its own system, so your pods can request only what is allocatable: What a worker keeps for itself gives the figures for each size. The cluster's own services also run on workers as pods, in kube-system and in the managed gateway's namespaces: DNS, the metrics API, pod networking and network policy, the Service proxy, the link to the control plane, and the managed gateway when it is on. They use part of what is allocatable.

To see what is left, read a worker's Allocatable and Allocated resources:

kubectl describe node <worker>
kubectl top nodes

The cluster's Nodes section shows the same live usage against what pods may use. Size your pods' requests to fit, and leave room for the system pods, above all on a 2 GB worker.

Built in

• kubectl top and the metrics API, so a HorizontalPodAutoscaler can scale on CPU.
• NetworkPolicy is enforced. No policy is created for you, so every pod can reach every other until you add one.
• Pod security refuses privileged pods: every namespace except the system ones enforces the baseline standard and warns on restricted. Clusters created before these defaults keep running without them.
• There is no node autoscaler: change a pool's number of workers yourself.

Limits

Per plan

A cluster still being deleted counts towards the clusters per workspace, and its name stays taken until it is gone. The workspace owner's plan decides the column, and workers of a paid size also use its quota: see Plans and limits.

For every cluster

Request limits

Past a limit the console says Too many requests. Please wait before trying again., and a Service's settings apply a few seconds later.

Troubleshooting

Cluster status

While a cluster is created or changed, its Overview can also say Waiting for capacity to place the cluster's nodes. or Setting up the cluster hit a problem and is retrying.: the platform keeps trying on its own. A worker stopped responding and is being replaced. means the platform is replacing a worker: see When a worker stops working.

A LoadBalancer Service has no external IP

Read the Service's events:

kubectl describe service <name>

A Service with allocateLoadBalancerNodePorts: false never gets a load balancer: leave that field out. Once it has one, clients connect on the public port in its service.sahabti.com/public-ports annotation, which the Load balancers page also shows.

A claim stays Pending

• It waits for its first pod: a volume is created on the worker where a pod using the claim is scheduled.
• Only the sahabti-delete and sahabti-retain classes, filesystem volumes, and ReadWriteOnce or ReadWriteOncePod access are provided, and a volume cannot be created from a snapshot or another claim.
• A ProvisioningFailed event names any other cause, such as a size above your plan's largest volume or no disk quota left:

kubectl describe pvc <claim>

A pod stays Pending

kubectl describe pod <pod> shows why the scheduler cannot place it:

• Not enough CPU or memory: lower its requests, add workers, or add a pool of a larger size.
• Taints it does not tolerate: add a toleration, or schedule it on another pool. The platform always keeps one pool without NoSchedule or NoExecute taints for the cluster's own services.
• A volume in another zone: a pod with an existing volume runs only on a worker of the volume's zone, and only while that worker has fewer than 24 volumes attached.

A hostname is not served

• A HostnameNotRouted event on a route or an Ingress means the cluster does not hold that hostname yet: point its CNAME record at the cluster, bind it through a verified domain, or use an extra name. See Hostnames.
• A managed gateway error naming Port 30080 means one of your Services took that node port: give it another one.
• A TLSAtEdge event on an Ingress is expected: HTTPS is served at the platform's edge, and the Ingress's tls section is not used.

kubectl cannot connect

• It times out: the API only answers the addresses its API access setting allows. Check API access in the cluster's Access section, or connect from inside your workspace network if the cluster is private.
• The certificate has expired: a 12-hour kubeconfig stops working when its certificate does. Download a new one.
• A self-renewing kubeconfig fails: it needs sh and curl 7.55 or later. It stops within the hour when its key is revoked or you lose your role on the cluster, and at once when the cluster is deleted.
• A certificate name error: the kubeconfig tells clients to check the API's certificate against the name kubernetes with tls-server-name. Use a client that honours it, as kubectl does.

See Kubernetes cluster access for both kinds of kubeconfig and the API access setting.