K
Q

How can I determine the current ephemeral-storage usage of a running Kubernetes pod?

November 8, 2018

How can I tell with 

kubectl
how much <a href="https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#local-ephemeral-storage">ephemeral storage</a> a pod is currently using?

In a Kubernetes pod spec, I can specify resource requests and limits for CPU, memory, and ephemeral storage:

resources:

  requests:

    memory: "60Mi"

    cpu: "70m"
    ephemeral-storage: "2Gi"

  limits:

    memory: "65Mi"

    cpu: "75m"
    ephemeral-storage: "4Gi"

However, to set good requests and limits on ephemeral storage, I need to know what this value actually is for a running pod, which I can't figure out. I can get CPU and memory usage using 

kubectl top pod
, but, from what I can tell, <a href="https://github.com/kubernetes/kubernetes/blob/30a06af453059ffe636f483b7db914d51757139a/pkg/kubelet/eviction/eviction_manager.go#L503">ephemeral storage usage is only actually calculated when making an actual eviction decision</a>.

-- bskaggs
kubernetes
kubectl
disk-access

Similar Questions

kubeadm init shows kubelet isn't running or healthy
shortcut for typing kubectl --all-namespaces everytime
Why do I get a ValidationError from my Kubernetes configuration file, even though it's valid YAML checked with a linter?
Minikube default CPU/Memory
How to check k8s deploy history?

2 Answers

July 7, 2022

You can do this through the raw command.

kubectl get --raw "/api/v1/nodes/(your-node-name)/proxy/stats/summary"

There is also this

kubectl get --raw "/api/v1/nodes/(your-node-name)/proxy/metrics/cadvisor"

EDIT:

I've created a Prometheus exporter for this now.

https://github.com/jmcgrath207/k8s-ephemeral-storage-metrics

-- jmcgrath207
Source: StackOverflow
November 8, 2018

The pure raw approach for this is to use the disk usage (du) Unix command line.

Shell into your pod:

$ kubectl exec -it <pod-id> sh

Change dirs to the mount point of your ephemeral-storage (if you are using volume mounts):

$ mount # check mount points if you'd like

$ cd /mnt/of/ephemeral

$ du .

If you are not using volume mounts:

$ du .

There are other tools that you can use to get metrics:

  • cAdvisor also embedded into the kubelet code, exposed under the 

    /stats/summary
    or 
    /metrics
    endpoint. More info here. An example output:

      $ curl -k -H 'Authorization: Bearer <Redacted>' \
  https://node-hostname:10250/stats/summary

  {
   "node": {
     "nodeName": "node-hostname",
     "systemContainers": [
      {
       "name": "kubelet",
      ...
      "volume": [
       {
        "time": "2018-11-08T23:52:03Z",
        "availableBytes": 1969168384,
        "capacityBytes": 1969180672,
        "usedBytes": 12288,
        "inodesFree": 480748,
        "inodes": 480757,
        "inodesUsed": 9,
        "name": "kube-proxy-token-pprwb"
       }
      ],
      "ephemeral-storage": {
       "time": "2018-11-09T00:05:10Z",
       "availableBytes": 31057477632,
       "capacityBytes": 41567858688,
       "inodesFree": 4873887,
       "inodes": 5120000
      }
  ...
  }

    Similarly:

      $ curl -k -H 'Authorization: Bearer <Redacted>' \
  https://node-hostname:10250/stats/summary

  # HELP apiserver_audit_event_total Counter of audit events generated and sent to the audit backend.
  # TYPE apiserver_audit_event_total counter
  apiserver_audit_event_total 0
  # HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request.
  # TYPE apiserver_client_certificate_expiration_seconds histogram
  apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0
  apiserver_client_certificate_expiration_seconds_bucket{le="21600"} 0
  apiserver_client_certificate_expiration_seconds_bucket{le="43200"} 0
  ...

More info on [kubelet authentication/authorization](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/).

  • Prometheus

More info on K8s metrics here.

-- Rico
Source: StackOverflow