Inspecting Pod Resources with `kubectl top` and `kubectl describe`

Introduction

In our last lesson, we established the theoretical foundation for running applications reliably on Kubernetes by defining resource requests and limits. We concluded that the only way to set these values correctly is to first measure how your application behaves under a realistic load.

This lesson transitions from theory to practice. We will cover the primary kubectl commands for doing exactly that. You will learn to use kubectl top to get a real-time snapshot of your pod's resource consumption and kubectl describe to inspect its configuration and event history.

Mastering these two commands provides the data-gathering toolkit essential for "right-sizing" your application. This is a crucial step before we can confidently configure the autoscaling necessary to handle your Pub/Sub message backlog.

kubectl top: Your Real-Time Resource Monitor

The kubectl top command is your window into the live resource consumption of your cluster's components. It provides a point-in-time snapshot of CPU and memory usage, which is invaluable for identifying immediate performance bottlenecks.

From your statistical background, you can think of kubectl top as taking a single sample from the time series of your application's resource usage.

Prerequisite: The Metrics Server

For kubectl top to function, a cluster add-on called the Metrics Server must be running. It collects resource metrics from each node's kubelet and exposes them through the Kubernetes Metrics API, which kubectl top then queries.

Most managed Kubernetes services, including GKE, install the Metrics Server by default. The following article from Datadog explains this prerequisite and shows you how to check if it's running.

Viewing Resource Usage

Once the Metrics Server is confirmed to be running, you can inspect resource usage at both the node and pod level. The Last9 blog provides an excellent, concise guide to these commands and how to interpret their output.

The key commands are:

• kubectl top nodes: Shows resource usage for all nodes in the cluster.
• kubectl top pods: Shows resource usage for pods in the current namespace.
• kubectl top pods -n <namespace>: Shows usage for pods in a specific namespace.
• kubectl top pod <pod-name> --containers: Drills down to show usage for each container within a single pod. This is especially useful for pods with sidecars.

kubectl describe: Inspecting Configuration and Events

While kubectl top shows you what's happening right now, kubectl describe tells you the configuration and history of a resource. For our purposes, it's the perfect counterpart to kubectl top. It allows you to see the requests and limits you configured in your Deployment YAML and view a log of important events that have affected the pod.

The most critical distinction is:

• kubectl top shows actual, real-time usage.
• kubectl describe shows allocated/configured resources (requests and limits).

The Datadog article you looked at earlier explains this difference very clearly.

The two most valuable sections in the kubectl describe pod <pod-name> output for our diagnosis are:

1. Containers: Under each container, you'll find the Requests and Limits for CPU and memory.
2. Events: At the very bottom, you'll find a chronological list of events. This is where you would see if a pod was OOMKilled for exceeding its memory limit or failed to schedule due to insufficient resources.

This short video demonstrates how to use the command for general troubleshooting.

The Diagnostic Workflow: Combining top and describe

The real power comes from using these two commands together. This workflow is central to diagnosing resource-related issues and right-sizing your applications.

Let's apply this to your situation: you've observed your service has high memory usage. Here is how you would investigate:

1. Get live data: Run kubectl top pod <your-service-pod-name>. You see that memory usage is, for example, 480Mi.
2. Check the configuration: Run kubectl describe pod <your-service-pod-name>. You look at the Limits section and see that the memory limit is 512Mi.
3. Synthesize: You can now conclude that your pod is operating at over 93% of its memory limit. This is a high-risk situation. Any small spike in message processing could push it over the limit and cause an OOMKill.
4. Check history: In the same describe output, you scan the Events table. If you see recent OOMKilled events, you have confirmed that the pod is unstable due to insufficient memory limits.

This process transforms a vague observation ("high memory usage") into a concrete, actionable insight ("the pod is at risk of OOMKill because its usage is too close to its limit").

The Last9 article details exactly this process of comparing usage with requests and limits.

Advanced Tips for Practical Monitoring

For a dynamic environment, a single snapshot is often not enough. To get a better feel for resource consumption over time, you can combine kubectl top with the standard Unix watch command.

watch -n 5 kubectl top pods -n <your-namespace>

This command will re-run kubectl top pods every 5 seconds, giving you a simple, real-time dashboard in your terminal. This is extremely useful for observing how resource usage changes as your application processes a workload.

Additionally, you can use standard shell commands to sort the output and quickly identify the most resource-intensive pods in a busy namespace.

kubectl top pods --all-namespaces | sort -k3 -nr | head -10

This command sorts all pods across all namespaces by the third column (CPU) in numeric, reverse order (-nr) and shows the top 10. To sort by memory, you would use -k4. Given your comfort with the command line, these kinds of compositions will likely feel natural.

Conclusion

In this lesson, you've learned how to use two of the most fundamental kubectl commands for operational insight.

Key Takeaways:

• kubectl top provides a real-time snapshot of actual CPU and memory usage, powered by the Metrics Server.
• kubectl describe provides a detailed static view of a resource's configuration, including its requests and limits, and a history of significant Events.
• The core diagnostic workflow is to compare the live usage from kubectl top with the configured limits from kubectl describe to assess the risk of CPU throttling or memory OOMKills.
• The Events section of the describe output is your first stop for debugging pod lifecycle issues.

You now have the tools to measure your subscriber service's baseline performance, which will allow you to confidently set its resource requests and limits.

With a stable, well-configured pod "template," we are finally ready to tackle scaling. In the next lesson, we will dive into the Horizontal Pod Autoscaler (HPA), the Kubernetes component that automatically adjusts the number of replicas based on observed metrics—including the custom Pub/Sub metrics you need.