Debug in Kubernetes
Using ephemeral containers to interactively troubleshoot Kubernetes production environments
For your Kubernetes deployment , you can use ephemeral debug containers to debug distroless Minimus images. Ephemeral debug containers can be temporarily attached to existing Pods to troubleshoot and inspect running services and are commonly used to inspect and troubleshoot running services.
Ephemeral debug containers may be necessary, since copying debugging tools into running containers on-demand with kubectl cp
is not possible for Minimus production images without tar
included.
Process namespace sharing
The ephemeral container needs to connect to the namespace of the Minimus container in order to sideload debugging tools that aren’t available in the Minimus container itself.
To simplify the debugging process, it is recommended customers enable process namespace sharing in your Pod settings. If enabled, you will be able to access processes running in other containers on the Pod without having to specify a target. Access to the filesystem may also be affected, due to default user permissions. The attribute ephemeralContainers
in the Pod spec can also be modified for existing Pod instances. Learn more from Kubernetes
Nginx example
In this example we have a Ngnix container in a Kubernetes cluster:
The Minimus Nginx production image is distroless, and does not contain troubleshooting tools to limit the attack surface, including a shell - so you can’t use kubectl exec
to troubleshoot the container.
The solution is to add a debugging sidecar. The Minimus generic base image is designed just for that purpose, and is ideal for running as an ephemeral debug container along with another Minimus container. Run the following to add the image as an ephemeral container and connect it to the namespaces of your already-running nginx container:
You can now inspect the container and its open ports:
You are now ready to sideload debugging tools and get to work.