Avoid kubectl's global state

## The problem

kubectl is the new, cloud-native ssh. That is, it’s the low-level tool to break out when you need to manually inspect or manipulate the state of your running applications.

ssh makes it easy to pop open a few virtual terminal windows to perform different tasks on several systems at once. Each virtual terminal window and its PTY encapsulates its own running state. There’s no way that cd-ing into a directory in one window affects the current working directory in another window.

kubectl breaks that paradigm. By default there is a single global ~/.kube/config file that stores the state of your authenticated session with a cluster. When you authenticate with a new k8s cluster, kubectl mutates this state by adding another context and switching to it.

Mutable global state, you say? Isn’t that… bad?

Yes, I think it is. And while the golden rule of global state being evil has traditionally referred to software engineering practices, I think it applies more generally.

Imagine you’re running a script that uses kubectl to do something. Label some namespaces, kill some pods, whatever. That takes a non-zero amount of time to run. Then you pop open another virtual terminal to do some work in the meantime and switch context into another cluster. Boom, your script in the other virtual terminal is now running against the wrong cluster!

Not only that, but often OIDC providers for k8s are configured to use very long lasting, or even everlasting, tokens that are stored in ~/.kube/config. This is like configuring passwordless root access via ssh to your production server. At any moment, any old script can just ssh root@prod rm -rf / without any interaction.

## A solution

I wanted to change the default behaviour of kubectl to behave more like good old ssh:

No persistent state anywhere.
Every virtual terminal has its own kubectl state.

For the first part, I created an empty file in ~/.kube/config, and set permissions to 0400.

-r-------- 1 scott scott 0 Oct 11  2019 /home/scott/.kube/config

This makes it impossible for kubectl to store global state in the default location.

For the second part, I added this snippet to my ~/.bashrc.

# dynamic kubeconfig per virtual terminal to avoid mutating global state
if [ -z "$KUBECONFIG" ]; then
	export KUBECONFIG=$(mktemp --tmpdir kubeconfig.XXXXXXXX)
	trap "rm -f $KUBECONFIG" EXIT
fi

This uses $KUBECONFIG to make it so that every new virtual terminal gets is own kubectl state. All state, including authentication, is cleared when the shell exits.

This changes the usual kubectl workflow because you have to authenticate anew each time you open a new shell. I see this as an improvment because it forces a workflow that makes you think about which cluster you’re authenticating to every time.

Edit: 2024-06-14

You can also set the immutable attribute to make the file extra difficult to change:

$ sudo chattr +i ~/.kube/config
$ lsattr ~/.kube/config
----i----------------- /home/scott/.kube/config

# Avoid kubectl's global state

## The problem

## A solution