The wrong cluster.
five things broken on my daily driver, and none of them said so.

For weeks every command I ran went through the control plane over SSH. The first time a checklist asked my own laptop to do the work, five problems showed up in a row.

How this started

Sunday night I sat down with a checklist for the Brasil homelab. Eight blocks, top to bottom, and each one says which machine to run it on.

Block 1 was all server work. Three nodes Ready, three pods Running, Flux green. It took ten minutes and I felt great about it.

Block 2 started with a grep against the repo:

grep -n "ingressClassName\|traefik" clusters/staging/brasil/ingress.yaml
#    -> grep: clusters/staging/brasil/ingress.yaml: No such file or directory

My first guess was that the file was missing. It was not. I was still SSHed into the control plane, and the repo does not live there. It lives on my Mac. The checklist said “(Mac)” right in the heading. I read past it.

That grep was the first sign. Once I switched over to the daily driver, it got worse.

git would not run on my own Mac

On the daily driver, in the right folder this time:

git remote -v
#    -> -bash: /usr/local/bin/git: Bad CPU type in executable

I had never seen that one before. It means macOS was asked to run a program built for a different kind of processor.

uname -m
#    -> arm64

file /usr/local/bin/git
#    -> /usr/local/bin/git: Mach-O 64-bit executable x86_64

My daily driver is Apple Silicon. That git was built for Intel.

Homebrew installs to different places depending on the chip. Intel Macs get /usr/local, Apple Silicon gets /opt/homebrew. I had an old Intel install sitting in /usr/local, and it was first in my PATH. The strange part is that I also had a working Apple Silicon Homebrew. My shell was running from /opt/homebrew/bin/bash the whole time. Two Homebrews, and the broken one kept getting asked first.

So I tried to reinstall git with Homebrew:

brew install git
#    -> .../portable-ruby/current/bin/ruby: Bad CPU type in executable
#    -> Error: 'git' must be installed and in your PATH!

Homebrew was the Intel one too, so its own Ruby would not run. And installing a fresh Homebrew needs git. I was stuck in a loop.

The fix was to leave Homebrew out of it. Apple ships a git with the Command Line Tools at /usr/bin/git, and it runs on both chips. xcode-select —install told me the tools were already installed. So a working git had been there the whole time. It was just standing behind the broken one.

# put Apple's directories ahead of the old Intel ones
export PATH="/usr/bin:/bin:/usr/sbin:/sbin:$PATH"

which git
#    -> /usr/bin/git

Now I check the architecture of anything I depend on:

file $(which kubectl) $(which flux) $(which rsync)

If it says x86_64 on an arm64 Mac, it is going to break at some point.

Three nodes, three usernames

My README says you can reach the nodes with ssh dell and ssh biggie. When I opened ~/.ssh/config, there was one entry in it, for a machine that is not even in this cluster. Neither alias existed.

I added the Dell and gave it the same username as the control plane, because I assumed they matched.

ssh dell "hostname"
#    -> Permission denied, please try again.
#    -> Permission denied, please try again.
#    -> Permission denied, please try again.

Then I tried ssh-copy-id to install my key. That needs the password once to get in, so it failed three more times.

ssh biggie failed a different way:

ssh biggie "hostname"
#    -> ssh: Could not resolve hostname biggie: nodename nor servname provided, or not known

That one was simple. I had only added the Dell, so the biggie alias did not exist yet.

For the Dell, I started guessing. I tried the username from my other server, and it worked. On the MacBook Air. The Dell had a third username. Three nodes, three different accounts. I knew that at some point. I did not remember it at 9pm.

Once I had the right username for each box, every node got an entry like this:

Host dell
    HostName <dell-ip>
    User op
    IdentityFile ~/.ssh/id_ed25519

Then one ssh-copy-id per node, typing the password one time:

ssh-copy-id -i ~/.ssh/id_ed25519.pub op@<dell-ip>
#    -> Number of key(s) added:        1

ssh dell "hostname"
#    -> dell-node

ssh biggie "hostname"
#    -> biggie-smalls

No password prompts. That matters for the next block, which rsyncs my whole music library to the Dell. I am not typing a password for that.

kubectl was talking to the wrong cluster

This is the one I learned the most from.

I ran a Flux check from the daily driver:

kubectl -n flux-system get gitrepository flux-system
#    -> error: the server doesn't have a resource type "gitrepository"

My cluster has that resource. Flux made it. Earlier that night, on the server, Flux was green. So I tried the Flux CLI instead:

flux get sources git -A
#    -> -bash: flux: command not found

Two problems. The Flux CLI was not installed, and something was off with kubectl. I checked which cluster it was talking to:

kubectl config current-context
#    -> rancher-desktop

Rancher Desktop. A small local cluster that lives on the laptop itself. The kubeconfig on my daily driver had never pointed at the real cluster.

kubectl did not fail. It connected to a real Kubernetes API and gave me a correct answer about a different cluster. If I had asked for my pods, I would have gotten No resources found and could have spent a long time looking for pods that were perfectly fine.

kubectl config current-context is now the first thing I run when something looks wrong.

Getting the kubeconfig off the server

k3s keeps its admin kubeconfig at /etc/rancher/k3s/k3s.yaml, and only root can read it. I tried to pull it over in one line:

ssh op@<server-ip> "sudo cat /etc/rancher/k3s/k3s.yaml" > ~/.kube/k3s-config
#    -> sudo: a terminal is required to read the password

sudo wants a terminal to ask for a password, and a command sent over SSH does not get one by default. Adding -t gives it one.

With -t, it asked for my SSH password and then just sat there. Nothing moved. I thought it had frozen.

It had not. It was waiting for my sudo password, but the prompt was going into the file because of the > redirect, so I never saw it.

I opened the file in vim to see what landed. This was the whole file:

[sudo] password for op:
sudo: a password is required

My own redirect had replaced the kubeconfig with an error message. I went to quit without saving and typed :q1. That is a one, not an exclamation mark. Vim had no idea what I meant. :q! on the second try.

Next I copied the kubeconfig text out of another window and pasted it in. YAML depends on indentation, and the paste lost all of it. Every - list marker came through as a bullet point too. kubectl was never going to read that.

What worked was doing it in plain steps.

# on the server: make a copy your user can read
sudo cp /etc/rancher/k3s/k3s.yaml ~/k3s.yaml
sudo chown op ~/k3s.yaml

# on the daily driver: copy it across, then delete the copy
scp op@<server-ip>:k3s.yaml ~/.kube/k3s-config
ssh op@<server-ip> "rm ~/k3s.yaml"

Two edits after that. The file says server: https://127.0.0.1:6443, which only works when you are on the server itself, so it gets the server’s real address. k3s also names everything default, and I renamed that to brasil so I always know which context I am on.

sed -i '' 's|https://127.0.0.1:6443|https://<server-ip>:6443|' ~/.kube/k3s-config
sed -i '' 's|: default|: brasil|g; s|name: default|name: brasil|g' ~/.kube/k3s-config
chmod 600 ~/.kube/k3s-config

export KUBECONFIG=~/.kube/config:~/.kube/k3s-config
kubectl config use-context brasil
#    -> Switched to context "brasil".

kubectl get nodes
#    -> biggie-smalls    Ready    worker
#    -> craventhegreat   Ready    control-plane
#    -> dell-node        Ready    worker

Three nodes, all Ready. The same thing the server showed me at the start of the night, but from my daily driver this time. I added the KUBECONFIG line to my shell profile so a new terminal does not drift back to Rancher Desktop.

Last, the missing Flux CLI:

curl -s https://fluxcd.io/install.sh | sudo bash
#    -> Downloading binary .../flux_2.9.5_darwin_arm64.tar.gz

flux version --client
#    -> flux: v2.9.5

It picked the Apple Silicon build on its own. After the rest of that night, I appreciated it.

Why none of this showed up sooner

For weeks I ran every command on the control plane over SSH. On the server, kubectl reads the right kubeconfig and every binary matches the CPU. My daily driver was only a way to get a terminal on the server, so none of its own tools ever had to work.

The first time the checklist asked the daily driver to do the work itself, five problems showed up one after another. None of them were new.

And almost none of them failed in a way that pointed at the cause. kubectl answered. The SSH config was valid. The redirect did what I told it to do. Every time, the fix started with checking where I actually was: which machine, which cluster, which user.

Final checklist: can your daily driver reach the cluster

If all of these pass, your daily driver can do the work on its own.

# 1. The tools match your CPU
uname -m
file $(which kubectl) $(which flux)
#    -> arm64 for both, not x86_64

# 2. kubectl is pointed at the right cluster
kubectl config current-context
#    -> brasil

# 3. The nodes answer from here
kubectl get nodes
#    -> three nodes, all Ready

# 4. SSH uses the right account and your key
ssh -G dell | grep -E '^(user|hostname) '
ssh dell "hostname"
#    -> dell-node, no password prompt

# 5. Flux is installed
flux version --client

Next is back on the checklist: two Navidrome settings that have been quietly doing nothing, and finally getting my music onto the Dell.