Where the Grafana password.
actually comes from.

I couldn’t log into my own Grafana. Three guesses, all wrong, and the reason why turned out to be more interesting than the fix.

I installed kube-prometheus-stack, port-forwarded to Grafana, typed admin / prom-operator, and got told my password was wrong.

Tried it again slower, in case my fingers had betrayed me. Same thing. So I went looking for the real password, and I spent twenty minutes looking in every place it could not possibly have been.

Here’s what I did wrong, what was actually going on underneath, and how to set your own password without leaving it somewhere embarrassing.

The setup

Local k3s in Rancher Desktop. One node, lives on my laptop, exists to be broken. The install was as boring as installs get:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install monitoring prometheus-community/kube-prometheus-stack -n monitoring --create-namespace

Six pods, all healthy. Prometheus, Alertmanager, Grafana, the operator, kube-state-metrics, node-exporter. Nothing was broken. There was just a login form and I couldn’t get past it.

Guess one: prom-operator

This is the answer you get from every search result and half the tutorials. It’s also the default from an older version of the chart, which I didn’t know at the time. It wasn’t mine.

Guess two: strongpassword

I figured the answer had to be in the chart’s defaults somewhere, so I dumped them and went digging.

helm show values prometheus-community/kube-prometheus-stack > prometheus-default-values.yaml
grep -n "adminPassword" prometheus-default-values.yaml

One hit, line 1480:

  # adminPassword: strongpassword

I typed strongpassword into that login form with real hope in my heart.

Look at the line again. It’s commented out. strongpassword isn’t a password, it’s a placeholder showing you the shape of a thing you could set yourself. I was reading documentation as if it were configuration, which is a fun mistake to catch yourself making in writing.

Guess three: the wrong secret entirely

Eventually I worked out the password lives in a Kubernetes Secret, which was correct and felt like progress. Then:

kubectl get secret -n monitoring prometheus-stack-grafana -o jsonpath="{.data.admin-password}" | base64 -d

Error from server (NotFound): secrets "prometheus-stack-grafana" not found

I’d copied that secret name off a screenshot of somebody else’s cluster. Helm names things after your release. I called mine monitoring, so my secret was monitoring-grafana. His release was called something else, so his wasn’t.

Small mistake, but it’s the kind that keeps happening. Nearly every name a chart produces comes from the release name you picked at install time. Copying resource names out of a tutorial works right up until the author picked a different name than you did, and then it fails in a way that looks like the thing doesn’t exist.

So where was it?

This is the part I didn’t understand at the time, and it’s the part that makes all three guesses look a bit silly in hindsight.

kube-prometheus-stack doesn’t ship Grafana itself. It pulls in the official Grafana chart as a subchart, which is why everything you configure sits under a grafana: key. Whatever you put there gets handed down.

That subchart builds its admin Secret from a template that goes roughly like this:

{{- if .Values.adminPassword }}
admin-password: {{ .Values.adminPassword | b64enc | quote }}
{{- else }}
admin-password: {{ randAlphaNum 40 | b64enc | quote }}
{{- end }}

Read the else. If no password is set, the chart makes up forty random characters right there at render time.

And kube-prometheus-stack ships adminPassword commented out. That’s the line I found at 1480. Commented out means unset, unset means the else fires.

So what actually happened was:

1. I ran helm install
2. Helm rendered the templates
3. randAlphaNum 40 produced a string that had never existed before
4. It went into the Secret
5. That was the only copy

There was nothing to look up. prom-operator was a value from a chart version I wasn’t running. strongpassword was a comment. Neither was ever going to work, and I could have read documentation all afternoon without finding the real one, because it didn’t exist until the moment I hit enter.

The Secret in my cluster wasn’t one source of truth among several. It was the only place that string had ever been.

The answer had been on my screen the whole time

Helm prints a NOTES.txt after every install. Mine scrolled past in the wall of output and I never read a word of it. You can get it back whenever you want:

helm get notes monitoring -n monitoring

And there it was, shipped with the chart, waiting the entire time:

kubectl get secret --namespace monitoring -l app.kubernetes.io/component=admin-secret \
  -o jsonpath="{.items[0].data.admin-password}" | base64 --decode ; echo

That’s the most useful command in this whole post and it came free with the install.

Two small things about it. The ; echo on the end is there because base64 --decode doesn’t print a newline, so without it the password runs into your next shell prompt and is very easy to mis-copy. And notice it looks the secret up by label instead of by name, which is why it works no matter what you called your release. That’s the exact mistake I made in guess three, already solved for me, in a file I didn’t read.

The items[0] is the only awkward bit. A label can match more than one thing, so kubectl hands back a list and you take the first entry.

One more reason to fix this properly

Once I understood the randAlphaNum branch, something slightly uncomfortable followed from it.

That template runs on every render, not just the first install. So every helm upgrade re-runs it, and if adminPassword is still unset, it generates a brand new forty-character string and writes that into the Secret.

Meanwhile Grafana has stored the admin user in its own database and doesn’t care what the Secret says anymore. So over time the two drift apart, and the command everyone tells you to run starts returning something that doesn’t log you in. There’s a long trail of GitHub issues from people hitting exactly that.

A password nobody chose, that quietly changes on upgrade, living in one place I could delete by accident. It worked, but only by accident, and I’d rather it work on purpose.

Setting your own password

Two ways to do this. I’ll show you both, but I’m only recommending one.

The way you’ll find everywhere

## values.yaml
grafana:
  adminPassword: my-password-here

Then helm upgrade. Done, works, three lines.

It also puts a plaintext password in a file, and the trouble isn’t really the file, it’s everywhere the file ends up. It gets committed, so now the password is in your git history permanently and anyone who clones the repo has it, including future you who forgot it was there. It shows up in CI logs. It sits in helm get values output for anyone with cluster read access. And unlike a leaked Secret, you can’t rotate your way out of git history without rewriting it.

On a laptop cluster with no repo behind it, honestly, it’s fine. I’m not going to pretend otherwise. But this is the kind of thing you do once on a throwaway cluster and then do again somewhere it matters, so I’d rather learn the other way while nothing is at stake.

The way I went with

You create the Secret yourself and point the chart at it. Your values file ends up holding a name instead of a password.

Everything below assumes a release called monitoring in a namespace called monitoring. Swap in yours.

1. Check you can read the current password.

If this works, the rest will.

kubectl get secret --namespace monitoring -l app.kubernetes.io/component=admin-secret \
  -o jsonpath="{.items[0].data.admin-password}" | base64 --decode ; echo

2. Make your own Secret.

kubectl create secret generic grafana-admin \
  -n monitoring \
  --from-literal=admin-user=admin \
  --from-literal=admin-password='my-password-here'

Single quotes around the password, otherwise your shell will cheerfully eat any $, !, or backtick you were counting on.

3. Point the chart at it.

## values.yaml
grafana:
  admin:
    existingSecret: grafana-admin
    userKey: admin-user
    passwordKey: admin-password

This part is safe to commit. It says “go find the Secret called grafana-admin” and nothing else. It also switches off the randAlphaNum branch, since the chart skips generating its own Secret entirely when existingSecret is set.

4. Upgrade.

helm upgrade monitoring prometheus-community/kube-prometheus-stack \
  -n monitoring --values values.yaml

5. Restart Grafana.

kubectl rollout restart deployment/monitoring-grafana -n monitoring

Don’t skip this. Grafana reads its admin credentials from environment variables at startup, so the pod has to cycle before anything changes. Skip it and you’ll swear the upgrade did nothing.

6. Confirm it’s using your Secret.

kubectl get deployment monitoring-grafana -n monitoring \
  -o jsonpath='{.spec.template.spec.containers[0].env}' ; echo

You should see grafana-admin in there rather than the chart’s generated secret.

If the new password still doesn’t work

This one surprised me and it’s worth knowing before it happens to you.

Grafana keeps its admin user in its own database, not in Kubernetes. On an install that’s already been running, changing the Secret doesn’t always make it through to the stored user. You can end up in the genuinely baffling state where the Secret says one thing and the login form disagrees with it.

Reset it directly and they line up again:

kubectl exec -n monitoring deploy/monitoring-grafana -c grafana -- \
  grafana-cli admin reset-admin-password 'my-password-here'

The way I think about it now: the Secret is what the chart intends. Grafana’s database is what actually lets you in. Usually those agree. When they don’t, the database wins.

This still isn’t the finish line

Worth being straight about where this stops, because it’d be easy to end on existingSecret and call it solved.

I made that Secret with an imperative kubectl create, so it only exists in the cluster. Not in git, not reproducible, not backed up, and gone the day I tear the cluster down. I’ve traded “password in the repo” for “password nowhere at all,” which is better for security and worse for basically everything else. It’s a problem I’ve deferred, not one I’ve fixed.

The real answer for a GitOps setup is an encrypted secret that does live in the repo, using something like SOPS or External Secrets, so you can rebuild the cluster from git without the plaintext ever being in it. That gets you both things at once, which neither option above manages on its own.

I run Flux at home, so that’s next on my list, and it’ll be its own post.

What I’d tell myself at the start

Ask the cluster, not the internet. When a chart generates a credential, there is no document anywhere that contains it.

Read NOTES.txt. It scrolls past, and helm get notes brings it back, written for your specific release.

Look things up by label rather than by name, because names depend on choices you made and labels come from the chart.

And read the template, not just the values file. The values told me adminPassword was unset. Only the template told me what happens when it is. My entire twenty minutes lived in that gap.

Oh, and strongpassword is not a password. Neither is changeme or your-value-here. I know that. I typed it anyway.