Files
Mikael BergandPaperclip 5ec56f20d5 feat: k8s manifests, Plausible self-host, Dockerfile, CI, and deploy runbook for using.cash
- k8s/using-cash/: namespace, deployment (2 replicas, rolling), service, ingress
  - apex (using.cash) with Let's Encrypt TLS via cert-manager HTTP-01
  - www.using.cash permanent-redirect → using.cash
  - app.using.cash placeholder ingress (DNS slot reserved for product app)
- k8s/analytics/: Plausible CE + Postgres 16 + ClickHouse 24.3 with PVCs
  - stats.using.cash ingress with TLS
  - init containers wait for Postgres and ClickHouse before Plausible starts
  - Secrets created out-of-band; placeholder files document the kubectl commands
- Dockerfile: multi-stage Next.js static export → nginx:alpine (port 8080, non-root)
- nginx.conf: trailing-slash static export support, immutable asset caching, security headers
- .gitea/workflows/deploy.yml: CI build→push→rollout on push to main
- docs/deploy.md: first-apply checklist, DNS cutover procedure, rollback commands

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-20 13:56:07 +00:00

6.7 KiB
Raw Permalink Blame History

Deploy & Rollback Runbook — using.cash

Overview

Component Namespace Image / Service
Marketing website using-cash git.falkvinge.net/using-cash/website:<sha>
Plausible analytics analytics ghcr.io/plausible/community-edition:v2.1.4
Postgres (Plausible) analytics postgres:16-alpine
ClickHouse (Plausible) analytics clickhouse/clickhouse-server:24.3-alpine

First-Time Apply (cluster bootstrap)

Run once when setting up the cluster for the first time. After this, CI handles deploys.

1. Prerequisites — verify cluster dependencies

# Confirm nginx-ingress is running
kubectl get pods -n ingress-nginx

# Confirm cert-manager is running
kubectl get pods -n cert-manager

# Confirm letsencrypt-prod ClusterIssuer exists
kubectl get clusterissuer letsencrypt-prod

If any are missing, install them before proceeding:

  • nginx-ingress: helm install ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx --create-namespace
  • cert-manager: helm install cert-manager jetstack/cert-manager -n cert-manager --create-namespace --set installCRDs=true
  • letsencrypt-prod ClusterIssuer: see k8s/cluster/letsencrypt-prod-issuer.yaml (create if absent)

2. Confirm ingress LB external IP

kubectl get svc -n ingress-nginx ingress-nginx-controller
# Note the EXTERNAL-IP — this is what using.cash A record must point to

3. Create secrets (out-of-band — never committed to git)

# Postgres password
kubectl create secret generic postgres-secret \
  --from-literal=POSTGRES_PASSWORD="$(openssl rand -hex 32)" \
  -n analytics --dry-run=client -o yaml | kubectl apply -f -

# Plausible app secrets
kubectl create secret generic plausible-env \
  --from-literal=SECRET_KEY_BASE="$(openssl rand -hex 64)" \
  --from-literal=TOTP_VAULT_KEY="$(openssl rand -base64 32)" \
  -n analytics --dry-run=client -o yaml | kubectl apply -f -

4. Apply namespaces first

kubectl apply -f k8s/using-cash/namespace.yaml
kubectl apply -f k8s/analytics/namespace.yaml

5. Apply website manifests

kubectl apply -f k8s/using-cash/deployment.yaml
kubectl apply -f k8s/using-cash/service.yaml
kubectl apply -f k8s/using-cash/ingress.yaml

6. Apply Plausible analytics manifests

kubectl apply -f k8s/analytics/pvcs.yaml
kubectl apply -f k8s/analytics/postgres.yaml
kubectl apply -f k8s/analytics/clickhouse.yaml
kubectl apply -f k8s/analytics/plausible.yaml

# Wait for all pods to be ready
kubectl rollout status deployment/postgres -n analytics
kubectl rollout status deployment/clickhouse -n analytics
kubectl rollout status deployment/plausible -n analytics

7. Create Plausible site + admin account

# Exec into the plausible pod
kubectl exec -it deploy/plausible -n analytics -- sh

# Create the admin user (first user becomes admin)
# Then log in at https://stats.using.cash and create site "using.cash"

DNS Cutover

Pre-cutover checklist

  • Ingress LB external IP confirmed (Step 2 above)
  • All manifests applied and pods healthy
  • Current using.cash A record documented (for rollback)
  • cert-manager + letsencrypt-prod ClusterIssuer confirmed
  • nginx-ingress confirmed running
  • Send-and-receive test from klaus.ubervold@using.cash to confirm mail works before cutover

Current DNS state (document before touching)

# Run before cutover:
dig using.cash A
dig www.using.cash A
dig using.cash MX
dig using.cash TXT   # SPF, DKIM, DMARC

Records to add/update

Record Type Value Notes
using.cash A <ingress LB external IP> Replace existing A if any
www.using.cash CNAME using.cash Adds / replaces
stats.using.cash CNAME using.cash New — Plausible
app.using.cash CNAME using.cash New — reserved slot, placeholder only

Records to NOT touch

  • using.cash MX — mail delivery
  • using.cash TXT (SPF) — mail auth
  • _domainkey.using.cash TXT (DKIM) — mail signing
  • _dmarc.using.cash TXT (DMARC) — mail policy

Post-cutover verification

# DNS propagation (may take 0–60 min depending on TTL)
dig using.cash A @8.8.8.8
dig www.using.cash CNAME @8.8.8.8

# TLS cert issuance (cert-manager HTTP-01 — happens automatically)
kubectl describe certificate using-cash-tls -n using-cash
kubectl describe certificaterequest -n using-cash

# Site up
curl -I https://using.cash
curl -I https://www.using.cash   # should redirect 301 → https://using.cash

# Plausible up
curl -I https://stats.using.cash/api/health

# Mail — send + receive test
# Send from/to: klaus.ubervold@using.cash, henrik.larsson@using.cash

DNS rollback

If the cutover causes issues, revert the A record to its previous value:

# Previous A record: <document before cutover>
# DNS TTL: set low (300s) before cutover to allow fast rollback

Ongoing Deploys (CI)

CI runs automatically on push to main via .gitea/workflows/deploy.yml.

Manual deploy (if CI is unavailable):

SHA=$(git rev-parse --short HEAD)
docker build -t git.falkvinge.net/using-cash/website:$SHA .
docker push git.falkvinge.net/using-cash/website:$SHA

kubectl set image deployment/website \
  website=git.falkvinge.net/using-cash/website:$SHA \
  -n using-cash

kubectl rollout status deployment/website -n using-cash

Rollback

Website rollback

# Immediate rollback to previous image
kubectl rollout undo deployment/website -n using-cash

# Verify
kubectl rollout status deployment/website -n using-cash
kubectl get pods -n using-cash

Rollback to specific SHA

# List rollout history
kubectl rollout history deployment/website -n using-cash

# Roll back to specific revision
kubectl rollout undo deployment/website -n using-cash --to-revision=<N>

Plausible rollback

Plausible uses Recreate strategy. Rolling back the deployment image is sufficient:

kubectl rollout undo deployment/plausible -n analytics

Postgres and ClickHouse data is persisted on PVCs and is unaffected by pod restarts.


Monitoring

# Website pods
kubectl get pods -n using-cash
kubectl logs deploy/website -n using-cash --tail=50

# Analytics pods
kubectl get pods -n analytics
kubectl logs deploy/plausible -n analytics --tail=50

# Cert status
kubectl get certificates -A
kubectl get certificaterequests -A

# Ingress
kubectl get ingress -A

CI Secrets (Gitea repository settings)

Secret name Value
REGISTRY_USERNAME Gitea username with write access to using-cash packages
REGISTRY_PASSWORD Gitea password or access token
KUBECONFIG Full kubeconfig YAML for the cluster (base64 or raw)