# Deploy & Rollback Runbook — using.cash ## Overview | Component | Namespace | Image / Service | |---|---|---| | Marketing website | `using-cash` | `git.falkvinge.net/using-cash/website:` | | Plausible analytics | `analytics` | `ghcr.io/plausible/community-edition:v2.1.4` | | Postgres (Plausible) | `analytics` | `postgres:16-alpine` | | ClickHouse (Plausible) | `analytics` | `clickhouse/clickhouse-server:24.3-alpine` | --- ## First-Time Apply (cluster bootstrap) Run once when setting up the cluster for the first time. After this, CI handles deploys. ### 1. Prerequisites — verify cluster dependencies ```bash # Confirm nginx-ingress is running kubectl get pods -n ingress-nginx # Confirm cert-manager is running kubectl get pods -n cert-manager # Confirm letsencrypt-prod ClusterIssuer exists kubectl get clusterissuer letsencrypt-prod ``` If any are missing, install them before proceeding: - nginx-ingress: `helm install ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx --create-namespace` - cert-manager: `helm install cert-manager jetstack/cert-manager -n cert-manager --create-namespace --set installCRDs=true` - letsencrypt-prod ClusterIssuer: see `k8s/cluster/letsencrypt-prod-issuer.yaml` (create if absent) ### 2. Confirm ingress LB external IP ```bash kubectl get svc -n ingress-nginx ingress-nginx-controller # Note the EXTERNAL-IP — this is what using.cash A record must point to ``` ### 3. Create secrets (out-of-band — never committed to git) ```bash # Postgres password kubectl create secret generic postgres-secret \ --from-literal=POSTGRES_PASSWORD="$(openssl rand -hex 32)" \ -n analytics --dry-run=client -o yaml | kubectl apply -f - # Plausible app secrets kubectl create secret generic plausible-env \ --from-literal=SECRET_KEY_BASE="$(openssl rand -hex 64)" \ --from-literal=TOTP_VAULT_KEY="$(openssl rand -base64 32)" \ -n analytics --dry-run=client -o yaml | kubectl apply -f - ``` ### 4. Apply namespaces first ```bash kubectl apply -f k8s/using-cash/namespace.yaml kubectl apply -f k8s/analytics/namespace.yaml ``` ### 5. Apply website manifests ```bash kubectl apply -f k8s/using-cash/deployment.yaml kubectl apply -f k8s/using-cash/service.yaml kubectl apply -f k8s/using-cash/ingress.yaml ``` ### 6. Apply Plausible analytics manifests ```bash kubectl apply -f k8s/analytics/pvcs.yaml kubectl apply -f k8s/analytics/postgres.yaml kubectl apply -f k8s/analytics/clickhouse.yaml kubectl apply -f k8s/analytics/plausible.yaml # Wait for all pods to be ready kubectl rollout status deployment/postgres -n analytics kubectl rollout status deployment/clickhouse -n analytics kubectl rollout status deployment/plausible -n analytics ``` ### 7. Create Plausible site + admin account ```bash # Exec into the plausible pod kubectl exec -it deploy/plausible -n analytics -- sh # Create the admin user (first user becomes admin) # Then log in at https://stats.using.cash and create site "using.cash" ``` --- ## DNS Cutover ### Pre-cutover checklist - [ ] Ingress LB external IP confirmed (Step 2 above) - [ ] All manifests applied and pods healthy - [ ] Current `using.cash` A record documented (for rollback) - [ ] cert-manager + letsencrypt-prod ClusterIssuer confirmed - [ ] nginx-ingress confirmed running - [ ] Send-and-receive test from `klaus.ubervold@using.cash` to confirm mail works before cutover ### Current DNS state (document before touching) ``` # Run before cutover: dig using.cash A dig www.using.cash A dig using.cash MX dig using.cash TXT # SPF, DKIM, DMARC ``` ### Records to add/update | Record | Type | Value | Notes | |---|---|---|---| | `using.cash` | A | `` | Replace existing A if any | | `www.using.cash` | CNAME | `using.cash` | Adds / replaces | | `stats.using.cash` | CNAME | `using.cash` | New — Plausible | | `app.using.cash` | CNAME | `using.cash` | New — reserved slot, placeholder only | ### Records to NOT touch - `using.cash` MX — mail delivery - `using.cash` TXT (SPF) — mail auth - `_domainkey.using.cash` TXT (DKIM) — mail signing - `_dmarc.using.cash` TXT (DMARC) — mail policy ### Post-cutover verification ```bash # DNS propagation (may take 0–60 min depending on TTL) dig using.cash A @8.8.8.8 dig www.using.cash CNAME @8.8.8.8 # TLS cert issuance (cert-manager HTTP-01 — happens automatically) kubectl describe certificate using-cash-tls -n using-cash kubectl describe certificaterequest -n using-cash # Site up curl -I https://using.cash curl -I https://www.using.cash # should redirect 301 → https://using.cash # Plausible up curl -I https://stats.using.cash/api/health # Mail — send + receive test # Send from/to: klaus.ubervold@using.cash, henrik.larsson@using.cash ``` ### DNS rollback If the cutover causes issues, revert the A record to its previous value: ```bash # Previous A record: # DNS TTL: set low (300s) before cutover to allow fast rollback ``` --- ## Ongoing Deploys (CI) CI runs automatically on push to `main` via `.gitea/workflows/deploy.yml`. Manual deploy (if CI is unavailable): ```bash SHA=$(git rev-parse --short HEAD) docker build -t git.falkvinge.net/using-cash/website:$SHA . docker push git.falkvinge.net/using-cash/website:$SHA kubectl set image deployment/website \ website=git.falkvinge.net/using-cash/website:$SHA \ -n using-cash kubectl rollout status deployment/website -n using-cash ``` --- ## Rollback ### Website rollback ```bash # Immediate rollback to previous image kubectl rollout undo deployment/website -n using-cash # Verify kubectl rollout status deployment/website -n using-cash kubectl get pods -n using-cash ``` ### Rollback to specific SHA ```bash # List rollout history kubectl rollout history deployment/website -n using-cash # Roll back to specific revision kubectl rollout undo deployment/website -n using-cash --to-revision= ``` ### Plausible rollback Plausible uses Recreate strategy. Rolling back the deployment image is sufficient: ```bash kubectl rollout undo deployment/plausible -n analytics ``` Postgres and ClickHouse data is persisted on PVCs and is unaffected by pod restarts. --- ## Monitoring ```bash # Website pods kubectl get pods -n using-cash kubectl logs deploy/website -n using-cash --tail=50 # Analytics pods kubectl get pods -n analytics kubectl logs deploy/plausible -n analytics --tail=50 # Cert status kubectl get certificates -A kubectl get certificaterequests -A # Ingress kubectl get ingress -A ``` --- ## CI Secrets (Gitea repository settings) | Secret name | Value | |---|---| | `REGISTRY_USERNAME` | Gitea username with write access to `using-cash` packages | | `REGISTRY_PASSWORD` | Gitea password or access token | | `KUBECONFIG` | Full kubeconfig YAML for the cluster (base64 or raw) |