Files
Mikael BergandPaperclip eb156ab146 fix: address CTO review notes — Loops env var clarity + NFS build doc
- README: clarify NEXT_PUBLIC_LOOPS_FORM_ID is required and that omitting
  it produces a broken (404ing) form, not a graceful degradation
- docs/deploy.md: add Local Build Notes section explaining the config.cache=false
  NFS workaround, the ~15s penalty, and the two-run fallback

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-20 14:32:09 +00:00

7.5 KiB
Raw Permalink Blame History

Deploy & Rollback Runbook — using.cash

Overview

Component Namespace Image / Service
Marketing website using-cash git.falkvinge.net/using-cash/website:<sha>
Plausible analytics analytics ghcr.io/plausible/community-edition:v2.1.4
Postgres (Plausible) analytics postgres:16-alpine
ClickHouse (Plausible) analytics clickhouse/clickhouse-server:24.3-alpine

First-Time Apply (cluster bootstrap)

Run once when setting up the cluster for the first time. After this, CI handles deploys.

1. Prerequisites — verify cluster dependencies

# Confirm nginx-ingress is running
kubectl get pods -n ingress-nginx

# Confirm cert-manager is running
kubectl get pods -n cert-manager

# Confirm letsencrypt-prod ClusterIssuer exists
kubectl get clusterissuer letsencrypt-prod

If any are missing, install them before proceeding:

  • nginx-ingress: helm install ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx --create-namespace
  • cert-manager: helm install cert-manager jetstack/cert-manager -n cert-manager --create-namespace --set installCRDs=true
  • letsencrypt-prod ClusterIssuer: see k8s/cluster/letsencrypt-prod-issuer.yaml (create if absent)

2. Confirm ingress LB external IP

kubectl get svc -n ingress-nginx ingress-nginx-controller
# Note the EXTERNAL-IP — this is what using.cash A record must point to

3. Create secrets (out-of-band — never committed to git)

# Postgres password
kubectl create secret generic postgres-secret \
  --from-literal=POSTGRES_PASSWORD="$(openssl rand -hex 32)" \
  -n analytics --dry-run=client -o yaml | kubectl apply -f -

# Plausible app secrets
kubectl create secret generic plausible-env \
  --from-literal=SECRET_KEY_BASE="$(openssl rand -hex 64)" \
  --from-literal=TOTP_VAULT_KEY="$(openssl rand -base64 32)" \
  -n analytics --dry-run=client -o yaml | kubectl apply -f -

4. Apply namespaces first

kubectl apply -f k8s/using-cash/namespace.yaml
kubectl apply -f k8s/analytics/namespace.yaml

5. Apply website manifests

kubectl apply -f k8s/using-cash/deployment.yaml
kubectl apply -f k8s/using-cash/service.yaml
kubectl apply -f k8s/using-cash/ingress.yaml

6. Apply Plausible analytics manifests

kubectl apply -f k8s/analytics/pvcs.yaml
kubectl apply -f k8s/analytics/postgres.yaml
kubectl apply -f k8s/analytics/clickhouse.yaml
kubectl apply -f k8s/analytics/plausible.yaml

# Wait for all pods to be ready
kubectl rollout status deployment/postgres -n analytics
kubectl rollout status deployment/clickhouse -n analytics
kubectl rollout status deployment/plausible -n analytics

7. Create Plausible site + admin account

# Exec into the plausible pod
kubectl exec -it deploy/plausible -n analytics -- sh

# Create the admin user (first user becomes admin)
# Then log in at https://stats.using.cash and create site "using.cash"

DNS Cutover

Pre-cutover checklist

  • Ingress LB external IP confirmed (Step 2 above)
  • All manifests applied and pods healthy
  • Current using.cash A record documented (for rollback)
  • cert-manager + letsencrypt-prod ClusterIssuer confirmed
  • nginx-ingress confirmed running
  • Send-and-receive test from klaus.ubervold@using.cash to confirm mail works before cutover

Current DNS state (document before touching)

# Run before cutover:
dig using.cash A
dig www.using.cash A
dig using.cash MX
dig using.cash TXT   # SPF, DKIM, DMARC

Records to add/update

Record Type Value Notes
using.cash A <ingress LB external IP> Replace existing A if any
www.using.cash CNAME using.cash Adds / replaces
stats.using.cash CNAME using.cash New — Plausible
app.using.cash CNAME using.cash New — reserved slot, placeholder only

Records to NOT touch

  • using.cash MX — mail delivery
  • using.cash TXT (SPF) — mail auth
  • _domainkey.using.cash TXT (DKIM) — mail signing
  • _dmarc.using.cash TXT (DMARC) — mail policy

Post-cutover verification

# DNS propagation (may take 0–60 min depending on TTL)
dig using.cash A @8.8.8.8
dig www.using.cash CNAME @8.8.8.8

# TLS cert issuance (cert-manager HTTP-01 — happens automatically)
kubectl describe certificate using-cash-tls -n using-cash
kubectl describe certificaterequest -n using-cash

# Site up
curl -I https://using.cash
curl -I https://www.using.cash   # should redirect 301 → https://using.cash

# Plausible up
curl -I https://stats.using.cash/api/health

# Mail — send + receive test
# Send from/to: klaus.ubervold@using.cash, henrik.larsson@using.cash

DNS rollback

If the cutover causes issues, revert the A record to its previous value:

# Previous A record: <document before cutover>
# DNS TTL: set low (300s) before cutover to allow fast rollback

Local Build Notes (NFS environments)

The Next.js build pipeline writes and immediately re-reads webpack manifest files. On NFS-backed workspaces (e.g. Paperclip agent workspaces), write-visibility lag causes the first cold build to fail at the "Collecting page data" step with an ENOENT on a manifest file.

The fix is already in next.config.mjs:

webpack(config) {
  config.cache = false  // disables webpack persistent cache to avoid NFS race
  ...
}

This adds ~15 seconds to each build (no cache warm-up) but makes the build reliable on NFS. In Docker/CI the build runs on local ephemeral storage so this penalty does not apply there.

If you see ENOENT: .next/server/pages-manifest.json or similar on a fresh build, run npm run build a second time — the second run will succeed as the cache is already warm.


Ongoing Deploys (CI)

CI runs automatically on push to main via .gitea/workflows/deploy.yml.

Manual deploy (if CI is unavailable):

SHA=$(git rev-parse --short HEAD)
docker build -t git.falkvinge.net/using-cash/website:$SHA .
docker push git.falkvinge.net/using-cash/website:$SHA

kubectl set image deployment/website \
  website=git.falkvinge.net/using-cash/website:$SHA \
  -n using-cash

kubectl rollout status deployment/website -n using-cash

Rollback

Website rollback

# Immediate rollback to previous image
kubectl rollout undo deployment/website -n using-cash

# Verify
kubectl rollout status deployment/website -n using-cash
kubectl get pods -n using-cash

Rollback to specific SHA

# List rollout history
kubectl rollout history deployment/website -n using-cash

# Roll back to specific revision
kubectl rollout undo deployment/website -n using-cash --to-revision=<N>

Plausible rollback

Plausible uses Recreate strategy. Rolling back the deployment image is sufficient:

kubectl rollout undo deployment/plausible -n analytics

Postgres and ClickHouse data is persisted on PVCs and is unaffected by pod restarts.


Monitoring

# Website pods
kubectl get pods -n using-cash
kubectl logs deploy/website -n using-cash --tail=50

# Analytics pods
kubectl get pods -n analytics
kubectl logs deploy/plausible -n analytics --tail=50

# Cert status
kubectl get certificates -A
kubectl get certificaterequests -A

# Ingress
kubectl get ingress -A

CI Secrets (Gitea repository settings)

Secret name Value
REGISTRY_USERNAME Gitea username with write access to using-cash packages
REGISTRY_PASSWORD Gitea password or access token
KUBECONFIG Full kubeconfig YAML for the cluster (base64 or raw)