- README: clarify NEXT_PUBLIC_LOOPS_FORM_ID is required and that omitting it produces a broken (404ing) form, not a graceful degradation - docs/deploy.md: add Local Build Notes section explaining the config.cache=false NFS workaround, the ~15s penalty, and the two-run fallback Co-Authored-By: Paperclip <noreply@paperclip.ing>
7.5 KiB
Deploy & Rollback Runbook — using.cash
Overview
| Component | Namespace | Image / Service |
|---|---|---|
| Marketing website | using-cash |
git.falkvinge.net/using-cash/website:<sha> |
| Plausible analytics | analytics |
ghcr.io/plausible/community-edition:v2.1.4 |
| Postgres (Plausible) | analytics |
postgres:16-alpine |
| ClickHouse (Plausible) | analytics |
clickhouse/clickhouse-server:24.3-alpine |
First-Time Apply (cluster bootstrap)
Run once when setting up the cluster for the first time. After this, CI handles deploys.
1. Prerequisites — verify cluster dependencies
# Confirm nginx-ingress is running
kubectl get pods -n ingress-nginx
# Confirm cert-manager is running
kubectl get pods -n cert-manager
# Confirm letsencrypt-prod ClusterIssuer exists
kubectl get clusterissuer letsencrypt-prod
If any are missing, install them before proceeding:
- nginx-ingress:
helm install ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx --create-namespace - cert-manager:
helm install cert-manager jetstack/cert-manager -n cert-manager --create-namespace --set installCRDs=true - letsencrypt-prod ClusterIssuer: see
k8s/cluster/letsencrypt-prod-issuer.yaml(create if absent)
2. Confirm ingress LB external IP
kubectl get svc -n ingress-nginx ingress-nginx-controller
# Note the EXTERNAL-IP — this is what using.cash A record must point to
3. Create secrets (out-of-band — never committed to git)
# Postgres password
kubectl create secret generic postgres-secret \
--from-literal=POSTGRES_PASSWORD="$(openssl rand -hex 32)" \
-n analytics --dry-run=client -o yaml | kubectl apply -f -
# Plausible app secrets
kubectl create secret generic plausible-env \
--from-literal=SECRET_KEY_BASE="$(openssl rand -hex 64)" \
--from-literal=TOTP_VAULT_KEY="$(openssl rand -base64 32)" \
-n analytics --dry-run=client -o yaml | kubectl apply -f -
4. Apply namespaces first
kubectl apply -f k8s/using-cash/namespace.yaml
kubectl apply -f k8s/analytics/namespace.yaml
5. Apply website manifests
kubectl apply -f k8s/using-cash/deployment.yaml
kubectl apply -f k8s/using-cash/service.yaml
kubectl apply -f k8s/using-cash/ingress.yaml
6. Apply Plausible analytics manifests
kubectl apply -f k8s/analytics/pvcs.yaml
kubectl apply -f k8s/analytics/postgres.yaml
kubectl apply -f k8s/analytics/clickhouse.yaml
kubectl apply -f k8s/analytics/plausible.yaml
# Wait for all pods to be ready
kubectl rollout status deployment/postgres -n analytics
kubectl rollout status deployment/clickhouse -n analytics
kubectl rollout status deployment/plausible -n analytics
7. Create Plausible site + admin account
# Exec into the plausible pod
kubectl exec -it deploy/plausible -n analytics -- sh
# Create the admin user (first user becomes admin)
# Then log in at https://stats.using.cash and create site "using.cash"
DNS Cutover
Pre-cutover checklist
- Ingress LB external IP confirmed (Step 2 above)
- All manifests applied and pods healthy
- Current
using.cashA record documented (for rollback) - cert-manager + letsencrypt-prod ClusterIssuer confirmed
- nginx-ingress confirmed running
- Send-and-receive test from
klaus.ubervold@using.cashto confirm mail works before cutover
Current DNS state (document before touching)
# Run before cutover:
dig using.cash A
dig www.using.cash A
dig using.cash MX
dig using.cash TXT # SPF, DKIM, DMARC
Records to add/update
| Record | Type | Value | Notes |
|---|---|---|---|
using.cash |
A | <ingress LB external IP> |
Replace existing A if any |
www.using.cash |
CNAME | using.cash |
Adds / replaces |
stats.using.cash |
CNAME | using.cash |
New — Plausible |
app.using.cash |
CNAME | using.cash |
New — reserved slot, placeholder only |
Records to NOT touch
using.cashMX — mail deliveryusing.cashTXT (SPF) — mail auth_domainkey.using.cashTXT (DKIM) — mail signing_dmarc.using.cashTXT (DMARC) — mail policy
Post-cutover verification
# DNS propagation (may take 0–60 min depending on TTL)
dig using.cash A @8.8.8.8
dig www.using.cash CNAME @8.8.8.8
# TLS cert issuance (cert-manager HTTP-01 — happens automatically)
kubectl describe certificate using-cash-tls -n using-cash
kubectl describe certificaterequest -n using-cash
# Site up
curl -I https://using.cash
curl -I https://www.using.cash # should redirect 301 → https://using.cash
# Plausible up
curl -I https://stats.using.cash/api/health
# Mail — send + receive test
# Send from/to: klaus.ubervold@using.cash, henrik.larsson@using.cash
DNS rollback
If the cutover causes issues, revert the A record to its previous value:
# Previous A record: <document before cutover>
# DNS TTL: set low (300s) before cutover to allow fast rollback
Local Build Notes (NFS environments)
The Next.js build pipeline writes and immediately re-reads webpack manifest files. On NFS-backed workspaces (e.g. Paperclip agent workspaces), write-visibility lag causes the first cold build to fail at the "Collecting page data" step with an ENOENT on a manifest file.
The fix is already in next.config.mjs:
webpack(config) {
config.cache = false // disables webpack persistent cache to avoid NFS race
...
}
This adds ~15 seconds to each build (no cache warm-up) but makes the build reliable on NFS. In Docker/CI the build runs on local ephemeral storage so this penalty does not apply there.
If you see ENOENT: .next/server/pages-manifest.json or similar on a fresh build, run npm run build a second time — the second run will succeed as the cache is already warm.
Ongoing Deploys (CI)
CI runs automatically on push to main via .gitea/workflows/deploy.yml.
Manual deploy (if CI is unavailable):
SHA=$(git rev-parse --short HEAD)
docker build -t git.falkvinge.net/using-cash/website:$SHA .
docker push git.falkvinge.net/using-cash/website:$SHA
kubectl set image deployment/website \
website=git.falkvinge.net/using-cash/website:$SHA \
-n using-cash
kubectl rollout status deployment/website -n using-cash
Rollback
Website rollback
# Immediate rollback to previous image
kubectl rollout undo deployment/website -n using-cash
# Verify
kubectl rollout status deployment/website -n using-cash
kubectl get pods -n using-cash
Rollback to specific SHA
# List rollout history
kubectl rollout history deployment/website -n using-cash
# Roll back to specific revision
kubectl rollout undo deployment/website -n using-cash --to-revision=<N>
Plausible rollback
Plausible uses Recreate strategy. Rolling back the deployment image is sufficient:
kubectl rollout undo deployment/plausible -n analytics
Postgres and ClickHouse data is persisted on PVCs and is unaffected by pod restarts.
Monitoring
# Website pods
kubectl get pods -n using-cash
kubectl logs deploy/website -n using-cash --tail=50
# Analytics pods
kubectl get pods -n analytics
kubectl logs deploy/plausible -n analytics --tail=50
# Cert status
kubectl get certificates -A
kubectl get certificaterequests -A
# Ingress
kubectl get ingress -A
CI Secrets (Gitea repository settings)
| Secret name | Value |
|---|---|
REGISTRY_USERNAME |
Gitea username with write access to using-cash packages |
REGISTRY_PASSWORD |
Gitea password or access token |
KUBECONFIG |
Full kubeconfig YAML for the cluster (base64 or raw) |