Files
Mikael BergandPaperclip eb156ab146 fix: address CTO review notes — Loops env var clarity + NFS build doc
- README: clarify NEXT_PUBLIC_LOOPS_FORM_ID is required and that omitting
  it produces a broken (404ing) form, not a graceful degradation
- docs/deploy.md: add Local Build Notes section explaining the config.cache=false
  NFS workaround, the ~15s penalty, and the two-run fallback

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-20 14:32:09 +00:00

271 lines
7.5 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Deploy & Rollback Runbook — using.cash
## Overview
| Component | Namespace | Image / Service |
|---|---|---|
| Marketing website | `using-cash` | `git.falkvinge.net/using-cash/website:<sha>` |
| Plausible analytics | `analytics` | `ghcr.io/plausible/community-edition:v2.1.4` |
| Postgres (Plausible) | `analytics` | `postgres:16-alpine` |
| ClickHouse (Plausible) | `analytics` | `clickhouse/clickhouse-server:24.3-alpine` |
---
## First-Time Apply (cluster bootstrap)
Run once when setting up the cluster for the first time. After this, CI handles deploys.
### 1. Prerequisites — verify cluster dependencies
```bash
# Confirm nginx-ingress is running
kubectl get pods -n ingress-nginx
# Confirm cert-manager is running
kubectl get pods -n cert-manager
# Confirm letsencrypt-prod ClusterIssuer exists
kubectl get clusterissuer letsencrypt-prod
```
If any are missing, install them before proceeding:
- nginx-ingress: `helm install ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx --create-namespace`
- cert-manager: `helm install cert-manager jetstack/cert-manager -n cert-manager --create-namespace --set installCRDs=true`
- letsencrypt-prod ClusterIssuer: see `k8s/cluster/letsencrypt-prod-issuer.yaml` (create if absent)
### 2. Confirm ingress LB external IP
```bash
kubectl get svc -n ingress-nginx ingress-nginx-controller
# Note the EXTERNAL-IP — this is what using.cash A record must point to
```
### 3. Create secrets (out-of-band — never committed to git)
```bash
# Postgres password
kubectl create secret generic postgres-secret \
--from-literal=POSTGRES_PASSWORD="$(openssl rand -hex 32)" \
-n analytics --dry-run=client -o yaml | kubectl apply -f -
# Plausible app secrets
kubectl create secret generic plausible-env \
--from-literal=SECRET_KEY_BASE="$(openssl rand -hex 64)" \
--from-literal=TOTP_VAULT_KEY="$(openssl rand -base64 32)" \
-n analytics --dry-run=client -o yaml | kubectl apply -f -
```
### 4. Apply namespaces first
```bash
kubectl apply -f k8s/using-cash/namespace.yaml
kubectl apply -f k8s/analytics/namespace.yaml
```
### 5. Apply website manifests
```bash
kubectl apply -f k8s/using-cash/deployment.yaml
kubectl apply -f k8s/using-cash/service.yaml
kubectl apply -f k8s/using-cash/ingress.yaml
```
### 6. Apply Plausible analytics manifests
```bash
kubectl apply -f k8s/analytics/pvcs.yaml
kubectl apply -f k8s/analytics/postgres.yaml
kubectl apply -f k8s/analytics/clickhouse.yaml
kubectl apply -f k8s/analytics/plausible.yaml
# Wait for all pods to be ready
kubectl rollout status deployment/postgres -n analytics
kubectl rollout status deployment/clickhouse -n analytics
kubectl rollout status deployment/plausible -n analytics
```
### 7. Create Plausible site + admin account
```bash
# Exec into the plausible pod
kubectl exec -it deploy/plausible -n analytics -- sh
# Create the admin user (first user becomes admin)
# Then log in at https://stats.using.cash and create site "using.cash"
```
---
## DNS Cutover
### Pre-cutover checklist
- [ ] Ingress LB external IP confirmed (Step 2 above)
- [ ] All manifests applied and pods healthy
- [ ] Current `using.cash` A record documented (for rollback)
- [ ] cert-manager + letsencrypt-prod ClusterIssuer confirmed
- [ ] nginx-ingress confirmed running
- [ ] Send-and-receive test from `klaus.ubervold@using.cash` to confirm mail works before cutover
### Current DNS state (document before touching)
```
# Run before cutover:
dig using.cash A
dig www.using.cash A
dig using.cash MX
dig using.cash TXT # SPF, DKIM, DMARC
```
### Records to add/update
| Record | Type | Value | Notes |
|---|---|---|---|
| `using.cash` | A | `<ingress LB external IP>` | Replace existing A if any |
| `www.using.cash` | CNAME | `using.cash` | Adds / replaces |
| `stats.using.cash` | CNAME | `using.cash` | New — Plausible |
| `app.using.cash` | CNAME | `using.cash` | New — reserved slot, placeholder only |
### Records to NOT touch
- `using.cash` MX — mail delivery
- `using.cash` TXT (SPF) — mail auth
- `_domainkey.using.cash` TXT (DKIM) — mail signing
- `_dmarc.using.cash` TXT (DMARC) — mail policy
### Post-cutover verification
```bash
# DNS propagation (may take 0–60 min depending on TTL)
dig using.cash A @8.8.8.8
dig www.using.cash CNAME @8.8.8.8
# TLS cert issuance (cert-manager HTTP-01 — happens automatically)
kubectl describe certificate using-cash-tls -n using-cash
kubectl describe certificaterequest -n using-cash
# Site up
curl -I https://using.cash
curl -I https://www.using.cash # should redirect 301 → https://using.cash
# Plausible up
curl -I https://stats.using.cash/api/health
# Mail — send + receive test
# Send from/to: klaus.ubervold@using.cash, henrik.larsson@using.cash
```
### DNS rollback
If the cutover causes issues, revert the A record to its previous value:
```bash
# Previous A record: <document before cutover>
# DNS TTL: set low (300s) before cutover to allow fast rollback
```
---
## Local Build Notes (NFS environments)
The Next.js build pipeline writes and immediately re-reads webpack manifest files. On NFS-backed workspaces (e.g. Paperclip agent workspaces), write-visibility lag causes the first cold build to fail at the "Collecting page data" step with an ENOENT on a manifest file.
**The fix is already in `next.config.mjs`:**
```js
webpack(config) {
config.cache = false // disables webpack persistent cache to avoid NFS race
...
}
```
This adds ~15 seconds to each build (no cache warm-up) but makes the build reliable on NFS. In Docker/CI the build runs on local ephemeral storage so this penalty does not apply there.
If you see `ENOENT: .next/server/pages-manifest.json` or similar on a fresh build, run `npm run build` a second time — the second run will succeed as the cache is already warm.
---
## Ongoing Deploys (CI)
CI runs automatically on push to `main` via `.gitea/workflows/deploy.yml`.
Manual deploy (if CI is unavailable):
```bash
SHA=$(git rev-parse --short HEAD)
docker build -t git.falkvinge.net/using-cash/website:$SHA .
docker push git.falkvinge.net/using-cash/website:$SHA
kubectl set image deployment/website \
website=git.falkvinge.net/using-cash/website:$SHA \
-n using-cash
kubectl rollout status deployment/website -n using-cash
```
---
## Rollback
### Website rollback
```bash
# Immediate rollback to previous image
kubectl rollout undo deployment/website -n using-cash
# Verify
kubectl rollout status deployment/website -n using-cash
kubectl get pods -n using-cash
```
### Rollback to specific SHA
```bash
# List rollout history
kubectl rollout history deployment/website -n using-cash
# Roll back to specific revision
kubectl rollout undo deployment/website -n using-cash --to-revision=<N>
```
### Plausible rollback
Plausible uses Recreate strategy. Rolling back the deployment image is sufficient:
```bash
kubectl rollout undo deployment/plausible -n analytics
```
Postgres and ClickHouse data is persisted on PVCs and is unaffected by pod restarts.
---
## Monitoring
```bash
# Website pods
kubectl get pods -n using-cash
kubectl logs deploy/website -n using-cash --tail=50
# Analytics pods
kubectl get pods -n analytics
kubectl logs deploy/plausible -n analytics --tail=50
# Cert status
kubectl get certificates -A
kubectl get certificaterequests -A
# Ingress
kubectl get ingress -A
```
---
## CI Secrets (Gitea repository settings)
| Secret name | Value |
|---|---|
| `REGISTRY_USERNAME` | Gitea username with write access to `using-cash` packages |
| `REGISTRY_PASSWORD` | Gitea password or access token |
| `KUBECONFIG` | Full kubeconfig YAML for the cluster (base64 or raw) |