- README: clarify NEXT_PUBLIC_LOOPS_FORM_ID is required and that omitting it produces a broken (404ing) form, not a graceful degradation - docs/deploy.md: add Local Build Notes section explaining the config.cache=false NFS workaround, the ~15s penalty, and the two-run fallback Co-Authored-By: Paperclip <noreply@paperclip.ing>
271 lines
7.5 KiB
Markdown
271 lines
7.5 KiB
Markdown
# Deploy & Rollback Runbook — using.cash
|
||
|
||
## Overview
|
||
|
||
| Component | Namespace | Image / Service |
|
||
|---|---|---|
|
||
| Marketing website | `using-cash` | `git.falkvinge.net/using-cash/website:<sha>` |
|
||
| Plausible analytics | `analytics` | `ghcr.io/plausible/community-edition:v2.1.4` |
|
||
| Postgres (Plausible) | `analytics` | `postgres:16-alpine` |
|
||
| ClickHouse (Plausible) | `analytics` | `clickhouse/clickhouse-server:24.3-alpine` |
|
||
|
||
---
|
||
|
||
## First-Time Apply (cluster bootstrap)
|
||
|
||
Run once when setting up the cluster for the first time. After this, CI handles deploys.
|
||
|
||
### 1. Prerequisites — verify cluster dependencies
|
||
|
||
```bash
|
||
# Confirm nginx-ingress is running
|
||
kubectl get pods -n ingress-nginx
|
||
|
||
# Confirm cert-manager is running
|
||
kubectl get pods -n cert-manager
|
||
|
||
# Confirm letsencrypt-prod ClusterIssuer exists
|
||
kubectl get clusterissuer letsencrypt-prod
|
||
```
|
||
|
||
If any are missing, install them before proceeding:
|
||
- nginx-ingress: `helm install ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx --create-namespace`
|
||
- cert-manager: `helm install cert-manager jetstack/cert-manager -n cert-manager --create-namespace --set installCRDs=true`
|
||
- letsencrypt-prod ClusterIssuer: see `k8s/cluster/letsencrypt-prod-issuer.yaml` (create if absent)
|
||
|
||
### 2. Confirm ingress LB external IP
|
||
|
||
```bash
|
||
kubectl get svc -n ingress-nginx ingress-nginx-controller
|
||
# Note the EXTERNAL-IP — this is what using.cash A record must point to
|
||
```
|
||
|
||
### 3. Create secrets (out-of-band — never committed to git)
|
||
|
||
```bash
|
||
# Postgres password
|
||
kubectl create secret generic postgres-secret \
|
||
--from-literal=POSTGRES_PASSWORD="$(openssl rand -hex 32)" \
|
||
-n analytics --dry-run=client -o yaml | kubectl apply -f -
|
||
|
||
# Plausible app secrets
|
||
kubectl create secret generic plausible-env \
|
||
--from-literal=SECRET_KEY_BASE="$(openssl rand -hex 64)" \
|
||
--from-literal=TOTP_VAULT_KEY="$(openssl rand -base64 32)" \
|
||
-n analytics --dry-run=client -o yaml | kubectl apply -f -
|
||
```
|
||
|
||
### 4. Apply namespaces first
|
||
|
||
```bash
|
||
kubectl apply -f k8s/using-cash/namespace.yaml
|
||
kubectl apply -f k8s/analytics/namespace.yaml
|
||
```
|
||
|
||
### 5. Apply website manifests
|
||
|
||
```bash
|
||
kubectl apply -f k8s/using-cash/deployment.yaml
|
||
kubectl apply -f k8s/using-cash/service.yaml
|
||
kubectl apply -f k8s/using-cash/ingress.yaml
|
||
```
|
||
|
||
### 6. Apply Plausible analytics manifests
|
||
|
||
```bash
|
||
kubectl apply -f k8s/analytics/pvcs.yaml
|
||
kubectl apply -f k8s/analytics/postgres.yaml
|
||
kubectl apply -f k8s/analytics/clickhouse.yaml
|
||
kubectl apply -f k8s/analytics/plausible.yaml
|
||
|
||
# Wait for all pods to be ready
|
||
kubectl rollout status deployment/postgres -n analytics
|
||
kubectl rollout status deployment/clickhouse -n analytics
|
||
kubectl rollout status deployment/plausible -n analytics
|
||
```
|
||
|
||
### 7. Create Plausible site + admin account
|
||
|
||
```bash
|
||
# Exec into the plausible pod
|
||
kubectl exec -it deploy/plausible -n analytics -- sh
|
||
|
||
# Create the admin user (first user becomes admin)
|
||
# Then log in at https://stats.using.cash and create site "using.cash"
|
||
```
|
||
|
||
---
|
||
|
||
## DNS Cutover
|
||
|
||
### Pre-cutover checklist
|
||
|
||
- [ ] Ingress LB external IP confirmed (Step 2 above)
|
||
- [ ] All manifests applied and pods healthy
|
||
- [ ] Current `using.cash` A record documented (for rollback)
|
||
- [ ] cert-manager + letsencrypt-prod ClusterIssuer confirmed
|
||
- [ ] nginx-ingress confirmed running
|
||
- [ ] Send-and-receive test from `klaus.ubervold@using.cash` to confirm mail works before cutover
|
||
|
||
### Current DNS state (document before touching)
|
||
|
||
```
|
||
# Run before cutover:
|
||
dig using.cash A
|
||
dig www.using.cash A
|
||
dig using.cash MX
|
||
dig using.cash TXT # SPF, DKIM, DMARC
|
||
```
|
||
|
||
### Records to add/update
|
||
|
||
| Record | Type | Value | Notes |
|
||
|---|---|---|---|
|
||
| `using.cash` | A | `<ingress LB external IP>` | Replace existing A if any |
|
||
| `www.using.cash` | CNAME | `using.cash` | Adds / replaces |
|
||
| `stats.using.cash` | CNAME | `using.cash` | New — Plausible |
|
||
| `app.using.cash` | CNAME | `using.cash` | New — reserved slot, placeholder only |
|
||
|
||
### Records to NOT touch
|
||
|
||
- `using.cash` MX — mail delivery
|
||
- `using.cash` TXT (SPF) — mail auth
|
||
- `_domainkey.using.cash` TXT (DKIM) — mail signing
|
||
- `_dmarc.using.cash` TXT (DMARC) — mail policy
|
||
|
||
### Post-cutover verification
|
||
|
||
```bash
|
||
# DNS propagation (may take 0–60 min depending on TTL)
|
||
dig using.cash A @8.8.8.8
|
||
dig www.using.cash CNAME @8.8.8.8
|
||
|
||
# TLS cert issuance (cert-manager HTTP-01 — happens automatically)
|
||
kubectl describe certificate using-cash-tls -n using-cash
|
||
kubectl describe certificaterequest -n using-cash
|
||
|
||
# Site up
|
||
curl -I https://using.cash
|
||
curl -I https://www.using.cash # should redirect 301 → https://using.cash
|
||
|
||
# Plausible up
|
||
curl -I https://stats.using.cash/api/health
|
||
|
||
# Mail — send + receive test
|
||
# Send from/to: klaus.ubervold@using.cash, henrik.larsson@using.cash
|
||
```
|
||
|
||
### DNS rollback
|
||
|
||
If the cutover causes issues, revert the A record to its previous value:
|
||
|
||
```bash
|
||
# Previous A record: <document before cutover>
|
||
# DNS TTL: set low (300s) before cutover to allow fast rollback
|
||
```
|
||
|
||
---
|
||
|
||
## Local Build Notes (NFS environments)
|
||
|
||
The Next.js build pipeline writes and immediately re-reads webpack manifest files. On NFS-backed workspaces (e.g. Paperclip agent workspaces), write-visibility lag causes the first cold build to fail at the "Collecting page data" step with an ENOENT on a manifest file.
|
||
|
||
**The fix is already in `next.config.mjs`:**
|
||
|
||
```js
|
||
webpack(config) {
|
||
config.cache = false // disables webpack persistent cache to avoid NFS race
|
||
...
|
||
}
|
||
```
|
||
|
||
This adds ~15 seconds to each build (no cache warm-up) but makes the build reliable on NFS. In Docker/CI the build runs on local ephemeral storage so this penalty does not apply there.
|
||
|
||
If you see `ENOENT: .next/server/pages-manifest.json` or similar on a fresh build, run `npm run build` a second time — the second run will succeed as the cache is already warm.
|
||
|
||
---
|
||
|
||
## Ongoing Deploys (CI)
|
||
|
||
CI runs automatically on push to `main` via `.gitea/workflows/deploy.yml`.
|
||
|
||
Manual deploy (if CI is unavailable):
|
||
|
||
```bash
|
||
SHA=$(git rev-parse --short HEAD)
|
||
docker build -t git.falkvinge.net/using-cash/website:$SHA .
|
||
docker push git.falkvinge.net/using-cash/website:$SHA
|
||
|
||
kubectl set image deployment/website \
|
||
website=git.falkvinge.net/using-cash/website:$SHA \
|
||
-n using-cash
|
||
|
||
kubectl rollout status deployment/website -n using-cash
|
||
```
|
||
|
||
---
|
||
|
||
## Rollback
|
||
|
||
### Website rollback
|
||
|
||
```bash
|
||
# Immediate rollback to previous image
|
||
kubectl rollout undo deployment/website -n using-cash
|
||
|
||
# Verify
|
||
kubectl rollout status deployment/website -n using-cash
|
||
kubectl get pods -n using-cash
|
||
```
|
||
|
||
### Rollback to specific SHA
|
||
|
||
```bash
|
||
# List rollout history
|
||
kubectl rollout history deployment/website -n using-cash
|
||
|
||
# Roll back to specific revision
|
||
kubectl rollout undo deployment/website -n using-cash --to-revision=<N>
|
||
```
|
||
|
||
### Plausible rollback
|
||
|
||
Plausible uses Recreate strategy. Rolling back the deployment image is sufficient:
|
||
|
||
```bash
|
||
kubectl rollout undo deployment/plausible -n analytics
|
||
```
|
||
|
||
Postgres and ClickHouse data is persisted on PVCs and is unaffected by pod restarts.
|
||
|
||
---
|
||
|
||
## Monitoring
|
||
|
||
```bash
|
||
# Website pods
|
||
kubectl get pods -n using-cash
|
||
kubectl logs deploy/website -n using-cash --tail=50
|
||
|
||
# Analytics pods
|
||
kubectl get pods -n analytics
|
||
kubectl logs deploy/plausible -n analytics --tail=50
|
||
|
||
# Cert status
|
||
kubectl get certificates -A
|
||
kubectl get certificaterequests -A
|
||
|
||
# Ingress
|
||
kubectl get ingress -A
|
||
```
|
||
|
||
---
|
||
|
||
## CI Secrets (Gitea repository settings)
|
||
|
||
| Secret name | Value |
|
||
|---|---|
|
||
| `REGISTRY_USERNAME` | Gitea username with write access to `using-cash` packages |
|
||
| `REGISTRY_PASSWORD` | Gitea password or access token |
|
||
| `KUBECONFIG` | Full kubeconfig YAML for the cluster (base64 or raw) |
|