# Uptime Sentinel — Product Requirements Document

**Status:** Production-bound at Evidence Action (first real-environment customer).
**Last updated:** 2026-05-31 (post executive-perf + analytics-retention + authz/secret sweep; planning/audit notes retired — this PRD is now the canonical project doc).

---

## 1. Purpose

Uptime Sentinel is a self-hosted network and service monitoring platform built for organisations that need real auditability, real outbound-egress safety, and real operational discipline — without the cost or telemetry exposure of Datadog/PagerDuty SaaS.

The first deployment target is **Evidence Action**, an NGO running production services across multiple countries. The product must run on a single-instance VPS (or a modest Kubernetes pod) against a shared MySQL database, be operable by one ops engineer, and never silently miss an outage.

Multi-tenant SaaS is explicitly out of scope. Future tenancy work, if any, is a separate project.

---

## 2. Target users

| User | Primary jobs-to-be-done |
|---|---|
| **Ops / SRE** | Configure monitors, view real-time status, get paged on outage, run postmortems, export audit logs for compliance. |
| **Executive** | Opens the Executive Command dashboard (or a status page); sees uptime, incidents, regional health, and a live world map at a glance. Views in-session (ADMIN/EDITOR) or via a login-less, device-locked display token. |
| **Display / wallboard** | An unattended TV or kiosk authenticated once with a device-locked `executive_display` token; renders `/executive/wall`. |
| **Status-page subscriber** (public) | Self-serves email subscription to incident updates. Does not have an account. |
| **AI assistant** (internal) | Read-only operational queries against monitors, maintenance windows, audit log, outbox status via permission-scoped tool API. |

---

## 3. Core capabilities (shipped)

### 3.1 Monitor types

- **HTTP/HTTPS** — status code assertion, keyword assertion (bounded body read), TLS cert validation + expiry tracking, native redirect follower with re-validation on every 3xx hop.
- **TCP** — port connectivity.
- **Ping/ICMP** — uses host `ping` binary. Container must run with `--cap-add=NET_RAW` or be privileged; otherwise probes fail and report as down.
- **DNS** — record presence and value assertion.
- **Heartbeat (dead-man's-switch)** — external job POSTs to `/api/heartbeat/<token>` on schedule; missed window triggers down.
- **Domain expiry** — WHOIS-based.

_(Synthetic-test feature was scoped in earlier drafts but dropped in audit2-17e (PR #71) — 0 rows live, no runner code; candidate for re-introduction if customer demand emerges.)_

### 3.2 Alerting / notification

- **Channels:** Email (SMTP), Slack, Discord webhooks, Telegram, custom webhooks (HMAC-SHA256 signed with `X-Sentinel-Signature: sha256=<hex>` + timestamp envelope), PagerDuty Events v2, Microsoft Teams.
- **Routing:** `NotificationRule` table maps monitors/regions/severity to channels with conditional logic.
- **Reliability primitives:**
  - Outbox table with exponential backoff (PR-11) and dead-channel meta-alert.
  - Per-monitor dedup window (PR-B).
  - Up-hysteresis: `requiredUps` consecutive recovery checks before emitting `monitor.up`.
  - Down-hysteresis: `Monitor.retries` (default 2) consecutive failures before emitting `monitor.down`.
  - Escalation: `NotificationRule.escalationDelay` defers a follow-up alert; cancels automatically on recovery (PR-26).

### 3.3 Authentication and authorisation

- Username/password with **bcrypt** hashing.
- **TOTP 2FA** with enforcement gate (`AUTH_2FA_ENFORCED`); enrolment allowlist for setup flow.
- **Entra ID SSO** (PR-29) via NextAuth provider for Evidence Action staff.
- **Executive display tokens:** admin-minted, device-locked `executive_display` tokens (SHA-256-hashed at rest, `exec_` prefix, optional expiry) authorise the login-less Executive dashboard/wallboard. Replaces the earlier `executive_session` JWT handoff, which minted a token for any authenticated user with no role claim — a privilege-escalation path now removed.
- Session JWT refresh on 2FA enable/disable (hotfix 2026-05-23) so middleware doesn't bounce users back to `/enroll-2fa` with stale tokens.
- Login form structurally cannot leak credentials to the URL (5 layered defences asserted by the `hotfix-login-credential-leak.ts` smoke).
- Roles: `ADMIN`, `EDITOR`, `VIEWER`, `ADMIN_READ_ONLY`, `PENDING` (pending-approval flow).
- Brute-force protection on `/api/auth/callback/credentials`: 5 attempts per IP per hour.

### 3.4 Outbound-egress security (SSRF defence)

- All outbound monitoring + notification requests route through `pinnedFetch`:
  - Single DNS resolution per request, pinned IP handed to `http.request`.
  - Host header + TLS SNI preserved for vhost compatibility.
  - No TOCTOU window between resolve and connect (closed by EA-2/PR-33).
- `isAlwaysBlocked` ranges (link-local IPv4 `169.254/16` + IPv6 `fe80::/10`) cannot be unlocked by `ALLOW_PRIVATE_NETWORK`. Cloud metadata endpoints stay blocked.
- Redirect re-validation: every 3xx hop runs through `resolveAndValidate` again. The "200 OK → 302 to `http://127.0.0.1`" exploit class is closed.
- `allowedHosts` bypass exists for Telegram + PagerDuty (operator-controlled allowlist; SaaS endpoints where DNS rebinding is not in the threat model).

### 3.5 Auditability and tamper-evidence

- **`AuditLog` table:** every state-changing action records an entry with actor, action, resource, payload, IP, user-agent.
- **Hash chain (EA-1/PR-32):** every row carries `sha256(prev_hash || canonical(this_row))`. BEFORE UPDATE/DELETE triggers raise `SIGNAL SQLSTATE '45000'` to make rows immutable at the DB layer. The chain is validable end-to-end; tampering anywhere breaks subsequent hashes.
- **Export (PR-23):** CSV and CEF formats with local rotation; routes through `AuditExportService`. The chain `verifyChain` CLI validates end-to-end integrity.

### 3.6 Observability

- **Prometheus metrics:** `/api/metrics` exposes monitor counts, check latencies (histograms), outbox depth, dedup hits, SSE listener count.
- **Health endpoints:**
  - `/api/health` — shallow liveness (no auth; orchestrator-safe).
  - `/api/health/deep` — schema + connectivity + outbox + worker + rollup; requires `Authorization: Bearer $CRON_SECRET`.
- **OpenTelemetry (PR-28):** opt-in via `OTEL_TRACES_ENABLED=true`. Exporter endpoint configurable via `OTEL_EXPORTER_OTLP_ENDPOINT`. Auto-instrumentation patches at startup; no-op when disabled.
- **Sentry:** error capture wired through `@/lib/observability/sentry` init in `instrumentation.ts`.

### 3.7 Maintenance windows

- `MaintenanceWindow` table suppresses alerts during scheduled work. Affected monitors flagged via M-N table. Status page renders the window. Notification engine consults the window before emitting alerts.

### 3.8 Reporting and dashboards

- **Dashboard:** real-time KPIs (uptime %, MTTR, current incidents), 90-day uptime ribbons, region heatmap, latency sparklines. _(SLA error budget widget was removed in audit2-18 / PR #72 along with the SLATarget + SLAReport tables — feature was unused.)_
- **Executive Command dashboard (`/executive`):** a Mission Control rail with four screens — Overview (KPI bento), Network, Incidents, Performance — plus an auto-rotating 1080p wallboard at `/executive/wall`. The **Network screen renders a real vector world map** (d3-geo + Natural Earth topojson via `world-atlas`) that plots each monitor at its resolved geolocation (host→IP→lat/lon/city via `GeoLocationService`, written to the `Monitor` row) and zooms into a country on click to show where its connections sit; monitors with no resolved geo fall back to a country centroid. Authorised in-session (ADMIN/EDITOR) or via a device-locked `executive_display` token. Built on d3-geo directly because react-simple-maps caps its React peer dep at 18.x (this repo is React 19).
- **Scheduled reports (P1-10):** weekly PDF executive summaries to configured recipients.
- **Public status page:** per-monitor status + incident timeline + maintenance windows. Email subscribers (PR-19): self-serve subscribe/confirm/unsubscribe + incident fan-out.

### 3.9 Executive display mode

- Login-less display at `/executive/*` (rail dashboard) and `/executive/wall` (auto-rotating wallboard), authorised by an admin-minted, device-locked `executive_display` token (usable only on the device it first authenticated from). Intended for unattended TVs / kiosks.
- _(Supersedes the earlier `/noc/*` NOC display + `noc_display` token subsystem, removed 2026-05-30 to eliminate two parallel display-token systems. The `executive_display` token is now the single display credential.)_

### 3.10 AI chat (PR-D)

- Internal `/dashboard/ai` chat surface with permission-scoped tool calls:
  - List/inspect monitors, maintenance windows, notification rules and channels.
  - Read audit log entries.
  - Inspect outbox status.
  - Query monitor latency history.
- Tool access enforced by role; `ADMIN_READ_ONLY` consistency-fixed in PR-D.

---

## 4. Architecture

### 4.1 Topology (ADR-0001)

- **Web/API process:** Next.js 16 App Router on Node 24. Handles dashboard, REST API, SSE, status page.
- **Worker process:** standalone `scripts/worker.ts` runs the monitor tick loop, outbox drain, rollup tick, heartbeat sweeper. Shares the MySQL DB with the web process.
- **Single-instance guard (EA-8/PR-39):** on startup, `instrumentation.ts` acquires MySQL `GET_LOCK('uptime-sentinel-web', 0)`. A second web process against the same DB refuses to boot. Disable via `SINGLE_INSTANCE_GUARD=false` only for emergency debug on a sacrificial DB. Runbook: [docs/deploy/single-instance.md](deploy/single-instance.md).
- **Database:** MySQL 8 (Prisma 5 client). SQLite supported for local dev only.

### 4.2 Monitor engine (ADR-0001)

- **Claim model (P2-1):** `claimMonitors` uses `SELECT ... FOR UPDATE SKIP LOCKED` inside a `ReadCommitted` transaction to grab a batch (default 20) of pending checks. Each row's `nextCheckAt` advances atomically.
- **Strategy pattern:** `MonitorStrategy` interface implemented per type (`http.strategy.ts`, `ping.strategy.ts`, `tcp.strategy.ts`, etc.).
- **Hysteresis:** `requiredDowns` (= `Monitor.retries`, default 2) and `requiredUps` (= `Monitor.requiredUps`) gate state transitions to suppress flap.
- **Per-monitor dedup (PR-B):** `OutboxDedupKey` prevents two `monitor.down` events from emitting the same notification within `DEDUP_WINDOW_SECONDS`.

### 4.3 Notification pipeline

- `monitor-engine` writes a `Heartbeat` row and (on state change) emits to `EventBus`.
- `notification-worker.ts` registers async EventBus listeners. On `monitor.down`/`monitor.up`/`user.login`, dispatches via `NotificationDispatcher`.
- Dispatcher writes a `NotificationOutbox` row, then the outbox tick drains it through the channel layer with exponential backoff.
- Channel layer: `email.channel`, `slack.channel`, `webhook.channel` (HMAC-signed), `pagerduty.channel`, `teams.channel`, `telegram.channel`.

### 4.4 Encryption at rest (ADR-0003)

- `secret-vault.ts` provides `encrypt()` / `decrypt()` using AES-256-GCM with 12-byte IV, 16-byte auth tag, `v1:` envelope prefix.
- `CHANNEL_ENCRYPTION_KEY` env var (base64-decoded to 32 bytes) is required at process start.
- Currently encrypted: `NotificationChannel.config`.
- **Not yet encrypted (see [audit-findings](#7-known-issues--open-hardening)):** `User.totpSecret`, `SystemSetting.value` (incl. SMTP password), `MonitorHeader.value`, `Monitor.heartbeatToken`.

### 4.5 CSP, headers, and middleware (PR-18)

- Per-request CSP nonce generated in middleware, propagated to layout via `x-nonce` request header. `script-src` drops `unsafe-inline` and `unsafe-eval` in production.
- HSTS, X-Content-Type-Options, X-Frame-Options, Referrer-Policy, Permissions-Policy all set.
- `/login` gets `Referrer-Policy: no-referrer` to prevent same-origin URL leakage.
- Static-asset extensions (`*.png/jpg/svg/webp/woff/etc.`) excluded from the auth matcher so `/public/` files load without 307→signin redirect (fix PR-40).

---

## 5. Non-functional requirements

| Concern | Target |
|---|---|
| **Footprint** | One Next.js process + one worker process + one MySQL DB. Single VPS deployable. Operator runs `docker-compose up`. |
| **Scale** | Designed for 100–500 monitors per instance at 30–60s tick. >500 monitors needs concurrency tuning (see §7 architecture follow-ups). |
| **Liveness floor** | A real outage MUST emit at least one notification within `intervalSeconds + retries × retryIntervalSeconds + outbox_tick`. Worker resilience guarantees: `isRunningMonitorTick` skip on overrun is logged. |
| **Auditability** | Every state-changing admin action recorded. Audit log is tamper-evident (hash chain + BEFORE-trigger immutability). |
| **SSRF posture** | Default: no probes hit RFC1918, link-local, or cloud metadata. `ALLOW_PRIVATE_NETWORK=true` unlocks RFC1918 only — link-local stays blocked unconditionally. |
| **Recovery time** | Worker restart should resume within one tick interval. Outbox entries survive process death (durable in MySQL). Single-instance lock auto-releases on connection close. |

---

## 6. Hardening work completed

Highlights merged to main (full detail in git history):

### Phase 0 (2026-05-21, PR #1) — 12 critical security fixes
P0-1 cron auth, P0-2 SSRF v1, P0-3 channel encrypt, P0-4 AI scope, P0-5 monitor scope, P0-6 2FA brute, P0-7 users list, P0-8 strict env validation, P0-9 soft-delete, P0-10 Prisma log redaction, P0-11 security headers, P0-12 body-size limits.

### Phase 1 (2026-05-21, PR #4) — Reliability + dedup
P1-1 retry interval, P1-2 dispatcher context, P1-7 email dedup, P1-11 outbox backoff + meta-alert.

### Phase 2 (2026-05-21, PR #3) — Worker hardening
P2-1 SKIP LOCKED claim, P2-8 worker resilience, P2-10 deep-health endpoint, P2-5 rollup tick (raw heartbeats → HeartbeatHourly with default 7-day retention).

### Themes A–D
A: 2FA-on-login modal flow. B: `errorClass` classification. C: maintenance windows. D: observability primitives.

### May 2026 series
- **PR-A**: IDOR fixes on notification-rules + synthetic-tests (synthetic side dropped in PR #71).
- **PR-B**: heartbeat sweeper, up-hysteresis, per-monitor dedup, `/api/health/deep` auth, worker lock TTL.
- **PR-C**: SSE listener metric.
- **PR-18**: CSP nonce wiring.
- **PR-D**: AI chat tool-calling (6 new operational-data tools).
- **PR-19**: public status-page email subscribers.
- **PR-23**: audit-log export (CSV + CEF).
- **PR-26**: escalation enforcement.
- **PR-27**: PagerDuty + Microsoft Teams channels.
- **PR-28**: OpenTelemetry bootstrap contract.
- **PR-29**: Entra ID SSO.
- **PR-30**: 2FA session-refresh on enable/disable.
- **Hotfix**: login credential URL-leak (5 layered defences).
- **Hotfix**: 2FA stale-JWT after enable/disable.

### EA-1..EA-8 series (May 2026, the "Evidence Action readiness" batch)
- **EA-1 / PR-32**: tamper-evident `AuditLog` — BEFORE UPDATE/DELETE triggers + SHA-256 hash chain.
- **EA-2 / PR-33**: DNS-rebinding TOCTOU closed in safe-fetch via pinned-IP + SNI preservation.
- **EA-3 / PR-34**: `/api/status` URL redaction + public `Monitor.displayName`.
- **EA-4 / PR-35**: dropped silently-broken `jsonPath`/`jsonExpectedValue` columns + `jsonpath-plus` dep.
- **EA-5 / PR-36**: Playwright spec for 2FA-disable JWT-rejection (test.fixme'd pending TOTP fixture).
- **EA-6 / PR-37**: `Monitor.retries` default lowered 3→2 + exposed in UI; guards engine hysteresis.
- **EA-7 / PR-38**: outbound webhook HMAC-SHA256 signing.
- **EA-8 / PR-39**: single-instance guard via MySQL `GET_LOCK` + deploy docs.

### PR-40 (open, awaiting review as of 2026-05-23)
- Login page brand redesign: Evidence Action navy/magenta on warm gradient, network packet animation reimagined (navy packets + magenta heartbeat + endpoint nodes).
- Middleware matcher exclusion for `/public/` static-asset extensions (PNG/JPG/SVG/etc.) — was 307'ing logo through auth.
- Hide "Welcome back" heading while preserving its vertical space.

---

## 7. Hardening status

The audit-1 + audit-2 + audit-2 sub-sprint findings (2026-05-21 → 2026-05-24) are now closed at the code level. Headlines:

### Closed (PR-32 .. #73)

| Original finding | PR | Notes |
|---|---|---|
| XFF spoofing on rate limiter | #42 (AUDIT-1) | Rightmost-hop only, default direct-mode bucket |
| `NEXTAUTH_SECRET` hardcoded fallback | #43 (AUDIT-2) | Deleted; throws on missing env |
| TOTP secrets plaintext at rest | #48 (AUDIT-3) | `secret-vault` envelope, lazy upgrade |
| SMTP password plaintext | #49 (AUDIT-4) | Encrypted; GET masks secrets |
| `MonitorHeader.value` plaintext | #50 (AUDIT-5) | Encrypted at rest |
| `Monitor.heartbeatToken` plaintext | #52 (AUDIT-6) | SHA-256 hash at rest |
| AES-GCM envelope missing AAD | #51 (AUDIT-7) | v2 envelope with AAD bound to row identifier |
| NOC device-lock pipe-packed | #53 (AUDIT-8) | Proper columns added. NOC subsystem since removed (2026-05-30); those columns now back the `executive_display` token. |
| `Promise.all` fan-out OOM risk | #55 (AUDIT-9) | Inline `boundedPool` helper (5-wide) |
| Missing `prisma migrate deploy` | #47 (AUDIT-10) → #61 (audit2-3) | Entrypoint runs deploy; manual SQL re-encoded as real migrations |
| Native `bcrypt^6` | #54 (AUDIT-11) | Swapped to `bcryptjs` |
| `ping.strategy` requires NET_RAW | #46 (AUDIT-12) | Runtime detection + runbook |
| `@types/*` in dependencies | #44 (AUDIT-13) | Moved to devDependencies |
| OpenTelemetry eagerly loaded | #45 (AUDIT-14) | Lazy import behind `OTEL_TRACES_ENABLED` |
| Audit-2 sub-sprint (perf + Tier-1 + Tier-2 + dead-code drops) | #62..#73 | Perf bundle + Tier-1/Tier-2 hardening + dead-code removal |

End-to-end Command Center load (against Evidence Action's WAN-DB deployment) improved from **~8 min → ~6 s** through the audit2-7/8/9/10/11 perf bundle.

### Still deferred

- **#10 KEYRING / key-rotation primitive** — security debt that only realises on compromise.
- **#11 `prisma migrate diff` CI guard** — would catch schema/migration drift before merge.
- **#12 EventBus async-on-sync refactor** — needs design discussion (BullMQ vs pg-boss vs Postgres LISTEN/NOTIFY). Highest-priority architectural debt remaining.
- **#14 Heartbeat partitioning** — DB-level work for the 21M+ rows/month retention.
- **#16 Per-tick `await import()` in monitor-engine** — refactor the circular-import root cause first.
- **#17 (rest)** dependency prune: `openai`+`ChatMessage` models, `leaflet`, `jspdf`, `framer-motion` — product-level decision.
- **#18 market features** — Cloudflare Tunnel integration, GitOps YAML, Wasm synthetics.
- **#19** SQLite option + zero-env defaults + single-container template (distribution friction).
- **#20** NextAuth → Auth.js 5 migration.

---

## 8. Out of scope (explicit non-goals)

- **Multi-tenant SaaS.** Tenancy work is deferred indefinitely (ADR-0002).
- **OneUptime / Datadog parity.** Uptime Sentinel intentionally stays narrow: monitoring, alerting, audit, status page. No APM, no log aggregation, no incident management beyond minimal timeline tracking.
- **Hosted offering by Evidence Action.** This is a self-hosted tool first.

---

## 9. Strategic positioning

| Beat | Lose | Net |
|---|---|---|
| Kuma on security primitives (SSRF, HMAC webhooks, tamper-evident audit log) | Kuma on footprint (one container vs Next.js + worker + MySQL) | Different segment — security-conscious operators |
| Gatus on usability (Kuma-style UI for monitor CRUD) | Gatus on GitOps (no YAML config sync today) | Adding `monitors.yaml → DB` sync is a wedge — proposed §7 follow-up |
| Checkly on price (self-host vs SaaS) | Checkly on synthetic-runner maturity | Synthetic-test feature was dropped in audit2-17e (0 rows live, no runner); could be re-introduced as a market wedge if customer demand emerges |
| BetterStack on transparency (audit hash chain) | BetterStack on polish | Trade-off accepted |

**Wedge feature candidates** (not built yet, ranked by leverage):
- Declarative `monitors.yaml → DB` sync for the GitOps-aware SRE segment.
- Native Tailscale / Cloudflare Tunnel egress for internal-network monitoring from a hosted instance.
- Programmable synthetics — let users upload `*.spec.ts` files as monitors.
- `--read-only` deploy mode for public status-page-only instances.

---

## 10. References

- Migration resolve sequence: [runbooks/prisma-migrations.md](runbooks/prisma-migrations.md).
- Secret rotation flow: [runbooks/key-rotation.md](runbooks/key-rotation.md).
- Webhook HMAC spec: [notification-channels/webhook-hmac.md](notification-channels/webhook-hmac.md).
- Single-instance guard: [deploy/single-instance.md](deploy/single-instance.md).
- REST API: [openapi.yaml](../openapi.yaml).
- Smoke regression gate: `scripts/smoke/run-all.ts` (91 offline checks; 1 live-only).
- Project memory (per-AI): `~/.claude/projects/e--Running-Apps-Network-Monitor-App-uptime-sentinel/memory/`.

_(The earlier HARDENING_PLAN, ADRs, threat-model, audit-followup, and most ops runbooks were retired on 2026-05-31 to keep this PRD the single canonical doc; their decisions are folded into §4 and §7 above and remain in git history.)_