Skip to content

ERP Plus Observability & Operational Model

This document defines how ERP Plus is observed, monitored, debugged, and operated in production.

It ensures the system is not only well-designed, but also fully operable at scale.

Core Principle

ERP Plus is not a black box.

It is:

a fully observable system where every request, engine, and job can be traced end-to-end.

Observability Layers

ERP Plus observability is divided into 4 layers:

1. Request Tracing Layer

Every request MUST be traceable across:

User Request
→ Authentication (erp_users)
→ Account Resolution (erp_accounts)
→ Engine Execution
→ Background Jobs (erp_workers)
→ Response

Each request MUST have a:

  • trace_id
  • account_id
  • user_id

2. Engine-Level Observability

Each engine MUST expose:

  • logs
  • execution metrics
  • service traces

Example:

erp_inventory:
  - product creation time
  - stock update latency
  - service execution logs

3. Background Job Observability

All async jobs (erp_workers) MUST include:

  • job_id
  • engine_origin
  • retry count
  • execution status
  • failure reason (if any)

4. System Metrics Layer

ERP Plus tracks:

  • request latency
  • engine execution time
  • job queue depth
  • error rates
  • account usage distribution

Logging Strategy

Logs MUST be:

  • structured (JSON preferred)
  • trace-aware
  • account-aware
  • engine-tagged

Example:

{
  "trace_id": "abc123",
  "engine": "erp_inventory",
  "account_id": "account_01",
  "event": "product_created",
  "status": "success"
}

Debugging Flow (CRITICAL)

When something breaks:

1. Identify trace_id
2. Locate engine responsible
3. Inspect logs
4. Check background jobs
5. Validate account context
6. Reproduce flow

Operational Model

ERP Plus production is operated via:

  • Docker containers
  • Kamal deployments
  • GitHub Actions pipelines
  • PostgreSQL logs
  • system monitoring tools (future integration)

Error Handling Strategy

Errors MUST:

  • be logged with trace_id
  • include engine context
  • include account context
  • never leak sensitive data
  • be categorized (system / domain / infra)

Security Observability

Security events MUST be tracked:

  • authentication failures
  • unauthorized access attempts
  • account isolation violations
  • suspicious request patterns

Account Observability

Each account has:

  • usage metrics
  • request volume
  • engine activity breakdown
  • performance impact analysis

Engine Observability Contract

Each engine MUST implement:

  • structured logs
  • service-level metrics
  • trace propagation support
  • error classification

Operational Responsibilities

Backend Engineers

  • ensure logs are emitted correctly
  • trace services properly
  • handle domain errors

DevOps Engineers

  • monitor system health
  • configure alerts
  • maintain observability pipelines
  • manage deployments

QA Engineers

  • validate error scenarios
  • ensure trace completeness
  • reproduce system issues

Frontend Engineers

  • propagate trace_id
  • report errors with context
  • avoid breaking observability chain

Scaling Observability

As ERP Plus grows:

  • observability becomes cross-engine
  • metrics become account-aware
  • tracing becomes distributed
  • monitoring becomes predictive

Mental Model

ERP Plus is:

A system where every action can be traced, measured, and debugged across engines and accounts.

Why this matters

Without observability:

  • debugging becomes guesswork
  • multi-engine systems become opaque
  • production issues become hard to reproduce

With observability:

  • ✔ full system transparency
  • ✔ fast debugging
  • ✔ safe scaling
  • ✔ predictable operations

Final System State

ERP Plus is now:

a modular monolith + governance + evolution model + fully observable production system