task ep.1.4.1

2026-07-13 13:08:00 +07:00
parent 06af36156f
commit bba4c8d97d
18 changed files with 9388 additions and 192 deletions
--- a/plans/task-ep.1.4.1.md
+++ b/plans/task-ep.1.4.1.md
@@ -0,0 +1,909 @@
+# Task EP.1.4.1 – Transactional Outbox Activation & Worker Runtime
+
+Status: Completed
+Priority: Critical
+Type: Platform Runtime / Delivery Reliability / Background Processing
+
+## Depends On
+
+* EP.1.1 Activity Domain Foundation
+* EP.1.2 Activity Integration & Legacy Follow-up Adapter
+* EP.1.3 Business Event Foundation
+* EP.1.4 Projection Foundation & Delivery Reliability
+* AR.1 Architecture Transition Plan
+* AR.2 Epic & Technical Design
+* ENG.0 Engineering Constitution
+
+---
+
+# Objective
+
+Activate the Transactional Outbox delivery model introduced in EP.1.4 and provide a production-ready worker runtime for reliable Business Event dispatch.
+
+This task must ensure that:
+
+* source-domain mutations and Business Event outbox persistence occur atomically
+* only committed outbox events are dispatched
+* projection consumers process events independently and idempotently
+* worker execution is safe across multiple application instances
+* retryable failures are rescheduled
+* terminal failures are moved to governed dead-letter state
+* successful source mutations are never reported as failed merely because a projection consumer fails after commit
+* existing CRM APIs and business behavior remain backward compatible
+
+The first activated source domain is Activity.
+
+Other source domains may adopt the same transaction pattern in later controlled tasks.
+
+---
+
+# Background
+
+EP.1.3 introduced:
+
+* canonical `BusinessEvent`
+* Event Registry
+* Publisher and Dispatcher abstractions
+* Activity lifecycle event publication
+* Event Sequence Matrix
+* Event Consumer Matrix
+
+EP.1.4 introduced:
+
+* `business_event_outbox`
+* projection consumer checkpoints
+* projection dead letters
+* projection rebuild runs
+* projection health snapshots
+* persistent projection runtime contracts
+* retry policy
+* Projection Registry
+* no-op skeleton consumers
+
+However, production delivery guarantees are not active yet because:
+
+1. Activity source mutations do not yet persist events atomically through the outbox in every mutation path.
+2. No worker currently drains pending outbox events.
+3. No multi-instance-safe event claiming process exists.
+4. No operational retry or dead-letter processing loop is active.
+5. Manual retry and drain contracts are not yet available.
+
+EP.1.4.1 activates these foundations.
+
+---
+
+# Architecture Principles
+
+## 1. Atomic Source Mutation and Event Persistence
+
+The source mutation and related outbox event must use the same database transaction.
+
+Required pattern:
+
+```text
+BEGIN
+
+Update source-domain state
+
+Insert Business Event Outbox record
+
+Write mandatory Audit Log where required
+
+COMMIT
+```
+
+If outbox persistence fails inside the transaction, the source mutation must roll back.
+
+---
+
+## 2. Post-Commit Consumer Processing
+
+Projection consumers process events only after the source transaction commits.
+
+A projection failure must not change the result of the already committed source mutation.
+
+Example:
+
+```text
+Activity completed
+        ↓
+Activity row + Outbox event committed
+        ↓
+API returns success
+        ↓
+Worker dispatches event
+        ↓
+Timeline consumer fails
+        ↓
+Checkpoint schedules retry
+```
+
+---
+
+## 3. At-Least-Once Delivery
+
+The runtime must assume that an event may be delivered more than once.
+
+Exactly-once transport is not required.
+
+Consumer idempotency and persistent checkpoints provide duplicate protection.
+
+---
+
+## 4. Consumer Isolation
+
+An event may have multiple consumers.
+
+Example:
+
+```text
+activity.completed
+├── Timeline
+├── Calendar
+├── Dashboard
+└── Notification
+```
+
+One failed consumer must not erase successful processing by other consumers.
+
+Retries must target only incomplete consumers.
+
+---
+
+## 5. Multi-Instance Safety
+
+More than one application or worker process may run concurrently.
+
+Only one worker may claim a given event delivery unit at a time.
+
+The design must support:
+
+* database-level claiming
+* lease expiration
+* stale claim recovery
+* duplicate-safe retries
+* graceful worker shutdown
+
+---
+
+## 6. Preserve Existing Behavior
+
+Existing Activity APIs and source-domain behavior remain unchanged from the caller’s perspective.
+
+This task must not replace:
+
+* existing follow-up APIs
+* Dashboard datasets
+* Report datasets
+* existing Notification behavior
+* existing Approval notification flow
+* recent-activity compatibility reads
+
+---
+
+# Review Required
+
+## Governance and Architecture
+
+* `AGENTS.md`
+* `docs/standards/engineering-constitution.md`
+* `docs/standards/project-foundations.md`
+* `docs/standards/architecture-rules.md`
+* `docs/standards/task-review-checklist.md`
+* `docs/security/crm-authorization-boundaries.md`
+* AR.1 Architecture Transition Plan
+* AR.2 Epic & Technical Design
+
+## Existing Implementations
+
+* EP.1.3 Business Event Foundation report and code
+* EP.1.4 Projection Foundation report and code
+* Activity service mutation paths
+* Business Event publisher
+* persistent outbox publisher
+* projection runtime
+* projection consumer checkpoints
+* projection dead letters
+* projection health snapshots
+* current Drizzle transaction conventions
+* existing scheduled-job or worker patterns
+* current application startup and shutdown behavior
+* current deployment topology and Docker runtime, where documented
+
+---
+
+# Scope
+
+## Part 1 — Activity Transaction Boundary Refactor
+
+Refactor Activity mutation paths to support transaction-scoped persistence.
+
+Minimum operations:
+
+* create
+* update
+* assign
+* reassign
+* start
+* reschedule
+* complete
+* cancel
+* delete, if governed deletion remains supported
+
+Each operation must persist:
+
+* Activity mutation
+* Business Event Outbox record
+* required Audit Log record
+
+inside one shared transaction where applicable.
+
+### Rules
+
+* Route Handlers remain thin.
+* Activity Service owns business transitions.
+* Transaction object must be propagated through approved service and persistence seams.
+* No nested independent transaction may break atomicity.
+* Event payload is created from the successful mutation result.
+* An event must never be inserted before business validation completes.
+
+---
+
+## Part 2 — Transaction-Aware Outbox Publisher
+
+Implement or complete a transaction-aware outbox publisher.
+
+Recommended contract:
+
+```ts
+export interface TransactionalBusinessEventPublisher {
+  enqueue(
+    transaction: DatabaseTransaction,
+    event: BusinessEvent
+  ): Promise<void>;
+
+  enqueueMany(
+    transaction: DatabaseTransaction,
+    events: BusinessEvent[]
+  ): Promise<void>;
+}
+```
+
+### Responsibilities
+
+* validate registry entry
+* validate schema version
+* validate payload
+* persist event using the supplied transaction
+* preserve correlation and causation metadata
+* prevent duplicate event insertion where an operation idempotency key exists
+* not dispatch consumers synchronously
+
+---
+
+## Part 3 — Outbox Worker Runtime
+
+Introduce a worker that drains committed outbox events.
+
+Worker lifecycle:
+
+```text
+poll
+  ↓
+claim batch
+  ↓
+load active consumers
+  ↓
+dispatch each incomplete consumer
+  ↓
+record checkpoints
+  ↓
+mark event delivery state
+  ↓
+sleep / repeat
+```
+
+### Worker Configuration
+
+Support configurable:
+
+* enabled state
+* polling interval
+* batch size
+* maximum concurrent events
+* maximum concurrent consumers
+* claim lease duration
+* stale claim threshold
+* worker instance identifier
+* graceful shutdown timeout
+
+Configuration must use existing environment and configuration conventions.
+
+---
+
+## Part 4 — Multi-Instance Event Claiming
+
+Implement a database-safe claiming strategy.
+
+Approved approaches may include:
+
+* `FOR UPDATE SKIP LOCKED`
+* atomic conditional update with lease token
+* another PostgreSQL-safe claiming mechanism
+
+Each claimed record should track, as appropriate:
+
+* claimed by
+* claimed at
+* lease expires at
+* processing started at
+* attempt count
+
+### Rules
+
+* two workers must not actively process the same event claim concurrently
+* expired claims must be recoverable
+* a crashed worker must not permanently block delivery
+* claiming must be organization-safe but may process multiple organizations in one batch
+
+---
+
+## Part 5 — Event and Consumer Completion Semantics
+
+Freeze and implement event completion rules.
+
+Recommended model:
+
+* Outbox state represents event dispatch lifecycle.
+* Consumer checkpoint represents per-consumer result.
+* Event is terminally completed only when all applicable active consumers reach a terminal state.
+
+Terminal consumer states:
+
+* completed
+* skipped unsupported version, when governed as terminal
+* dead letter
+* explicitly ignored by registry policy
+
+### Important Rule
+
+A successful consumer must not be called again merely because another consumer failed.
+
+---
+
+## Part 6 — Retry Processor
+
+Activate retry processing using the EP.1.4 policy.
+
+Default policy:
+
+```text
+attempt 1: immediate
+attempt 2: +1 minute
+attempt 3: +5 minutes
+attempt 4: +30 minutes
+attempt 5: dead letter
+```
+
+Implement:
+
+* retry scheduling
+* retryable-error classification
+* non-retryable-error classification
+* next-attempt selection
+* attempt count update
+* consumer-specific retry
+* jitter where appropriate
+* stale-processing recovery
+
+Retry values must be configurable or governed centrally.
+
+---
+
+## Part 7 — Dead-Letter Runtime
+
+Activate dead-letter creation when:
+
+* maximum retries are exceeded
+* failure is explicitly non-retryable
+* event version is unsupported and policy requires review
+* registry or payload incompatibility cannot be resolved automatically
+
+Dead-letter records must include:
+
+* event reference
+* consumer name
+* event type
+* organization id
+* attempt count
+* sanitized error code
+* sanitized error message
+* first failure time
+* final failure time
+* resolution status
+
+Do not duplicate unrestricted event payloads into dead-letter storage.
+
+---
+
+## Part 8 — Manual Retry and Drain Service Contracts
+
+Provide protected service contracts for operations support.
+
+Minimum capabilities:
+
+* drain pending outbox events
+* retry one failed consumer checkpoint
+* retry all failed consumers for one event
+* retry one dead-letter item
+* mark dead letter resolved with reason
+* inspect sanitized processing status
+* recover stale claims
+
+Recommended service methods:
+
+```ts
+drainOutbox(options)
+retryConsumer(eventId, consumerName, actor)
+retryEvent(eventId, actor)
+retryDeadLetter(deadLetterId, actor)
+resolveDeadLetter(deadLetterId, reason, actor)
+recoverStaleClaims(options)
+```
+
+Public UI is not required.
+
+Admin API routes are optional and should only be added when authorization and operational need are clearly defined.
+
+---
+
+## Part 9 — Worker Startup Strategy
+
+Define how the worker runs in current deployment environments.
+
+Evaluate:
+
+* worker inside application process
+* dedicated Node worker process
+* scheduled CLI drain command
+* separate Docker service using the same application image
+
+Preferred production direction:
+
+> Dedicated worker process or service using shared runtime modules.
+
+The implementation must not accidentally start duplicate unmanaged polling loops during:
+
+* Next.js hot reload
+* serverless request execution
+* build phase
+* migration commands
+* test execution
+
+---
+
+## Part 10 — Graceful Shutdown
+
+Worker shutdown must:
+
+* stop claiming new events
+* allow active handlers to finish within timeout
+* release or expire unfinished leases safely
+* flush structured logs
+* report final worker status
+
+Handle common termination signals according to runtime conventions.
+
+---
+
+## Part 11 — Projection Health Updates
+
+Update projection health from real worker processing.
+
+Track:
+
+* last event observed
+* last successful event
+* pending count
+* processing count
+* retry count
+* dead-letter count
+* unsupported-version count
+* last failure time
+* processing lag
+* worker heartbeat
+* last successful drain
+
+Health storage must remain payload-free.
+
+---
+
+## Part 12 — Delivery SLA Baseline
+
+Freeze initial operational targets.
+
+Suggested targets:
+
+| Capability                                     |                                  Initial Target |
+| ---------------------------------------------- | ----------------------------------------------: |
+| Outbox event available after commit            |                                       immediate |
+| Worker polling interval                        |                                     1–5 seconds |
+| Activity projection delivery under normal load |                                under 10 seconds |
+| Retry scheduling accuracy                      |                     within one polling interval |
+| Stale claim recovery                           | within lease duration plus one polling interval |
+| Worker health heartbeat                        |                             every 30–60 seconds |
+
+Final values must reflect deployment constraints and can be classified as initial non-binding SLOs.
+
+---
+
+## Part 13 — Security and Authorization
+
+Worker processing must:
+
+* maintain organization scoping
+* not bypass pricing-sensitive event restrictions
+* not expose internal-only Activity details
+* use consumer-specific security rules
+* avoid persisting decrypted credentials or secrets
+* sanitize operational errors
+* audit manual retry and dead-letter resolution actions
+
+Manual operations must require explicit foundation or system administration permission.
+
+---
+
+## Part 14 — Existing Publisher Compatibility
+
+Preserve the existing in-process publisher where needed for:
+
+* tests
+* non-production isolated execution
+* compatibility consumers not yet cut over
+
+However, Activity production mutation paths must use the transactional outbox publisher after activation.
+
+Do not publish the same Activity event through both in-process and outbox paths.
+
+Add a guard or configuration strategy preventing accidental double publication.
+
+---
+
+## Part 15 — Consumer Registration Snapshot
+
+Freeze the consumer set used when determining applicable checkpoints.
+
+Define behavior when:
+
+* a new consumer is registered after an event was completed
+* a consumer is disabled
+* a consumer is renamed
+* a consumer’s supported versions change
+* an event has no consumers
+* a projection remains contract-only
+
+Recommended approach:
+
+* persist or deterministically resolve applicable consumers at dispatch time
+* document replay or backfill behavior for newly introduced consumers
+* avoid reopening old completed events automatically without explicit rebuild
+
+---
+
+## Part 16 — Delivery State Machine
+
+Freeze the outbox delivery lifecycle.
+
+Recommended model:
+
+```text
+pending
+  ↓
+claimed
+  ↓
+processing
+  ├── completed
+  ├── retry_scheduled
+  └── dead_letter
+```
+
+Possible recovery paths:
+
+```text
+claimed / processing
+  ↓ lease expired
+pending
+```
+
+Consumer checkpoint lifecycle:
+
+```text
+pending
+  ↓
+processing
+  ├── completed
+  ├── retry_scheduled
+  ├── skipped_unsupported_version
+  └── dead_letter
+```
+
+Implement valid transitions and reject invalid direct state jumps.
+
+---
+
+## Part 17 — End-to-End Reliability Scenario
+
+Implement an end-to-end validation path for at least one Activity operation.
+
+Required scenario:
+
+```text
+Complete Activity
+        ↓
+Activity mutation persisted
+        ↓
+activity.completed Outbox event persisted in same transaction
+        ↓
+API returns success
+        ↓
+Worker claims event
+        ↓
+Skeleton consumers execute
+        ↓
+Consumer checkpoints become completed
+        ↓
+Duplicate dispatch is skipped
+```
+
+Failure variation:
+
+```text
+One consumer fails
+        ↓
+Activity remains completed
+        ↓
+Other consumers remain completed
+        ↓
+Failed consumer schedules retry
+        ↓
+Retry completes or moves to dead letter
+```
+
+---
+
+# Deliverables
+
+## 1. Activity Transactional Outbox Adoption
+
+Atomic source mutation, audit, and event enqueue.
+
+## 2. Transaction-Aware Outbox Publisher
+
+Shared transaction-scoped persistence contract.
+
+## 3. Outbox Worker Runtime
+
+Polling, claiming, dispatch, and completion processing.
+
+## 4. Multi-Instance Claiming Strategy
+
+Lease-based or lock-based safe batch processing.
+
+## 5. Per-Consumer Dispatch Semantics
+
+Independent checkpoints and isolated retries.
+
+## 6. Retry Processor
+
+Configured retry and backoff implementation.
+
+## 7. Dead-Letter Runtime
+
+Terminal failure persistence and recovery contract.
+
+## 8. Manual Retry and Drain Services
+
+Protected operational service interfaces.
+
+## 9. Worker Startup and Deployment Strategy
+
+Current development, UAT, and production execution model.
+
+## 10. Graceful Shutdown Handling
+
+Safe worker termination behavior.
+
+## 11. Projection Health Integration
+
+Real worker health and lag metrics.
+
+## 12. Delivery SLA Baseline
+
+Initial delivery and recovery targets.
+
+## 13. Security and Operations Matrix
+
+Worker, retry, dead-letter, and administrative access rules.
+
+## 14. Existing Publisher Compatibility Strategy
+
+No-double-publication and transition rules.
+
+## 15. Consumer Registration Strategy
+
+Rules for new, disabled, renamed, and version-changed consumers.
+
+## 16. Delivery State Machine
+
+Outbox and checkpoint lifecycle rules.
+
+## 17. End-to-End Reliability Tests
+
+Atomicity, worker delivery, retry, duplicate, and recovery scenarios.
+
+## 18. EP.1.5 Readiness Report
+
+Confirm that Timeline can rely on durable incremental Activity events.
+
+---
+
+# Constraints
+
+Must preserve:
+
+* existing Activity API contracts
+* Activity lifecycle business behavior
+* legacy follow-up APIs and storage
+* recent-activity compatibility layer
+* existing Dashboard and Report consumers
+* existing Approval and Notification behavior
+* existing Audit Log behavior
+* existing CRM security and pricing boundaries
+* EP.1.3 Event Contract and Registry
+* EP.1.4 Projection Registry and checkpoint model
+
+Do not implement:
+
+* final Timeline projection
+* Timeline UI
+* final Calendar projection
+* Calendar UI
+* Notification event expansion
+* My Day
+* Manager Workspace
+* Executive Workspace
+* Relationship Health
+* Forecast replacement
+* external Kafka, RabbitMQ, or Redis Streams
+* legacy follow-up migration
+* follow-up dual-write
+* Dashboard source replacement
+
+Do not start unmanaged worker loops inside request handlers or Client Components.
+
+No breaking API changes.
+
+---
+
+# Compatibility Strategy
+
+Activity is the first source domain activated with transactional outbox delivery.
+
+Other source domains remain on their current event or notification paths until dedicated cutover tasks are approved.
+
+During transition:
+
+* Activity uses transactional outbox.
+* Existing Approval notifications remain unchanged.
+* Existing in-process publisher remains available for tests and approved compatibility use.
+* No event may be emitted through both outbox and in-process production paths.
+* Existing projection skeletons remain non-user-facing.
+
+---
+
+# Testing Requirements
+
+## Atomicity Tests
+
+* Activity mutation and outbox insert commit together
+* outbox insert failure rolls back Activity mutation
+* validation failure creates neither mutation nor event
+* audit failure behavior follows existing transaction governance
+* multiple events enqueue in deterministic sequence
+
+## Worker Claim Tests
+
+* one worker claims one event
+* two workers do not process the same active lease
+* stale lease becomes recoverable
+* batch size is honored
+* claim order is deterministic where required
+* graceful shutdown stops new claims
+
+## Consumer Tests
+
+* all active consumers receive the event
+* one failure does not invalidate successful consumers
+* completed checkpoint skips duplicate execution
+* unsupported version follows policy
+* no-consumer event terminates safely
+* disabled consumer behavior follows registry rule
+
+## Retry Tests
+
+* retryable failure schedules next attempt
+* non-retryable failure creates dead letter
+* maximum attempts create dead letter
+* retry affects only incomplete consumer
+* manual retry does not rerun completed consumers
+* dead-letter retry preserves original event id
+
+## Restart and Recovery Tests
+
+* pending event survives process restart simulation
+* retry-scheduled checkpoint survives restart
+* stale processing claim is recovered
+* duplicate delivery after restart creates no duplicate effect
+
+## Security Tests
+
+* organization isolation
+* sensitive payload remains minimized
+* dead-letter error is sanitized
+* manual retry requires authorization
+* internal-only Activity does not leak through operational status APIs
+
+## Compatibility Tests
+
+* Activity API response remains unchanged
+* existing recent-activity reads remain unchanged
+* existing follow-up APIs remain unchanged
+* Dashboard and Reports remain unchanged
+* Approval notifications remain unchanged
+* no duplicate Activity Business Event publication occurs
+
+## End-to-End Tests
+
+* Activity create through outbox to consumer completion
+* Activity complete through outbox to consumer completion
+* consumer failure followed by successful retry
+* consumer failure reaching dead letter
+* duplicate dispatch skipped
+* stale claim recovery
+
+---
+
+# Acceptance Criteria
+
+* Activity source mutations persist outbox events atomically.
+* Event persistence failure inside the transaction rolls back the Activity mutation.
+* Worker dispatch occurs only after transaction commit.
+* Activity APIs return success even when a non-owning consumer later fails.
+* Outbox events are claimed safely in multi-instance execution.
+* Consumer checkpoints are persistent and idempotent.
+* Successful consumers are not rerun because another consumer failed.
+* Retry scheduling and backoff operate according to policy.
+* Terminal failures create sanitized dead-letter records.
+* Manual retry and stale-claim recovery service contracts exist.
+* Worker can shut down gracefully.
+* Projection health reflects actual worker state.
+* No Activity event is double-published through in-process and outbox publishers.
+* Existing Follow-up, Dashboard, Report, Approval, Notification, and recent-activity behavior remains unchanged.
+* End-to-end atomicity, dispatch, duplicate, retry, and recovery tests pass.
+* TypeScript, migration checks, targeted lint, and worker/runtime tests pass.
+
+---
+
+# Success Criteria
+
+ALLA OS has an active, production-ready Transactional Outbox delivery path for Activity Business Events.
+
+Source mutations and event persistence are atomic.
+
+Projection delivery is post-commit, independently retryable, idempotent, observable, and safe across process restarts and multiple worker instances.
+
+Future Timeline and Calendar projections can rely on durable incremental Activity events without coupling projection failures to source-domain mutation results.
+
+Ready for:
+
+# EP.1.5 – Timeline Projection Foundation