910 lines
22 KiB
Markdown
910 lines
22 KiB
Markdown
# Task EP.1.4.1 – Transactional Outbox Activation & Worker Runtime
|
||
|
||
Status: Completed
|
||
Priority: Critical
|
||
Type: Platform Runtime / Delivery Reliability / Background Processing
|
||
|
||
## Depends On
|
||
|
||
* EP.1.1 Activity Domain Foundation
|
||
* EP.1.2 Activity Integration & Legacy Follow-up Adapter
|
||
* EP.1.3 Business Event Foundation
|
||
* EP.1.4 Projection Foundation & Delivery Reliability
|
||
* AR.1 Architecture Transition Plan
|
||
* AR.2 Epic & Technical Design
|
||
* ENG.0 Engineering Constitution
|
||
|
||
---
|
||
|
||
# Objective
|
||
|
||
Activate the Transactional Outbox delivery model introduced in EP.1.4 and provide a production-ready worker runtime for reliable Business Event dispatch.
|
||
|
||
This task must ensure that:
|
||
|
||
* source-domain mutations and Business Event outbox persistence occur atomically
|
||
* only committed outbox events are dispatched
|
||
* projection consumers process events independently and idempotently
|
||
* worker execution is safe across multiple application instances
|
||
* retryable failures are rescheduled
|
||
* terminal failures are moved to governed dead-letter state
|
||
* successful source mutations are never reported as failed merely because a projection consumer fails after commit
|
||
* existing CRM APIs and business behavior remain backward compatible
|
||
|
||
The first activated source domain is Activity.
|
||
|
||
Other source domains may adopt the same transaction pattern in later controlled tasks.
|
||
|
||
---
|
||
|
||
# Background
|
||
|
||
EP.1.3 introduced:
|
||
|
||
* canonical `BusinessEvent`
|
||
* Event Registry
|
||
* Publisher and Dispatcher abstractions
|
||
* Activity lifecycle event publication
|
||
* Event Sequence Matrix
|
||
* Event Consumer Matrix
|
||
|
||
EP.1.4 introduced:
|
||
|
||
* `business_event_outbox`
|
||
* projection consumer checkpoints
|
||
* projection dead letters
|
||
* projection rebuild runs
|
||
* projection health snapshots
|
||
* persistent projection runtime contracts
|
||
* retry policy
|
||
* Projection Registry
|
||
* no-op skeleton consumers
|
||
|
||
However, production delivery guarantees are not active yet because:
|
||
|
||
1. Activity source mutations do not yet persist events atomically through the outbox in every mutation path.
|
||
2. No worker currently drains pending outbox events.
|
||
3. No multi-instance-safe event claiming process exists.
|
||
4. No operational retry or dead-letter processing loop is active.
|
||
5. Manual retry and drain contracts are not yet available.
|
||
|
||
EP.1.4.1 activates these foundations.
|
||
|
||
---
|
||
|
||
# Architecture Principles
|
||
|
||
## 1. Atomic Source Mutation and Event Persistence
|
||
|
||
The source mutation and related outbox event must use the same database transaction.
|
||
|
||
Required pattern:
|
||
|
||
```text
|
||
BEGIN
|
||
|
||
Update source-domain state
|
||
|
||
Insert Business Event Outbox record
|
||
|
||
Write mandatory Audit Log where required
|
||
|
||
COMMIT
|
||
```
|
||
|
||
If outbox persistence fails inside the transaction, the source mutation must roll back.
|
||
|
||
---
|
||
|
||
## 2. Post-Commit Consumer Processing
|
||
|
||
Projection consumers process events only after the source transaction commits.
|
||
|
||
A projection failure must not change the result of the already committed source mutation.
|
||
|
||
Example:
|
||
|
||
```text
|
||
Activity completed
|
||
↓
|
||
Activity row + Outbox event committed
|
||
↓
|
||
API returns success
|
||
↓
|
||
Worker dispatches event
|
||
↓
|
||
Timeline consumer fails
|
||
↓
|
||
Checkpoint schedules retry
|
||
```
|
||
|
||
---
|
||
|
||
## 3. At-Least-Once Delivery
|
||
|
||
The runtime must assume that an event may be delivered more than once.
|
||
|
||
Exactly-once transport is not required.
|
||
|
||
Consumer idempotency and persistent checkpoints provide duplicate protection.
|
||
|
||
---
|
||
|
||
## 4. Consumer Isolation
|
||
|
||
An event may have multiple consumers.
|
||
|
||
Example:
|
||
|
||
```text
|
||
activity.completed
|
||
├── Timeline
|
||
├── Calendar
|
||
├── Dashboard
|
||
└── Notification
|
||
```
|
||
|
||
One failed consumer must not erase successful processing by other consumers.
|
||
|
||
Retries must target only incomplete consumers.
|
||
|
||
---
|
||
|
||
## 5. Multi-Instance Safety
|
||
|
||
More than one application or worker process may run concurrently.
|
||
|
||
Only one worker may claim a given event delivery unit at a time.
|
||
|
||
The design must support:
|
||
|
||
* database-level claiming
|
||
* lease expiration
|
||
* stale claim recovery
|
||
* duplicate-safe retries
|
||
* graceful worker shutdown
|
||
|
||
---
|
||
|
||
## 6. Preserve Existing Behavior
|
||
|
||
Existing Activity APIs and source-domain behavior remain unchanged from the caller’s perspective.
|
||
|
||
This task must not replace:
|
||
|
||
* existing follow-up APIs
|
||
* Dashboard datasets
|
||
* Report datasets
|
||
* existing Notification behavior
|
||
* existing Approval notification flow
|
||
* recent-activity compatibility reads
|
||
|
||
---
|
||
|
||
# Review Required
|
||
|
||
## Governance and Architecture
|
||
|
||
* `AGENTS.md`
|
||
* `docs/standards/engineering-constitution.md`
|
||
* `docs/standards/project-foundations.md`
|
||
* `docs/standards/architecture-rules.md`
|
||
* `docs/standards/task-review-checklist.md`
|
||
* `docs/security/crm-authorization-boundaries.md`
|
||
* AR.1 Architecture Transition Plan
|
||
* AR.2 Epic & Technical Design
|
||
|
||
## Existing Implementations
|
||
|
||
* EP.1.3 Business Event Foundation report and code
|
||
* EP.1.4 Projection Foundation report and code
|
||
* Activity service mutation paths
|
||
* Business Event publisher
|
||
* persistent outbox publisher
|
||
* projection runtime
|
||
* projection consumer checkpoints
|
||
* projection dead letters
|
||
* projection health snapshots
|
||
* current Drizzle transaction conventions
|
||
* existing scheduled-job or worker patterns
|
||
* current application startup and shutdown behavior
|
||
* current deployment topology and Docker runtime, where documented
|
||
|
||
---
|
||
|
||
# Scope
|
||
|
||
## Part 1 — Activity Transaction Boundary Refactor
|
||
|
||
Refactor Activity mutation paths to support transaction-scoped persistence.
|
||
|
||
Minimum operations:
|
||
|
||
* create
|
||
* update
|
||
* assign
|
||
* reassign
|
||
* start
|
||
* reschedule
|
||
* complete
|
||
* cancel
|
||
* delete, if governed deletion remains supported
|
||
|
||
Each operation must persist:
|
||
|
||
* Activity mutation
|
||
* Business Event Outbox record
|
||
* required Audit Log record
|
||
|
||
inside one shared transaction where applicable.
|
||
|
||
### Rules
|
||
|
||
* Route Handlers remain thin.
|
||
* Activity Service owns business transitions.
|
||
* Transaction object must be propagated through approved service and persistence seams.
|
||
* No nested independent transaction may break atomicity.
|
||
* Event payload is created from the successful mutation result.
|
||
* An event must never be inserted before business validation completes.
|
||
|
||
---
|
||
|
||
## Part 2 — Transaction-Aware Outbox Publisher
|
||
|
||
Implement or complete a transaction-aware outbox publisher.
|
||
|
||
Recommended contract:
|
||
|
||
```ts
|
||
export interface TransactionalBusinessEventPublisher {
|
||
enqueue(
|
||
transaction: DatabaseTransaction,
|
||
event: BusinessEvent
|
||
): Promise<void>;
|
||
|
||
enqueueMany(
|
||
transaction: DatabaseTransaction,
|
||
events: BusinessEvent[]
|
||
): Promise<void>;
|
||
}
|
||
```
|
||
|
||
### Responsibilities
|
||
|
||
* validate registry entry
|
||
* validate schema version
|
||
* validate payload
|
||
* persist event using the supplied transaction
|
||
* preserve correlation and causation metadata
|
||
* prevent duplicate event insertion where an operation idempotency key exists
|
||
* not dispatch consumers synchronously
|
||
|
||
---
|
||
|
||
## Part 3 — Outbox Worker Runtime
|
||
|
||
Introduce a worker that drains committed outbox events.
|
||
|
||
Worker lifecycle:
|
||
|
||
```text
|
||
poll
|
||
↓
|
||
claim batch
|
||
↓
|
||
load active consumers
|
||
↓
|
||
dispatch each incomplete consumer
|
||
↓
|
||
record checkpoints
|
||
↓
|
||
mark event delivery state
|
||
↓
|
||
sleep / repeat
|
||
```
|
||
|
||
### Worker Configuration
|
||
|
||
Support configurable:
|
||
|
||
* enabled state
|
||
* polling interval
|
||
* batch size
|
||
* maximum concurrent events
|
||
* maximum concurrent consumers
|
||
* claim lease duration
|
||
* stale claim threshold
|
||
* worker instance identifier
|
||
* graceful shutdown timeout
|
||
|
||
Configuration must use existing environment and configuration conventions.
|
||
|
||
---
|
||
|
||
## Part 4 — Multi-Instance Event Claiming
|
||
|
||
Implement a database-safe claiming strategy.
|
||
|
||
Approved approaches may include:
|
||
|
||
* `FOR UPDATE SKIP LOCKED`
|
||
* atomic conditional update with lease token
|
||
* another PostgreSQL-safe claiming mechanism
|
||
|
||
Each claimed record should track, as appropriate:
|
||
|
||
* claimed by
|
||
* claimed at
|
||
* lease expires at
|
||
* processing started at
|
||
* attempt count
|
||
|
||
### Rules
|
||
|
||
* two workers must not actively process the same event claim concurrently
|
||
* expired claims must be recoverable
|
||
* a crashed worker must not permanently block delivery
|
||
* claiming must be organization-safe but may process multiple organizations in one batch
|
||
|
||
---
|
||
|
||
## Part 5 — Event and Consumer Completion Semantics
|
||
|
||
Freeze and implement event completion rules.
|
||
|
||
Recommended model:
|
||
|
||
* Outbox state represents event dispatch lifecycle.
|
||
* Consumer checkpoint represents per-consumer result.
|
||
* Event is terminally completed only when all applicable active consumers reach a terminal state.
|
||
|
||
Terminal consumer states:
|
||
|
||
* completed
|
||
* skipped unsupported version, when governed as terminal
|
||
* dead letter
|
||
* explicitly ignored by registry policy
|
||
|
||
### Important Rule
|
||
|
||
A successful consumer must not be called again merely because another consumer failed.
|
||
|
||
---
|
||
|
||
## Part 6 — Retry Processor
|
||
|
||
Activate retry processing using the EP.1.4 policy.
|
||
|
||
Default policy:
|
||
|
||
```text
|
||
attempt 1: immediate
|
||
attempt 2: +1 minute
|
||
attempt 3: +5 minutes
|
||
attempt 4: +30 minutes
|
||
attempt 5: dead letter
|
||
```
|
||
|
||
Implement:
|
||
|
||
* retry scheduling
|
||
* retryable-error classification
|
||
* non-retryable-error classification
|
||
* next-attempt selection
|
||
* attempt count update
|
||
* consumer-specific retry
|
||
* jitter where appropriate
|
||
* stale-processing recovery
|
||
|
||
Retry values must be configurable or governed centrally.
|
||
|
||
---
|
||
|
||
## Part 7 — Dead-Letter Runtime
|
||
|
||
Activate dead-letter creation when:
|
||
|
||
* maximum retries are exceeded
|
||
* failure is explicitly non-retryable
|
||
* event version is unsupported and policy requires review
|
||
* registry or payload incompatibility cannot be resolved automatically
|
||
|
||
Dead-letter records must include:
|
||
|
||
* event reference
|
||
* consumer name
|
||
* event type
|
||
* organization id
|
||
* attempt count
|
||
* sanitized error code
|
||
* sanitized error message
|
||
* first failure time
|
||
* final failure time
|
||
* resolution status
|
||
|
||
Do not duplicate unrestricted event payloads into dead-letter storage.
|
||
|
||
---
|
||
|
||
## Part 8 — Manual Retry and Drain Service Contracts
|
||
|
||
Provide protected service contracts for operations support.
|
||
|
||
Minimum capabilities:
|
||
|
||
* drain pending outbox events
|
||
* retry one failed consumer checkpoint
|
||
* retry all failed consumers for one event
|
||
* retry one dead-letter item
|
||
* mark dead letter resolved with reason
|
||
* inspect sanitized processing status
|
||
* recover stale claims
|
||
|
||
Recommended service methods:
|
||
|
||
```ts
|
||
drainOutbox(options)
|
||
retryConsumer(eventId, consumerName, actor)
|
||
retryEvent(eventId, actor)
|
||
retryDeadLetter(deadLetterId, actor)
|
||
resolveDeadLetter(deadLetterId, reason, actor)
|
||
recoverStaleClaims(options)
|
||
```
|
||
|
||
Public UI is not required.
|
||
|
||
Admin API routes are optional and should only be added when authorization and operational need are clearly defined.
|
||
|
||
---
|
||
|
||
## Part 9 — Worker Startup Strategy
|
||
|
||
Define how the worker runs in current deployment environments.
|
||
|
||
Evaluate:
|
||
|
||
* worker inside application process
|
||
* dedicated Node worker process
|
||
* scheduled CLI drain command
|
||
* separate Docker service using the same application image
|
||
|
||
Preferred production direction:
|
||
|
||
> Dedicated worker process or service using shared runtime modules.
|
||
|
||
The implementation must not accidentally start duplicate unmanaged polling loops during:
|
||
|
||
* Next.js hot reload
|
||
* serverless request execution
|
||
* build phase
|
||
* migration commands
|
||
* test execution
|
||
|
||
---
|
||
|
||
## Part 10 — Graceful Shutdown
|
||
|
||
Worker shutdown must:
|
||
|
||
* stop claiming new events
|
||
* allow active handlers to finish within timeout
|
||
* release or expire unfinished leases safely
|
||
* flush structured logs
|
||
* report final worker status
|
||
|
||
Handle common termination signals according to runtime conventions.
|
||
|
||
---
|
||
|
||
## Part 11 — Projection Health Updates
|
||
|
||
Update projection health from real worker processing.
|
||
|
||
Track:
|
||
|
||
* last event observed
|
||
* last successful event
|
||
* pending count
|
||
* processing count
|
||
* retry count
|
||
* dead-letter count
|
||
* unsupported-version count
|
||
* last failure time
|
||
* processing lag
|
||
* worker heartbeat
|
||
* last successful drain
|
||
|
||
Health storage must remain payload-free.
|
||
|
||
---
|
||
|
||
## Part 12 — Delivery SLA Baseline
|
||
|
||
Freeze initial operational targets.
|
||
|
||
Suggested targets:
|
||
|
||
| Capability | Initial Target |
|
||
| ---------------------------------------------- | ----------------------------------------------: |
|
||
| Outbox event available after commit | immediate |
|
||
| Worker polling interval | 1–5 seconds |
|
||
| Activity projection delivery under normal load | under 10 seconds |
|
||
| Retry scheduling accuracy | within one polling interval |
|
||
| Stale claim recovery | within lease duration plus one polling interval |
|
||
| Worker health heartbeat | every 30–60 seconds |
|
||
|
||
Final values must reflect deployment constraints and can be classified as initial non-binding SLOs.
|
||
|
||
---
|
||
|
||
## Part 13 — Security and Authorization
|
||
|
||
Worker processing must:
|
||
|
||
* maintain organization scoping
|
||
* not bypass pricing-sensitive event restrictions
|
||
* not expose internal-only Activity details
|
||
* use consumer-specific security rules
|
||
* avoid persisting decrypted credentials or secrets
|
||
* sanitize operational errors
|
||
* audit manual retry and dead-letter resolution actions
|
||
|
||
Manual operations must require explicit foundation or system administration permission.
|
||
|
||
---
|
||
|
||
## Part 14 — Existing Publisher Compatibility
|
||
|
||
Preserve the existing in-process publisher where needed for:
|
||
|
||
* tests
|
||
* non-production isolated execution
|
||
* compatibility consumers not yet cut over
|
||
|
||
However, Activity production mutation paths must use the transactional outbox publisher after activation.
|
||
|
||
Do not publish the same Activity event through both in-process and outbox paths.
|
||
|
||
Add a guard or configuration strategy preventing accidental double publication.
|
||
|
||
---
|
||
|
||
## Part 15 — Consumer Registration Snapshot
|
||
|
||
Freeze the consumer set used when determining applicable checkpoints.
|
||
|
||
Define behavior when:
|
||
|
||
* a new consumer is registered after an event was completed
|
||
* a consumer is disabled
|
||
* a consumer is renamed
|
||
* a consumer’s supported versions change
|
||
* an event has no consumers
|
||
* a projection remains contract-only
|
||
|
||
Recommended approach:
|
||
|
||
* persist or deterministically resolve applicable consumers at dispatch time
|
||
* document replay or backfill behavior for newly introduced consumers
|
||
* avoid reopening old completed events automatically without explicit rebuild
|
||
|
||
---
|
||
|
||
## Part 16 — Delivery State Machine
|
||
|
||
Freeze the outbox delivery lifecycle.
|
||
|
||
Recommended model:
|
||
|
||
```text
|
||
pending
|
||
↓
|
||
claimed
|
||
↓
|
||
processing
|
||
├── completed
|
||
├── retry_scheduled
|
||
└── dead_letter
|
||
```
|
||
|
||
Possible recovery paths:
|
||
|
||
```text
|
||
claimed / processing
|
||
↓ lease expired
|
||
pending
|
||
```
|
||
|
||
Consumer checkpoint lifecycle:
|
||
|
||
```text
|
||
pending
|
||
↓
|
||
processing
|
||
├── completed
|
||
├── retry_scheduled
|
||
├── skipped_unsupported_version
|
||
└── dead_letter
|
||
```
|
||
|
||
Implement valid transitions and reject invalid direct state jumps.
|
||
|
||
---
|
||
|
||
## Part 17 — End-to-End Reliability Scenario
|
||
|
||
Implement an end-to-end validation path for at least one Activity operation.
|
||
|
||
Required scenario:
|
||
|
||
```text
|
||
Complete Activity
|
||
↓
|
||
Activity mutation persisted
|
||
↓
|
||
activity.completed Outbox event persisted in same transaction
|
||
↓
|
||
API returns success
|
||
↓
|
||
Worker claims event
|
||
↓
|
||
Skeleton consumers execute
|
||
↓
|
||
Consumer checkpoints become completed
|
||
↓
|
||
Duplicate dispatch is skipped
|
||
```
|
||
|
||
Failure variation:
|
||
|
||
```text
|
||
One consumer fails
|
||
↓
|
||
Activity remains completed
|
||
↓
|
||
Other consumers remain completed
|
||
↓
|
||
Failed consumer schedules retry
|
||
↓
|
||
Retry completes or moves to dead letter
|
||
```
|
||
|
||
---
|
||
|
||
# Deliverables
|
||
|
||
## 1. Activity Transactional Outbox Adoption
|
||
|
||
Atomic source mutation, audit, and event enqueue.
|
||
|
||
## 2. Transaction-Aware Outbox Publisher
|
||
|
||
Shared transaction-scoped persistence contract.
|
||
|
||
## 3. Outbox Worker Runtime
|
||
|
||
Polling, claiming, dispatch, and completion processing.
|
||
|
||
## 4. Multi-Instance Claiming Strategy
|
||
|
||
Lease-based or lock-based safe batch processing.
|
||
|
||
## 5. Per-Consumer Dispatch Semantics
|
||
|
||
Independent checkpoints and isolated retries.
|
||
|
||
## 6. Retry Processor
|
||
|
||
Configured retry and backoff implementation.
|
||
|
||
## 7. Dead-Letter Runtime
|
||
|
||
Terminal failure persistence and recovery contract.
|
||
|
||
## 8. Manual Retry and Drain Services
|
||
|
||
Protected operational service interfaces.
|
||
|
||
## 9. Worker Startup and Deployment Strategy
|
||
|
||
Current development, UAT, and production execution model.
|
||
|
||
## 10. Graceful Shutdown Handling
|
||
|
||
Safe worker termination behavior.
|
||
|
||
## 11. Projection Health Integration
|
||
|
||
Real worker health and lag metrics.
|
||
|
||
## 12. Delivery SLA Baseline
|
||
|
||
Initial delivery and recovery targets.
|
||
|
||
## 13. Security and Operations Matrix
|
||
|
||
Worker, retry, dead-letter, and administrative access rules.
|
||
|
||
## 14. Existing Publisher Compatibility Strategy
|
||
|
||
No-double-publication and transition rules.
|
||
|
||
## 15. Consumer Registration Strategy
|
||
|
||
Rules for new, disabled, renamed, and version-changed consumers.
|
||
|
||
## 16. Delivery State Machine
|
||
|
||
Outbox and checkpoint lifecycle rules.
|
||
|
||
## 17. End-to-End Reliability Tests
|
||
|
||
Atomicity, worker delivery, retry, duplicate, and recovery scenarios.
|
||
|
||
## 18. EP.1.5 Readiness Report
|
||
|
||
Confirm that Timeline can rely on durable incremental Activity events.
|
||
|
||
---
|
||
|
||
# Constraints
|
||
|
||
Must preserve:
|
||
|
||
* existing Activity API contracts
|
||
* Activity lifecycle business behavior
|
||
* legacy follow-up APIs and storage
|
||
* recent-activity compatibility layer
|
||
* existing Dashboard and Report consumers
|
||
* existing Approval and Notification behavior
|
||
* existing Audit Log behavior
|
||
* existing CRM security and pricing boundaries
|
||
* EP.1.3 Event Contract and Registry
|
||
* EP.1.4 Projection Registry and checkpoint model
|
||
|
||
Do not implement:
|
||
|
||
* final Timeline projection
|
||
* Timeline UI
|
||
* final Calendar projection
|
||
* Calendar UI
|
||
* Notification event expansion
|
||
* My Day
|
||
* Manager Workspace
|
||
* Executive Workspace
|
||
* Relationship Health
|
||
* Forecast replacement
|
||
* external Kafka, RabbitMQ, or Redis Streams
|
||
* legacy follow-up migration
|
||
* follow-up dual-write
|
||
* Dashboard source replacement
|
||
|
||
Do not start unmanaged worker loops inside request handlers or Client Components.
|
||
|
||
No breaking API changes.
|
||
|
||
---
|
||
|
||
# Compatibility Strategy
|
||
|
||
Activity is the first source domain activated with transactional outbox delivery.
|
||
|
||
Other source domains remain on their current event or notification paths until dedicated cutover tasks are approved.
|
||
|
||
During transition:
|
||
|
||
* Activity uses transactional outbox.
|
||
* Existing Approval notifications remain unchanged.
|
||
* Existing in-process publisher remains available for tests and approved compatibility use.
|
||
* No event may be emitted through both outbox and in-process production paths.
|
||
* Existing projection skeletons remain non-user-facing.
|
||
|
||
---
|
||
|
||
# Testing Requirements
|
||
|
||
## Atomicity Tests
|
||
|
||
* Activity mutation and outbox insert commit together
|
||
* outbox insert failure rolls back Activity mutation
|
||
* validation failure creates neither mutation nor event
|
||
* audit failure behavior follows existing transaction governance
|
||
* multiple events enqueue in deterministic sequence
|
||
|
||
## Worker Claim Tests
|
||
|
||
* one worker claims one event
|
||
* two workers do not process the same active lease
|
||
* stale lease becomes recoverable
|
||
* batch size is honored
|
||
* claim order is deterministic where required
|
||
* graceful shutdown stops new claims
|
||
|
||
## Consumer Tests
|
||
|
||
* all active consumers receive the event
|
||
* one failure does not invalidate successful consumers
|
||
* completed checkpoint skips duplicate execution
|
||
* unsupported version follows policy
|
||
* no-consumer event terminates safely
|
||
* disabled consumer behavior follows registry rule
|
||
|
||
## Retry Tests
|
||
|
||
* retryable failure schedules next attempt
|
||
* non-retryable failure creates dead letter
|
||
* maximum attempts create dead letter
|
||
* retry affects only incomplete consumer
|
||
* manual retry does not rerun completed consumers
|
||
* dead-letter retry preserves original event id
|
||
|
||
## Restart and Recovery Tests
|
||
|
||
* pending event survives process restart simulation
|
||
* retry-scheduled checkpoint survives restart
|
||
* stale processing claim is recovered
|
||
* duplicate delivery after restart creates no duplicate effect
|
||
|
||
## Security Tests
|
||
|
||
* organization isolation
|
||
* sensitive payload remains minimized
|
||
* dead-letter error is sanitized
|
||
* manual retry requires authorization
|
||
* internal-only Activity does not leak through operational status APIs
|
||
|
||
## Compatibility Tests
|
||
|
||
* Activity API response remains unchanged
|
||
* existing recent-activity reads remain unchanged
|
||
* existing follow-up APIs remain unchanged
|
||
* Dashboard and Reports remain unchanged
|
||
* Approval notifications remain unchanged
|
||
* no duplicate Activity Business Event publication occurs
|
||
|
||
## End-to-End Tests
|
||
|
||
* Activity create through outbox to consumer completion
|
||
* Activity complete through outbox to consumer completion
|
||
* consumer failure followed by successful retry
|
||
* consumer failure reaching dead letter
|
||
* duplicate dispatch skipped
|
||
* stale claim recovery
|
||
|
||
---
|
||
|
||
# Acceptance Criteria
|
||
|
||
* Activity source mutations persist outbox events atomically.
|
||
* Event persistence failure inside the transaction rolls back the Activity mutation.
|
||
* Worker dispatch occurs only after transaction commit.
|
||
* Activity APIs return success even when a non-owning consumer later fails.
|
||
* Outbox events are claimed safely in multi-instance execution.
|
||
* Consumer checkpoints are persistent and idempotent.
|
||
* Successful consumers are not rerun because another consumer failed.
|
||
* Retry scheduling and backoff operate according to policy.
|
||
* Terminal failures create sanitized dead-letter records.
|
||
* Manual retry and stale-claim recovery service contracts exist.
|
||
* Worker can shut down gracefully.
|
||
* Projection health reflects actual worker state.
|
||
* No Activity event is double-published through in-process and outbox publishers.
|
||
* Existing Follow-up, Dashboard, Report, Approval, Notification, and recent-activity behavior remains unchanged.
|
||
* End-to-end atomicity, dispatch, duplicate, retry, and recovery tests pass.
|
||
* TypeScript, migration checks, targeted lint, and worker/runtime tests pass.
|
||
|
||
---
|
||
|
||
# Success Criteria
|
||
|
||
ALLA OS has an active, production-ready Transactional Outbox delivery path for Activity Business Events.
|
||
|
||
Source mutations and event persistence are atomic.
|
||
|
||
Projection delivery is post-commit, independently retryable, idempotent, observable, and safe across process restarts and multiple worker instances.
|
||
|
||
Future Timeline and Calendar projections can rely on durable incremental Activity events without coupling projection failures to source-domain mutation results.
|
||
|
||
Ready for:
|
||
|
||
# EP.1.5 – Timeline Projection Foundation
|