alla-allaos-fullstack/plans/task-ep.1.4.1.md

Task EP.1.4.1 – Transactional Outbox Activation & Worker Runtime

Status: Completed Priority: Critical Type: Platform Runtime / Delivery Reliability / Background Processing

Depends On

• EP.1.1 Activity Domain Foundation
• EP.1.2 Activity Integration & Legacy Follow-up Adapter
• EP.1.3 Business Event Foundation
• EP.1.4 Projection Foundation & Delivery Reliability
• AR.1 Architecture Transition Plan
• AR.2 Epic & Technical Design
• ENG.0 Engineering Constitution

---

Objective

Activate the Transactional Outbox delivery model introduced in EP.1.4 and provide a production-ready worker runtime for reliable Business Event dispatch.

This task must ensure that:

• source-domain mutations and Business Event outbox persistence occur atomically
• only committed outbox events are dispatched
• projection consumers process events independently and idempotently
• worker execution is safe across multiple application instances
• retryable failures are rescheduled
• terminal failures are moved to governed dead-letter state
• successful source mutations are never reported as failed merely because a projection consumer fails after commit
• existing CRM APIs and business behavior remain backward compatible

The first activated source domain is Activity.

Other source domains may adopt the same transaction pattern in later controlled tasks.

---

Background

EP.1.3 introduced:

• canonical BusinessEvent
• Event Registry
• Publisher and Dispatcher abstractions
• Activity lifecycle event publication
• Event Sequence Matrix
• Event Consumer Matrix

EP.1.4 introduced:

• business_event_outbox
• projection consumer checkpoints
• projection dead letters
• projection rebuild runs
• projection health snapshots
• persistent projection runtime contracts
• retry policy
• Projection Registry
• no-op skeleton consumers

However, production delivery guarantees are not active yet because:

1. Activity source mutations do not yet persist events atomically through the outbox in every mutation path.
2. No worker currently drains pending outbox events.
3. No multi-instance-safe event claiming process exists.
4. No operational retry or dead-letter processing loop is active.
5. Manual retry and drain contracts are not yet available.

EP.1.4.1 activates these foundations.

---

Architecture Principles

1. Atomic Source Mutation and Event Persistence

The source mutation and related outbox event must use the same database transaction.

Required pattern:

BEGIN

Update source-domain state

Insert Business Event Outbox record

Write mandatory Audit Log where required

COMMIT

If outbox persistence fails inside the transaction, the source mutation must roll back.

---

2. Post-Commit Consumer Processing

Projection consumers process events only after the source transaction commits.

A projection failure must not change the result of the already committed source mutation.

Example:

Activity completed
        ↓
Activity row + Outbox event committed
        ↓
API returns success
        ↓
Worker dispatches event
        ↓
Timeline consumer fails
        ↓
Checkpoint schedules retry

---

3. At-Least-Once Delivery

The runtime must assume that an event may be delivered more than once.

Exactly-once transport is not required.

Consumer idempotency and persistent checkpoints provide duplicate protection.

---

4. Consumer Isolation

An event may have multiple consumers.

Example:

activity.completed
├── Timeline
├── Calendar
├── Dashboard
└── Notification

One failed consumer must not erase successful processing by other consumers.

Retries must target only incomplete consumers.

---

5. Multi-Instance Safety

More than one application or worker process may run concurrently.

Only one worker may claim a given event delivery unit at a time.

The design must support:

• database-level claiming
• lease expiration
• stale claim recovery
• duplicate-safe retries
• graceful worker shutdown

---

6. Preserve Existing Behavior

Existing Activity APIs and source-domain behavior remain unchanged from the caller’s perspective.

This task must not replace:

• existing follow-up APIs
• Dashboard datasets
• Report datasets
• existing Notification behavior
• existing Approval notification flow
• recent-activity compatibility reads

---

Review Required

Governance and Architecture

• AGENTS.md
• docs/standards/engineering-constitution.md
• docs/standards/project-foundations.md
• docs/standards/architecture-rules.md
• docs/standards/task-review-checklist.md
• docs/security/crm-authorization-boundaries.md
• AR.1 Architecture Transition Plan
• AR.2 Epic & Technical Design

Existing Implementations

• EP.1.3 Business Event Foundation report and code
• EP.1.4 Projection Foundation report and code
• Activity service mutation paths
• Business Event publisher
• persistent outbox publisher
• projection runtime
• projection consumer checkpoints
• projection dead letters
• projection health snapshots
• current Drizzle transaction conventions
• existing scheduled-job or worker patterns
• current application startup and shutdown behavior
• current deployment topology and Docker runtime, where documented

---

Scope

Part 1 — Activity Transaction Boundary Refactor

Refactor Activity mutation paths to support transaction-scoped persistence.

Minimum operations:

• create
• update
• assign
• reassign
• start
• reschedule
• complete
• cancel
• delete, if governed deletion remains supported

Each operation must persist:

• Activity mutation
• Business Event Outbox record
• required Audit Log record

inside one shared transaction where applicable.

Rules

• Route Handlers remain thin.
• Activity Service owns business transitions.
• Transaction object must be propagated through approved service and persistence seams.
• No nested independent transaction may break atomicity.
• Event payload is created from the successful mutation result.
• An event must never be inserted before business validation completes.

---

Part 2 — Transaction-Aware Outbox Publisher

Implement or complete a transaction-aware outbox publisher.

Recommended contract:

export interface TransactionalBusinessEventPublisher {
  enqueue(
    transaction: DatabaseTransaction,
    event: BusinessEvent
  ): Promise<void>;

  enqueueMany(
    transaction: DatabaseTransaction,
    events: BusinessEvent[]
  ): Promise<void>;
}

Responsibilities

• validate registry entry
• validate schema version
• validate payload
• persist event using the supplied transaction
• preserve correlation and causation metadata
• prevent duplicate event insertion where an operation idempotency key exists
• not dispatch consumers synchronously

---

Part 3 — Outbox Worker Runtime

Introduce a worker that drains committed outbox events.

Worker lifecycle:

poll
  ↓
claim batch
  ↓
load active consumers
  ↓
dispatch each incomplete consumer
  ↓
record checkpoints
  ↓
mark event delivery state
  ↓
sleep / repeat

Worker Configuration

Support configurable:

• enabled state
• polling interval
• batch size
• maximum concurrent events
• maximum concurrent consumers
• claim lease duration
• stale claim threshold
• worker instance identifier
• graceful shutdown timeout

Configuration must use existing environment and configuration conventions.

---

Part 4 — Multi-Instance Event Claiming

Implement a database-safe claiming strategy.

Approved approaches may include:

• FOR UPDATE SKIP LOCKED
• atomic conditional update with lease token
• another PostgreSQL-safe claiming mechanism

Each claimed record should track, as appropriate:

• claimed by
• claimed at
• lease expires at
• processing started at
• attempt count

Rules

• two workers must not actively process the same event claim concurrently
• expired claims must be recoverable
• a crashed worker must not permanently block delivery
• claiming must be organization-safe but may process multiple organizations in one batch

---

Part 5 — Event and Consumer Completion Semantics

Freeze and implement event completion rules.

Recommended model:

• Outbox state represents event dispatch lifecycle.
• Consumer checkpoint represents per-consumer result.
• Event is terminally completed only when all applicable active consumers reach a terminal state.

Terminal consumer states:

• completed
• skipped unsupported version, when governed as terminal
• dead letter
• explicitly ignored by registry policy

Important Rule

A successful consumer must not be called again merely because another consumer failed.

---

Part 6 — Retry Processor

Activate retry processing using the EP.1.4 policy.

Default policy:

attempt 1: immediate
attempt 2: +1 minute
attempt 3: +5 minutes
attempt 4: +30 minutes
attempt 5: dead letter

Implement:

• retry scheduling
• retryable-error classification
• non-retryable-error classification
• next-attempt selection
• attempt count update
• consumer-specific retry
• jitter where appropriate
• stale-processing recovery

Retry values must be configurable or governed centrally.

---

Part 7 — Dead-Letter Runtime

Activate dead-letter creation when:

• maximum retries are exceeded
• failure is explicitly non-retryable
• event version is unsupported and policy requires review
• registry or payload incompatibility cannot be resolved automatically

Dead-letter records must include:

• event reference
• consumer name
• event type
• organization id
• attempt count
• sanitized error code
• sanitized error message
• first failure time
• final failure time
• resolution status

Do not duplicate unrestricted event payloads into dead-letter storage.

---

Part 8 — Manual Retry and Drain Service Contracts

Provide protected service contracts for operations support.

Minimum capabilities:

• drain pending outbox events
• retry one failed consumer checkpoint
• retry all failed consumers for one event
• retry one dead-letter item
• mark dead letter resolved with reason
• inspect sanitized processing status
• recover stale claims

Recommended service methods:

drainOutbox(options)
retryConsumer(eventId, consumerName, actor)
retryEvent(eventId, actor)
retryDeadLetter(deadLetterId, actor)
resolveDeadLetter(deadLetterId, reason, actor)
recoverStaleClaims(options)

Public UI is not required.

Admin API routes are optional and should only be added when authorization and operational need are clearly defined.

---

Part 9 — Worker Startup Strategy

Define how the worker runs in current deployment environments.

Evaluate:

• worker inside application process
• dedicated Node worker process
• scheduled CLI drain command
• separate Docker service using the same application image

Preferred production direction:

Dedicated worker process or service using shared runtime modules.

The implementation must not accidentally start duplicate unmanaged polling loops during:

• Next.js hot reload
• serverless request execution
• build phase
• migration commands
• test execution

---

Part 10 — Graceful Shutdown

Worker shutdown must:

• stop claiming new events
• allow active handlers to finish within timeout
• release or expire unfinished leases safely
• flush structured logs
• report final worker status

Handle common termination signals according to runtime conventions.

---

Part 11 — Projection Health Updates

Update projection health from real worker processing.

Track:

• last event observed
• last successful event
• pending count
• processing count
• retry count
• dead-letter count
• unsupported-version count
• last failure time
• processing lag
• worker heartbeat
• last successful drain

Health storage must remain payload-free.

---

Part 12 — Delivery SLA Baseline

Freeze initial operational targets.

Suggested targets:

Capability	Initial Target
Outbox event available after commit	immediate
Worker polling interval	1–5 seconds
Activity projection delivery under normal load	under 10 seconds
Retry scheduling accuracy	within one polling interval
Stale claim recovery	within lease duration plus one polling interval
Worker health heartbeat	every 30–60 seconds

Final values must reflect deployment constraints and can be classified as initial non-binding SLOs.

---

Part 13 — Security and Authorization

Worker processing must:

• maintain organization scoping
• not bypass pricing-sensitive event restrictions
• not expose internal-only Activity details
• use consumer-specific security rules
• avoid persisting decrypted credentials or secrets
• sanitize operational errors
• audit manual retry and dead-letter resolution actions

Manual operations must require explicit foundation or system administration permission.

---

Part 14 — Existing Publisher Compatibility

Preserve the existing in-process publisher where needed for:

• tests
• non-production isolated execution
• compatibility consumers not yet cut over

However, Activity production mutation paths must use the transactional outbox publisher after activation.

Do not publish the same Activity event through both in-process and outbox paths.

Add a guard or configuration strategy preventing accidental double publication.

---

Part 15 — Consumer Registration Snapshot

Freeze the consumer set used when determining applicable checkpoints.

Define behavior when:

• a new consumer is registered after an event was completed
• a consumer is disabled
• a consumer is renamed
• a consumer’s supported versions change
• an event has no consumers
• a projection remains contract-only

Recommended approach:

• persist or deterministically resolve applicable consumers at dispatch time
• document replay or backfill behavior for newly introduced consumers
• avoid reopening old completed events automatically without explicit rebuild

---

Part 16 — Delivery State Machine

Freeze the outbox delivery lifecycle.

Recommended model:

pending
  ↓
claimed
  ↓
processing
  ├── completed
  ├── retry_scheduled
  └── dead_letter

Possible recovery paths:

claimed / processing
  ↓ lease expired
pending

Consumer checkpoint lifecycle:

pending
  ↓
processing
  ├── completed
  ├── retry_scheduled
  ├── skipped_unsupported_version
  └── dead_letter

Implement valid transitions and reject invalid direct state jumps.

---

Part 17 — End-to-End Reliability Scenario

Implement an end-to-end validation path for at least one Activity operation.

Required scenario:

Complete Activity
        ↓
Activity mutation persisted
        ↓
activity.completed Outbox event persisted in same transaction
        ↓
API returns success
        ↓
Worker claims event
        ↓
Skeleton consumers execute
        ↓
Consumer checkpoints become completed
        ↓
Duplicate dispatch is skipped

Failure variation:

One consumer fails
        ↓
Activity remains completed
        ↓
Other consumers remain completed
        ↓
Failed consumer schedules retry
        ↓
Retry completes or moves to dead letter

---

Deliverables

1. Activity Transactional Outbox Adoption

Atomic source mutation, audit, and event enqueue.

2. Transaction-Aware Outbox Publisher

Shared transaction-scoped persistence contract.

3. Outbox Worker Runtime

Polling, claiming, dispatch, and completion processing.

4. Multi-Instance Claiming Strategy

Lease-based or lock-based safe batch processing.

5. Per-Consumer Dispatch Semantics

Independent checkpoints and isolated retries.

6. Retry Processor

Configured retry and backoff implementation.

7. Dead-Letter Runtime

Terminal failure persistence and recovery contract.

8. Manual Retry and Drain Services

Protected operational service interfaces.

9. Worker Startup and Deployment Strategy

Current development, UAT, and production execution model.

10. Graceful Shutdown Handling

Safe worker termination behavior.

11. Projection Health Integration

Real worker health and lag metrics.

12. Delivery SLA Baseline

Initial delivery and recovery targets.

13. Security and Operations Matrix

Worker, retry, dead-letter, and administrative access rules.

14. Existing Publisher Compatibility Strategy

No-double-publication and transition rules.

15. Consumer Registration Strategy

Rules for new, disabled, renamed, and version-changed consumers.

16. Delivery State Machine

Outbox and checkpoint lifecycle rules.

17. End-to-End Reliability Tests

Atomicity, worker delivery, retry, duplicate, and recovery scenarios.

18. EP.1.5 Readiness Report

Confirm that Timeline can rely on durable incremental Activity events.

---

Constraints

Must preserve:

• existing Activity API contracts
• Activity lifecycle business behavior
• legacy follow-up APIs and storage
• recent-activity compatibility layer
• existing Dashboard and Report consumers
• existing Approval and Notification behavior
• existing Audit Log behavior
• existing CRM security and pricing boundaries
• EP.1.3 Event Contract and Registry
• EP.1.4 Projection Registry and checkpoint model

Do not implement:

• final Timeline projection
• Timeline UI
• final Calendar projection
• Calendar UI
• Notification event expansion
• My Day
• Manager Workspace
• Executive Workspace
• Relationship Health
• Forecast replacement
• external Kafka, RabbitMQ, or Redis Streams
• legacy follow-up migration
• follow-up dual-write
• Dashboard source replacement

Do not start unmanaged worker loops inside request handlers or Client Components.

No breaking API changes.

---

Compatibility Strategy

Activity is the first source domain activated with transactional outbox delivery.

Other source domains remain on their current event or notification paths until dedicated cutover tasks are approved.

During transition:

• Activity uses transactional outbox.
• Existing Approval notifications remain unchanged.
• Existing in-process publisher remains available for tests and approved compatibility use.
• No event may be emitted through both outbox and in-process production paths.
• Existing projection skeletons remain non-user-facing.

---

Testing Requirements

Atomicity Tests

• Activity mutation and outbox insert commit together
• outbox insert failure rolls back Activity mutation
• validation failure creates neither mutation nor event
• audit failure behavior follows existing transaction governance
• multiple events enqueue in deterministic sequence

Worker Claim Tests

• one worker claims one event
• two workers do not process the same active lease
• stale lease becomes recoverable
• batch size is honored
• claim order is deterministic where required
• graceful shutdown stops new claims

Consumer Tests

• all active consumers receive the event
• one failure does not invalidate successful consumers
• completed checkpoint skips duplicate execution
• unsupported version follows policy
• no-consumer event terminates safely
• disabled consumer behavior follows registry rule

Retry Tests

• retryable failure schedules next attempt
• non-retryable failure creates dead letter
• maximum attempts create dead letter
• retry affects only incomplete consumer
• manual retry does not rerun completed consumers
• dead-letter retry preserves original event id

Restart and Recovery Tests

• pending event survives process restart simulation
• retry-scheduled checkpoint survives restart
• stale processing claim is recovered
• duplicate delivery after restart creates no duplicate effect

Security Tests

• organization isolation
• sensitive payload remains minimized
• dead-letter error is sanitized
• manual retry requires authorization
• internal-only Activity does not leak through operational status APIs

Compatibility Tests

• Activity API response remains unchanged
• existing recent-activity reads remain unchanged
• existing follow-up APIs remain unchanged
• Dashboard and Reports remain unchanged
• Approval notifications remain unchanged
• no duplicate Activity Business Event publication occurs

End-to-End Tests

• Activity create through outbox to consumer completion
• Activity complete through outbox to consumer completion
• consumer failure followed by successful retry
• consumer failure reaching dead letter
• duplicate dispatch skipped
• stale claim recovery

---

Acceptance Criteria

• Activity source mutations persist outbox events atomically.
• Event persistence failure inside the transaction rolls back the Activity mutation.
• Worker dispatch occurs only after transaction commit.
• Activity APIs return success even when a non-owning consumer later fails.
• Outbox events are claimed safely in multi-instance execution.
• Consumer checkpoints are persistent and idempotent.
• Successful consumers are not rerun because another consumer failed.
• Retry scheduling and backoff operate according to policy.
• Terminal failures create sanitized dead-letter records.
• Manual retry and stale-claim recovery service contracts exist.
• Worker can shut down gracefully.
• Projection health reflects actual worker state.
• No Activity event is double-published through in-process and outbox publishers.
• Existing Follow-up, Dashboard, Report, Approval, Notification, and recent-activity behavior remains unchanged.
• End-to-end atomicity, dispatch, duplicate, retry, and recovery tests pass.
• TypeScript, migration checks, targeted lint, and worker/runtime tests pass.

---

Success Criteria

ALLA OS has an active, production-ready Transactional Outbox delivery path for Activity Business Events.

Source mutations and event persistence are atomic.

Projection delivery is post-commit, independently retryable, idempotent, observable, and safe across process restarts and multiple worker instances.

Future Timeline and Calendar projections can rely on durable incremental Activity events without coupling projection failures to source-domain mutation results.

Ready for:

EP.1.5 – Timeline Projection Foundation