alla-allaos-fullstack/docs/implementation/task-ep1.4.1-transactional-outbox-activation-worker-runtime-2026-07-13.md

Task EP.1.4.1 Transactional Outbox Activation & Worker Runtime - 2026-07-13

Scope

• activated Activity Business Event publication through transaction-aware outbox enqueue
• added lease-based outbox claiming fields to business_event_outbox
• added projection outbox worker runtime
• added manual projection operations service contracts
• added PostgreSQL FOR UPDATE SKIP LOCKED claim strategy in Drizzle store
• preserved existing in-process Business Event publisher for compatibility and tests
• preserved current Activity API response contracts, follow-up APIs, dashboard/report datasets, notifications, approvals, and recent-activity behavior

Review Summary

Reviewed before and during implementation:

• AGENTS.md
• plans/task-ep.1.4.1.md
• docs/implementation/task-ep1.3-business-event-foundation-2026-07-13.md
• docs/implementation/task-ep1.4-projection-foundation-delivery-reliability-2026-07-13.md
• docs/standards/engineering-constitution.md
• docs/standards/project-foundations.md
• docs/standards/architecture-rules.md
• docs/standards/task-review-checklist.md
• docs/security/crm-authorization-boundaries.md
• existing Activity service mutation paths
• existing Business Event publisher and projection runtime

Implementation Summary

Activity Transactional Outbox Adoption

Activity mutation paths now enqueue Business Events to the outbox inside the same db.transaction() as the Activity row mutation:

• create
• update
• assign/reassign
• reschedule
• complete
• cancel
• delete

The Activity API still returns hydrated Activity records after commit. Projection consumers are not run synchronously from Activity source mutations.

Transaction-Aware Publisher

Added:

• src/features/foundation/projections/server/transactional-publisher.ts

The publisher validates registry and payload contracts, then inserts into business_event_outbox using the supplied transaction/client. This prevents accidental double-publication through the in-process dispatcher.

Worker Runtime

Added:

• src/features/foundation/projections/worker.ts

Worker behavior:

• claims available events in batches
• applies a lease with claimedBy, claimedAt, and claimExpiresAt
• dispatches claimed committed events through ProjectionConsumerRuntime
• does not auto-start unless PROJECTION_WORKER_ENABLED=true
• supports graceful stop semantics for the in-process loop

Default operational config:

• polling interval: 3000ms
• batch size: 25
• max concurrent events: 1
• max concurrent consumers: 4
• claim lease: 60000ms
• stale claim: 90000ms
• graceful shutdown: 30000ms

Multi-Instance Claiming

Added schema fields:

• business_event_outbox.claimed_by
• business_event_outbox.claimed_at
• business_event_outbox.claim_expires_at

Generated migration:

• drizzle/0004_sharp_mercury.sql

Database store uses PostgreSQL-safe claiming with FOR UPDATE SKIP LOCKED. Expired claimed or processing leases are recoverable.

Manual Operations Contracts

Added:

• src/features/foundation/projections/operations.ts

Contracts:

• drainOutbox
• retryEvent
• retryConsumer
• retryDeadLetter
• resolveDeadLetter

Manual operations require systemRole: "super_admin" at the service-contract boundary. No public UI/API route was added in this task.

Delivery State Machine

Outbox lifecycle:

pending -> claimed -> processing -> completed
pending -> claimed -> processing -> retry_scheduled
pending -> claimed -> processing -> dead_letter
claimed / processing -> lease expired -> claimable again

Checkpoint lifecycle remains:

pending -> processing -> completed
pending -> processing -> retry_scheduled
pending -> processing -> skipped_unsupported_version
pending -> processing -> dead_letter

Completed checkpoints are not rerun when another consumer fails.

Worker Startup Strategy

Current strategy:

• no automatic worker startup in Next.js request/build/test paths
• dedicated process/service should instantiate ProjectionOutboxWorker and call start()
• scheduled CLI drain can call drainOnce() for UAT or operational repair

This avoids duplicate unmanaged loops during hot reload, build, migrations, tests, or serverless request execution.

SLA Baseline

Initial non-binding targets:

• event available after commit: immediate
• polling interval: 1-5 seconds, default 3 seconds
• normal Activity projection delivery: under 10 seconds
• retry scheduling accuracy: within one polling interval
• stale claim recovery: lease duration plus one polling interval
• worker heartbeat/health integration: future operational surface

Compatibility Notes

• Activity events are not emitted through both outbox and in-process publisher in production service paths.
• Existing in-process publisher remains available for tests and explicitly approved compatibility paths.
• Existing Dashboard, Report, Notification, Approval, Follow-up, and recent-activity behavior remains unchanged.
• No final Timeline, Calendar, Notification expansion, My Day, Manager Workspace, Executive Workspace, Forecast, or Relationship Health feature was implemented.

Verification

• node --disable-warning=ExperimentalWarning --disable-warning=MODULE_TYPELESS_PACKAGE_JSON --experimental-strip-types --test src/features/foundation/business-events/*.test.ts src/features/foundation/projections/*.test.ts src/features/crm/activities/server/business-events.test.ts
• npx oxlint src/features/foundation/projections src/features/crm/activities/server/business-events.ts src/features/crm/activities/server/repository.ts src/features/crm/activities/server/service.ts src/db/schema.ts
• npm run db:generate

Typecheck Note

npm run typecheck is currently blocked by generated .next/dev/types/routes.d.ts syntax errors unrelated to the EP.1.4.1 source changes. Targeted runtime tests and scoped lint passed for the changed files.

Residual Risks / Follow-up

• Worker process entrypoint/CLI command is not yet wired into package.json; runtime modules are ready for a dedicated process or scheduled drain task.
• Manual retry/dead-letter APIs are intentionally not exposed publicly yet.
• Health snapshots are recorded by runtime; a user-facing operational health page remains future work.
• Future source domains must adopt the same transaction-aware publisher through dedicated cutover tasks.