alla-allaos-fullstack/plans/task-ep.1.4.md

# Task EP.1.4 – Projection Foundation & Delivery Reliability

Status: Completed
Priority: Critical
Type: Platform Foundation / Projection Runtime / Delivery Reliability

## Depends On

* EP.1.1 Activity Domain Foundation
* EP.1.2 Activity Integration & Legacy Follow-up Adapter
* EP.1.3 Business Event Foundation
* BU-R.0 Business Blueprint Freeze
* BU-R.0.1 Workspace & Activity Business Blueprint Completion
* BU-R.1 Business Capability Audit
* AR.1 Architecture Transition Plan
* AR.2 Epic & Technical Design
* ENG.0 Engineering Constitution

---

# Objective

Introduce the shared Projection Foundation and reliable Business Event delivery model for ALLA OS.

This task establishes the runtime required for future Timeline, Calendar, Dashboard, Notification, My Day, Manager Workspace, Executive Workspace, Forecast, and Relationship Health projections.

The task must ensure that:

* successful source-domain mutations are not reported as failed merely because a projection consumer fails
* projection consumers process events idempotently
* failed projection processing is observable and retryable
* projections can be rebuilt from governed source data
* projection state remains derived and non-authoritative
* current production CRM behavior remains backward compatible

This task does not implement final Timeline, Calendar, My Day, Manager Workspace, Executive Workspace, or dashboard replacement features.

---

# Background

EP.1.3 introduced:

* canonical `BusinessEvent`
* machine-readable Event Registry
* event ownership rules
* in-process publisher and dispatcher
* Activity lifecycle event publishing
* Event Sequence Matrix
* Event Consumer Matrix
* replay contracts
* versioning and idempotency contracts

The remaining reliability gaps are:

1. subscriber failure may currently cause publisher dispatch to fail after the source mutation has already succeeded
2. idempotency is in-memory and does not survive process restart or multi-instance deployment
3. no persistent event delivery or projection processing state exists
4. replay is contract-only because there is no persistent event source
5. projections do not yet have a shared runtime, checkpoint, rebuild, retry, or health model

EP.1.4 resolves these gaps before business-facing projection features are implemented.

---

# Architecture Principles

## 1. Source Mutation Independence

A committed source-domain mutation must not be rolled back or reported as unsuccessful solely because a non-owning projection consumer failed.

Example:

```text
Activity completed successfully
        ↓
activity.completed emitted
        ↓
Timeline projection fails
        ↓
Activity remains completed
Projection failure is recorded for retry
```

---

## 2. Eventual Consistency

Timeline, Calendar, Dashboard, Notification, My Day, Forecast, and other projections are eventually consistent read models.

They must not become authoritative owners of source-domain lifecycle state.

---

## 3. Persistent Idempotency

Projection consumers must prevent duplicate side effects across:

* retries
* process restarts
* multi-instance deployment
* repeated event delivery
* manual rebuild operations

In-memory duplicate detection is insufficient for production projection consumers.

---

## 4. Rebuildable Projections

Every projection must be reproducible from governed source data, persisted business events, or an approved hybrid strategy.

Deleting or rebuilding a projection must not delete operational business records.

---

## 5. Preserve Before Replace

Current dashboard, report, follow-up, approval, notification, and detail-page behavior remains operational.

Projection consumers are introduced additively before any existing consumer is replaced.

---

## 6. Security at Projection Boundaries

Projection reads and writes must preserve:

* organization isolation
* branch and product scope
* CRM record visibility
* internal-only visibility
* pricing-sensitive redaction
* source-domain authorization boundaries

Projection storage must not become a route around source security rules.

---

# Review Required

## Governance

* `AGENTS.md`
* `docs/standards/engineering-constitution.md`
* `docs/standards/project-foundations.md`
* `docs/standards/architecture-rules.md`
* `docs/standards/task-review-checklist.md`
* `docs/security/crm-authorization-boundaries.md`

## Business and Architecture

* Relationship & Sales Workspace Blueprint
* BU-R.1 Business Capability Audit
* AR.1 Architecture Transition Plan
* AR.2 Epic & Technical Design

## Existing Implementation

* EP.1.1 Activity Domain Foundation
* EP.1.2 Activity Integration & Legacy Follow-up Adapter
* EP.1.3 Business Event Foundation
* Business Event Registry
* Event Sequence Matrix
* Event Consumer Matrix
* in-process dispatcher
* Activity event publishers
* current Audit Log foundation
* current Notification foundation
* current Dashboard and Report services
* current Approval event behavior
* existing database transaction patterns
* current background-job or scheduled-task foundations, if any

---

# Scope

## Part 1 — Delivery Semantics

Freeze the official event delivery semantics.

### Required Rules

* Business mutation commits first.
* Non-owning consumer failure must not undo a committed source mutation.
* Event delivery status must be distinguishable from mutation status.
* Consumers may be synchronous internally, but projection processing must follow post-commit semantics.
* Source services must not return a misleading mutation failure after successful commit.
* Critical synchronous side effects, if any, must be explicitly approved and documented.

### Required Decision

Select and document the initial delivery strategy:

* Transactional Outbox
* Post-commit persistent delivery record
* Hybrid source-query plus event checkpoint strategy
* another approach explicitly justified against AR.1 and ENG.0

Preferred direction:

> Transactional Outbox or an equivalent persistent post-commit delivery mechanism.

---

## Part 2 — Persistent Event Delivery Model

Introduce persistent delivery records for Business Events.

Recommended responsibilities:

* persist canonical event envelope
* record publication state
* record availability for dispatch
* record dispatch attempts
* record successful completion
* record dead-letter or terminal failure
* preserve correlation and causation metadata
* support organization-scoped querying

Recommended conceptual states:

```text
pending
processing
completed
failed
dead_letter
```

### Rules

* Event records are immutable business facts after publication.
* Delivery metadata may change independently.
* Event payload must continue respecting data minimization and pricing sensitivity.
* A persistent delivery record does not make Event storage the owner of source lifecycle state.

---

## Part 3 — Transactional Outbox Decision and Foundation

Assess and, if approved, implement the Outbox foundation.

### Outbox Requirements

* source mutation and event persistence occur in the same database transaction
* dispatcher processes only committed events
* retries are safe
* duplicate delivery is expected and handled
* outbox cleanup or retention is governed
* events are organization scoped
* no external broker is required

### Deliverables

* Outbox architecture decision
* schema or storage contract
* transaction integration pattern
* dispatcher polling or draining strategy
* failure recovery behavior
* retention recommendation

If Outbox implementation is deferred, an equivalent persistent reliability mechanism must be approved and documented.

---

## Part 4 — Projection Consumer Runtime

Introduce the shared Projection Consumer Runtime.

Minimum contract:

```ts
export interface ProjectionConsumer<TEvent extends BusinessEvent = BusinessEvent> {
  readonly consumerName: string;
  readonly supportedEventTypes: string[];
  readonly supportedVersions: Record<string, number[]>;

  handle(event: TEvent): Promise<ProjectionHandleResult>;
}
```

Runtime responsibilities:

* resolve registered consumers
* validate supported event version
* acquire idempotency guard
* execute consumer
* record result
* release or finalize checkpoint
* record duration and failure details
* schedule retry when appropriate

---

## Part 5 — Persistent Consumer Checkpoints

Introduce persistent consumer processing state.

Minimum uniqueness:

```text
consumerName + eventId
```

Recommended data:

* consumer name
* event id
* event type
* schema version
* organization id
* processing status
* attempt count
* first attempted at
* last attempted at
* completed at
* next retry at
* error code
* sanitized error message
* processing duration

### Required States

```text
pending
processing
completed
retry_scheduled
failed
dead_letter
skipped_unsupported_version
```

### Rules

* completed checkpoints prevent duplicate projection effects
* failed consumers may retry independently
* one consumer failure must not block unrelated consumers permanently
* checkpoints remain technical delivery state, not business lifecycle state

---

## Part 6 — Retry Policy

Freeze retry behavior.

Determine:

* retryable error categories
* non-retryable error categories
* maximum attempts
* exponential or fixed backoff
* jitter
* retry scheduling
* dead-letter threshold
* manual retry support
* organization-level safety limits

Recommended defaults must be documented rather than hidden in code.

### Example

```text
attempt 1: immediate
attempt 2: +1 minute
attempt 3: +5 minutes
attempt 4: +30 minutes
attempt 5: dead letter
```

Exact values must align with existing operational standards.

---

## Part 7 — Dead-Letter and Failure Records

Introduce a governed dead-letter model for projection failures that cannot be processed automatically.

Required capabilities:

* retain failed event reference
* retain consumer name
* retain sanitized failure information
* retain attempt history
* support manual review
* support manual retry
* support mark-resolved with reason
* prevent sensitive payload exposure

No user-facing operations UI is required in this task unless explicitly approved.

An admin/service interface and documentation are sufficient.

---

## Part 8 — Projection Registry

Create the machine-readable Projection Registry.

Every projection definition must document:

* projection name
* owning feature
* consumed event types
* supported versions
* rebuild source
* persistence strategy
* security classification
* idempotency key
* retry policy
* retention policy
* implementation status

Minimum planned projections:

* Timeline
* Calendar
* Dashboard
* Notification
* My Day
* Manager Workspace
* Executive Workspace
* Relationship Health
* Forecast

### Example

```ts
export type ProjectionDefinition = {
  projectionName: string;
  consumerName: string;
  consumedEventTypes: string[];
  supportedVersions: Record<string, number[]>;
  rebuildStrategy: "source-query" | "event-replay" | "hybrid";
  persistence: "query-time" | "cached-read-model" | "materialized";
  securityClassification: string;
  implementationStatus: "contract-only" | "active" | "deprecated";
};
```

---

## Part 9 — Projection Storage Standards

Freeze the allowed projection storage patterns.

Supported patterns:

### Query-Time Projection

* calculated from source domains when requested
* no persisted read-model table
* suitable for low-volume or early rollout

### Cached Read Model

* persisted derivative data
* invalidated or updated by events
* fully rebuildable

### Materialized Projection

* optimized projection table or materialized view
* suitable for high-volume Timeline, Calendar, Dashboard, or reporting use cases
* never authoritative

### Rules

* projection tables must include organization scope
* projection data must be rebuildable
* source IDs and event provenance must be retained
* unique constraints must protect against duplicate projection rows
* deletion of projection data must not delete source records
* projection schema changes require rebuild compatibility analysis

---

## Part 10 — Initial Projection Skeletons

Create contracts and no-op or test projection consumers for:

* Timeline
* Calendar
* Notification
* Dashboard

These consumers must validate:

* registration
* version support
* idempotency
* checkpointing
* retry behavior
* consumer isolation

They must not yet implement business-facing projection behavior.

No final projection tables or UI are required.

---

## Part 11 — Hybrid Rebuild Strategy

Freeze the initial rebuild strategy.

Recommended strategy:

```text
Initial build
    ↓
Governed source queries + legacy adapters

Incremental updates
    ↓
Business Events

Recovery
    ↓
Source rebuild + event checkpoint reconciliation
```

This hybrid strategy is preferred because:

* legacy follow-up events were not historically persisted
* current source tables remain authoritative
* Activity and new domain events can support incremental projection updates
* migration risk remains controlled

### Required Outputs

* source-query rebuild contract
* event-based incremental contract
* legacy adapter role
* reconciliation rules
* cutover criteria
* rollback behavior

---

## Part 12 — Projection Rebuild Contract

Implement or define a shared rebuild contract.

Suggested interface:

```ts
export type ProjectionRebuildRequest = {
  projectionName: string;
  organizationId: string;
  entityType?: string;
  entityId?: string;
  occurredFrom?: string;
  occurredTo?: string;
  dryRun?: boolean;
  resetExisting?: boolean;
};
```

Required behavior:

* organization scoped
* permission protected
* dry-run supported
* progress report supported
* idempotent
* restartable
* source strategy recorded
* existing projection reset explicitly controlled

No public UI is required.

---

## Part 13 — Projection Health Model

Define projection operational health.

Suggested statuses:

```text
healthy
lagging
degraded
failed
rebuilding
paused
```

Track:

* last processed event
* last successful processing time
* pending count
* retry count
* dead-letter count
* unsupported-version count
* processing lag
* average processing duration
* last rebuild time
* rebuild source

Projection health must be observable without exposing sensitive business payloads.

---

## Part 14 — Delivery Observability

Add structured observability for:

* event persisted
* event ready
* dispatch started
* dispatch completed
* consumer started
* consumer completed
* consumer retry scheduled
* consumer failed
* dead-letter created
* duplicate skipped
* unsupported version skipped
* projection rebuild started
* projection rebuild completed
* projection rebuild failed

Required metadata:

* event id
* event type
* schema version
* organization id
* correlation id
* causation id
* consumer name
* projection name
* attempt
* duration
* status
* sanitized error code

---

## Part 15 — Security and Data Redaction

Freeze projection security rules.

### Required Rules

* Projection persistence must not widen record visibility.
* Pricing-sensitive event payloads must remain redacted.
* Consumers needing full details must re-query through governed source services or security-aware query modules.
* Internal-only Activities must not leak into general projections.
* Organization scope must be mandatory.
* Manager and Executive consumers must still obey branch, product, role, and pricing boundaries.
* Dead-letter and failure records must not persist unrestricted sensitive payloads.

Produce a security matrix for every planned projection.

---

## Part 16 — Existing Consumer Compatibility

Document compatibility with:

* current approval notification events
* current notification inbox
* current Dashboard datasets
* current Report datasets
* legacy lead follow-ups
* legacy opportunity follow-ups
* legacy quotation follow-ups
* recent-activity compatibility reads

### Frozen Rule

No current production consumer is replaced in EP.1.4.

The new runtime operates beside existing consumers until later projection-specific tasks approve cutover.

---

## Part 17 — Projection Delivery Matrix

Create the official Projection Delivery Matrix.

Minimum columns:

| Projection          | Event Source                                  | Initial Build Source      | Incremental Strategy | Persistence                  | Idempotency | Retry    | Cutover Phase            |
| ------------------- | --------------------------------------------- | ------------------------- | -------------------- | ---------------------------- | ----------- | -------- | ------------------------ |
| Timeline            | Activity + CRM events                         | source queries + adapters | Business Events      | query-time or cached         | required    | required | Timeline epic            |
| Calendar            | Activity + milestones + approval              | source queries            | Business Events      | cached/query-time            | required    | required | Calendar epic            |
| Dashboard           | Opportunity + Quotation + Approval + Activity | existing datasets         | hybrid               | existing + future read model | required    | required | Dashboard enhancement    |
| Notification        | Activity + Approval + CRM events              | not applicable            | Business Events      | notification records         | required    | required | Notification expansion   |
| My Day              | Activity + approvals + Hot Project            | source composition        | hybrid               | query-time/cached            | required    | required | My Day epic              |
| Relationship Health | Customer + Activity + Opportunity             | source rebuild            | Business Events      | cached score                 | required    | required | Relationship Health epic |
| Forecast            | Opportunity events                            | source opportunity query  | hybrid               | report/read model            | required    | required | Forecast enhancement     |

The final matrix must reflect repository-specific decisions.

---

## Part 18 — Delivery Failure Matrix

Create the official Delivery Failure Matrix.

Minimum scenarios:

| Failure Scenario                                  | Source Mutation Result | Event State | Consumer State             | Retry          | User/API Result   |
| ------------------------------------------------- | ---------------------- | ----------- | -------------------------- | -------------- | ----------------- |
| Event persistence fails inside source transaction | rollback               | none        | none                       | no             | mutation fails    |
| Source mutation commits, consumer fails           | committed              | persisted   | retry scheduled            | yes            | mutation succeeds |
| One of multiple consumers fails                   | committed              | persisted   | isolated failure           | yes            | mutation succeeds |
| Unsupported event version                         | committed              | persisted   | skipped/failed by consumer | governed       | mutation succeeds |
| Duplicate event delivered                         | committed              | persisted   | duplicate skipped          | no side effect | unchanged         |
| Projection storage unavailable                    | committed              | persisted   | retry scheduled            | yes            | mutation succeeds |
| Dead-letter threshold reached                     | committed              | persisted   | dead letter                | manual         | mutation succeeds |
| Rebuild fails                                     | unchanged              | unchanged   | rebuild failed             | restartable    | no source impact  |

This matrix becomes the reliability baseline for future projection tasks.

---

# Deliverables

## 1. Delivery Semantics Decision

Official post-commit and eventual-consistency behavior.

## 2. Persistent Event Delivery Foundation

Event persistence and delivery state.

## 3. Transactional Outbox or Equivalent Reliability Mechanism

Approved and implemented foundation.

## 4. Projection Consumer Runtime

Shared processing runtime.

## 5. Persistent Consumer Checkpoints

Production-grade idempotency and processing state.

## 6. Retry Policy

Governed retry and backoff behavior.

## 7. Dead-Letter Foundation

Terminal failure handling and manual recovery contract.

## 8. Projection Registry

Machine-readable projection catalog.

## 9. Projection Storage Standards

Query-time, cached, and materialized projection rules.

## 10. Initial Projection Consumer Skeletons

Timeline, Calendar, Dashboard, and Notification runtime validation consumers.

## 11. Hybrid Rebuild Strategy

Source-query initial build plus event-driven incremental update.

## 12. Projection Rebuild Contract

Dry-run, scoped, restartable rebuild foundation.

## 13. Projection Health Model

Lag, failure, retry, dead-letter, and rebuild visibility.

## 14. Delivery Observability

Structured event and consumer processing telemetry.

## 15. Projection Security Matrix

Visibility, pricing, internal-only, and organization-boundary rules.

## 16. Existing Consumer Compatibility Report

No-regression and cutover readiness documentation.

## 17. Projection Delivery Matrix

Official source, persistence, retry, and cutover mapping.

## 18. Delivery Failure Matrix

Official behavior for mutation, dispatch, consumer, and rebuild failures.

## 19. Projection Readiness Report

Readiness for Timeline, Calendar, Notification, Dashboard, My Day, Manager, Executive, Forecast, and Relationship Health tasks.

---

# Constraints

Must preserve:

* Customer
* Contact
* Lead
* Opportunity
* Quotation
* Approval
* Activity
* current Activity APIs
* legacy follow-up APIs and storage
* recent-activity compatibility reads
* current Dashboard and Report datasets
* current Notification behavior
* current Approval notification publishing
* current Audit Log behavior
* current security and pricing visibility boundaries

Do not implement:

* final Timeline projection
* Timeline UI
* final Calendar projection
* Calendar UI
* Dashboard source replacement
* Notification business-event expansion
* My Day
* Manager Workspace
* Executive Workspace
* Relationship Health calculation
* Forecast replacement
* Automation rules
* external Kafka, RabbitMQ, or Redis Streams infrastructure
* public replay UI
* legacy follow-up migration
* dual-write from follow-up to Activity
* source-domain replacement

No breaking API changes.

---

# Compatibility Strategy

The Projection Foundation is additive.

Current production consumers remain active.

Business Events may be persisted and processed through the new delivery runtime without changing existing API responses.

Projection skeleton consumers must not produce user-visible duplicate records.

Any future cutover from legacy datasets to a projection requires:

* parity validation
* regression tests
* reconciliation report
* feature flag or controlled rollout
* rollback strategy
* explicit implementation task approval

---

# Testing Requirements

## Delivery Tests

* source transaction plus event persistence succeeds atomically
* failed event persistence rolls back source transaction
* committed source mutation survives consumer failure
* multiple consumers process independently
* no-subscriber event completes safely
* persistent event resumes after process restart simulation

## Checkpoint Tests

* first processing succeeds
* duplicate event is skipped
* failed attempt schedules retry
* completed checkpoint prevents duplicate side effect
* unsupported version is recorded
* concurrent processing does not execute twice

## Retry Tests

* retryable failure
* non-retryable failure
* backoff scheduling
* maximum-attempt behavior
* dead-letter creation
* manual retry contract

## Projection Runtime Tests

* consumer registration
* consumer version support
* consumer isolation
* projection registry lookup
* projection health updates
* sanitized failure recording

## Rebuild Tests

* organization-scoped dry run
* source-query rebuild
* restartable rebuild
* reset-existing protection
* duplicate-safe rebuild
* failed rebuild does not affect source data

## Security Tests

* organization isolation
* branch/product scope enforcement
* pricing-sensitive redaction
* internal-only activity exclusion
* unauthorized rebuild rejection
* sensitive payload not exposed in dead-letter records

## Compatibility Tests

* existing Activity API behavior unchanged
* current follow-up APIs unchanged
* existing Dashboard datasets unchanged
* existing Reports unchanged
* existing approval notifications unchanged
* existing recent-activity reads unchanged

---

# Acceptance Criteria

* Source mutations and non-owning projection failures are operationally separated.
* A committed source mutation is not reported as failed because a projection consumer failed.
* Business Events have a persistent delivery or outbox mechanism.
* Projection consumers use persistent checkpoints.
* Duplicate event processing does not create duplicate side effects.
* Retry and dead-letter behavior is implemented and tested.
* Projection consumers remain independently recoverable.
* Projection Registry is machine-readable.
* Projection storage rules are frozen.
* Timeline, Calendar, Dashboard, and Notification skeleton consumers validate the runtime without creating final business-facing projections.
* Hybrid source rebuild and event-driven incremental strategy is documented.
* Projection rebuild contract supports dry-run and organization scope.
* Projection health is measurable.
* Delivery observability captures processing and failure state.
* Projection security rules preserve all current CRM authorization boundaries.
* Existing Dashboard, Report, Notification, Approval, Follow-up, and Activity behavior remains unchanged.
* No final workspace or projection UI is introduced.
* TypeScript, migration generation, targeted lint, and projection/runtime tests pass.

---

# Success Criteria

ALLA OS has a reliable, production-ready Projection Foundation.

Business Events can be delivered and processed without coupling projection failure to source-domain mutation success.

Projection consumers can process events idempotently, retry safely, expose health, recover from failure, and rebuild from governed source data.

Future Timeline, Calendar, Notification, Dashboard, My Day, Manager Workspace, Executive Workspace, Relationship Health, and Forecast tasks can focus on business read models rather than re-solving event delivery reliability.

Ready for:

# EP.1.5 – Timeline Projection Foundation