494 lines
9.6 KiB
Markdown
494 lines
9.6 KiB
Markdown
# Task D.7.5 – CSV Commit Import Engine
|
||
|
||
## Objective
|
||
|
||
Implement the CSV Commit Import Engine for ALLA OS Initial Setup.
|
||
|
||
This task must take a previously validated CSV preview result from D.7.4, verify its `previewHash`, and commit supported CSV imports safely using transactions and idempotent upsert behavior.
|
||
|
||
D.7.5 must reuse the D.7.4 preview/validation engine and must not duplicate validation logic.
|
||
|
||
---
|
||
|
||
## Review Before Implementation
|
||
|
||
Review:
|
||
|
||
- AGENTS.md
|
||
- plans/task-d.7.md
|
||
- plans/task-d.7.1.md
|
||
- plans/task-d.7.2.md
|
||
- plans/task-d.7.3.md
|
||
- plans/task-d.7.4.md
|
||
- docs/setup/seed-matrix.md
|
||
- docs/setup/csv-template-spec.md
|
||
- docs/setup/import-mapping.md
|
||
- docs/setup/setup-wizard-blueprint.md
|
||
- docs/setup/verification-matrix.md
|
||
- setup/templates/*
|
||
- setup/templates/templates.json
|
||
- src/features/setup/server/csv-preview.service.ts
|
||
- src/features/setup/server/csv/**
|
||
- src/app/api/setup/import/preview/route.ts
|
||
- src/db/schema.ts
|
||
- src/db/seeds/**/*
|
||
- Existing CRM/foundation services for organizations, users, master options, document sequences, approval, customers, contacts, leads, opportunities, and quotations.
|
||
|
||
---
|
||
|
||
## Implementation Scope
|
||
|
||
Create:
|
||
|
||
src/features/setup/server/csv-commit.service.ts
|
||
|
||
Supporting modules if needed:
|
||
|
||
src/features/setup/server/csv/commit-builder.ts
|
||
src/features/setup/server/csv/upsert-executor.ts
|
||
src/features/setup/server/csv/import-report.ts
|
||
|
||
Create API:
|
||
|
||
POST /api/setup/import/commit
|
||
|
||
---
|
||
|
||
## Core Rule
|
||
|
||
D.7.5 must use the exact same parsing, template detection, validation, dependency resolution, and preview report behavior from D.7.4.
|
||
|
||
Do not create a second validation engine.
|
||
|
||
Commit flow:
|
||
|
||
1. Receive CSV files and `previewHash`.
|
||
2. Re-run preview validation using D.7.4 engine.
|
||
3. Compare new preview hash with submitted `previewHash`.
|
||
4. Reject commit if hashes do not match.
|
||
5. Reject commit if preview contains blocking errors.
|
||
6. Commit valid files in dependency/import order.
|
||
7. Return structured import report.
|
||
|
||
---
|
||
|
||
## Input
|
||
|
||
Accept `multipart/form-data`.
|
||
|
||
Required:
|
||
|
||
- CSV files
|
||
- `previewHash`
|
||
|
||
Optional:
|
||
|
||
- `mode`: `strict | permissive`
|
||
- `organizationId`
|
||
- `organizationCode`
|
||
- `dryRun`
|
||
- `allowOptionCreate`
|
||
- `skipWarnings`
|
||
- `replaceExisting`
|
||
|
||
Default behavior:
|
||
|
||
- `mode = strict`
|
||
- `dryRun = false`
|
||
- `allowOptionCreate = false`
|
||
- `skipWarnings = false`
|
||
- `replaceExisting = false`
|
||
|
||
---
|
||
|
||
## Commit Rules
|
||
|
||
### General
|
||
|
||
- Commit only rows classified as `create` or `update`.
|
||
- Do not commit rows classified as `error`.
|
||
- In strict mode, any blocking error prevents the whole commit.
|
||
- Warnings do not block commit unless `skipWarnings = false` and the warning is configured as blocking.
|
||
- Unsupported templates must never commit.
|
||
- Unknown templates must never commit.
|
||
|
||
### Transaction
|
||
|
||
- Use one database transaction for all DB-only writes in the import batch.
|
||
- If any DB write fails, rollback the whole batch.
|
||
- Storage writes are out of scope for D.7.5.
|
||
- Attachments are still unsupported/deferred.
|
||
|
||
### Upsert
|
||
|
||
Every importer must be idempotent.
|
||
|
||
Re-running the same CSV should produce either:
|
||
|
||
- `updated`
|
||
- `skipped`
|
||
- or no-op
|
||
|
||
Never create duplicate records.
|
||
|
||
---
|
||
|
||
## Supported Commit Templates
|
||
|
||
Implement commit for templates currently supported by D.7.3/D.7.4:
|
||
|
||
Foundation / Organization:
|
||
|
||
- users.csv
|
||
- organizations.csv
|
||
- memberships.csv
|
||
- crm-role-assignments.csv
|
||
- master-options.csv
|
||
- branches.csv
|
||
- product-types.csv
|
||
- document-sequences.csv
|
||
- approval-workflows.csv
|
||
- approval-steps.csv
|
||
- approval-matrix.csv
|
||
|
||
Business:
|
||
|
||
- customers.csv
|
||
- contacts.csv
|
||
- contact-shares.csv
|
||
- leads.csv
|
||
- opportunities.csv
|
||
- opportunity-parties.csv
|
||
- quotations.csv
|
||
- quotation-items.csv
|
||
- quotation-parties.csv
|
||
- quotation-topics.csv
|
||
- quotation-topic-items.csv
|
||
|
||
If a template cannot yet be safely committed because the service contract is unclear, mark it as blocked with a clear reason. Do not fake success.
|
||
|
||
---
|
||
|
||
## Required Import Order
|
||
|
||
Use import order from `setup/templates/templates.json`.
|
||
|
||
Commit order must follow resolved dependency order, not upload order.
|
||
|
||
---
|
||
|
||
## Importer Responsibilities
|
||
|
||
Each importer must define:
|
||
|
||
- template name
|
||
- target table or feature service
|
||
- business key
|
||
- create behavior
|
||
- update behavior
|
||
- skip behavior
|
||
- conflict behavior
|
||
- reference resolution
|
||
- audit behavior if existing service supports it
|
||
|
||
---
|
||
|
||
## Data Mapping
|
||
|
||
Follow:
|
||
|
||
docs/setup/import-mapping.md
|
||
|
||
Important examples:
|
||
|
||
branches.csv
|
||
|
||
maps to:
|
||
|
||
ms_options
|
||
category = crm_branch
|
||
|
||
product-types.csv
|
||
|
||
maps to:
|
||
|
||
ms_options
|
||
category = crm_product_type
|
||
|
||
document-sequences.csv
|
||
|
||
resolves:
|
||
|
||
organization_code
|
||
branch_code
|
||
product_type_code
|
||
document_type
|
||
period
|
||
prefix
|
||
|
||
quotation-topic-items.csv
|
||
|
||
uses `topic_key` from uploaded batch to resolve committed quotation topic ids.
|
||
|
||
---
|
||
|
||
## Security
|
||
|
||
- API must be protected.
|
||
- Require `super_admin` for setup-wide import.
|
||
- Organization admin may be supported later, but D.7.5 can restrict to `super_admin`.
|
||
- Do not return raw SQL, stack traces, password hashes, or secrets.
|
||
- Raw `password` from users.csv must be hashed before storage.
|
||
- Never log raw passwords.
|
||
- Empty password must not overwrite existing password.
|
||
- Existing user password can be reset only when explicit reset flag exists.
|
||
|
||
---
|
||
|
||
## Password Rules
|
||
|
||
For `users.csv`:
|
||
|
||
- New credential user requires either:
|
||
- `password`, or
|
||
- future invitation flow support.
|
||
- Existing user:
|
||
- empty password = keep existing password
|
||
- password + reset_password=true = update password hash
|
||
- password + reset_password=false = ignore password with warning
|
||
- Never return password or hash in report.
|
||
|
||
---
|
||
|
||
## Report Contract
|
||
|
||
Return:
|
||
|
||
ImportReport
|
||
|
||
Fields:
|
||
|
||
- status: PASS | WARNING | FAIL
|
||
- generatedAt
|
||
- previewHash
|
||
- committedHash
|
||
- dryRun
|
||
- summary
|
||
- files
|
||
- errors
|
||
- warnings
|
||
|
||
File report:
|
||
|
||
- fileName
|
||
- template
|
||
- status
|
||
- imported
|
||
- updated
|
||
- skipped
|
||
- errors
|
||
- warnings
|
||
|
||
Row report:
|
||
|
||
- row
|
||
- action: create | update | skip | error
|
||
- businessKey
|
||
- entityId if safe
|
||
- message
|
||
- warnings
|
||
- errors
|
||
|
||
Example:
|
||
|
||
{
|
||
"status": "PASS",
|
||
"previewHash": "...",
|
||
"committedHash": "...",
|
||
"summary": {
|
||
"imported": 10,
|
||
"updated": 4,
|
||
"skipped": 2,
|
||
"errors": 0,
|
||
"warnings": 1
|
||
},
|
||
"files": []
|
||
}
|
||
|
||
---
|
||
|
||
## Error Contract
|
||
|
||
Every error must include:
|
||
|
||
- file
|
||
- row
|
||
- column
|
||
- code
|
||
- message
|
||
- suggestedFix
|
||
|
||
Example:
|
||
|
||
{
|
||
"file": "customers.csv",
|
||
"row": 12,
|
||
"column": "customer_code",
|
||
"code": "UPSERT_FAILED",
|
||
"message": "Customer could not be imported.",
|
||
"suggestedFix": "Review the customer row and retry after fixing related references."
|
||
}
|
||
|
||
---
|
||
|
||
## Dry Run
|
||
|
||
Support `dryRun=true`.
|
||
|
||
Dry run must:
|
||
|
||
- re-run preview
|
||
- build commit plan
|
||
- return planned actions
|
||
- not write database
|
||
- not run transaction writes
|
||
|
||
---
|
||
|
||
## Audit
|
||
|
||
If existing feature services automatically write audit logs, allow them.
|
||
|
||
If importing directly with Drizzle because no safe service exists, do not invent audit semantics in D.7.5.
|
||
|
||
Instead:
|
||
|
||
- include audit gap warning in report
|
||
- document follow-up
|
||
|
||
---
|
||
|
||
## Out Of Scope
|
||
|
||
- CSV preview validation logic duplication
|
||
- CSV template creation
|
||
- Setup Wizard UI
|
||
- Seed manifest persistence
|
||
- Setup state persistence
|
||
- Rollback after successful commit
|
||
- Storage import
|
||
- Attachment import
|
||
- Email settings import
|
||
- Unsupported templates
|
||
- Destructive reset
|
||
- PDF generation
|
||
- Approval workflow execution
|
||
- Document artifact creation
|
||
|
||
---
|
||
|
||
## Deliverables
|
||
|
||
Required:
|
||
|
||
src/features/setup/server/csv-commit.service.ts
|
||
src/features/setup/server/csv/commit-builder.ts
|
||
src/features/setup/server/csv/upsert-executor.ts
|
||
src/features/setup/server/csv/import-report.ts
|
||
src/app/api/setup/import/commit/route.ts
|
||
|
||
Optional:
|
||
|
||
src/features/setup/server/csv/importers/*.ts
|
||
|
||
Recommended importer organization:
|
||
|
||
src/features/setup/server/csv/importers/
|
||
users.importer.ts
|
||
organizations.importer.ts
|
||
memberships.importer.ts
|
||
crm-role-assignments.importer.ts
|
||
master-options.importer.ts
|
||
document-sequences.importer.ts
|
||
approval.importer.ts
|
||
customers.importer.ts
|
||
contacts.importer.ts
|
||
leads.importer.ts
|
||
opportunities.importer.ts
|
||
quotations.importer.ts
|
||
|
||
---
|
||
|
||
## Acceptance Criteria
|
||
|
||
✓ POST /api/setup/import/commit exists.
|
||
|
||
✓ Commit requires `previewHash`.
|
||
|
||
✓ Commit re-runs D.7.4 preview engine.
|
||
|
||
✓ Commit rejects mismatched preview hash.
|
||
|
||
✓ Commit rejects preview with blocking errors.
|
||
|
||
✓ Commit follows dependency/import order from templates.json.
|
||
|
||
✓ Commit is transaction-safe for DB writes.
|
||
|
||
✓ Re-running the same valid CSV does not duplicate records.
|
||
|
||
✓ users.csv password handling is safe.
|
||
|
||
✓ branches.csv imports into ms_options category crm_branch.
|
||
|
||
✓ product-types.csv imports into ms_options category crm_product_type.
|
||
|
||
✓ quotation-topic-items.csv resolves topic_key from committed/imported quotation topics.
|
||
|
||
✓ Unsupported templates do not commit.
|
||
|
||
✓ dryRun returns planned actions without writing DB.
|
||
|
||
✓ Structured import report is returned.
|
||
|
||
✓ No storage/binary attachment import is implemented.
|
||
|
||
✓ Typecheck passes or unrelated failures are documented.
|
||
|
||
---
|
||
|
||
## Suggested Verification
|
||
|
||
Run:
|
||
|
||
npm run typecheck
|
||
|
||
Run scoped lint:
|
||
|
||
npx oxlint src/features/setup/server/csv src/features/setup/server/csv-commit.service.ts src/app/api/setup/import/commit
|
||
|
||
Manual test flow:
|
||
|
||
1. Call POST /api/setup/import/preview with valid CSV batch.
|
||
2. Copy previewHash.
|
||
3. Call POST /api/setup/import/commit with same files and previewHash.
|
||
4. Confirm rows are created/updated.
|
||
5. Re-run same commit.
|
||
6. Confirm no duplicates.
|
||
7. Modify CSV after preview.
|
||
8. Confirm commit rejects previewHash mismatch.
|
||
9. Submit CSV with blocking errors.
|
||
10. Confirm commit fails without partial writes.
|
||
|
||
---
|
||
|
||
## Follow-up
|
||
|
||
After this task:
|
||
|
||
D.7.6 – Dashboard Setup Wizard UI
|
||
|
||
D.7.8 – Seed Manifest and Setup State Persistence
|
||
|
||
D.7.7 – Scoped Reset and Reseed Engine
|