task-d.7.5
This commit is contained in:
phaichayon
493
plans/task-d.7.5.md
Normal file
493
plans/task-d.7.5.md
Normal file
@@ -0,0 +1,493 @@
|
||||
# Task D.7.5 – CSV Commit Import Engine
|
||||
|
||||
## Objective
|
||||
|
||||
Implement the CSV Commit Import Engine for ALLA OS Initial Setup.
|
||||
|
||||
This task must take a previously validated CSV preview result from D.7.4, verify its `previewHash`, and commit supported CSV imports safely using transactions and idempotent upsert behavior.
|
||||
|
||||
D.7.5 must reuse the D.7.4 preview/validation engine and must not duplicate validation logic.
|
||||
|
||||
---
|
||||
|
||||
## Review Before Implementation
|
||||
|
||||
Review:
|
||||
|
||||
- AGENTS.md
|
||||
- plans/task-d.7.md
|
||||
- plans/task-d.7.1.md
|
||||
- plans/task-d.7.2.md
|
||||
- plans/task-d.7.3.md
|
||||
- plans/task-d.7.4.md
|
||||
- docs/setup/seed-matrix.md
|
||||
- docs/setup/csv-template-spec.md
|
||||
- docs/setup/import-mapping.md
|
||||
- docs/setup/setup-wizard-blueprint.md
|
||||
- docs/setup/verification-matrix.md
|
||||
- setup/templates/*
|
||||
- setup/templates/templates.json
|
||||
- src/features/setup/server/csv-preview.service.ts
|
||||
- src/features/setup/server/csv/**
|
||||
- src/app/api/setup/import/preview/route.ts
|
||||
- src/db/schema.ts
|
||||
- src/db/seeds/**/*
|
||||
- Existing CRM/foundation services for organizations, users, master options, document sequences, approval, customers, contacts, leads, opportunities, and quotations.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Scope
|
||||
|
||||
Create:
|
||||
|
||||
src/features/setup/server/csv-commit.service.ts
|
||||
|
||||
Supporting modules if needed:
|
||||
|
||||
src/features/setup/server/csv/commit-builder.ts
|
||||
src/features/setup/server/csv/upsert-executor.ts
|
||||
src/features/setup/server/csv/import-report.ts
|
||||
|
||||
Create API:
|
||||
|
||||
POST /api/setup/import/commit
|
||||
|
||||
---
|
||||
|
||||
## Core Rule
|
||||
|
||||
D.7.5 must use the exact same parsing, template detection, validation, dependency resolution, and preview report behavior from D.7.4.
|
||||
|
||||
Do not create a second validation engine.
|
||||
|
||||
Commit flow:
|
||||
|
||||
1. Receive CSV files and `previewHash`.
|
||||
2. Re-run preview validation using D.7.4 engine.
|
||||
3. Compare new preview hash with submitted `previewHash`.
|
||||
4. Reject commit if hashes do not match.
|
||||
5. Reject commit if preview contains blocking errors.
|
||||
6. Commit valid files in dependency/import order.
|
||||
7. Return structured import report.
|
||||
|
||||
---
|
||||
|
||||
## Input
|
||||
|
||||
Accept `multipart/form-data`.
|
||||
|
||||
Required:
|
||||
|
||||
- CSV files
|
||||
- `previewHash`
|
||||
|
||||
Optional:
|
||||
|
||||
- `mode`: `strict | permissive`
|
||||
- `organizationId`
|
||||
- `organizationCode`
|
||||
- `dryRun`
|
||||
- `allowOptionCreate`
|
||||
- `skipWarnings`
|
||||
- `replaceExisting`
|
||||
|
||||
Default behavior:
|
||||
|
||||
- `mode = strict`
|
||||
- `dryRun = false`
|
||||
- `allowOptionCreate = false`
|
||||
- `skipWarnings = false`
|
||||
- `replaceExisting = false`
|
||||
|
||||
---
|
||||
|
||||
## Commit Rules
|
||||
|
||||
### General
|
||||
|
||||
- Commit only rows classified as `create` or `update`.
|
||||
- Do not commit rows classified as `error`.
|
||||
- In strict mode, any blocking error prevents the whole commit.
|
||||
- Warnings do not block commit unless `skipWarnings = false` and the warning is configured as blocking.
|
||||
- Unsupported templates must never commit.
|
||||
- Unknown templates must never commit.
|
||||
|
||||
### Transaction
|
||||
|
||||
- Use one database transaction for all DB-only writes in the import batch.
|
||||
- If any DB write fails, rollback the whole batch.
|
||||
- Storage writes are out of scope for D.7.5.
|
||||
- Attachments are still unsupported/deferred.
|
||||
|
||||
### Upsert
|
||||
|
||||
Every importer must be idempotent.
|
||||
|
||||
Re-running the same CSV should produce either:
|
||||
|
||||
- `updated`
|
||||
- `skipped`
|
||||
- or no-op
|
||||
|
||||
Never create duplicate records.
|
||||
|
||||
---
|
||||
|
||||
## Supported Commit Templates
|
||||
|
||||
Implement commit for templates currently supported by D.7.3/D.7.4:
|
||||
|
||||
Foundation / Organization:
|
||||
|
||||
- users.csv
|
||||
- organizations.csv
|
||||
- memberships.csv
|
||||
- crm-role-assignments.csv
|
||||
- master-options.csv
|
||||
- branches.csv
|
||||
- product-types.csv
|
||||
- document-sequences.csv
|
||||
- approval-workflows.csv
|
||||
- approval-steps.csv
|
||||
- approval-matrix.csv
|
||||
|
||||
Business:
|
||||
|
||||
- customers.csv
|
||||
- contacts.csv
|
||||
- contact-shares.csv
|
||||
- leads.csv
|
||||
- opportunities.csv
|
||||
- opportunity-parties.csv
|
||||
- quotations.csv
|
||||
- quotation-items.csv
|
||||
- quotation-parties.csv
|
||||
- quotation-topics.csv
|
||||
- quotation-topic-items.csv
|
||||
|
||||
If a template cannot yet be safely committed because the service contract is unclear, mark it as blocked with a clear reason. Do not fake success.
|
||||
|
||||
---
|
||||
|
||||
## Required Import Order
|
||||
|
||||
Use import order from `setup/templates/templates.json`.
|
||||
|
||||
Commit order must follow resolved dependency order, not upload order.
|
||||
|
||||
---
|
||||
|
||||
## Importer Responsibilities
|
||||
|
||||
Each importer must define:
|
||||
|
||||
- template name
|
||||
- target table or feature service
|
||||
- business key
|
||||
- create behavior
|
||||
- update behavior
|
||||
- skip behavior
|
||||
- conflict behavior
|
||||
- reference resolution
|
||||
- audit behavior if existing service supports it
|
||||
|
||||
---
|
||||
|
||||
## Data Mapping
|
||||
|
||||
Follow:
|
||||
|
||||
docs/setup/import-mapping.md
|
||||
|
||||
Important examples:
|
||||
|
||||
branches.csv
|
||||
|
||||
maps to:
|
||||
|
||||
ms_options
|
||||
category = crm_branch
|
||||
|
||||
product-types.csv
|
||||
|
||||
maps to:
|
||||
|
||||
ms_options
|
||||
category = crm_product_type
|
||||
|
||||
document-sequences.csv
|
||||
|
||||
resolves:
|
||||
|
||||
organization_code
|
||||
branch_code
|
||||
product_type_code
|
||||
document_type
|
||||
period
|
||||
prefix
|
||||
|
||||
quotation-topic-items.csv
|
||||
|
||||
uses `topic_key` from uploaded batch to resolve committed quotation topic ids.
|
||||
|
||||
---
|
||||
|
||||
## Security
|
||||
|
||||
- API must be protected.
|
||||
- Require `super_admin` for setup-wide import.
|
||||
- Organization admin may be supported later, but D.7.5 can restrict to `super_admin`.
|
||||
- Do not return raw SQL, stack traces, password hashes, or secrets.
|
||||
- Raw `password` from users.csv must be hashed before storage.
|
||||
- Never log raw passwords.
|
||||
- Empty password must not overwrite existing password.
|
||||
- Existing user password can be reset only when explicit reset flag exists.
|
||||
|
||||
---
|
||||
|
||||
## Password Rules
|
||||
|
||||
For `users.csv`:
|
||||
|
||||
- New credential user requires either:
|
||||
- `password`, or
|
||||
- future invitation flow support.
|
||||
- Existing user:
|
||||
- empty password = keep existing password
|
||||
- password + reset_password=true = update password hash
|
||||
- password + reset_password=false = ignore password with warning
|
||||
- Never return password or hash in report.
|
||||
|
||||
---
|
||||
|
||||
## Report Contract
|
||||
|
||||
Return:
|
||||
|
||||
ImportReport
|
||||
|
||||
Fields:
|
||||
|
||||
- status: PASS | WARNING | FAIL
|
||||
- generatedAt
|
||||
- previewHash
|
||||
- committedHash
|
||||
- dryRun
|
||||
- summary
|
||||
- files
|
||||
- errors
|
||||
- warnings
|
||||
|
||||
File report:
|
||||
|
||||
- fileName
|
||||
- template
|
||||
- status
|
||||
- imported
|
||||
- updated
|
||||
- skipped
|
||||
- errors
|
||||
- warnings
|
||||
|
||||
Row report:
|
||||
|
||||
- row
|
||||
- action: create | update | skip | error
|
||||
- businessKey
|
||||
- entityId if safe
|
||||
- message
|
||||
- warnings
|
||||
- errors
|
||||
|
||||
Example:
|
||||
|
||||
{
|
||||
"status": "PASS",
|
||||
"previewHash": "...",
|
||||
"committedHash": "...",
|
||||
"summary": {
|
||||
"imported": 10,
|
||||
"updated": 4,
|
||||
"skipped": 2,
|
||||
"errors": 0,
|
||||
"warnings": 1
|
||||
},
|
||||
"files": []
|
||||
}
|
||||
|
||||
---
|
||||
|
||||
## Error Contract
|
||||
|
||||
Every error must include:
|
||||
|
||||
- file
|
||||
- row
|
||||
- column
|
||||
- code
|
||||
- message
|
||||
- suggestedFix
|
||||
|
||||
Example:
|
||||
|
||||
{
|
||||
"file": "customers.csv",
|
||||
"row": 12,
|
||||
"column": "customer_code",
|
||||
"code": "UPSERT_FAILED",
|
||||
"message": "Customer could not be imported.",
|
||||
"suggestedFix": "Review the customer row and retry after fixing related references."
|
||||
}
|
||||
|
||||
---
|
||||
|
||||
## Dry Run
|
||||
|
||||
Support `dryRun=true`.
|
||||
|
||||
Dry run must:
|
||||
|
||||
- re-run preview
|
||||
- build commit plan
|
||||
- return planned actions
|
||||
- not write database
|
||||
- not run transaction writes
|
||||
|
||||
---
|
||||
|
||||
## Audit
|
||||
|
||||
If existing feature services automatically write audit logs, allow them.
|
||||
|
||||
If importing directly with Drizzle because no safe service exists, do not invent audit semantics in D.7.5.
|
||||
|
||||
Instead:
|
||||
|
||||
- include audit gap warning in report
|
||||
- document follow-up
|
||||
|
||||
---
|
||||
|
||||
## Out Of Scope
|
||||
|
||||
- CSV preview validation logic duplication
|
||||
- CSV template creation
|
||||
- Setup Wizard UI
|
||||
- Seed manifest persistence
|
||||
- Setup state persistence
|
||||
- Rollback after successful commit
|
||||
- Storage import
|
||||
- Attachment import
|
||||
- Email settings import
|
||||
- Unsupported templates
|
||||
- Destructive reset
|
||||
- PDF generation
|
||||
- Approval workflow execution
|
||||
- Document artifact creation
|
||||
|
||||
---
|
||||
|
||||
## Deliverables
|
||||
|
||||
Required:
|
||||
|
||||
src/features/setup/server/csv-commit.service.ts
|
||||
src/features/setup/server/csv/commit-builder.ts
|
||||
src/features/setup/server/csv/upsert-executor.ts
|
||||
src/features/setup/server/csv/import-report.ts
|
||||
src/app/api/setup/import/commit/route.ts
|
||||
|
||||
Optional:
|
||||
|
||||
src/features/setup/server/csv/importers/*.ts
|
||||
|
||||
Recommended importer organization:
|
||||
|
||||
src/features/setup/server/csv/importers/
|
||||
users.importer.ts
|
||||
organizations.importer.ts
|
||||
memberships.importer.ts
|
||||
crm-role-assignments.importer.ts
|
||||
master-options.importer.ts
|
||||
document-sequences.importer.ts
|
||||
approval.importer.ts
|
||||
customers.importer.ts
|
||||
contacts.importer.ts
|
||||
leads.importer.ts
|
||||
opportunities.importer.ts
|
||||
quotations.importer.ts
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
✓ POST /api/setup/import/commit exists.
|
||||
|
||||
✓ Commit requires `previewHash`.
|
||||
|
||||
✓ Commit re-runs D.7.4 preview engine.
|
||||
|
||||
✓ Commit rejects mismatched preview hash.
|
||||
|
||||
✓ Commit rejects preview with blocking errors.
|
||||
|
||||
✓ Commit follows dependency/import order from templates.json.
|
||||
|
||||
✓ Commit is transaction-safe for DB writes.
|
||||
|
||||
✓ Re-running the same valid CSV does not duplicate records.
|
||||
|
||||
✓ users.csv password handling is safe.
|
||||
|
||||
✓ branches.csv imports into ms_options category crm_branch.
|
||||
|
||||
✓ product-types.csv imports into ms_options category crm_product_type.
|
||||
|
||||
✓ quotation-topic-items.csv resolves topic_key from committed/imported quotation topics.
|
||||
|
||||
✓ Unsupported templates do not commit.
|
||||
|
||||
✓ dryRun returns planned actions without writing DB.
|
||||
|
||||
✓ Structured import report is returned.
|
||||
|
||||
✓ No storage/binary attachment import is implemented.
|
||||
|
||||
✓ Typecheck passes or unrelated failures are documented.
|
||||
|
||||
---
|
||||
|
||||
## Suggested Verification
|
||||
|
||||
Run:
|
||||
|
||||
npm run typecheck
|
||||
|
||||
Run scoped lint:
|
||||
|
||||
npx oxlint src/features/setup/server/csv src/features/setup/server/csv-commit.service.ts src/app/api/setup/import/commit
|
||||
|
||||
Manual test flow:
|
||||
|
||||
1. Call POST /api/setup/import/preview with valid CSV batch.
|
||||
2. Copy previewHash.
|
||||
3. Call POST /api/setup/import/commit with same files and previewHash.
|
||||
4. Confirm rows are created/updated.
|
||||
5. Re-run same commit.
|
||||
6. Confirm no duplicates.
|
||||
7. Modify CSV after preview.
|
||||
8. Confirm commit rejects previewHash mismatch.
|
||||
9. Submit CSV with blocking errors.
|
||||
10. Confirm commit fails without partial writes.
|
||||
|
||||
---
|
||||
|
||||
## Follow-up
|
||||
|
||||
After this task:
|
||||
|
||||
D.7.6 – Dashboard Setup Wizard UI
|
||||
|
||||
D.7.8 – Seed Manifest and Setup State Persistence
|
||||
|
||||
D.7.7 – Scoped Reset and Reseed Engine
|
||||
Reference in New Issue
Block a user