allagroup/alla-allaos-fullstack
Template
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
3b5fede134f8c41b7921a0e2046805aca12ec6df
alla-allaos-fullstack/plans/task-d.7.5.md
phaichayon b7589017ab task-d.7.5
2026-07-03 09:30:57 +07:00

9.6 KiB
Raw Blame History

Task D.7.5 – CSV Commit Import Engine

Objective

Implement the CSV Commit Import Engine for ALLA OS Initial Setup.

This task must take a previously validated CSV preview result from D.7.4, verify its previewHash, and commit supported CSV imports safely using transactions and idempotent upsert behavior.

D.7.5 must reuse the D.7.4 preview/validation engine and must not duplicate validation logic.

Review Before Implementation

Review:

  • AGENTS.md
  • plans/task-d.7.md
  • plans/task-d.7.1.md
  • plans/task-d.7.2.md
  • plans/task-d.7.3.md
  • plans/task-d.7.4.md
  • docs/setup/seed-matrix.md
  • docs/setup/csv-template-spec.md
  • docs/setup/import-mapping.md
  • docs/setup/setup-wizard-blueprint.md
  • docs/setup/verification-matrix.md
  • setup/templates/*
  • setup/templates/templates.json
  • src/features/setup/server/csv-preview.service.ts
  • src/features/setup/server/csv/**
  • src/app/api/setup/import/preview/route.ts
  • src/db/schema.ts
  • src/db/seeds/**/*
  • Existing CRM/foundation services for organizations, users, master options, document sequences, approval, customers, contacts, leads, opportunities, and quotations.

Implementation Scope

Create:

src/features/setup/server/csv-commit.service.ts

Supporting modules if needed:

src/features/setup/server/csv/commit-builder.ts src/features/setup/server/csv/upsert-executor.ts src/features/setup/server/csv/import-report.ts

Create API:

POST /api/setup/import/commit

Core Rule

D.7.5 must use the exact same parsing, template detection, validation, dependency resolution, and preview report behavior from D.7.4.

Do not create a second validation engine.

Commit flow:

  1. Receive CSV files and previewHash.
  2. Re-run preview validation using D.7.4 engine.
  3. Compare new preview hash with submitted previewHash.
  4. Reject commit if hashes do not match.
  5. Reject commit if preview contains blocking errors.
  6. Commit valid files in dependency/import order.
  7. Return structured import report.

Input

Accept multipart/form-data.

Required:

  • CSV files
  • previewHash

Optional:

  • mode: strict | permissive
  • organizationId
  • organizationCode
  • dryRun
  • allowOptionCreate
  • skipWarnings
  • replaceExisting

Default behavior:

  • mode = strict
  • dryRun = false
  • allowOptionCreate = false
  • skipWarnings = false
  • replaceExisting = false

Commit Rules

General

  • Commit only rows classified as create or update.
  • Do not commit rows classified as error.
  • In strict mode, any blocking error prevents the whole commit.
  • Warnings do not block commit unless skipWarnings = false and the warning is configured as blocking.
  • Unsupported templates must never commit.
  • Unknown templates must never commit.

Transaction

  • Use one database transaction for all DB-only writes in the import batch.
  • If any DB write fails, rollback the whole batch.
  • Storage writes are out of scope for D.7.5.
  • Attachments are still unsupported/deferred.

Upsert

Every importer must be idempotent.

Re-running the same CSV should produce either:

  • updated
  • skipped
  • or no-op

Never create duplicate records.

Supported Commit Templates

Implement commit for templates currently supported by D.7.3/D.7.4:

Foundation / Organization:

  • users.csv
  • organizations.csv
  • memberships.csv
  • crm-role-assignments.csv
  • master-options.csv
  • branches.csv
  • product-types.csv
  • document-sequences.csv
  • approval-workflows.csv
  • approval-steps.csv
  • approval-matrix.csv

Business:

  • customers.csv
  • contacts.csv
  • contact-shares.csv
  • leads.csv
  • opportunities.csv
  • opportunity-parties.csv
  • quotations.csv
  • quotation-items.csv
  • quotation-parties.csv
  • quotation-topics.csv
  • quotation-topic-items.csv

If a template cannot yet be safely committed because the service contract is unclear, mark it as blocked with a clear reason. Do not fake success.

Required Import Order

Use import order from setup/templates/templates.json.

Commit order must follow resolved dependency order, not upload order.

Importer Responsibilities

Each importer must define:

  • template name
  • target table or feature service
  • business key
  • create behavior
  • update behavior
  • skip behavior
  • conflict behavior
  • reference resolution
  • audit behavior if existing service supports it

Data Mapping

Follow:

docs/setup/import-mapping.md

Important examples:

branches.csv

maps to:

ms_options category = crm_branch

product-types.csv

maps to:

ms_options category = crm_product_type

document-sequences.csv

resolves:

organization_code branch_code product_type_code document_type period prefix

quotation-topic-items.csv

uses topic_key from uploaded batch to resolve committed quotation topic ids.

Security

  • API must be protected.
  • Require super_admin for setup-wide import.
  • Organization admin may be supported later, but D.7.5 can restrict to super_admin.
  • Do not return raw SQL, stack traces, password hashes, or secrets.
  • Raw password from users.csv must be hashed before storage.
  • Never log raw passwords.
  • Empty password must not overwrite existing password.
  • Existing user password can be reset only when explicit reset flag exists.

Password Rules

For users.csv:

  • New credential user requires either:
    • password, or
    • future invitation flow support.
  • Existing user:
    • empty password = keep existing password
    • password + reset_password=true = update password hash
    • password + reset_password=false = ignore password with warning
  • Never return password or hash in report.

Report Contract

Return:

ImportReport

Fields:

  • status: PASS | WARNING | FAIL
  • generatedAt
  • previewHash
  • committedHash
  • dryRun
  • summary
  • files
  • errors
  • warnings

File report:

  • fileName
  • template
  • status
  • imported
  • updated
  • skipped
  • errors
  • warnings

Row report:

  • row
  • action: create | update | skip | error
  • businessKey
  • entityId if safe
  • message
  • warnings
  • errors

Example:

{ "status": "PASS", "previewHash": "...", "committedHash": "...", "summary": { "imported": 10, "updated": 4, "skipped": 2, "errors": 0, "warnings": 1 }, "files": [] }

Error Contract

Every error must include:

  • file
  • row
  • column
  • code
  • message
  • suggestedFix

Example:

{ "file": "customers.csv", "row": 12, "column": "customer_code", "code": "UPSERT_FAILED", "message": "Customer could not be imported.", "suggestedFix": "Review the customer row and retry after fixing related references." }

Dry Run

Support dryRun=true.

Dry run must:

  • re-run preview
  • build commit plan
  • return planned actions
  • not write database
  • not run transaction writes

Audit

If existing feature services automatically write audit logs, allow them.

If importing directly with Drizzle because no safe service exists, do not invent audit semantics in D.7.5.

Instead:

  • include audit gap warning in report
  • document follow-up

Out Of Scope

  • CSV preview validation logic duplication
  • CSV template creation
  • Setup Wizard UI
  • Seed manifest persistence
  • Setup state persistence
  • Rollback after successful commit
  • Storage import
  • Attachment import
  • Email settings import
  • Unsupported templates
  • Destructive reset
  • PDF generation
  • Approval workflow execution
  • Document artifact creation

Deliverables

Required:

src/features/setup/server/csv-commit.service.ts src/features/setup/server/csv/commit-builder.ts src/features/setup/server/csv/upsert-executor.ts src/features/setup/server/csv/import-report.ts src/app/api/setup/import/commit/route.ts

Optional:

src/features/setup/server/csv/importers/*.ts

Recommended importer organization:

src/features/setup/server/csv/importers/ users.importer.ts organizations.importer.ts memberships.importer.ts crm-role-assignments.importer.ts master-options.importer.ts document-sequences.importer.ts approval.importer.ts customers.importer.ts contacts.importer.ts leads.importer.ts opportunities.importer.ts quotations.importer.ts

Acceptance Criteria

✓ POST /api/setup/import/commit exists.

✓ Commit requires previewHash.

✓ Commit re-runs D.7.4 preview engine.

✓ Commit rejects mismatched preview hash.

✓ Commit rejects preview with blocking errors.

✓ Commit follows dependency/import order from templates.json.

✓ Commit is transaction-safe for DB writes.

✓ Re-running the same valid CSV does not duplicate records.

✓ users.csv password handling is safe.

✓ branches.csv imports into ms_options category crm_branch.

✓ product-types.csv imports into ms_options category crm_product_type.

✓ quotation-topic-items.csv resolves topic_key from committed/imported quotation topics.

✓ Unsupported templates do not commit.

✓ dryRun returns planned actions without writing DB.

✓ Structured import report is returned.

✓ No storage/binary attachment import is implemented.

✓ Typecheck passes or unrelated failures are documented.

Suggested Verification

Run:

npm run typecheck

Run scoped lint:

npx oxlint src/features/setup/server/csv src/features/setup/server/csv-commit.service.ts src/app/api/setup/import/commit

Manual test flow:

  1. Call POST /api/setup/import/preview with valid CSV batch.
  2. Copy previewHash.
  3. Call POST /api/setup/import/commit with same files and previewHash.
  4. Confirm rows are created/updated.
  5. Re-run same commit.
  6. Confirm no duplicates.
  7. Modify CSV after preview.
  8. Confirm commit rejects previewHash mismatch.
  9. Submit CSV with blocking errors.
  10. Confirm commit fails without partial writes.

Follow-up

After this task:

D.7.6 – Dashboard Setup Wizard UI

D.7.8 – Seed Manifest and Setup State Persistence

D.7.7 – Scoped Reset and Reseed Engine

Reference in New Issue View Git Blame Copy Permalink