Atlas Plan

Phase 1: @source/ Restructure

Purpose: Reorganize source files into raw/, clean/, config/ subdirectories and update all YAML config paths

T-001 - Create @source/ subdirectories and move xlsx files

Create @source/raw/, @source/clean/, and verify @source/config/ exists. Move all xlsx files from @source/ root into @source/raw/. Add .gitkeep to @source/clean/.

Status: completed
Priority: P0
Dependencies: none

Acceptance

@source/raw/ exists and contains all 8 xlsx files (4 LAPORAN + 4 DATABASE SISWA)
@source/clean/ exists with .gitkeep
@source/config/ unchanged
Original xlsx files removed from @source/ root
@source/Contoh punya TM WLC.pdf moved to @source/raw/ (keep with source data)

Files

@source/raw/ (new directory)
@source/clean/.gitkeep (new)

T-002 - Update @source/config YAML paths

Update all four @source/config/ions-YYYY.yaml files: change every path: @source/LAPORAN... and path: @source/DATABASE... to path: @source/raw/LAPORAN... and path: @source/raw/DATABASE....

Status: completed
Priority: P0
Dependencies: T-001

Acceptance

ions-2023.yaml all path: values reference @source/raw/
ions-2024.yaml all path: values reference @source/raw/
ions-2025.yaml all path: values reference @source/raw/
ions-2026.yaml all path: values reference @source/raw/
pnpm run sync -- --source xlsx --entity IONS --year 2026 still resolves paths correctly (loadConfig finds @source/raw/ files)

Files

@source/config/ions-2023.yaml
@source/config/ions-2024.yaml
@source/config/ions-2025.yaml
@source/config/ions-2026.yaml

Phase 2: @python/ Workspace

Purpose: Add @python/ as a new workspace prefix and move @packages/transform into it

T-003 - Add @python/ workspace prefix to pnpm-workspace.yaml

Add @python/** to the packages list in pnpm-workspace.yaml. Create the @python/ directory at monorepo root.

Status: completed
Priority: P0
Dependencies: none

Acceptance

pnpm-workspace.yaml packages list includes - '@python/**'
@python/ directory exists at monorepo root

Files

pnpm-workspace.yaml
@python/ (new directory)

T-004 - Move @packages/transform → @python/analytics

Copy the entire @packages/transform/ directory to @python/analytics/. Update pyproject.toml name from atlas-transform to atlas-analytics. Update profiles.yml if path depth changed (verify ../../atlas.db is still correct from @python/analytics/). Remove @packages/transform/ after verifying.

Status: completed
Priority: P0
Dependencies: T-003

Acceptance

@python/analytics/ exists with all dbt files at root: dbt_project.yml, profiles.yml, pyproject.toml, models/, macros/, tests/
pyproject.toml name = "atlas-analytics"
profiles.yml path: resolves correctly to atlas.db at monorepo root (two levels up from @python/analytics/)
cd @python/analytics && uv run dbt debug connects without error
cd @python/analytics && uv run dbt run runs all models successfully
@packages/transform/ removed

Notes

.venv/, target/, logs/, dbt_packages/, .user.yml, uv.lock from the old directory can be recreated — copy only the tracked files (pyproject.toml, requirements.txt, profiles.yml, dbt_project.yml, models/, macros/, tests/)
The .gitignore in @packages/transform/ should also be copied to @python/analytics/

Files

@python/analytics/pyproject.toml
@python/analytics/profiles.yml
@python/analytics/dbt_project.yml
@python/analytics/requirements.txt
@python/analytics/.gitignore

Phase 3: @packages/pipeline

Purpose: Create new unified TypeScript package merging sync + format sources

T-005 - Scaffold @packages/pipeline package

Create @packages/pipeline/ with package.json, tsconfig.json, eslint.config.mjs. Set up Vitest in this task as the test runner (dependency + config). The package replaces both @packages/sync and @packages/format.

Status: completed
Priority: P0
Dependencies: none

Acceptance

@packages/pipeline/package.json exists with name @packages/pipeline, scripts: build, dev, lint, lint:fix, test, test:type, start
test script uses vitest run (not node --test)
Dependencies include: @duckdb/node-api, @libsql/client, @packages/db, yaml, xlsx (all via catalog references)
DevDependencies include: @core/lint, @core/typescript, @types/node, tsx, typescript, vitest
tsconfig.json extends @core/typescript/lib (same as current sync)
eslint.config.mjs uses base() from @core/lint/base
Vitest catalog entry added to pnpm-workspace.yaml if not already present
pnpm install resolves without errors

Notes

Check if vitest is already in a catalog in pnpm-workspace.yaml before adding
xlsx catalog entry already exists as catalog:util in pnpm-workspace.yaml

Files

@packages/pipeline/package.json
@packages/pipeline/tsconfig.json
@packages/pipeline/eslint.config.mjs
pnpm-workspace.yaml (vitest catalog entry if needed)

T-006 - Migrate @packages/sync source into @packages/pipeline

Copy all source files from @packages/sync/@source/ into @packages/pipeline/@source/. Organize into subdirectories: config.ts and duck.ts stay at @source/ root; seed.ts stays at root; adapter moves to load/csv.ts (renamed from adapters/xlsx.ts). Fix all imports.

Status: completed
Priority: P0
Dependencies: T-005

Acceptance

@packages/pipeline/@source/config.ts exists (unchanged from sync)
@packages/pipeline/@source/duck.ts exists — merged from both sync's duck.ts and format's duck.ts; uses the richer DuckClient interface from format's version, extended with the raw-table DDL helpers from sync's version
@packages/pipeline/@source/seed.ts exists (unchanged from sync)
@packages/pipeline/@source/load/csv.ts exists (renamed from adapters/xlsx.ts; logic unchanged for now — still reads xlsx directly; Plan 009 will switch to CSV)
All imports updated to reflect new paths
pnpm --filter @packages/pipeline test:type passes

Notes

The two duck.ts files have different shapes: sync's duck.ts has raw DDL helpers (createRawTable, appendRawTable, etc.); format's duck.ts has the DuckClient query wrapper. Merge both into one @source/duck.ts — export all helpers from sync plus the DuckClient factory from format.

Files

@packages/pipeline/@source/config.ts
@packages/pipeline/@source/duck.ts
@packages/pipeline/@source/seed.ts
@packages/pipeline/@source/load/csv.ts

T-007 - Migrate @packages/format source into @packages/pipeline

Copy all source files from @packages/format/@source/ into @packages/pipeline/@source/format/. Fix imports (duck client now comes from ../duck). Preserve all exports.

Status: completed
Priority: P0
Dependencies: T-006

Acceptance

@packages/pipeline/@source/format/report.ts exists
@packages/pipeline/@source/format/types.ts exists
@packages/pipeline/@source/format/number.ts exists
@packages/pipeline/@source/format/sections/revenue.ts exists
@packages/pipeline/@source/format/sections/orders.ts exists
@packages/pipeline/@source/format/sections/marketing.ts exists
@packages/pipeline/@source/format/sections/schools.ts exists
All imports pointing to the old duck.ts updated to ../duck (or ../../duck as appropriate)
pnpm --filter @packages/pipeline test:type passes

Files

@packages/pipeline/@source/format/report.ts
@packages/pipeline/@source/format/types.ts
@packages/pipeline/@source/format/number.ts
@packages/pipeline/@source/format/sections/revenue.ts
@packages/pipeline/@source/format/sections/orders.ts
@packages/pipeline/@source/format/sections/marketing.ts
@packages/pipeline/@source/format/sections/schools.ts

T-008 - Create stub extract/ and validate/ directories

Create placeholder modules for the extract and validate steps that Plan 009 will implement. These stubs make the CLI wiring complete without requiring Plan 009's logic now.

Status: completed
Priority: P1
Dependencies: T-006

Acceptance

@packages/pipeline/@source/extract/index.ts exists — exports a extract() stub that logs "extract: not yet implemented" and returns
@packages/pipeline/@source/validate/index.ts exists — exports a validate() stub that logs "validate: not yet implemented" and returns
Both files type-check cleanly

Files

@packages/pipeline/@source/extract/index.ts
@packages/pipeline/@source/validate/index.ts

T-009 - Create unified CLI index.ts

Create @packages/pipeline/@source/index.ts with sub-command dispatch. Sub-commands: extract, load, validate, format, seed, run. Shared flags: --entity, --year, --month, --period, --mode. The run sub-command sequences: extract → load → validate → (dbt is external) → format.

Status: completed
Priority: P0
Dependencies: T-007, T-008

Acceptance

pnpm --filter @packages/pipeline start -- load --entity IONS --year 2026 runs the xlsx load (same behavior as old @packages/sync)
pnpm --filter @packages/pipeline start -- format --entity IONS --period 2026-02 runs report assembly (same behavior as old @packages/format)
pnpm --filter @packages/pipeline start -- seed --entity IONS runs seed (same behavior as old sync seed)
pnpm --filter @packages/pipeline start -- extract --entity IONS --year 2026 logs "extract: not yet implemented"
pnpm --filter @packages/pipeline start -- validate --entity IONS --year 2026 logs "validate: not yet implemented"
Unknown sub-command prints usage and exits non-zero
pnpm --filter @packages/pipeline test:type passes

Files

@packages/pipeline/@source/index.ts

Phase 4: Vitest Tests

Purpose: Add Vitest test scaffold with passing tests for pure utility functions

T-010 - Add Vitest scaffold and initial tests

Add a vitest.config.ts and write initial unit tests for the most testable pure functions: config.ts (path normalization), format/number.ts (toNumber, roundTo2), and the CLI arg parser in index.ts.

Status: completed
Priority: P1
Dependencies: T-009

Acceptance

@packages/pipeline/vitest.config.ts exists with correct config (points to @source/**/*.test.ts)
@packages/pipeline/@source/__tests__/number.test.ts exists — tests toNumber() and roundTo2() covering: null, undefined, bigint, string, NaN, Infinity, valid number
@packages/pipeline/@source/__tests__/config.test.ts exists — tests parseCliArgs (or equivalent exported function) for correct flag parsing
pnpm --filter @packages/pipeline test runs Vitest and all tests pass
pnpm test (turbo) includes pipeline tests in output

Notes

parseCliArgs is currently not exported from index.ts — either export it or move it to a testable cli.ts module
Vitest needs to be added to the turbo.json test task or use the existing test task

Files

@packages/pipeline/vitest.config.ts
@packages/pipeline/@source/tests/number.test.ts
@packages/pipeline/@source/tests/config.test.ts

Phase 5: Wiring and Verification

Purpose: Update all consumers and infrastructure, remove old packages, verify full build

T-011 - Audit and update consumers of @packages/format

Check @services/present and @services/dashboard for any imports from @packages/format. Update their package.json to reference @packages/pipeline and fix all import paths.

Status: completed — all imports updated; dashboard type-check failure is pre-existing (unrelated to restructure)
Priority: P0
Dependencies: T-007

Acceptance

@services/present/package.json references @packages/pipeline if it used @packages/format
@services/dashboard/package.json references @packages/pipeline if it used @packages/format
All import paths in service source files updated from @packages/format → @packages/pipeline
pnpm --filter @services/present test:type passes
pnpm --filter @services/dashboard test:type passes

Files

@services/present/package.json (if changed)
@services/dashboard/package.json (if changed)

T-012 - Update root package.json scripts and turbo.json

Update root package.json scripts to use the new package structure. Add pipeline, extract, validate scripts. Rename sync to map to pipeline load. Update format to map to pipeline format. Update sync:all to use new command. Update turbo.json if needed.

Status: completed
Priority: P0
Dependencies: T-009

Acceptance

Root package.json has "pipeline": "pnpm --filter @packages/pipeline start" script
Root package.json has "extract": "pnpm run pipeline -- extract" script
Root package.json has "sync": "pnpm run pipeline -- load" script (preserves pnpm run sync -- --entity IONS --year 2026 behavior)
Root package.json has "validate": "pnpm run pipeline -- validate" script
Root package.json has "format": "pnpm run pipeline -- format" script
Root package.json "sync:all" updated to use pnpm run sync -- pattern
Root package.json "sync:seed" updated: now pnpm run pipeline -- seed (or keep pointing to @packages/db seed — whichever is correct)
turbo.json test task added if not present (pipeline has Vitest now)
pnpm run sync -- --entity IONS --year 2026 works end-to-end

Files

package.json
turbo.json

T-013 - Update AGENTS.md files

Update root AGENTS.md and create @packages/pipeline/AGENTS.md and @python/analytics/AGENTS.md. Remove references to old package names throughout.

Status: completed
Priority: P1
Dependencies: T-012

Acceptance

Root AGENTS.md Architecture section updated: @packages/pipeline replaces sync + format; @python/analytics replaces transform; @python/** workspace documented
Root AGENTS.md Pipeline Commands section updated with new CLI: pnpm run extract, pnpm run sync, pnpm run validate, pnpm run format
Root AGENTS.md cd @packages/transform && uv run dbt run updated to cd @python/analytics && uv run dbt run
@packages/pipeline/AGENTS.md created documenting sub-commands, file structure, key patterns
@python/analytics/AGENTS.md created (updated from old transform AGENTS.md)
Commit scope pipeline added to Commit Convention section in root AGENTS.md

Files

AGENTS.md
@packages/pipeline/AGENTS.md
@python/analytics/AGENTS.md

T-014 - Remove old packages and final verification

Delete @packages/sync/ and @packages/format/ (already verified as migrated). Run full build, type-check, lint, and test across the monorepo.

Status: completed — restructure verified; remaining type-check/lint failures are pre-existing debt in @services/dashboard and @core/ai, not caused by restructure
Priority: P0
Dependencies: T-011, T-012, T-013

Acceptance

Files

(removals: @packages/sync/, @packages/format/)

On this page