Configuration Layout ==================== This chapter is the canonical map of the ``configs/`` tree and explains how to place, name, and maintain configuration files in a way that scales for production operations. If you only need key descriptions, use :doc:`full_settings_catalog`. If you need stage behavior, use :doc:`pleb_deep_dive`. Purpose ------- PLEB keeps authored configuration separate from generated state. The goal is to avoid common failure modes: - policy duplicated across many run files, - generated lock artifacts mixed with authored files, - unclear ownership between run profiles, catalogs, and workflow orchestration. Canonical Tree -------------- .. code-block:: text configs/ runs/ ingest/ fixdataset/ pqc/ pipeline/ workflows/ steps/ catalogs/ ingest/ public_releases/ system_flags/ system_tables/ variants/ rules/ overlap/ relabel/ pqc/ schemas/ state/ lockfiles/ settings/ # legacy Directory Roles --------------- ``configs/runs/`` ~~~~~~~~~~~~~~~~~ Runnable mode profiles passed to ``pleb --config``. Use this for execution context: - stage toggles, - branch names, - paths to catalogs/rules, - run-local choices (for example detection profile selection). Do not use this for reusable mapping data that many profiles share. ``configs/workflows/`` ~~~~~~~~~~~~~~~~~~~~~~ Top-level workflow orchestration files passed to ``pleb workflow --file ...``. Use this for: - stage sequencing (serial/parallel groups), - step-to-step branch hand-off, - step overrides. ``configs/workflows/steps/`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Reusable step configuration fragments used by workflows. Use this for shared step templates so workflows remain concise and easy to review. ``configs/catalogs/`` ~~~~~~~~~~~~~~~~~~~~~ Reusable data assets. These are stable tables/mappings that multiple runs consume. Subfolders: - ``ingest/``: ingest mapping JSON files. - ``public_releases/``: provider catalog for public release comparison. - ``system_flags/``: editable mapping/allowlist/rule sources for system/telescope/backend inference. - ``system_tables/``: lookup tables used by fix/legacy behaviors (bandwidths, jump maps, overlaps, PTA labels). - ``variants/``: backend classification and ``_all.variant.tim`` definitions. ``configs/rules/`` ~~~~~~~~~~~~~~~~~~ Declarative behavior policies. Subfolders: - ``overlap/``: overlap resolution policies. - ``relabel/``: backend/system relabel policies. - ``pqc/``: per-backend PQC profile overrides. ``configs/schemas/`` ~~~~~~~~~~~~~~~~~~~~ Machine-readable schema files for tooling and GUI integration. ``configs/state/`` ~~~~~~~~~~~~~~~~~~ Generated runtime state (currently ingest lock snapshots/validation). Treat this as generated reproducibility metadata, not authored policy. ``configs/settings/`` ~~~~~~~~~~~~~~~~~~~~~ Legacy location from older layouts. Do not place new config assets here. What Goes Where --------------- Use this decision rule: 1. Is it directly executed with ``--config``? Put it under ``configs/runs//``. 2. Is it a multi-step orchestration file? Put it under ``configs/workflows/``. 3. Is it static mapping/lookup data reused by many runs? Put it under ``configs/catalogs//``. 4. Is it a policy rule set (behavior choice)? Put it under ``configs/rules//``. 5. Is it generated by run tooling? Put it under ``configs/state/``. Naming Guidelines ----------------- Run profiles ~~~~~~~~~~~~ Use ``_.toml``. Examples: - ``fixdataset_par_defaults.toml`` - ``pqc_balanced.toml`` - ``ingest_epta_data.toml`` Workflow files ~~~~~~~~~~~~~~ Use ``_.toml``. Examples: - ``branch_chained_fix_pqc_variants.toml`` - ``j1713_j1022_stress_parallel_serial.toml`` Catalog and rule files ~~~~~~~~~~~~~~~~~~~~~~ Use ``_.`` where practical. Examples: - ``backend_classifications_legacy_new.toml`` - ``system_flag_mapping.ingest_epta_data.json`` - ``backend_profiles.example.toml`` Path Conventions ---------------- Prefer repository-relative paths for files inside this repo: - ``configs/catalogs/ingest/ingest_mapping_epta_data.json`` - ``configs/rules/pqc/backend_profiles.example.toml`` Use absolute paths only for external source trees and environment-specific locations. Recommended split: - external data roots: absolute, - repo catalogs/rules/workflows/runs: relative. Composition Model ----------------- PLEB resolves configuration in this order: 1. code defaults, 2. TOML config values, 3. CLI overrides (for example ``--set key=value``). Operationally: - keep baseline behavior in TOML, - use CLI overrides for temporary experiments, - persist successful override changes back to TOML. Workflow Layout Pattern ----------------------- For production-quality multi-stage execution: 1. workflow file in ``configs/workflows/`` references step files in ``configs/workflows/steps/``; 2. step A writes to branch ``X``; 3. step B sets ``fix_base_branch = "X"`` and writes to branch ``Y``; 4. step C sets base ``Y`` and applies final mutations. This keeps each transformation stage auditable. Catalog vs Rule Boundary ------------------------ A common confusion is where overlap/relabel data should live. Use this distinction: - If it is primarily lookup data (table-like, broadly reused), use ``configs/catalogs``. - If it encodes conditional behavior or strategy choice, use ``configs/rules``. Examples: - jump map table -> ``catalogs/system_tables``. - prefer-multichannel overlap action policy -> ``rules/overlap``. Generated State Policy ---------------------- ``configs/state/lockfiles`` contains generated lock artifacts. Recommended practice: - exploratory ingest: lock strictness relaxed, - production ingest: lock strictness enforced and lock artifacts tracked. Do not manually hand-edit lockfiles unless debugging. Migration Map ------------- If you still use historical paths, migrate using: - ``configs/system_tables/...`` -> ``configs/catalogs/system_tables/...`` - ``configs/runs/workflow_steps/...`` -> ``configs/workflows/steps/...`` - ``configs/settings/*.lock*.json`` -> ``configs/state/lockfiles/...`` Maintenance Checklist --------------------- When adding a new configuration file: 1. place it in the correct role directory, 2. choose a clear mode/intent name, 3. add a top comment showing intended invocation, 4. avoid absolute paths for repo-local assets, 5. update docs examples if user-facing. When changing layout: 1. update this chapter, 2. update ``configs/README.md``, 3. update path references in docs and tests, 4. update Python defaults that embed paths. Cross References ---------------- - Overview of settings strategy: :doc:`configuration` - Full key catalog: :doc:`full_settings_catalog` - Stage orchestration and behavior: :doc:`pleb_deep_dive` - Run mode usage: :doc:`running_modes`, :doc:`modes`