CortanaROI Brain

2026-05-15-mk3-data-foundation-constraint

6 min read

MK3 data-foundation constraint - what the corpus can and cannot validate

On 2026-05-15, while gating MK3 implementation behind a per-slice Nautilus-grounded design review (the S2.0 “data-inventory” gate), an audit of the actual recorded data established a load-bearing constraint that reshapes the entire MK3 validation strategy. It is filed here because it is empirical, non-obvious, and will be referenced by every future MK3 design decision - and because planning as if the data exists when it does not is the most expensive mistake available.

The finding (derived - decisions.db schema + machine-wide search, 2026-05-15)

~/cortanaroi-data/decisions.db is 24 GB but contains only four tables. The substantive one, scoring_events, holds 289,218 decision-cadence snapshots (one row per ~15s / per trigger, ≈5 weeks). Each row carries a replay_snapshot_json (the MK2 data_validator.snapshot_for_replay() payload - feed state + per-field timestamps at the decision moment), feature_json, ~80 derived signal columns, and outcomes. There is no raw UW WebSocket frame table. No raw .jsonl/.ndjson capture exists anywhere. MK2 never recorded the raw, continuous, sub-second UW stream - it recorded derived state sampled at scoring cadence.

Separately, a broad machine-wide search confirmed the 1-year UW data-shop bundle is absent from this machine. Per the MK2 build-week plan it was purchased 2026-05-07 (~$1,531, 10 datasets) and delivered via signed download links (which expire in hours/days). It was never downloaded here, or has been lost. The only historical UW-derived data that exists is the ~5-week scoring_events corpus.

What this means - the validation split

The MK3 architecture (the Setup Hunter spine) is sound. The validation substrate it assumed is partially missing. The honest split:

  • Scoring / composite / gate / setup logic - corpus-validatable (GREEN). 289K labelled events with replay snapshots, features, and outcomes is an excellent parity and conditional-EV oracle. This is the bulk of the port and is well-founded.
  • The impulse engine - forward-SHADOW only (RED for history). The MK2 live-impulse edge (HIRO-lite z-scores on 15s/30s/60s windows, sweep/block clustering, strike-stacking) operates on raw sub-second flow that was never recorded. It cannot be back-tested or parity-checked from history. It may run as SHADOW telemetry but may not anchor a LIVE setup on historical proof. Re-acquiring the data-shop bundle does not fix this - those datasets are aggregated products, not the raw stream. The only remedies are forward capture (start recording raw UW in MK2 now) or accepting it as unvalidatable-from-history and down-weighting it.
  • The 1-year OOS gate - blocked (RED). The spine’s §5 validation pipeline makes OOS_PASSED mandatory before LIVE, and explicitly says CORPUS_ONLY (the 5-week window) is necessary but insufficient because it overfits one regime. With the 1Y bundle absent, no setup can honestly reach OOS_PASSED, so LIVE is unreachable until the bundle is re-acquired from the UW account. This is a purchase/download prerequisite, not a build task.

Why it matters

This is the spine §7 “honest worst case” arriving early and concretely. It does not kill MK3 - the corpus-validatable bulk proceeds - but it fences two things hard: the impulse edge is forward-only, and live trading is gated on data that must be re-acquired. Discovering this at the planning stage, before a line of strategy code, is the validation discipline working as intended rather than failing three weeks into implementation. It also feeds the MK3 strategic stop/go gate: the framework migration is justified for the scoring/setup port; the impulse and OOS dimensions are explicitly constrained, not assumed.

Source

Derived - decisions.db PRAGMA table_info + row counts and a machine-wide filesystem search, 2026-05-15. Full record: ~/conductor/workspaces/cortana-mk3/indianapolis/plans/2026-05-15-mk3-slice-designs.md §S2.0 and .../plans/2026-05-15-mk3-architecture-map.md §6 “VALIDATION-STRATEGY DECISION”.

See Also

Update - 2026-05-21 (bundle acquired, partial unblock)

The 1Y UW data-shop bundle has been re-acquired and staged at ~/cortana-data/uw-historical/. Verified contents (du -sh per dataset): big_option_trades 265M, option_chains 177M, market_tide 7.9M, ohlc_1m 5.5M, net_flow_holdings 2.5M, net_flow 2.1M, ohlc_1d 36K, iv_rank/delta_exposure/gamma_exposure 12K each - 460 MB total, 10 parquet files. Same end-date (2026-05-07) and dataset list as the original 5-07 purchase.

Updated verdict:

  • 1Y OOS gate for scoring / composite / gate / setup logic - now EXECUTABLE (GREEN). OOS_PASSED is reachable for these once a slice’s Codex-authored uw_parquet_loader (Week-5-order) writes the parquets into a ParquetDataCatalog and the BacktestEngine replays them.
  • Impulse-engine raw-tick edge - still RED. The bundle is aggregated historical products (1y daily/intraday OHLC, big-option-trade prints, flow aggregates), not MK2’s raw sub-second WebSocket frames. No purchase brings back data that was never recorded. Impulse stays forward-SHADOW only; the MK2 raw-capture handoff (.context/codex-handoff-02-mk2-raw-uw-capture.md) remains the time-sensitive prerequisite for future impulse-OOS.

In short: the scoring/setup half of the validation pipeline is unblocked; the impulse-engine half still cannot be validated from history.

Update - 2026-05-21 (impulse-engine refinement after schema inspection)

After staging the bundle (prior update), an empirical schema inspection of big_option_trades/1y/2025-05-08-through-2026-05-07/SPY.parquet materially refines the impulse-engine verdict. The file is 3,151,260 per-trade rows over 1 year of SPY options (277 MB), with 30 columns including microsecond-UTC executed_at, nbbo_bid/nbbo_ask/ ewma_nbbo_* (sufficient for aggressor inference via Lee-Ready or EWMA-anchored variants), exchange (multi-exchange sweep clustering), price/size/premium/volume/open_interest, full per-trade Greeks (delta/gamma/theta/vega/rho/implied_volatility/theo), and report_flags/trade_code (almost certainly carrying UW’s sweep/block/opening/aggressor classification labels - to be confirmed when the Codex-authored loader reads them).

Refined verdict for the impulse-engine edge:

  • Updated from RED (forward-SHADOW only) → YELLOW (back-testable, pending classification validation). The trade-event sequence the impulse engine actually decides on (HIRO-lite z-score windows, sweep/block clustering, strike-stacking) is fully reconstructable from this 3.15M-row substrate. What is permanently lost is MK2’s specific receive-time / in-process artifacts - but those are not what the engine scored on; they were the engine’s runtime context, not its input.
  • The build-week plan’s classification-validation harness target (≥85% agreement between derived classifications and MK2 live labels on overlap days) is the gating step that flips YELLOW → GREEN (substrate good) or YELLOW → “buy Databento OPRA” (substrate insufficient, re-derive from raw OPRA).
  • Databento is therefore a conditional fallback, not a prerequisite. Saves significant budget while the cheaper UW-bundle path is exercised first.

This refines the §S2.0 verdict written into ~/conductor/workspaces/cortana-mk3/indianapolis/plans/2026-05-15-mk3-slice-designs.md. Scoring/setup OOS remains GREEN. The 1Y OOS_PASSED gate is unblocked (scoring) and conditionally unblocked (impulse, pending validation).

Graph View

Backlinks