Data Ingestion Readiness Plan

This document defines the preparatory plan for the reliable measurement backbone. It covers ENG-5169, ENG-5239, and ENG-5240.

It is deliberately a documentation and specification artifact. It does not change ingestion behavior, database schema, background jobs, or API responses yet.

Purpose

The measurement layer needs to answer one operational question before the model layer can be trusted:

Are we collecting, normalizing, and deriving the data needed for the model
and portfolio analytics every day, and can we explain every gap?

Gordon validated the dashboard as useful as a consolidated display, but the next phase depends on source completeness. The system should not simply say that a value is missing. It should explain whether the gap is caused by identity mapping, provider coverage, entitlement, market hours, adjusted contracts, expired horizons, unsupported fields, or a retryable ingestion failure.

Scope

Current Foundation

The system already captures daily Amberdata snapshots for the core crypto instruments:

• Greeks
• IV surface
• spot
• open interest
• BTC and ETH snapshot history
• Alpha trades for ICOI and IMST

This is the correct foundation. The next readiness layer should make the data capture auditable and repeatable enough that model pages can distinguish:

• data that is loaded and fresh
• data that is stale
• data that is missing because the provider returned no rows
• data that is missing because the contract identity is unresolved
• data that is unsupported by the chosen provider
• data that is pending because the horizon has not matured
• data that exists only as a proxy, fallback, or estimate

Source Domains

Durable Job Contract

Any broad historical or rolling ingestion process should run as a durable job, not as a browser-bound or interactive admin request.

Durable jobs should be:

• checkpointed: store progress by source, portfolio, instrument, date window, and job stage
• idempotent: rerunning a job should update the same logical observations without duplicating rows
• resumable: failures should restart from the last completed checkpoint
• bounded: request windows should respect provider limits, including one-hour quote windows where applicable
• auditable: record source URL family, provider status, row counts, timestamps, retry count, and final reason state
• non-blocking: long-running backfills should not depend on an open browser or a single request lifecycle
• source-labelled: every written observation should carry source and quality metadata

Job Matrix

Coverage Reason Taxonomy

Coverage probes should return explicit reason buckets. They should not mutate trade economics or invent marks.

Data Readiness Output

Every readiness output should be intelligible to an operator. A useful response is not just a percentage. It should say what is ready, what is missing, and what action is required.

Minimum readiness shape:

data_readiness = {
  scope,
  as_of,
  requested_rows,
  analyzable_rows,
  blocked_rows,
  pending_rows,
  estimated_rows,
  proxy_rows,
  reason_counts,
  source_counts,
  oldest_source_timestamp,
  newest_source_timestamp,
  next_action
}

Example operator wording:

Execution benchmark coverage is blocked mostly by adjusted-contract identity and
provider-empty quote responses. Forward 7d outcomes are additionally blocked by
expired-before-horizon rows that need terminal marks.

Provider Validation Questions

These questions can be investigated before Gordon makes business decisions because they do not force an economic assumption.

Decision Points

Implementation Boundaries

Preparatory work can proceed now:

• document source contracts and provider questions
• define durable job and checkpoint requirements
• define coverage reason buckets
• expose non-mutating readiness diagnostics in later implementation
• keep source-quality states separate from economic values

Implementation should wait for business/source answers before:

• treating proxy or fallback marks as decision-grade
• using adjusted standard-root fallback in sourced aggregates
• displaying realized P&L
• ingesting SMA trades into production scope
• making strong trade warnings dependent on incomplete source families

Acceptance For Preparatory Stage

Preparatory documentation is complete when:

• the reliable measurement backbone is described as a first-class product pillar
• every remaining data gap can be assigned to a named reason bucket
• durable backfill jobs have a clear non-interactive operating contract
• provider-validation questions are explicit
• no documentation implies missing economics should be shown as zero
• no documentation implies a proxy market context is the same as a traded-contract mark