Skip to content
citrusquant

Data layout

This is the canonical contract for data you feed into citrusquant. The engine does not ship market data. You bring your own files (or any ObjectSource that serves the same keys).

yuzu-core only sees in-memory Panel values. The on-disk layout below is what yuzu-data reads and what yuzu-cli / yuzu-server expect under a data root (local directory or S3-compatible object store via yuzu-source-s3).

For NAV / metrics / strategy semantics, see backtest-engine.md. For strategy syntax, see lemon.md. If you load data from an FMP Starter-class key: which library features you can honestly backtest (and which are blocked by missing panels) is documented in fmp-data-source.md — feature/series gaps, not a plan comparison table.

What you must provide

ResponsibilityWho
Prices, fundamentals, optional membership panelsYou (or your pipeline)
Symbol list for a run (symbols / CLI universe)You (request or files present under prices/)
Point-in-time index membership (e.g. “in S&P 500 that day”)You — supply a 0/1 series or filter symbols yourself
Sector map for industry opsYou — optional CSV (see Industry map)
Evaluating lemon → positions → NAVEngine

1. Directory tree

Default key prefixes (override with env on the server: YUZU_PRICES_DIR, YUZU_FUNDAMENTALS_DIR, YUZU_PANELS_DIR):

<data-root>/
  prices/              # per-symbol OHLCV (required for price strategies)
    AAPL.csv.gz
    MSFT.csv.gz
  fundamentals/        # per-symbol factors (optional)
    AAPL.csv.gz
  panels/              # wide combined series (optional fast path)
    close.csv.gz
    pe.csv.gz
  tracked/             # optional; sector snapshot for industry ops
    universe.csv.gz    # any name; see industry section
PathRequired?Role
prices/Yes for CLI / price-based runsPer-symbol OHLCV archives
fundamentals/NoPer-symbol fundamental columns
panels/NoOne wide file per series (preferred when present)
tracked/NoSymbol → sector (and optional market cap)

Object keys under S3 are the same relative paths (prices/AAPL.csv.gz, …).

CLI note: yuzu-cli --data <root> currently loads close (and volume / a benchmark symbol when those config flags need them) from prices/. It does not auto-load fundamentals or industry. Use yuzu-server, or build an EvalContext in library code, for full series sets.

Server note: yuzu-server loads only series referenced by the strategy Data nodes (plus always high/low for MAE/MFE, and volume when liquidity or impact knobs are on). See Series name map.

2. File encodings

Loaders detect format from content, not only the extension:

FormatMagic / notesTypical key suffix
gzip CSV1f 8b.csv.gz (default write format)
plain CSVUTF-8 text.csv
Apache ParquetPAR1.parquet (read-only; needs yuzu-data parquet feature)

Probe order per object: .csv.gz → .parquet (if feature enabled) → .csv.

CSV rules (prices and fundamentals):

  • No quoting required; simple comma-separated values.
  • Header row required.
  • Dates in the day column are YYYY-MM-DD strings in CSV.
  • In memory / API bounds (from / to), dates are packed i32 YYYYMMDD (e.g. 20240102).
  • Empty numeric cells and the token NaN become floating-point NaN.
  • Rows should be oldest first.

Parquet: column names match CSV (day, adj_close, pe, …; combined panels use one column per symbol). The day column may be integer YYYYMMDD, a YYYY-MM-DD string, or a logical date type.

3. Series names (lemon ↔ files)

Bare identifiers in lemon (e.g. close, pe) become Data { name } leaves. Loaders map those names as follows.

Price series

Series name in lemon / panels mapColumn in prices/{SYMBOL}.*
closeadj_close
openadj_open
highadj_high
lowadj_low
volumevolume

Default price_key for the NAV loop is "close".

Fundamental series (per-symbol files)

Column order in fundamentals/{SYMBOL}.* (after day):

pe, ps, pb, roe, net_margin, debt_to_equity, market_cap, gross_margin,
receivables_turnover, debt_to_assets, revenue, revenue_growth, eps_growth,
operating_income_growth, net_income_growth, gross_profit_growth, report_event
NameMeaning
pe … gross_profit_growthFactor values (see list above). Dense / forward-filled is the usual convention so every trading day has a row.
report_event1.0 on a day a new report was disclosed, else 0.0. Missing → NaN (treated as “no event” by the engine’s truthiness).

These names are the Data series names — keep them snake_case and in this set if you use yuzu-server’s automatic loaders.

Snapshot factor panels (combined only)

These names are recognized as fundamental-side series for routing, but they are not columns of the per-symbol fundamentals CSV. Supply them as wide files under panels/:

piotroski_score, altman_z, fcf_yield, pe_industry_pctile,
analyst_upside_pct, consensus_rating

Custom series

Any other name (e.g. a membership mask in_sp500) is a normal panel in EvalContext.panels when you insert it (library / WASM path).

yuzu-server only auto-loads known price + fundamental series names. Custom series are skipped at load time; if the strategy still references them, run_backtest fails with an unknown-series error. To use custom series today, build the context yourself or extend the server loader.

4. Per-symbol prices — prices/{SYMBOL}.*

prices/AAPL.csv.gz

CSV header and sample:

day,adj_open,adj_high,adj_low,adj_close,volume
2024-01-02,9.5,11.0,9.0,10.0,1000
2024-01-03,10.1,11.5,9.8,10.8,1200
ColumnTypeNotes
dayYYYY-MM-DDOne row per trading day, ascending
adj_open … adj_closefloatSplit/dividend-adjusted prices
volumefloatShare volume

Missing symbol file → all-NaN column for that symbol when assembling a panel. Corrupt file is treated like missing (does not abort the whole batch).

yuzu-cli discovers the universe by listing files under prices/ (suffixes .csv.gz, .parquet, .csv).

5. Per-symbol fundamentals — fundamentals/{SYMBOL}.*

fundamentals/AAPL.csv.gz

Header (single line):

day,pe,ps,pb,roe,net_margin,debt_to_equity,market_cap,gross_margin,receivables_turnover,debt_to_assets,revenue,revenue_growth,eps_growth,operating_income_growth,net_income_growth,gross_profit_growth,report_event

Example row:

2024-01-02,28.5,7.1,45.0,0.15,0.22,1.4,2.8e12,0.45,8.0,0.3,1.2e11,0.05,0.08,0.06,0.07,0.04,0.0

Same date and NaN conventions as prices. Column order must match FUNDAMENTAL_FIELDS + trailing report_event in crates/yuzu-data/src/fundamentals.rs.

6. Combined panels — panels/{series}.*

Optional wide files: one object per series for the whole universe (one GET instead of N per-symbol reads).

panels/close.csv.gz
panels/pe.csv.gz

CSV shape:

day,AAPL,MSFT,GOOGL
2024-01-02,10.0,20.0,15.0
2024-01-03,10.5,,15.2
  • Header: day, then symbol tickers (any order; loaders reindex to the requested symbols list).
  • Empty cell → NaN.
  • Symbols requested but absent from the header → NaN column.

Load order in yuzu-server: try panels/{name} first; if the combined object is absent, fall back to per-symbol prices/ or fundamentals/. A symbol missing from an existing combined file is NaN until you rebuild — the server does not merge per-symbol rows into a partial combined file.

Rebuild helper (native): rebuild_combined_panels in yuzu-data (also exposed as the server rebuild path) writes gzip CSV under panels/ from per-symbol archives. Snapshot factor names listed above are not produced by that rebuild; supply those panels yourself.

7. Industry map

Industry ops (neutralize_industry, industry_rank, groupby_category) need EvalContext.industry: symbol → sector string.

Helper: yuzu_data::parse_industry_csv accepts CSV text shaped like:

symbol,sector,market_cap
NVDA,Technology,5103000000000
XOM,Energy,470000000000
  • Header row with symbol is skipped.
  • Rows with empty sector are dropped.
  • Extra columns after sector (e.g. market_cap) are allowed; only the first two fields are used for the map.

Wiring: the parser is in yuzu-data; yuzu-cli currently leaves industry empty. Pass the map when calling the library/server integration yourself, or load a snapshot from e.g. tracked/*.csv.gz in your product code.

8. Universe and point-in-time membership

Universe

The set of columns in every panel is the symbols list for that run:

  • Server: BacktestRequest.symbols
  • CLI: every symbol that has a file under prices/ (optionally limit by only syncing the files you want)

There is no special “universe file” format inside the engine.

Point-in-time (PIT) index membership

The engine does not maintain historical index constituents.

ApproachHow
Fixed listPass today’s (or any fixed) ticker list as symbols — simple, not true PIT
PIT list per runYour code chooses “listed as of from” tickers, then passes that symbols
Membership panelLibrary path: put a dates×symbols 0/1 panel in context (e.g. name in_sp500) and mask(signal, in_sp500) in lemon — only if that series is loaded into the context

Delisted names: keep prices/{SYM}.* files that end on the last trading day. Pair with BacktestConfig.delist_after / delist_haircut so the NAV loop force-exits after a NaN streak. Survivorship (only shipping still-listed names) is a data choice; the engine cannot invent missing history.

9. In-memory model (after load)

Every series becomes a Panel:

  • dates: Vec<i32> — YYYYMMDD, sorted unique
  • symbols: Vec<String> — column order
  • data: Array2<f64> — NaN = missing
  • Booleans are 1.0 / 0.0 in the same matrix

EvalContext = panels: HashMap<String, Panel> + industry: HashMap<String, String>.

Binary ops align panels (union of dates, intersection of symbols). Details: backtest-engine.md.

10. Minimal offline example

Directory:

mydata/
  prices/
    AAA.csv.gz
    BBB.csv.gz

prices/AAA.csv (plain CSV also works):

day,adj_open,adj_high,adj_low,adj_close,volume
2024-01-02,10,11,9,10,1000
2024-01-03,10,12,10,11,1100
2024-01-04,11,12,10,11.5,1200

prices/BBB.csv — same dates, different closes.

Strategy as a JSON Expr file mom.json (what yuzu-cli run --spec accepts). Equivalent lemon: is_largest(pct_change(close, 1), 1).

{
  "op": "IsLargest",
  "n": 1,
  "of": {
    "op": "PctChange",
    "n": 1,
    "of": { "op": "Data", "name": "close" }
  }
}

CLI (after cargo build -p yuzu-cli):

Terminal window
# dates are YYYYMMDD; universe = every symbol under prices/
yuzu-cli run --data ./mydata --spec mom.json --from 20240102 --to 20240104

To author in lemon text, parse with the lemon crate / CLI first, then pass the JSON tree to yuzu-cli or run_backtest.

Library sketch (no server):

// load panels with yuzu_data::load_panel / load_fundamental_panel,
// insert into EvalContext, then:
yuzu_core::run_backtest(&spec_json, &ctx, "close", &BacktestConfig::default())?;

Server request shape: see crates/yuzu-server/examples/backtest-request.json. Data root via YUZU_DATA_DIR (or equivalent) must contain prices/ (and optionally fundamentals/, panels/).

11. Source of truth in code

ContractLocation
Price columns / Fieldcrates/yuzu-data/src/csv_io.rs
Fundamental column listcrates/yuzu-data/src/fundamentals.rs → FUNDAMENTAL_FIELDS, REPORT_EVENT_FIELD, FACTOR_PANEL_FIELDS
Combined wide panelscrates/yuzu-data/src/combined.rs
Industry CSV parsecrates/yuzu-data/src/industry.rs
Format probecrates/yuzu-data/src/format.rs
Server series routingcrates/yuzu-server/src/lib.rs → price_field, handle_backtest
CLI price discoverycrates/yuzu-cli/src/lib.rs → list_symbols, load_ctx
FMP builder (writes this tree)crates/yuzu-cli/src/fmp.rs → yuzu-cli fmp-sync

If this doc and the code disagree, trust the code and update this file.