caishen

PDSP — Refresh Engine CLI

RISE Framework Specification

Spec ID: 74 Version: 1.0 Document ID: caishen-rise-pdsp-cli-v1.0 Last Updated: 2026-08-01 Depends on: 72 (incremental update), 73 (events)

Creative Intent

What The Engine Enables:

Refreshing an arbitrary slice of the market — one instrument or every instrument, one timeframe or all of them — with a single short command
Scheduling those refreshes at cadences matched to each timeframe, so minute data updates often and monthly data rarely
Backfilling years of history and resuming that backfill after interruption

Desired Outcomes:

refresh CTX H4 LAST LAST brings the working set current, doing minimal work
The same command is safe to run on a timer, unattended, indefinitely
A named group of instruments can be refreshed without enumerating it
Long backfills survive interruption

Invocation Grammar

engine <instrumentSpec> <timeframeSpec> <from> <to> [exportFlags...]
engine <subcommand> [args...]

Positional arguments

Position	Name	Meaning
1	`instrumentSpec`	One instrument, a comma-separated list, or a group alias
2	`timeframeSpec`	One timeframe, a comma-separated list, or a group alias
3	`from`	Window start — see Window specification
4	`to`	Window end — see Window specification
5+	`exportFlags`	`avg`, `full`, or both, in any order

The refresh is the default action; anything not matching a known subcommand is parsed as a refresh.

Requirement: the original’s argument parsing indexes args[0..3] unconditionally and relies on exception handling for anything shorter, meaning a malformed invocation produces a stack trace rather than usage. Use a real parser with named options and a usage message. Keep the positional form as a compatibility shorthand if desired.

Instrument Specification

Spec	Expands to
`EUR/USD`	that instrument alone
`EUR/USD,GBP/USD,USD/CAD`	the listed instruments
`CTX`	the configured working set (the instruments currently under study)
`allforex`	every configured forex pair
`noctx` / `noctxforex`	every forex pair except the working set
`crsipairs` / `allcrsi`	the currency-strength basket
`allindices` / `indices`	every configured index
`allcommodities` / `commodities`	every configured commodity
`all` / `allinstruments`	indices + forex + commodities
`<CUR>.pair`	every forex pair containing that currency, e.g. `CAD.pair`

CTX — “context” — is the central idea. Rather than naming instruments in every scheduled command, the operator declares a working set once in configuration, and every scheduled job refreshes whatever is currently in it. Changing focus is a configuration edit, not a rewrite of every scheduled job.

The noctx complement exists so background maintenance of everything outside the working set can run on a slower cadence than the working set itself.

.pair is a substring filter over configured forex pairs. Note it matches by substring, so USD.pair matches every pair containing USD on either side — which is the intent, but it means a currency code that is a substring of another would over-match. Constrain to whole currency-code positions.

Timeframe Specification

Spec	Expands to
`H4`	that timeframe alone
`H4,H1,m15`	the listed timeframes
`CTX`	the configured working timeframe set
`alltf` / `alltimeframes`	every configured timeframe

The refresh iterates the Cartesian product of instruments × timeframes. A configuration flag selects whether the outer loop is instruments or timeframes.

Why the loop order matters. Instrument-outer completes each instrument across all its timeframes before moving on, so a perspective becomes fully current as early as possible — right for interactive work and for multi-timeframe strategy evaluation. Timeframe-outer completes each timeframe across all instruments, so comparisons across instruments at one timeframe become consistent as early as possible — right for the currency-strength basket. Keep both; make the choice explicit rather than an environment variable read at startup.

Window Specification

Both from and to accept:

Form	Meaning
`LAST` or `L`	Anchored mode — derive the window from the store’s state. See below.
`now`	The current instant
`up`	Synonym for `now`, additionally forcing a provider fetch (bypasses the disk cache)
`y`	One day ago (`from` only)
a negative integer	That many days back from now, e.g. `-2200`
a timestamp	Parsed as an absolute date-time

Anchored mode — the operational default

When either bound is LAST, the engine derives the window per series:

anchor = FindAnchor(instrument, timeframe)          -- spec 72

IF anchor exists:
    from = anchor.dt
    to   = now + oneperiod(timeframe)
ELSE:
    -- cold series: fall back to a per-timeframe default lookback
    from = now - defaultLookbackDays(timeframe)
    to   = now + oneperiod(timeframe)

defaultLookbackDays (original getNbPeriodForTF, PDSEngine2203/Program.cs:1393-1409):

Timeframe	Days back	Timeframe	Days back
`m1` / `mi1`	5	`H2`	990
`m5`	45	`H4`	1,400
`m15`	120	`H6`	2,000
`m30`	90	`H8`	2,200
`H1`	390	`D1`	3,500
		`W1`	8,500
unlisted code	11	`M1` (monthly)	12,500

min1 raises a deprecation error rather than returning a value.

The 11-day figure is only the fallback for an unrecognized timeframe code — it is not the value for “everything else”. Using it for H4 or above would cold-start a series with almost no history.

The lookbacks are generous by design: they are wall-clock spans chosen to guarantee enough trading periods after weekends and holidays are excluded (cf. the period arithmetic in spec 70). They are not tuned numbers and a re-implementation should derive them from a target bar count plus the market calendar rather than copying the table.

This is the mode used by every scheduled job. It is what makes the whole system cheap: the engine asks the provider for exactly the span that can still change, and Algorithm A (spec 72) then merges it with minimal work.

The to bound is deliberately pushed one period into the future so the currently forming period is included in the provider’s response.

Note the direct dependency on spec 72’s anchor. If a series has no forming bar, anchored mode falls back to a fixed lookback rather than resuming after the last closed bar. Once spec 72’s ResumeFrom is implemented, anchored mode should use it instead of the day-count table — the table is a workaround for the missing bootstrap path.

Negative-integer form and request sizing

from = -2200 means 2,200 days back. The engine scales the provider’s per-request bar limit as the span grows (adding 300 bars per threshold crossed at −3, −6, −9, −12, −22, −33 days, and 1,000 more beyond −190 days; minute data is capped at 2,000).

This step-function is a fitted workaround for the provider’s paging, not a principle. Requirement: replace with explicit pagination — request in fixed pages until the window is covered or the provider returns nothing new. The engine already has the machinery (the disk snapshot cache, spec 72) to make paged backfill resumable.

Monthly and weekly adjustment

For M1 and W1 in non-anchored mode, the window start is pushed back an extra month. Long periods need extra span to guarantee the requested number of complete periods is returned. Correct in intent; it belongs in the period arithmetic module, not inline in the engine.

Export Flags

Flag	Effect
`avg`	After a series commits, export the mid-price CSV (spec 75)
`full`	After a series commits, export the full compressed CSV (spec 75)
`noavg`	Explicitly disable the averaged export

Exports run on background tasks; the engine awaits all outstanding export tasks before exiting. Preserve this — the process must not exit with exports in flight.

Subcommands

Subcommand	Purpose
`reorg`	Reorganize storage indexes (spec 71)
`rebuild`	Rebuild storage indexes (spec 71)
`read` / `r` `<instrument> <tf> <from> [nb]`	Query the store and print a series — no provider call, no writes
`rmax <instrument> <tf> <from> <max>`	As `read`, with an explicit row cap
`dt` / `dt2 <args>`	Timestamp/period inspection utilities
`bc <args>`	Chart-annotation component inspection
`ls-iprop`	List stored instrument properties
`xavg`	Run the averaged export standalone
`orderdata`	Inspect strategy ordering data (reaches into the strategy store)
`linqtl` / `bclinqtl`	Timeline queries against the strategy store
`wait <seconds>` / `sleep <seconds>`	Sleep and exit — a scheduling primitive

wait exists so a batch script can space out invocations without depending on a platform sleep utility. Harmless, but a scheduler concern; drop it in favour of the scheduler.

orderdata, linqtl, and bclinqtl reach into the strategy database from the price engine. This is a layering violation — the price engine holds a connection to a store it has no business writing or reading. In the re-implementation these belong to a strategy CLI (spec 02 / 41).

Execution Flow

 Initialize configuration
 Read .env overrides from the working directory
 Parse arguments; on failure print usage and exit
 Handle subcommands; otherwise continue as a refresh
 Expand instrumentSpec  -> instruments[]
    Expand timeframeSpec   -> timeframes[]
 Pre-resolve instrument properties for every instrument (lazy upsert, spec 70)
 Open the provider session
 FOR each (instrument, timeframe) in the configured loop order:
        a.  Derive the window (anchored or explicit)
        b.  RefreshSeries(...)                            -- spec 72
        c.  Emit SeriesUpdateCompleted                    -- spec 73
        d.  Queue exports if flagged                      -- spec 75
 Close the provider session
Optionally reorganize indexes                          -- if configured
Await all outstanding export tasks
Exit

Step 6 is a deliberate pre-pass: resolving properties for all instruments up front means the per-series loop never blocks on a provider metadata call mid-refresh.

Configuration Sources

The original reads configuration from three places, with unclear precedence:

Source	Contents
Application config file	Connection strings, feature flags
`.env` in the working directory	`REORGINDEX`, `BYTIMEFRAMES`, `FULLEXPORT`, `BYPASSADDRANGE`
Static application-configuration object	Instrument groups, timeframe groups, working set, verbosity, hook switches

.env keys observed:

Key	Effect
`REORGINDEX`	Reorganize indexes at the end of the run — non-functional as written, see defect 11
`BYTIMEFRAMES`	Loop timeframes as the outer dimension
`FULLEXPORT`	Force the full CSV export on
`BYPASSADDRANGE`	Disable the bulk-insert fast path (spec 72, Algorithm B) — a debugging escape hatch for when bulk insert conflicts

Requirements:

One configuration model with documented precedence (defaults → file → environment → command line).
Instrument groups, timeframe groups, and the working set are data, not code. In the original they are properties on a static object, so changing the working set requires a rebuild.
Credentials come from the environment or a secret store, never from a config file committed to the repository. The original’s config files carry SQL Server credentials inline; treat those as compromised and rotate them.

Scheduling Pattern

The engine is designed to be driven by an external scheduler, one job per timeframe, at cadences matched to the period length. The pattern from the original’s own usage notes:

Timeframe	Suggested cadence	Command
`m5`	every 5 min	`engine CTX m5 LAST LAST avg`
`m15`	every 15 min	`engine CTX m15 LAST LAST avg`
`H1`	hourly	`engine CTX H1 LAST LAST avg`
`H4`	every 4 h	`engine CTX H4 LAST LAST avg`
`D1`	daily	`engine CTX D1 LAST LAST avg`
`W1`	weekly	`engine CTX W1 LAST LAST avg`
`M1`	monthly	`engine CTX M1 LAST LAST avg`

Backfill uses the day-count form instead of LAST:

engine CTX M1  -50000 now avg
engine CTX W1  -26666 now avg
engine CTX D1  -15000 now avg
engine CTX H4   -2200 now avg
engine CTX H1    -740 now avg
engine CTX m15   -200 now avg
engine CTX m5     -60 now avg

Requirement: anchored mode must be safe to run concurrently with itself for different series but not for the same series (spec 70, Concurrency model). Two scheduled jobs whose instrument sets overlap will race. Either partition the schedule by series or acquire a per-series lease.

Output and Observability

The original writes to the console using absolute cursor positioning, so progress overwrites in place. This is unreadable when redirected to a file and it makes the output useless for automated monitoring.

Requirements:

Structured output (one record per series) on the primary stream, human progress on the secondary stream.

A machine-readable run summary:

{ runId, startedAt, completedAt,
  series: [ { instrument, timeframe, inserted, updated, skipped,
              provider: {requests, retries}, durationMs, errors[] } ],
  exports: [ { instrument, timeframe, path, bytes } ] }

Exit code reflects the run: 0 all series succeeded; non-zero if any series failed. The original always exits 0.
Verbosity as a level, not scattered boolean checks.

Defects To Fix, Not Port

#	Defect	Fix
1	Argument parsing indexes `args[0..3]` unconditionally; short invocations produce stack traces	Real parser + usage message
2	Debug mode prints `args[0..3]` before validating length	Validate first
3	Always exits `0` regardless of failures	Meaningful exit codes
4	Cursor-positioned console output	Structured logging
5	Instrument groups hardcoded on a static object	Configuration data
6	Strategy-store subcommands in the price engine	Move to the strategy CLI
7	Provider request sizing is a fitted step-function	Explicit pagination
8	Two dated snapshots of the engine’s main program kept alongside the live one	Version control, not filenames
9	`.env` read from the working directory with no precedence rules	Unified configuration model
10	No guard against concurrent runs over the same series	Per-series lease
11	`REORGINDEX` never takes effect. The `else` branch resetting `autoREorgIndex = false` sits inside the per-key loop (`Program.cs:274-276`), so any `.env` key processed after `REORGINDEX` clears it. Only a file whose sole key is `REORGINDEX` works.	Parse each key independently; never let one key’s absence reset another’s value
12	Snapshot cache read/write key mismatch. The cache read uses the loop-local `_dtFromString`/`_dtToString` (`Program.cs:813-814`) while the write uses the statics `dtFromString`/`dtToString` (`Program.cs:910`). A snapshot can be written under a key that is never read back, so the cache silently under-hits.	One request-envelope value, used for both (spec 72, Disk snapshot cache)

Appendix A — Observed Group Memberships

The group aliases above resolve against configuration files, one per group, in src/Caishen/Common/DTS.Common.Configuration.Console/. Each file’s first line is the active value; the lines below it are earlier values kept as history, interleaved with # comment lines. Values are comma-separated.

These are the memberships as last committed. They are data, not specification — reproduced here so the example commands in this spec are runnable, and so a migration has a starting set.

Group	File	Active value
`allforex`	`all_forex.txt`	AUD/CAD, AUD/JPY, AUD/NZD, AUD/USD, CAD/CHF, CAD/JPY, CHF/JPY, EUR/AUD, EUR/CAD, EUR/CHF, EUR/GBP, EUR/JPY, EUR/NZD, EUR/USD, GBP/AUD, GBP/CAD, GBP/CHF, GBP/JPY, GBP/NZD, GBP/USD, NZD/CAD, NZD/CHF, NZD/JPY, NZD/USD, USD/CAD, USD/CHF, USD/JPY (27)
`allindices`	`all_indices.txt`	AUS200, CHN50, ESP35, EUSTX50, FRA40, GER30, HKG33, JPN225, NAS100, SPX500, UK100, US2000 (12)
`allcommodities`	`all_commodities.txt`	Copper, CORNF, NGAS, SOYF, UKOil, USOil, WHEATF, XAG/USD, XAU/USD (9)
treasury	`all_treasury.txt`	Bund (1)
crypto	`all_crypto.txt`	BTC/USD, ETH/USD, LTC/USD, XRP/USD, BCH/USD (5)
`CTX` (working set)	`activesymbol.txt`	SPX500 (1 — see note)
`alltf`	`alltimeframes.txt`	M1, W1, D1, H8, H6, H4, H2, H1, m30, m15, m5, m1 (12)
`CTX` (timeframes)	`activetimeframes.txt`	D1 (1 — see note)

Note on the two CTX files. Their first lines are single values, with far richer sets on the lines beneath — the working set was narrowed by editing the top line and pushing the previous value down. The engine even ships an --edit-activesymbol / --edit-activetimeframe switch for exactly this. Two representative historical working sets, useful as realistic test fixtures:

NAS100,SPX500,US30,FRA40,AUD/CAD,AUD/JPY,AUD/NZD,AUD/USD,CAD/CHF,CAD/JPY,
CHF/JPY,EUR/AUD,EUR/CAD,EUR/CHF,EUR/GBP,EUR/JPY,EUR/NZD,EUR/USD,GBP/AUD,
GBP/CAD,GBP/CHF,GBP/JPY,GBP/NZD,GBP/USD,NZD/CAD,NZD/CHF,NZD/JPY,NZD/USD,
USD/CAD,USD/CHF,USD/JPY          # "all forex with indices I check"

M1,W1,D1,H8,H6,H4,H2,H1,m15,m5   # a full multi-timeframe working set

The forex list carries a comment recording deliberate exclusions — “AUD/CHF, CNH were removed (Issues) + other I won’t trade” — so the active set is narrower than the broker’s full symbol list by choice, not by accident.

Requirement (restating spec 74’s configuration rule): these belong in configuration a re-implementation can edit at runtime. The first-line-wins-with- history-below convention is a workable poor-man’s version history; a target should either keep it deliberately or replace it with real versioning, not lose it silently.

Traceability

Concept	Original artifact
Argument grammar & dispatch	`PDSEngine2203/Program.cs`, `Main` and its `switch`
Group membership files	`Common/DTS.Common.Configuration.Console/{all_forex,all_indices,all_commodities,all_treasury,all_crypto,activesymbol,alltimeframes,activetimeframes}.txt`
Group accessors	`Common/PS.Common.Framework/Config/PistisConfigurationContext.cs` (`AllForex`, `AllIndices`, `AllCommodities`, `AllTreasury`, `AllTimeframes`)
Working-set edit switch	`SE/SEngine/Program.cs:161,1052` (`--edit-activesymbol`, `--edit-activetimeframe`)
Instrument/timeframe expansion	`PDSEngine2203/Program.cs`, `#region POVs CTX setup`
Anchored window derivation	`PDSEngine2203/Program.cs` → `StartGettingHistoryPrices`
Default lookback table	`PDSEngine2203/Program.cs` → `getNbPeriodForTF`
`.env` handling	`PDSEngine2203/Program.cs`, `#region dotEnv for Query`
Provider session & listeners	`PDSEngine2203/ResponseListener.cs`, `SessionStatusListener.cs`
Service-side launcher	`PDSP.Services/PDSPHistoryServices.cs` → `phPriceUpdate`
Historical snapshots (do not port)	`PDSEngine2203/Program-220913-maybe-stable-but-UTC-issue.cs`, `Program-220925-chg-with-conflict.cs`