caishen

PDSP — Persistence Architecture

RISE Framework Specification

Spec ID: 70 Version: 1.0 Document ID: caishen-rise-pdsp-arch-v1.0 Last Updated: 2026-08-01 Depends on: 03 (PDS acquisition), 10 (data schemas) Defines contracts for: 71, 72, 73, 74, 75, 76, 77

Creative Intent

What PDSP Enables Users to Create:

A durable, queryable market-price history that converges toward truth without ever re-downloading what it already holds
Point-in-time reconstruction of any chart, for any instrument/timeframe, as it would have appeared at a chosen moment
A single shared source of price truth that many independent processes — chart UIs, indicator engines, strategy runners — can read concurrently

Desired Outcomes:

Asking for “300 bars of EUR/USD H4 as of 2022-03-15 14:00” returns exactly that, cheaply, offline, and repeatably
A refresh run costs work proportional to what changed, not to the size of the history
The currently-forming bar is always identifiable and always current
Completion of a refresh is an event other systems can react to, not a condition they must poll for

Layer Decomposition

The original implementation separated five layers. The separation is worth preserving because each layer has a distinct reason to change.

┌─────────────────────────────────────────────────────────┐
│ CLI / Batch Engine            (spec 74)                 │
│   argument grammar, context expansion, orchestration    │
├─────────────────────────────────────────────────────────┤
│ Service Layer                                            │
│   coarse-grained operations, remoting surface,           │
│   process launching, DTO conversion                      │
├─────────────────────────────────────────────────────────┤
│ Business / Component Layer                               │
│   incremental update algorithm (spec 72),                │
│   lifecycle state machine, event emission (spec 73)      │
├─────────────────────────────────────────────────────────┤
│ Data Access Layer                                        │
│   query composition, identity resolution,                │
│   unit-of-work, stored-procedure bindings                │
├─────────────────────────────────────────────────────────┤
│ Entity Layer                                             │
│   bar, instrument property, collections, key derivation  │
└─────────────────────────────────────────────────────────┘

Rule: the Business layer never issues storage-engine-specific commands, and the Data layer never decides whether a bar should be written. Violating this is what made the original hardest to port.

Entity Layer

PricePoint (original: `PDSPPrice`)

The atomic unit: one OHLC bar, carrying both bid and ask sides.

Field	Type	Nullable	Meaning
`povTlid`	string(32)	no	Primary key. Composite natural key — see Bar Identity below
`instrument`	string(16)	no	Canonical instrument symbol, e.g. `EUR/USD`
`timeframe`	string(4)	yes	Canonical timeframe code, e.g. `H4`
`askO,askH,askL,askC`	float	no	Ask-side open/high/low/close
`bidO,bidH,bidL,bidC`	float	no	Bid-side open/high/low/close
`volume`	int	no	Tick volume for the period
`dt`	datetime	no	Period start timestamp (see Time semantics)
`isIncompleted`	bool	yes	`true` = the period is still forming. See spec 72
`instrumentPropertyRef`	string(16)	no	FK to `InstrumentProperty.instrument`

Notes for implementers:

Bid and ask are both stored, unaggregated. Downstream consumers derive mid, median, or spread as they need. Do not collapse to a single price at write time — the export layer (spec 75) and the charting layer disagree about which derivation they want, and both must remain possible.
Price numeric type is an open decision, not an inherited one. The original stores prices as binary double (float in the source schema, double in the application) throughout. This spec deliberately does not carry that forward as a requirement. A re-implementation must decide explicitly:
- Binary floating point — matches the original bit-for-bit, matches the broker’s own representation, and is what every indicator formula in specs 04/05 was tuned against. Accumulated error is irrelevant for OHLC storage because bars are stored, not summed.
- Fixed-point decimal, scaled per instrument — exact comparison and exact pip arithmetic, which matters for the ordering path (spec 76) where a breakout comparison against a stored trigger price must be deterministic. Requires precision from the instrument properties at every boundary.
The trade-off is real and differs by consumer. Recommendation: binary double for the price store (fidelity to source, no conversion on ingest), fixed-point decimal in the ordering domain where equality and rounding are decisions rather than approximations. Whichever is chosen, state it once and apply it consistently — the original never made the choice, it inherited it.
isIncompleted is nullable in the original schema and treated as “null means false” by every reader. A re-implementation should make it non-nullable with default false; the null case exists only in legacy rows.
The instrument property reference is a genuine foreign key, not a denormalized copy — bars cannot exist for an instrument whose trading properties are unknown, because pip size and precision are required to interpret the prices.

Bar Identity — the `povTlid`

This is the single most important design decision in the store, and the one most easily lost in a port.

Every bar’s primary key is derived deterministically from (instrument, timeframe, periodStart). No surrogate ID, no auto-increment.

povTlid  =  <fileNamePOV> "__" <tlid>

fileNamePOV = instrument with "/" → "-", then "_" + timeframe
tlid        = periodStart formatted per the timeframe's precision

Timeframe → timestamp format (the “tlid” reduction):

Timeframe	Format	Example key
`M1` (monthly)	`yyyyMM`	`EUR-USD_M1__202203`
`W1` (weekly)	`yyyyMMdd`	`EUR-USD_W1__20220314`
`D1` (daily)	`yyMMdd`	`EUR-USD_D1__220315`
`H4`, `H1`	`yyMMddHH`	`EUR-USD_H4__22031512`
`m15`, `m5`, `m1`	`yyMMddHHmm`	`EUR-USD_m5__2203151435`

Reference implementation: src/Caishen/Common/PS.Common.Framework/PovType/BarTlider.cs.

Why this matters:

Idempotent writes. Re-fetching the same period from the broker produces the same key, so “insert if absent, update if present” is a total function of the data — no dedup table, no lookup by timestamp range.
Human-legible keys. Operators can read a key and know exactly what it is, which is why the debugging and repair SQL in the repo works at all.
Precision matches the timeframe. A daily bar’s key does not carry hours, so two daily bars for the same day cannot both exist regardless of what timestamp the broker returned.

Constraint for re-implementation: the key derivation function must be pure, must live in a single shared module, and must be called with identical arguments by the writer and by every reader.

The original violates this, and it is a live defect rather than a stylistic one. MakePovTlid takes two independent reduction flags:

MakePovTlid(instrument, tf, dt, reduceHighTF = false, reduce_m15_m5 = false, ...)

and the call sites disagree on the second:

Role	Call site	Arguments	m15 key format
Writer	`PDSP.Data/Ctx/PDSPEntitiesExtensions.cs:30`, `PDSEntities2203.cs:298`	4 args → `reduce_m15_m5` defaults false	`yyMMddHHmm`
Reader	`PDSP.Business/PDSPPriceComponent.cs:92`, `PDSEngine2203/Program.cs:1062`	5 args → passes `ReducePovTlid` (true)	`yyMMddHHm`

For m15 the two differ by one character, so a key computed at read time can never equal the key that was written. Consequences are specified as defect 10 in spec 72. Both dated engine snapshots pass ReducePovTlid_m15_m5 (which is false) at the same site, so the author located this — the corrected snapshot was never promoted.

Two further sources of drift to remove:

A stale private mkPovTlidParts remains in PDSPPrice.cs:48-59 alongside the live path, with subtly different behaviour (see the space replacement note below).
BarTlider carries two competing reduction schemes: GetBarTlid (BarTlider.cs:22-42) reduces by substring length (W1 → 6 chars, M1 → 4), while getTlidFormatterStringByTimeframe (:105-125) reduces by format string (W1 → yyyyMMdd = 8, M1 → yyyyMM = 6). Only the second produces povTlid values, but a reimplementer reading legacy keys or other subsystems’ tlids needs to know both exist.

Requirement: one function, one flag set, one call signature. If two reduction policies are genuinely needed, make them a named enum argument that cannot be defaulted.

Note on the fileNamePOV rule. The live path (POV.ToFileNamePOV, PovType/POV.cs:223-230) performs only / → -. The space → _ replacement exists solely in the dead private mkPovTlidParts (PDSPPrice.cs:51) and is therefore not part of the key format as persisted. Do not implement it unless instrument symbols containing spaces are actually in scope.

Timeframe canonicalization

The original carried an unresolved naming collision: M1 means one month, while m1 means one minute. Case alone distinguished them, and several storage paths could not preserve case, producing the aliases mi1 and min1.

The code contains normalization at four separate points (PDSPPrice.OnTimeframeChanged, PDSPPrice.OnPovTlidChanged, PDSEntities2203.FixTF, and inline Replace calls in the engine and in the export procedures).

Requirement for re-implementation: define a closed enumeration of timeframes with unambiguous, case-insensitive codes. A suggested set that avoids the collision entirely:

Code	Period	Legacy forms to accept on read
`MN1`	1 month	`M1`
`W1`	1 week	—
`D1`	1 day	—
`H8`, `H6`, `H4`, `H2`, `H1`	hours	—
`m30`	30 minutes	—
`m15`	15 minutes	`min15`
`m5`	5 minutes	—
`m1`	1 minute	`mi1` (see below), `min1`

Accept legacy forms when reading migrated data; never write them. Normalize once, at the system boundary — not at four layers.

Migration-critical: mi1 is not a legacy alias — it is the canonical persisted form for 1-minute data. POV.ToFileNamePOV (PovType/POV.cs:226) rewrites m1 → mi1 when building every key, and PDSPPrice.OnTimeframeChanged (PDSPPrice.cs:17-18) rewrites the Timeframe column the same way. So stored rows carry Timeframe = 'mi1' and povTlid values containing _mi1__, never _m1__. Production queries in the repo confirm it (gia-mssql/issue__CADJPY_PDSP__221004.sql, gia-mssql/SQLQuery3.sql — both filter on 'mi1').

m1 is the input spelling accepted at the CLI and in the API; mi1 is the storage spelling. A migration that treats mi1 as a rare legacy value will miss the entire 1-minute dataset. min1 is genuinely deprecated — the engine raises an error on it (Program.cs:1397).

InstrumentProperty

Trading metadata required to interpret and act on prices.

Field	Type	Meaning
`instrument`	string(16)	Primary key. Canonical symbol
`pipSize`	float	Value of one pip in quote currency
`precision`	int	Decimal places for display/rounding
`mmr`	float	Minimum margin requirement
`lmr`	float	Liquidation margin requirement
`quantityMinimum` / `quantityMaximum`	int	Order size bounds
`baseUnitSize`	int	Contract base unit
`contractMultiplier`	float
`contractCurrency`	string
`trailingStepMinimum` / `trailingStepMaximum`	int	Trailing-stop bounds
`subscriptionStatus`	bool	Whether the feed is subscribed
`marketCode`	string(64)	Classification: `FOREX`, `COMMODITY`, `INDICE`, `TREASURY`

Resolution chain (original: PDSPInstrumentPropertyComponent.Get):

Look up in the database by canonical instrument
If absent → ask the acquisition layer (local cache on disk, then broker)
Persist the newly resolved property
Return it

This chain is a lazy upsert. It means a first-ever fetch of a new instrument transparently populates its properties. Preserve this — the alternative (requiring pre-registration of instruments) was tried and abandoned.

Defect to fix, not port: the marketCode classification loop compares prop.Instrument == "i" — a string literal where the loop variable i was intended. The classification therefore never fires. Both PDSPInstrumentPropertyComponent.updateMarketCode and the duplicate in PDSEntities2203.updateMarketCode carry this bug. The intent is clear and should be implemented correctly: classify the instrument by membership in the configured commodity / index / treasury sets, defaulting to forex.

Collections

PriceSeries (original PDSPPriceCollection) — an ordered set of PricePoint for one instrument/timeframe, the unit returned by reads.
RawPriceSnapshot (original PDSPRawPriceCollection) — an ordered set plus the request envelope (dtFrom, dtTo, instrument, timeframe). This is the form serialized to disk as a fetch cache; see spec 72, Disk snapshot cache.

Instrument symbol normalization

Two spellings exist throughout: EUR/USD (canonical, used in the database and in queries) and EUR-USD (filesystem-safe, used in keys, filenames, and CLI arguments). Conversion is a bare Replace("-", "/") / Replace("/", "-") scattered across layers.

Requirement: a single InstrumentSymbol value type with two explicit renderings (canonical and fileSafe), converted once at the boundary.

Data Access Layer

Responsibilities

The data layer owns: query composition, identity lookup, unit-of-work batching, and bindings to storage-side procedures. It owns no business decisions.

Required operations

Interface PriceStore:

  GetByKey(povTlid) -> PricePoint | null
      Exact identity lookup.

  GetOldestIncomplete(instrument, timeframe) -> PricePoint | null
      Returns the most recent bar flagged incomplete for this series,
      or null if none. Despite the name, ordering is DESCENDING by dt —
      "oldest" here means "the oldest point we must refresh from",
      which is the newest incomplete bar. See spec 72.

  GetSeries(instrument, timeframe, dtLatest, nbPeriods) -> ordered PricePoint[]
      Point-in-time window. Semantics:
        - filter: dt <= dtLatest
        - order DESCENDING by dt, take nbPeriods
        - then re-order ASCENDING for return
      This "take from the end, then flip" shape is required: it returns the
      N bars immediately preceding the cut-off, regardless of gaps (weekends,
      holidays, missing data). A naive BETWEEN range query returns the wrong
      count whenever the market was closed.

  GetPeriodTimestamps(instrument, timeframe, dtFrom, dtTo) -> ordered datetime[]
      Returns only the timestamps present in a range. Used as a cheap
      existence probe (spec 72, range fast-path) and to drive point-in-time
      replay without loading bar bodies.

  GetLast(instrument, timeframe) -> PricePoint | null
      Single most recent bar.

  Upsert(PricePoint)         -- insert if key absent, else update in place
  UpsertRange(PricePoint[])  -- bulk path; see spec 72 for the fallback rule
  Commit()                   -- flush the unit of work

Query composition rules

Always filter on (instrument, timeframe) as a pair, never on a LIKE pattern over povTlid. The original mixed both; the LIKE 'AUD/USD_H4%' form appears in several views and procedures and is a prefix scan that defeats indexing and mis-matches similar symbols. Filter on the columns.
Time window is inclusive of the cut-off. When a caller supplies an explicit dtLatest, the original adds one second before comparing so that a bar exactly at the cut-off is included. Prefer an explicit <= comparison over timestamp arithmetic.
Never assume contiguity. Markets close. Bars are missing by design, not by fault. Every “N periods back” computation must be a row-count operation, not a wall-clock subtraction.

Period arithmetic

Converting “N periods” to a wall-clock offset is needed only for requesting data from the broker, never for reading from the store. The original mapping (DtChartingHelper):

Timeframe	Minutes per period
`m1`	1
`m5`	5
`m15`	15
`H1`	60
`H4`	240
`D1`	1,440
`W1`	10,080
`M1` (monthly)	43,200 (30 days, approximated)

Plus a weekend-padding fudge factor when computing a start date, so that requesting 300 periods actually spans enough calendar time to contain 300 trading periods.

Requirement: keep this arithmetic confined to the acquisition request path and label it clearly as an approximation. It must never be used to decide what exists in the store — that is what GetPeriodTimestamps is for.

Business / Component Layer

PriceComponent (original: `PDSPPriceComponent`)

The component owns the incremental update algorithm and the lifecycle events.

Interface PriceComponent:

  UpdateFrom(instrument, timeframe, freshHistory) -> PricePoint[]
      THE incremental operation. Fully specified in spec 72.

  GetAsPriceHistory(instrument, timeframe, dtFrom="now", nbPeriods=300)
      -> PriceHistory
      Reads a window. Applies timeframe-aware cut-off adjustment before
      querying (see below).

  GetTimerange(instrument, timeframe, dtPoint1, dtPoint2)
      -> map<datetime, PriceHistory>
      Point-in-time replay: for every period timestamp in the range, return
      the full N-bar window as it stood at that moment. This is what makes
      historical strategy replay possible. Cost is O(periods x nbPeriods);
      treat it as a batch operation, not an interactive one.

  IsTimerangeInStore(instrument, timeframe, dtPoint1, dtPoint2) -> bool
      Cheap existence probe over timestamps only.

  ReorganizeIndex() / RebuildIndex()
      Storage maintenance passthrough. See spec 71.

Cut-off adjustment. Before reading, the component advances the requested timestamp by exactly one period of the requested timeframe (Adjust_Dt_for_Timeframe_request). This is because callers say “as of 2022-03-15” meaning “including the bar that covers 2022-03-15”, and the raw dt stored is the period start. Without the adjustment, the caller silently loses the bar they were asking about.

Requirement: make this adjustment explicit and testable rather than implicit. A cleaner target formulation: readers specify whether the cut-off is periodStartAtOrBefore or periodCoveringInstant, and the store handles both.

Lifecycle state machine

The component carries an explicit state machine (PDSPPriceComponentStateEnum, driven by the generated PDSPPriceComponentFsm):

Instanciated ──▶ Idle ──▶ Running ──┬──▶ Running_LoadingPrices
                   ▲                 │            │
                   │                 │            ▼
                   └─────────────────┴──── Running_PriceLoaded
                                     │
                                     ▼
                                    End

State	Meaning
`Instanciated`	Constructed, machine not started
`Idle`	Started, no work in flight
`Running`	Composite state; a work cycle is active
`Running_LoadingPrices`	Substate: a read/fetch is in flight
`Running_PriceLoaded`	Substate: data present, ready to emit completion
`End`	Terminated; waiters released

Transitions emit StateChanging / StateChanged notifications carrying the from/to pair.

Assessment for re-implementation. The state machine as built is thin — it tracks load progress and little else, and 1,264 lines of it are generated boilerplate. Its genuinely valuable properties are:

Component lifecycle is observable — external code can subscribe rather than poll
A synchronization primitive lets a caller block until the component reaches End
Events fire at defined lifecycle points (spec 73)

Reproduce those three properties. Do not reproduce the code-generated machinery. The richer, genuinely essential state machine in this platform is the strategy one (spec 02), not this one.

Component events

Event	Payload	Fired when
`Initialized`	component identity	Machine started
`PriceLoadingStarted`	`(pov, dtFrom, nbBars)`	Before a read begins
`PriceLoadingCompleted`	`PriceSeries`	After a read returns
`PriceLoadingError`	error detail	Read failed
`StateChanging` / `StateChanged`	from/to state	Any transition

These are in-process events. The out-of-process propagation contract is spec 73.

Service Layer

PriceHistoryService (original: `PDSPHistoryServices`)

Coarse-grained façade. Adds nothing but composition and remoting-friendliness.

Interface PriceHistoryService:

  GetAsPriceHistory(instrument, timeframe, dtFrom, nbPeriods) -> PriceHistory
  GetAsPriceHistory(request) -> PriceHistory
  GetInstrumentProperties(instrument) -> InstrumentProperty
  GetTimerange(instrument, timeframe, dtPoint1, dtPoint2) -> map<datetime, PriceHistory>
  GetTimerangeTimestamps(instrument, timeframe, dtPoint1, dtPoint2) -> datetime[]
  IsTimerangeInStore(instrument, timeframe, dtPoint1, dtPoint2) -> bool
  GetOldestIncomplete(instrument, timeframe) -> PricePoint
  RequestRefresh(refreshRequest) -> Acknowledgement

RequestRefresh — the process-launch seam

RequestRefresh in the original spawns the CLI engine as a child process and optionally waits for it (PDSPHistoryServices.phPriceUpdate). The engine path, executable name, and working directory are hardcoded.

This is the seam where “service” and “batch job” meet, and it is the weakest point in the original design:

Hardcoded absolute paths make the service non-portable
Arguments are assembled by string concatenation with quoting that breaks on instruments containing spaces
Failure is observable only as a process exit code and console output
A fixed timeout (default 220s) silently abandons long runs

Requirement for re-implementation: replace process-spawning with an explicit job contract:

SubmitRefreshJob(instruments[], timeframes[], dtFrom, dtTo, options) -> jobId
GetJobStatus(jobId) -> {state, progress, startedAt, completedAt, errors[]}

The job may still be executed by a separate worker — that part of the design is sound, and it is what let refreshes run without blocking interactive charting. What must change is that the boundary becomes a typed contract with observable status, rather than a process launch with a timeout.

Remote service contract

The original exposed a small remoting surface (IPDSPService): a health check, a live-price read, and a test operation. It is thin because the real integration mechanism was the hook system (spec 73) and the shared database, not RPC.

A re-implementation should expose the read operations of PriceHistoryService over whatever transport suits the target stack, plus the job contract above. There is no need to preserve the original’s SOAP-era shape.

DTO conversion

A dedicated converter exists per entity (Converting/*Converter.cs), splitting generated mapping from hand-written overrides. The pattern is sound: keep entity shapes and wire shapes separate so storage changes do not break consumers.

Time Semantics

This is the highest-risk area of the whole system and the original never fully resolved it.

Observed facts from the source:

The engine sets a global flag SE_PDSP_Use_UTC_DateTime = true at startup, with an in-code note that it “requires to be set elsewhere for all components” — i.e. the setting is known to be applied inconsistently.
GetAsPriceHistory converts "now" to UTC only when that flag is set, but parses caller-supplied timestamps as local time regardless.
The stored dt column is a naive datetime with no zone.
Dedicated remediation artifacts exist in the repo: gia-mssql/tests/220925-validate_UTC_fixed.sql, gia-mssql/issue220810__Extra_Period_compared_to_FXTradeStation.sql, and two dated snapshots of the engine’s Program.cs named for UTC problems.

Requirement for re-implementation — non-negotiable:

Store every timestamp as an explicit instant in UTC.
dt is the period start, always. Document it in the schema.
Accept timezone-aware input at the boundary; reject naive input rather than guessing.
Render in local/broker time only at presentation.
Anchor period boundaries to the broker’s trading-day convention, not to the server’s midnight. The “extra period compared to the trading station” defect above is exactly this: the daily bar boundary is a broker convention, not a calendar fact.

Carrying the original’s ambiguity forward will reproduce its bugs.

Concurrency Model

The original is single-writer by construction: one CLI engine process at a time performs updates for a given series, while many readers query concurrently. There is no locking, no optimistic concurrency token, and no transaction spanning more than one save.

This worked because refresh runs were serialized by the operator. It is not safe under concurrent writers: two engines updating the same series would race on the incomplete bar.

Requirement: either

(a) preserve single-writer-per-series explicitly, with a lease or advisory lock keyed on (instrument, timeframe), or
(b) make the upsert genuinely atomic at the storage layer (a real insert-or-update on the natural key), which the deterministic key design already makes possible.

Option (b) is preferred and is nearly free given the key design. Readers need no changes under either option.

Error Handling Posture

The original’s posture is “log and continue” almost everywhere: individual bar failures are caught and skipped, save failures are caught and retried once, export failures are swallowed entirely, and hook failures are ignored.

This is a deliberate and correct choice for a market-data refresher — a single malformed bar must not abort a run covering dozens of instruments. But it was implemented without observability: failures printed to a console nobody read.

Requirement: preserve the resilience, add the accounting.

Every refresh run produces a structured result:
  { instrument, timeframe,
    barsInserted, barsUpdated, barsSkipped,
    errors[{ stage, key, message }],
    durationMs, startedAt }

A run that skipped 4,000 bars and a run that skipped none must be distinguishable without reading logs.

Traceability

This spec’s concept	Original artifact
PricePoint	`PDSP.Entities/Entities/PDSDataProvider2203Model.PDSPPrice.cs`
Bar identity / povTlid	`Common/PS.Common.Framework/PovType/BarTlider.cs`
InstrumentProperty + resolution chain	`PDSP.Business/PDSPInstrumentPropertyComponent.cs`
PriceStore operations	`PDSP.Data/Ctx/PDSDataProvider2203Model.PDSEntities2203.cs`
PriceComponent	`PDSP.Business/PDSPPriceComponent.cs`, `PDSPPriceComponentDbContext.cs`
Lifecycle state machine	`PDSP.Business/PDSPPriceComponentFsm.cs`, `PDSPPriceComponentStateEnum.cs`
PriceHistoryService	`PDSP.Services/PDSPHistoryServices.cs`
Remote contract	`PDSP.Contracts/SvcContracts/IPDSPService.cs`
Period arithmetic	`Common/PS.Common.Charting/DateTimeUtilities/DtChartingHelper.cs`