caishen

SpiderDb — Strategy & Analysis Store Schema

RISE Framework Specification

Spec ID: 76 Version: 1.0 Document ID: caishen-rise-spiderdb-v1.0 Last Updated: 2026-08-01 Source of truth: src/Caishen/Databases/SpiderDbDatabase/gia-mssql/SpiderDb__scriptExport__220907.sql (UTF-16LE) Related: 02 (SDS), 10 (data schemas), 21 (ordering campaign), 40 (fractal breakout workflow)


Creative Intent

What SpiderDb Enables:

Desired Outcomes:

  1. A strategy created in the chart UI persists, and a headless runner can act on it
  2. Every state transition is recorded with its cause, in order, permanently
  3. Orders sent and responses received are correlated to the strategy that caused them
  4. Annotations and analyses reference specific bars, unambiguously

Relationship to PDSDb

SpiderDb and PDSDb are separate databases in the same server instance. There are no cross-database foreign keys; the join is by convention, in application code.

  PDSDb (spec 71) SpiderDb
Content Price bars, indicator panels Strategies, orders, timelines, annotations
Write pattern High-volume, machine-generated, incremental Low-volume, human- and event-driven
Row lifetime Immutable once closed Mutable through a lifecycle
Identity Content-derived key (povTlid) Assigned unique identifier (Idug)
Loss tolerance Recoverable — re-fetch from provider Not recoverable — it is the record of intent and action

Keep them separate. The retention, backup, and audit requirements are genuinely different. SpiderDb is small and irreplaceable; PDSDb is large and reconstructible.

Shared vocabulary: both use the povTlid / tlid bar-identity scheme from spec 70. That is the cross-database join key, and it is why annotations and strategies can point at exact bars without storing timestamps that might drift.


Schema Organization

66 tables across 17 namespaces. The namespaces are meaningful and worth preserving.

Namespace Domain Status
pl Strategy lifecycle — strategies, orders, trades, journals Core
tl Timelines — generic strategy event history Core
adsann Chart annotations Core
chart Chart bars, POV metadata, instrument properties Superseded by PDSDb
chart2 Second-generation chart data Superseded
neptune Instrument/POV analyses, observations, price alerts Active
mm Market monitoring — divergence scanning, market overview Active
gt Goal tracking — structural-tension goal hierarchy Active, separate concern
c Campaigns Emerging
ca Case analyses Active
asmpgc Pattern grammar — patterns, steps, terms Experimental
order Generic strategy ordering Superseded by pl
tdso Trading-data-service ordering Superseded by pl
plx Experimental exit-strategy modelling Experimental
conf Configuration data Support
security Identity — users, roles, claims, logins, profiles Support
dbo Views, procedures, sequence tables Mixed

Requirement: a re-implementation should carry forward pl, tl, adsann, neptune, mm, gt, ca and c. The chart/chart2, order, and tdso namespaces are superseded generations kept alongside their replacements — the same concept implemented three times. Migrate any surviving data and drop them.


Core: Strategy Lifecycle (pl)

pl.BDBOStrategies — the central entity

A breakout entry strategy: an intention to enter the market when price crosses a level. Fully specified behaviourally in spec 02 and spec 40; this is its persisted shape.

TABLE Strategies
  idug                      uuid          NOT NULL   -- PRIMARY KEY
  instrument                varchar(8)    NOT NULL
  timeframe                 varchar(8)    NOT NULL
  buySell                   varchar       NOT NULL   -- direction: B | S
  breakoutPrice             float         NOT NULL   -- the trigger level
  breakoutPriceReached      bit           NOT NULL   -- has it triggered?
  active                    bit           NOT NULL
  isNew                     bit           NOT NULL

  -- lifecycle
  status                    int           NOT NULL   -- numeric state
  stateId                   int           NOT NULL   -- state machine state
  stateName                 varchar(128)  NOT NULL   -- state machine state, named
  dt                        datetime      NOT NULL   -- created
  dtModified                datetime      NOT NULL
  dtFirstAction             datetime      NOT NULL

  -- sizing and risk
  k                         int           NOT NULL   -- position size (lots)
  dp                        float         NOT NULL   -- delta/pip parameter
  mpr                       float         NOT NULL   -- money/pip ratio
  vcp                       int           NOT NULL

  -- counters and limits
  cft                       int           NOT NULL   -- count fractal tries
  ctc                       int           NOT NULL   -- count trade cycles
  ceoc                      int           NOT NULL   -- count exit-order cycles
  maxTC                     int           NOT NULL
  maxEOC                    int           NOT NULL
  maxPUI                    int           NOT NULL

  -- signal validity
  diValid                   int           NOT NULL   -- divergence-indicator validity

  -- trailing
  tt                        bit           NOT NULL   -- trailing enabled
  tto                       int           NOT NULL   -- trailing offset

  -- exit
  exitStrategyType          int           NOT NULL
  exitStrategyTargetPrice   float         NOT NULL

  -- provenance
  correlationId             uuid          NOT NULL
  note                      text          NOT NULL
  meta                      varchar(4096) NOT NULL   -- JSON

Observations and requirements:

pl.StrategyTimelineItems — the event history

Every state transition, appended, never modified.

TABLE StrategyTimelineItems
  idug              uuid          NOT NULL   -- PRIMARY KEY
  strategyIdug      uuid          NOT NULL   -- FK → Strategies
  dt                datetime      NOT NULL
  statusPrevious    int           NOT NULL
  statusNew         int           NOT NULL
  eventName         varchar(255)  NOT NULL   -- what caused the transition
  itemType          varchar(128)  NOT NULL
  message           varchar(255)  NOT NULL
  note              varchar(4096) NOT NULL
  meta              varchar(8000) NULL       -- JSON

This is the most valuable table in the database. It is a genuine event log: append-only, ordered, carrying both the transition and its cause. It makes post-hoc analysis of “why did this strategy do that” possible.

Requirements:

pl.BDBOStrategyOrderRequests / pl.BDBOStrategyOrderResponses

The request/response pair for order placement — a durable record of what was sent and what came back.

TABLE OrderRequests
  idug            uuid           NOT NULL   -- PRIMARY KEY
  strategyIdug    uuid           NOT NULL   -- FK → Strategies
  dtRequested     datetime       NOT NULL
  buySell         varchar(1)     NOT NULL
  entryPrice      float          NOT NULL
  exitPrice       float          NOT NULL
  limitPrice      float          NOT NULL
  k               int            NOT NULL   -- size
  barJSONData     varchar(2048)  NULL       -- the bar that triggered it
  meta            varchar(4096)  NOT NULL

TABLE OrderResponses
  idug            uuid           NOT NULL   -- PRIMARY KEY
  requestIdug     uuid           NOT NULL   -- FK → OrderRequests
  strategyIdug    uuid           NOT NULL   -- FK → Strategies
  dtOrderCreated  datetime       NOT NULL
  orderID         varchar(32)    NOT NULL   -- broker's order identifier
  tradeId         varchar(64)    NOT NULL   -- broker's trade identifier
  meta            varchar(4096)  NOT NULL

The request/response split is the right pattern and must be preserved. A request exists the moment the intention is formed; a response exists only if the broker replied. A request with no response is a detectable, actionable state — an order that may or may not have reached the market. Collapsing these into one row loses that.

Requirements:

pl.BDBOCancelOrderRequests / pl.BDBOCancelOrderResponses

Same pattern for cancellation. Same requirements.

pl.BDBOTrades

Executed trades, linked to their strategy. The realized outcome.

pl.TraderJournals

Free-form notes attached to a strategy, positioned on the chart:

TABLE TraderJournals
  idug            uuid          NOT NULL   -- PRIMARY KEY
  strategyIdug    uuid          NOT NULL   -- FK → Strategies
  dt              datetime      NOT NULL
  summary         varchar(128)  NOT NULL
  note            text          NOT NULL
  xTlidPosition   varchar(32)   NOT NULL   -- horizontal anchor: a bar identity
  yPricePosition  float         NOT NULL   -- vertical anchor: a price

xTlidPosition is a bar identity, not a pixel coordinate — so a journal note stays anchored to the bar it concerns regardless of zoom, pan, or timeframe. This is the right way to anchor chart annotations and it should be adopted everywhere.


Core: Generic Timelines (tl)

A second, more general timeline implementation:

TABLE StrategyTimelines
  idug                  uuid          NOT NULL   -- PRIMARY KEY
  strategyIdug          uuid          NOT NULL
  pov                   varchar(64)   NOT NULL
  statusCurrent         int           NOT NULL
  dtCreated             datetime      NOT NULL
  dtModified            datetime      NOT NULL
  dtStrategyCreated     datetime      NULL
  note                  varchar(4096) NOT NULL
  initialJSONMetaData   text          NULL

TABLE TimelineItems
  idug                    uuid          NOT NULL   -- PRIMARY KEY
  strategyTimelineIdug    uuid          NOT NULL   -- FK → StrategyTimelines
  dt                      datetime      NOT NULL
  statusPrevious          int           NOT NULL
  statusNew               int           NOT NULL
  eventName               varchar(255)  NOT NULL
  itemType                varchar(128)  NOT NULL
  message                 varchar(255)  NOT NULL
  note                    varchar(4096) NOT NULL
  jsonMetaData            varchar(8000) NULL
  jsonChartMetaData       varchar(8000) NULL

This duplicates pl.StrategyTimelineItems with an added grouping level and a separate chart-metadata field. Two timeline implementations coexist and several views query each.

Requirement: unify. The tl shape is the better one — a timeline is a first-class object with its own identity and a POV — but pl’s items are the ones actually written by the strategy service. Pick the tl structure, migrate pl data into it, and drop the duplicate.


Chart Annotations (adsann, neptune)

TABLE POVAnnotations
  idug             uuid            NOT NULL   -- PRIMARY KEY
  pov              nvarchar(16)    NULL       -- e.g. "EUR/USD_H4"
  tlid             nvarchar(16)    NULL       -- the bar being annotated
  tag              nvarchar(128)   NULL
  annotationText   nvarchar(1024)  NULL
  fileTagName      nvarchar(255)   NULL       -- associated file/screenshot
  dtContext        datetime        NOT NULL   -- the moment being annotated
  dtCreated        datetime        NOT NULL
  dtModified       datetime        NOT NULL
  archived         bit             NOT NULL

Again the (pov, tlid) pair anchors the annotation to an exact bar.

Two copies exist — adsann.POVAnnotation and neptune.POVAnnotation — with slightly different nullability and a wider fileTagName in the neptune copy. neptune has generated CRUD procedures; adsann has seven per-timeframe views over it. Consolidate to one table.

Requirement: the seven vAnnotation__<TF> views plus two per-instrument variants are the same anti-pattern as spec 71’s per-instrument views. Replace with a parameterized query filtering on the POV’s timeframe component — and store instrument and timeframe as separate columns rather than parsing them out of a concatenated pov string with LIKE '%H4'.


Market Monitoring (mm)

Scans the market for divergence conditions across instruments and timeframes.

Table Purpose
ChaosStatus Per-bar market condition: divergence indicator, bullish divergent bar, divergence A/B, trend of last wave, strength ordering
MarketOverViewers Historical snapshots of the same, plus isDiverging and mouth-openness validation
MarketOverViewersCurrents Current snapshot only
VDivergenceIndicatorCurrents Current divergence indicator per series
InstrumentPerspectives Per-instrument divergence sequence across timeframes, encoded as a compact string
VInstrumentDivergenceIndicatorSequenceLasts Latest such sequence per instrument
TradingPerspectives Scan summary: counts of divergent bars and divergences at a moment
ChaosMessages Free-form messages keyed by bar identity
GetAOPeakDivRequests Oscillator peak-divergence query parameters

The core idea worth preserving: InstrumentPerspectives.DivergenceIndicatorSequenceData encodes one instrument’s divergence state across all timeframes as a single short string. That makes “show me every instrument whose weekly, daily and four-hourly all agree” a string match rather than a multi-way join. It is a deliberate and effective denormalization.

Requirements:


Goal Tracking (gt)

A structural-tension goal hierarchy — Robert Fritz’s methodology, the same framework behind the TandT application. It shares the database with trading but is a separate bounded context.

Table Purpose
Goals The goal itself: text, acronym tag, due date, completion, state, telescope level
CurrentRealities The current-reality side of the structural tension
GoalImages Visual representation of the desired outcome (binary + text)
MomentOfTruths A four-step review: difference from expectation → how it got that way → plan for next time → feedback system for the new plan
ConceptNotes Notes attached to any goal entity
Resources URIs and metadata attached to any goal entity
Tags Tags attached to any goal entity

Goals.IsATelescope / TelescopeLevel implement goal nesting — a goal that “telescopes” contains sub-goals at a finer level. This is the structural-tension hierarchy, and it is the same pattern the pattern-grammar and segment tables use.

MomentOfTruths is the review discipline made structural: its four columns are the four steps of the method, so a review cannot be recorded incompletely. This is a good example of encoding a practice in a schema.

Requirement: extract gt into its own database or service. It is coupled to trading only by convenience of hosting. All seven tables share a common base shape (idug, dtCreated, dtModified, sortOrder, note, textContext, correlationId, itemHistoryIdug, parentIdug) — that base belongs in a shared entity definition, and itemHistoryIdug suggests a versioning scheme that was planned but is not visible in the schema. Clarify or remove it.


Case Analysis & Campaigns (ca, c)

TABLE CACases
  idug            uuid          NOT NULL   -- PRIMARY KEY
  instrument      varchar       NOT NULL
  timeframe       varchar       NOT NULL
  dtPoint         datetime      NOT NULL   -- the moment being studied
  strategyIdug    uuid          NULL       -- optional link to a strategy
  note            varchar(4000) NOT NULL
  dtCreated       datetime      NOT NULL
  meta            varchar(8000) NULL

TABLE CAItems           -- observations within a case

TABLE Campaigns
  idug                       uuid          NOT NULL   -- PRIMARY KEY
  title                      varchar(255)  NOT NULL
  sortOrder                  int           NULL
  correlationIdug            uuid          NULL
  financialActionStepIdug    uuid          NOT NULL   -- links to gt.Goals
  targetBudget               float         NOT NULL
  targetBudgetMandatory      float         NOT NULL
  targetBudgetOptional       float         NOT NULL
  dtCreated, dtModified      datetime      NOT NULL

Campaigns is the bridge between the two halves of the system: a campaign links a financial goal (gt) to the trading activity meant to fund it, with a budget split into mandatory and optional targets. Even though gt should be extracted, this relationship is the reason the two ever shared a database. Model it as an explicit cross-context reference.

Note CACases.instrument/timeframe are varchar(8000). Size them properly.


Pattern Grammar (asmpgc)

An experiment in describing market behaviour as an ordered sequence of defined terms.

TABLE Patterns      { idug, patternType, patternTitle, patternNote }
TABLE PatternSteps  { idug, patternIdug, stepTermIdug, order, stepNote }
TABLE StepTerms     { idug, termText, termDefinition }

A pattern is an ordered sequence of steps; each step references a term drawn from a controlled vocabulary that carries its own definition.

Worth preserving as a concept. A vocabulary whose terms are defined in the data means the language of analysis is extensible without a schema change, and every use of a term is traceable to one definition. Only three tables and a handful of rows, but the idea is the most reusable one in the database.


Superseded Namespaces

chart, chart2 — replaced by PDSDb

chart.Bars holds price data and computed indicators in one row (alligator lines, oscillators, fractals, distances, bar classification). chart.PriceBars holds prices only. chart.InstrumentProperties, chart.POVData, and chart.POVPrices are three near-identical copies of instrument metadata — POVData and POVPrices differ only in name.

The evolution is legible: prices and indicators together (chart.Bars) → prices alone (chart.PriceBars) → a second attempt (chart2) → separation into PDSDb’s PDSPPrices + PanoIndicators.

The separation was the right conclusion. Prices are provider facts; indicators are computed and change when the computation changes. Do not re-merge them.

chart.Bars does carry fields absent from PDSDb’s panel and worth keeping: Median, BarHeight, IsBullish/IsBearish, GatorPricePosition, GatorMouthMaxOpenness, PriceRelativeDistancePercent.

order, tdso — replaced by pl

Two earlier generations of the order request/response pattern, structurally similar to pl’s. Migrate and drop.

plx — experimental exits

EntryStrategicTradeSets, StrategicTrades, StrategicExitModels, and a placeholder Class1s (an unrenamed template class that reached the database). An unfinished attempt at modelling multi-leg entries with per-leg trailing exits.

The idea is real and appears in spec 02’s exit strategies. Take it from there, not from these tables. Drop Class1s.


Support Namespaces

security

Identity tables — users, roles, claims, logins, profiles — in the shape of a standard framework’s identity schema.

Requirement: do not re-implement. Use the target platform’s identity system. If any of these rows survive, treat every stored credential as compromised.

conf

DTSConfigDatas — application configuration in the database. Combined with the .env files (spec 74) and the application config files, configuration lives in at least four places. Consolidate (spec 74, Configuration sources).


Views

26 views, in four families:

Family Examples Disposition
Strategy + timeline joins vBDBOTimelineItems, vBDBOTimelineItemsV2, vBDBOTimelineItemsV3, pl.vBDBOStrategyOrderingHistory Keep the intent; consolidate the three versions
Active-strategy filters vBDBOActiveStrategies, vBDBOActiveStrategies_Active, vBDBOActiveStrategiesCount Parameterized query
Annotation filters vAnnotation__<TF> ×7, vPovAnnotation__<INSTRUMENT> ×2 Parameterized query
Dated/temporary vBDBOStrategy_tmp__220303, vTMP__BDBBO_Observe_220303, cavSAMPLE2206101723_*, vSDSTimelineItems220511, tl.old_vStrategyTimeline Delete

vBDBOTimelineItemsV3 is the most complete and shows what a strategy history report needs: strategy identity and direction, breakout price and whether it was reached, current state, every timeline transition with old and new status, the order request (direction, entry, exit, size, triggering bar) and the order response (broker order and trade identifiers, creation time), plus decomposed date parts for grouping.

Requirement: this is a report, not a view. Its date-part decomposition (DATEPART for year, month, day, hour, minute) exists only to support grouping in a reporting tool. Implement it as a query in the reporting layer. Note also that its four-way INNER JOIN silently drops any strategy that has no order — which is most of them. Use outer joins.


Stored Procedures

42 procedures. Almost all are generated CRUD: ctpr<Entity>_{InsertOne, SelectOne, SelectAll, SelectAllByCriteria, SelectAllByCriteriaCount, SelectAllByCriteriaProjection, SelectAllCount, UpdateOne, DeleteOne} for Goals, POVAnnotation, BDBOStrategies, BDBOStrategyOrderRequests, and BDBOStrategyOrderResponses.

Requirement: do not port. Generated CRUD in the database adds a deployment artifact per entity with no behaviour the data layer cannot express. Two are hand-written and carry real logic:


Required Indexes

The original defines almost none beyond primary keys. Every query below is a scan.

Strategies              (instrument, timeframe, active, status)
Strategies              (correlationId)
StrategyTimelineItems   (strategyIdug, dt)
OrderRequests           (strategyIdug, dtRequested)
OrderResponses          (strategyIdug)
OrderResponses          (orderID)
OrderResponses          (requestIdug)
POVAnnotations          (pov, tlid)
POVAnnotations          (dtContext)
ChaosStatus             (instrument, timeframe, dt DESC)
MarketOverViewers       (instrument, timeframe, dt DESC)
CACases                 (instrument, timeframe, dtPoint)
Goals                   (parentIdug, sortOrder)

Migration Notes

  1. The backups are the real source of truth for data. db/SpiderDb.bak and db/RESTORE/*.bak contain the actual strategy history. The BACPAC at gia-mssql/exports/dac-220211/spiderdb-dac-220211.bacpac is the most portable starting point for schema+data extraction.
  2. Restore before migrating. The schema exports show structure; only a restore shows which tables carry data and which are empty experiments. Expect pl, tl, adsann, and mm to be populated and much of plx, asmpgc, order, tdso to be near-empty.
  3. Migration order: instrument properties → strategies → timeline items → order requests → order responses → trades → annotations → everything else.
  4. Idug is a GUID throughout. Preserve the values; they are referenced from exported files, hook arguments, and journal notes outside the database.
  5. Timestamps carry the same zone ambiguity as PDSDb (spec 70, Time semantics). Determine the zone empirically per table during migration by correlating strategy creation times against the price bars they reference.
  6. Rotate every credential found in security or in configuration.

Traceability

Object Source
Full schema export gia-mssql/SpiderDb__scriptExport__220907.sql (UTF-16LE, 7,253 lines)
Earlier export gia-mssql/SpiderDb__scriptExport__2208091252.sql
Genesis creation script SpiderDbDatabase/Create_SpiderDb__200424.sql
Portable package gia-mssql/exports/dac-220211/spiderdb-dac-220211.bacpac
Data backups db/SpiderDb.bak, db/RESTORE/SpiderDb.bak, db/RESTORE/spiderdb_backup_2018_09_20_180829.bak
Strategy views SpiderDbDatabase/sdsV_RecentStrategy.sql, vSTC.sql, STCQueryView__2206.sql, STCGoal__in_Strategies__220607.sql
Timeline queries gia-mssql/s__StrategyTimeLine.sql, tl__EURUSD__220318.sql, gia-mssql/tst_another_TL.sql
Active-strategy queries gia-mssql/ls_BDBOStrategy_ACTIVE.sql, gia-mssql/ls__BDBOStrategie.sql, gia-mssql/y_st__Active_WaitingBreakout.sql
Annotation queries gia-mssql/s__POVAnnotations_.sql, Pov__Select_As__xAI2206.sql
Order/trade diagnostics gia-mssql/tst__orderID_vs_TradeID.sql, gia-mssql/trade_2201312103-EURUSD.sql
Sample dataset SpiderDbDatabase/SDS.SAMPLE2206101723/
Strategy business logic src/Caishen/SDS/SDS.Services/BDBOStrategyService.cs