RISE Framework Specification
Spec ID: 76
Version: 1.0
Document ID: caishen-rise-spiderdb-v1.0
Last Updated: 2026-08-01
Source of truth: src/Caishen/Databases/SpiderDbDatabase/gia-mssql/SpiderDb__scriptExport__220907.sql (UTF-16LE)
Related: 02 (SDS), 10 (data schemas), 21 (ordering campaign), 40 (fractal breakout workflow)
What SpiderDb Enables:
Desired Outcomes:
SpiderDb and PDSDb are separate databases in the same server instance. There are no cross-database foreign keys; the join is by convention, in application code.
| PDSDb (spec 71) | SpiderDb | |
|---|---|---|
| Content | Price bars, indicator panels | Strategies, orders, timelines, annotations |
| Write pattern | High-volume, machine-generated, incremental | Low-volume, human- and event-driven |
| Row lifetime | Immutable once closed | Mutable through a lifecycle |
| Identity | Content-derived key (povTlid) |
Assigned unique identifier (Idug) |
| Loss tolerance | Recoverable — re-fetch from provider | Not recoverable — it is the record of intent and action |
Keep them separate. The retention, backup, and audit requirements are genuinely different. SpiderDb is small and irreplaceable; PDSDb is large and reconstructible.
Shared vocabulary: both use the povTlid / tlid bar-identity scheme from
spec 70. That is the cross-database join key, and it is why annotations and
strategies can point at exact bars without storing timestamps that might drift.
66 tables across 17 namespaces. The namespaces are meaningful and worth preserving.
| Namespace | Domain | Status |
|---|---|---|
pl |
Strategy lifecycle — strategies, orders, trades, journals | Core |
tl |
Timelines — generic strategy event history | Core |
adsann |
Chart annotations | Core |
chart |
Chart bars, POV metadata, instrument properties | Superseded by PDSDb |
chart2 |
Second-generation chart data | Superseded |
neptune |
Instrument/POV analyses, observations, price alerts | Active |
mm |
Market monitoring — divergence scanning, market overview | Active |
gt |
Goal tracking — structural-tension goal hierarchy | Active, separate concern |
c |
Campaigns | Emerging |
ca |
Case analyses | Active |
asmpgc |
Pattern grammar — patterns, steps, terms | Experimental |
order |
Generic strategy ordering | Superseded by pl |
tdso |
Trading-data-service ordering | Superseded by pl |
plx |
Experimental exit-strategy modelling | Experimental |
conf |
Configuration data | Support |
security |
Identity — users, roles, claims, logins, profiles | Support |
dbo |
Views, procedures, sequence tables | Mixed |
Requirement: a re-implementation should carry forward pl, tl, adsann,
neptune, mm, gt, ca and c. The chart/chart2, order, and tdso
namespaces are superseded generations kept alongside their replacements — the
same concept implemented three times. Migrate any surviving data and drop them.
pl)pl.BDBOStrategies — the central entityA breakout entry strategy: an intention to enter the market when price crosses a level. Fully specified behaviourally in spec 02 and spec 40; this is its persisted shape.
TABLE Strategies
idug uuid NOT NULL -- PRIMARY KEY
instrument varchar(8) NOT NULL
timeframe varchar(8) NOT NULL
buySell varchar NOT NULL -- direction: B | S
breakoutPrice float NOT NULL -- the trigger level
breakoutPriceReached bit NOT NULL -- has it triggered?
active bit NOT NULL
isNew bit NOT NULL
-- lifecycle
status int NOT NULL -- numeric state
stateId int NOT NULL -- state machine state
stateName varchar(128) NOT NULL -- state machine state, named
dt datetime NOT NULL -- created
dtModified datetime NOT NULL
dtFirstAction datetime NOT NULL
-- sizing and risk
k int NOT NULL -- position size (lots)
dp float NOT NULL -- delta/pip parameter
mpr float NOT NULL -- money/pip ratio
vcp int NOT NULL
-- counters and limits
cft int NOT NULL -- count fractal tries
ctc int NOT NULL -- count trade cycles
ceoc int NOT NULL -- count exit-order cycles
maxTC int NOT NULL
maxEOC int NOT NULL
maxPUI int NOT NULL
-- signal validity
diValid int NOT NULL -- divergence-indicator validity
-- trailing
tt bit NOT NULL -- trailing enabled
tto int NOT NULL -- trailing offset
-- exit
exitStrategyType int NOT NULL
exitStrategyTargetPrice float NOT NULL
-- provenance
correlationId uuid NOT NULL
note text NOT NULL
meta varchar(4096) NOT NULL -- JSON
Observations and requirements:
The abbreviated column names (K, DP, MPR, CFT, CTC, CEOC, VCP,
TT, TTO, DiValid) are a real obstacle. Their meanings live only in the
business code. Expand them, and document each in the schema. The expansions
above are reconstructed from usage and should be confirmed against
src/Caishen/SDS/SDS.Services/BDBOStrategyService.cs during implementation.
Both status and stateId/stateName exist, and they can disagree. The
numeric status predates the state machine; stateId/stateName came with it.
Views join on all three. Keep one: the state machine’s state, stored as a
string, with the numeric form derived if needed for ordering.
correlationId links a strategy to the analysis session that produced it —
this is the identifier that spec 73 asks to be propagated through the event
chain. It exists here and nowhere else; make it universal.
meta is unstructured JSON in a varchar(4096). Use a real document column
with a declared schema, or promote its contents to columns.
pl.StrategyTimelineItems — the event historyEvery state transition, appended, never modified.
TABLE StrategyTimelineItems
idug uuid NOT NULL -- PRIMARY KEY
strategyIdug uuid NOT NULL -- FK → Strategies
dt datetime NOT NULL
statusPrevious int NOT NULL
statusNew int NOT NULL
eventName varchar(255) NOT NULL -- what caused the transition
itemType varchar(128) NOT NULL
message varchar(255) NOT NULL
note varchar(4096) NOT NULL
meta varchar(8000) NULL -- JSON
This is the most valuable table in the database. It is a genuine event log: append-only, ordered, carrying both the transition and its cause. It makes post-hoc analysis of “why did this strategy do that” possible.
Requirements:
(strategyIdug, dt) — every read is “this strategy’s history in order”.statusPrevious/statusNew as the state machine’s named states, in step
with the change to Strategies above.pl.BDBOStrategyOrderRequests / pl.BDBOStrategyOrderResponsesThe request/response pair for order placement — a durable record of what was sent and what came back.
TABLE OrderRequests
idug uuid NOT NULL -- PRIMARY KEY
strategyIdug uuid NOT NULL -- FK → Strategies
dtRequested datetime NOT NULL
buySell varchar(1) NOT NULL
entryPrice float NOT NULL
exitPrice float NOT NULL
limitPrice float NOT NULL
k int NOT NULL -- size
barJSONData varchar(2048) NULL -- the bar that triggered it
meta varchar(4096) NOT NULL
TABLE OrderResponses
idug uuid NOT NULL -- PRIMARY KEY
requestIdug uuid NOT NULL -- FK → OrderRequests
strategyIdug uuid NOT NULL -- FK → Strategies
dtOrderCreated datetime NOT NULL
orderID varchar(32) NOT NULL -- broker's order identifier
tradeId varchar(64) NOT NULL -- broker's trade identifier
meta varchar(4096) NOT NULL
The request/response split is the right pattern and must be preserved. A request exists the moment the intention is formed; a response exists only if the broker replied. A request with no response is a detectable, actionable state — an order that may or may not have reached the market. Collapsing these into one row loses that.
Requirements:
barJSONData — the triggering bar captured inline. Replace with a povTlid
reference (spec 70) plus, if a snapshot is genuinely needed for audit, a
structured document column.(strategyIdug, dtRequested) and (orderID).pending | acknowledged | rejected | timedOut)
so an unanswered request is queryable rather than inferred from a missing join.pl.BDBOCancelOrderRequests / pl.BDBOCancelOrderResponsesSame pattern for cancellation. Same requirements.
pl.BDBOTradesExecuted trades, linked to their strategy. The realized outcome.
pl.TraderJournalsFree-form notes attached to a strategy, positioned on the chart:
TABLE TraderJournals
idug uuid NOT NULL -- PRIMARY KEY
strategyIdug uuid NOT NULL -- FK → Strategies
dt datetime NOT NULL
summary varchar(128) NOT NULL
note text NOT NULL
xTlidPosition varchar(32) NOT NULL -- horizontal anchor: a bar identity
yPricePosition float NOT NULL -- vertical anchor: a price
xTlidPosition is a bar identity, not a pixel coordinate — so a journal note
stays anchored to the bar it concerns regardless of zoom, pan, or timeframe. This
is the right way to anchor chart annotations and it should be adopted everywhere.
tl)A second, more general timeline implementation:
TABLE StrategyTimelines
idug uuid NOT NULL -- PRIMARY KEY
strategyIdug uuid NOT NULL
pov varchar(64) NOT NULL
statusCurrent int NOT NULL
dtCreated datetime NOT NULL
dtModified datetime NOT NULL
dtStrategyCreated datetime NULL
note varchar(4096) NOT NULL
initialJSONMetaData text NULL
TABLE TimelineItems
idug uuid NOT NULL -- PRIMARY KEY
strategyTimelineIdug uuid NOT NULL -- FK → StrategyTimelines
dt datetime NOT NULL
statusPrevious int NOT NULL
statusNew int NOT NULL
eventName varchar(255) NOT NULL
itemType varchar(128) NOT NULL
message varchar(255) NOT NULL
note varchar(4096) NOT NULL
jsonMetaData varchar(8000) NULL
jsonChartMetaData varchar(8000) NULL
This duplicates pl.StrategyTimelineItems with an added grouping level and a
separate chart-metadata field. Two timeline implementations coexist and several
views query each.
Requirement: unify. The tl shape is the better one — a timeline is a
first-class object with its own identity and a POV — but pl’s items are the
ones actually written by the strategy service. Pick the tl structure, migrate
pl data into it, and drop the duplicate.
adsann, neptune)TABLE POVAnnotations
idug uuid NOT NULL -- PRIMARY KEY
pov nvarchar(16) NULL -- e.g. "EUR/USD_H4"
tlid nvarchar(16) NULL -- the bar being annotated
tag nvarchar(128) NULL
annotationText nvarchar(1024) NULL
fileTagName nvarchar(255) NULL -- associated file/screenshot
dtContext datetime NOT NULL -- the moment being annotated
dtCreated datetime NOT NULL
dtModified datetime NOT NULL
archived bit NOT NULL
Again the (pov, tlid) pair anchors the annotation to an exact bar.
Two copies exist — adsann.POVAnnotation and neptune.POVAnnotation — with
slightly different nullability and a wider fileTagName in the neptune copy.
neptune has generated CRUD procedures; adsann has seven per-timeframe views
over it. Consolidate to one table.
Requirement: the seven vAnnotation__<TF> views plus two per-instrument
variants are the same anti-pattern as spec 71’s per-instrument views. Replace
with a parameterized query filtering on the POV’s timeframe component — and store
instrument and timeframe as separate columns rather than parsing them out of a
concatenated pov string with LIKE '%H4'.
mm)Scans the market for divergence conditions across instruments and timeframes.
| Table | Purpose |
|---|---|
ChaosStatus |
Per-bar market condition: divergence indicator, bullish divergent bar, divergence A/B, trend of last wave, strength ordering |
MarketOverViewers |
Historical snapshots of the same, plus isDiverging and mouth-openness validation |
MarketOverViewersCurrents |
Current snapshot only |
VDivergenceIndicatorCurrents |
Current divergence indicator per series |
InstrumentPerspectives |
Per-instrument divergence sequence across timeframes, encoded as a compact string |
VInstrumentDivergenceIndicatorSequenceLasts |
Latest such sequence per instrument |
TradingPerspectives |
Scan summary: counts of divergent bars and divergences at a moment |
ChaosMessages |
Free-form messages keyed by bar identity |
GetAOPeakDivRequests |
Oscillator peak-divergence query parameters |
The core idea worth preserving: InstrumentPerspectives.DivergenceIndicatorSequenceData
encodes one instrument’s divergence state across all timeframes as a single
short string. That makes “show me every instrument whose weekly, daily and
four-hourly all agree” a string match rather than a multi-way join. It is a
deliberate and effective denormalization.
Requirements:
Currents tables are materialized “latest row per series” projections.
Keep them (the query is hot) but derive them, don’t maintain them by hand.varchar(8) for instrument here versus varchar(16) in PDSDb — unify.StrenghtOrder [sic] — fix the spelling.gt)A structural-tension goal hierarchy — Robert Fritz’s methodology, the same framework behind the TandT application. It shares the database with trading but is a separate bounded context.
| Table | Purpose |
|---|---|
Goals |
The goal itself: text, acronym tag, due date, completion, state, telescope level |
CurrentRealities |
The current-reality side of the structural tension |
GoalImages |
Visual representation of the desired outcome (binary + text) |
MomentOfTruths |
A four-step review: difference from expectation → how it got that way → plan for next time → feedback system for the new plan |
ConceptNotes |
Notes attached to any goal entity |
Resources |
URIs and metadata attached to any goal entity |
Tags |
Tags attached to any goal entity |
Goals.IsATelescope / TelescopeLevel implement goal nesting — a goal that
“telescopes” contains sub-goals at a finer level. This is the structural-tension
hierarchy, and it is the same pattern the pattern-grammar and segment tables use.
MomentOfTruths is the review discipline made structural: its four columns
are the four steps of the method, so a review cannot be recorded incompletely.
This is a good example of encoding a practice in a schema.
Requirement: extract gt into its own database or service. It is coupled to
trading only by convenience of hosting. All seven tables share a common base
shape (idug, dtCreated, dtModified, sortOrder, note, textContext,
correlationId, itemHistoryIdug, parentIdug) — that base belongs in a shared
entity definition, and itemHistoryIdug suggests a versioning scheme that was
planned but is not visible in the schema. Clarify or remove it.
ca, c)TABLE CACases
idug uuid NOT NULL -- PRIMARY KEY
instrument varchar NOT NULL
timeframe varchar NOT NULL
dtPoint datetime NOT NULL -- the moment being studied
strategyIdug uuid NULL -- optional link to a strategy
note varchar(4000) NOT NULL
dtCreated datetime NOT NULL
meta varchar(8000) NULL
TABLE CAItems -- observations within a case
TABLE Campaigns
idug uuid NOT NULL -- PRIMARY KEY
title varchar(255) NOT NULL
sortOrder int NULL
correlationIdug uuid NULL
financialActionStepIdug uuid NOT NULL -- links to gt.Goals
targetBudget float NOT NULL
targetBudgetMandatory float NOT NULL
targetBudgetOptional float NOT NULL
dtCreated, dtModified datetime NOT NULL
Campaigns is the bridge between the two halves of the system: a campaign
links a financial goal (gt) to the trading activity meant to fund it, with a
budget split into mandatory and optional targets. Even though gt should be
extracted, this relationship is the reason the two ever shared a database. Model
it as an explicit cross-context reference.
Note CACases.instrument/timeframe are varchar(8000). Size them properly.
asmpgc)An experiment in describing market behaviour as an ordered sequence of defined terms.
TABLE Patterns { idug, patternType, patternTitle, patternNote }
TABLE PatternSteps { idug, patternIdug, stepTermIdug, order, stepNote }
TABLE StepTerms { idug, termText, termDefinition }
A pattern is an ordered sequence of steps; each step references a term drawn from a controlled vocabulary that carries its own definition.
Worth preserving as a concept. A vocabulary whose terms are defined in the data means the language of analysis is extensible without a schema change, and every use of a term is traceable to one definition. Only three tables and a handful of rows, but the idea is the most reusable one in the database.
chart, chart2 — replaced by PDSDbchart.Bars holds price data and computed indicators in one row (alligator
lines, oscillators, fractals, distances, bar classification). chart.PriceBars
holds prices only. chart.InstrumentProperties, chart.POVData, and
chart.POVPrices are three near-identical copies of instrument metadata —
POVData and POVPrices differ only in name.
The evolution is legible: prices and indicators together (chart.Bars) → prices
alone (chart.PriceBars) → a second attempt (chart2) → separation into PDSDb’s
PDSPPrices + PanoIndicators.
The separation was the right conclusion. Prices are provider facts; indicators are computed and change when the computation changes. Do not re-merge them.
chart.Bars does carry fields absent from PDSDb’s panel and worth keeping:
Median, BarHeight, IsBullish/IsBearish, GatorPricePosition,
GatorMouthMaxOpenness, PriceRelativeDistancePercent.
order, tdso — replaced by plTwo earlier generations of the order request/response pattern, structurally
similar to pl’s. Migrate and drop.
plx — experimental exitsEntryStrategicTradeSets, StrategicTrades, StrategicExitModels, and a
placeholder Class1s (an unrenamed template class that reached the database).
An unfinished attempt at modelling multi-leg entries with per-leg trailing exits.
The idea is real and appears in spec 02’s exit strategies. Take it from there,
not from these tables. Drop Class1s.
securityIdentity tables — users, roles, claims, logins, profiles — in the shape of a standard framework’s identity schema.
Requirement: do not re-implement. Use the target platform’s identity system. If any of these rows survive, treat every stored credential as compromised.
confDTSConfigDatas — application configuration in the database. Combined with the
.env files (spec 74) and the application config files, configuration lives in
at least four places. Consolidate (spec 74, Configuration sources).
26 views, in four families:
| Family | Examples | Disposition |
|---|---|---|
| Strategy + timeline joins | vBDBOTimelineItems, vBDBOTimelineItemsV2, vBDBOTimelineItemsV3, pl.vBDBOStrategyOrderingHistory |
Keep the intent; consolidate the three versions |
| Active-strategy filters | vBDBOActiveStrategies, vBDBOActiveStrategies_Active, vBDBOActiveStrategiesCount |
Parameterized query |
| Annotation filters | vAnnotation__<TF> ×7, vPovAnnotation__<INSTRUMENT> ×2 |
Parameterized query |
| Dated/temporary | vBDBOStrategy_tmp__220303, vTMP__BDBBO_Observe_220303, cavSAMPLE2206101723_*, vSDSTimelineItems220511, tl.old_vStrategyTimeline |
Delete |
vBDBOTimelineItemsV3 is the most complete and shows what a strategy history
report needs: strategy identity and direction, breakout price and whether it was
reached, current state, every timeline transition with old and new status, the
order request (direction, entry, exit, size, triggering bar) and the order
response (broker order and trade identifiers, creation time), plus decomposed
date parts for grouping.
Requirement: this is a report, not a view. Its date-part decomposition
(DATEPART for year, month, day, hour, minute) exists only to support grouping in
a reporting tool. Implement it as a query in the reporting layer. Note also that
its four-way INNER JOIN silently drops any strategy that has no order — which
is most of them. Use outer joins.
42 procedures. Almost all are generated CRUD:
ctpr<Entity>_{InsertOne, SelectOne, SelectAll, SelectAllByCriteria,
SelectAllByCriteriaCount, SelectAllByCriteriaProjection, SelectAllCount,
UpdateOne, DeleteOne} for Goals, POVAnnotation, BDBOStrategies,
BDBOStrategyOrderRequests, and BDBOStrategyOrderResponses.
Requirement: do not port. Generated CRUD in the database adds a deployment artifact per entity with no behaviour the data layer cannot express. Two are hand-written and carry real logic:
pl.CountActiveByPov — count active strategies for a series. Genuinely useful
(it enforces a per-series concurrency limit); implement in the service layer.dbo.sp_ls_py_pack — lists installed Python packages. Diagnostic; drop.The original defines almost none beyond primary keys. Every query below is a scan.
Strategies (instrument, timeframe, active, status)
Strategies (correlationId)
StrategyTimelineItems (strategyIdug, dt)
OrderRequests (strategyIdug, dtRequested)
OrderResponses (strategyIdug)
OrderResponses (orderID)
OrderResponses (requestIdug)
POVAnnotations (pov, tlid)
POVAnnotations (dtContext)
ChaosStatus (instrument, timeframe, dt DESC)
MarketOverViewers (instrument, timeframe, dt DESC)
CACases (instrument, timeframe, dtPoint)
Goals (parentIdug, sortOrder)
db/SpiderDb.bak and
db/RESTORE/*.bak contain the actual strategy history. The BACPAC at
gia-mssql/exports/dac-220211/spiderdb-dac-220211.bacpac is the most portable
starting point for schema+data extraction.pl, tl, adsann, and mm to be populated and much of plx, asmpgc,
order, tdso to be near-empty.Idug is a GUID throughout. Preserve the values; they are referenced from
exported files, hook arguments, and journal notes outside the database.security or in configuration.| Object | Source |
|---|---|
| Full schema export | gia-mssql/SpiderDb__scriptExport__220907.sql (UTF-16LE, 7,253 lines) |
| Earlier export | gia-mssql/SpiderDb__scriptExport__2208091252.sql |
| Genesis creation script | SpiderDbDatabase/Create_SpiderDb__200424.sql |
| Portable package | gia-mssql/exports/dac-220211/spiderdb-dac-220211.bacpac |
| Data backups | db/SpiderDb.bak, db/RESTORE/SpiderDb.bak, db/RESTORE/spiderdb_backup_2018_09_20_180829.bak |
| Strategy views | SpiderDbDatabase/sdsV_RecentStrategy.sql, vSTC.sql, STCQueryView__2206.sql, STCGoal__in_Strategies__220607.sql |
| Timeline queries | gia-mssql/s__StrategyTimeLine.sql, tl__EURUSD__220318.sql, gia-mssql/tst_another_TL.sql |
| Active-strategy queries | gia-mssql/ls_BDBOStrategy_ACTIVE.sql, gia-mssql/ls__BDBOStrategie.sql, gia-mssql/y_st__Active_WaitingBreakout.sql |
| Annotation queries | gia-mssql/s__POVAnnotations_.sql, Pov__Select_As__xAI2206.sql |
| Order/trade diagnostics | gia-mssql/tst__orderID_vs_TradeID.sql, gia-mssql/trade_2201312103-EURUSD.sql |
| Sample dataset | SpiderDbDatabase/SDS.SAMPLE2206101723/ |
| Strategy business logic | src/Caishen/SDS/SDS.Services/BDBOStrategyService.cs |