🪣 @backtest-kit/minio

import { setup } from '@backtest-kit/minio';
setup({
  CC_MINIO_ENDPOINT: 'minio', CC_MINIO_PORT: 9000,
  CC_MINIO_ACCESSKEY: 'minioadmin', CC_MINIO_SECRETKEY: 'secret',
  CC_REDIS_HOST: 'redis', CC_REDIS_PORT: 6379, CC_REDIS_PASSWORD: 'secret',
});
Copy

Values passed to setup() / setConfig() always take precedence over env vars. Within the CLI, put setup() in config/setup.config.ts — when present, the CLI skips its default file-adapter registration and your config owns persistence.

• Candle is immutable — first write wins; the insert is a stat + PUT pair, so a repeated write of the same (exchange, symbol, interval, timestamp) costs one HEAD and never downloads or rewrites the body (historical OHLCV never changes).
• Log / Notification entries are immutable events — the library re-sends the whole accumulated list on every write, so an in-process FIFO-capped index of persisted keys skips the PUT entirely when the key already exists: re-sending costs zero network in steady state.
• All others rewrite the object — each write is one idempotent PUT under the stable context key.
• Measure / Interval / Memory delete physically — removed: true translates to a DELETE instead of a tombstone. listKeys is then a pure prefix LIST with zero body reads, and reads of removed entries return null by construction.

backtest-kit has a write durability contract: after writeXData(...) returns, the very next readXData(...) must see the just-written value. S3 gives strong read-after-write consistency for single objects, so the contract holds with plain object semantics — no transactions needed:

1. Deterministic keys. Every record's object key is a pure function of its context (symbol/strategy/exchange/…), so an upsert is a single idempotent PUT — no read-before-write, no duplicate-key races.
2. Immutable entities never rewrite. Candles use a stat + PUT insert-only pair; log entries and notifications skip the PUT entirely when the key already exists.
3. Write order: MinIO first, Redis second. A crash between the two leaves an object readable by key but invisible to listings — never a phantom entry pointing at nothing.
4. removed means absent. Soft-delete entities (Measure, Interval, Memory) physically delete the object instead of writing a tombstone.

// src/lib/services/data/CandleDataService.ts — insert-only, one stat + one PUT
public create = async (dto: ICandleDto): Promise<ICandleRow> => {
  const key = GET_STORAGE_KEY_FN(dto.exchangeName, dto.symbol, dto.interval, dto.timestamp);
  const row: ICandleRow = { id: key, ...dto, /* dates */ };
  if (await this.has(key)) return row;   // candles are immutable — no body download
  await this.set(key, row);
  return row;
};
Copy

S3 can list keys only in lexicographic order and cannot answer "what was created last" without walking the bucket. For the three entities that need newest-first listings (Log, Notification, Storage), a *ConnectionService maintains a Redis index:

• One Redis SET per minute: <entity>-connection:<aligned-minute> → object names. register() is a single pipeline (SADD + SETNX of the floor marker). Timestamps are minute-aligned, so re-registering within a minute deduplicates by construction.
• listNewest(limit, prefix) walks backwards from the current minute — direct key lookups, no SCAN over the keyspace. Minutes are probed in pipelines of 1000; a cheap SCARD pass skips empty minutes without transferring a single member; hot minutes (a fast backtest replay packs many records into one wall-clock minute) are paged via SSCAN with early exit at limit.
• Cold-index fallback. If Redis was flushed, listings fall back to the bucket LIST (inverted-timestamp keys are already newest-first) and warm the index back up.

Steady-state cost of readLogData at startup: 1 RTT for the floor + 1–2 pipeline RTTs + ≤200 point GETs for bodies — independent of how many objects the bucket holds.

// src/lib/services/data/LogDataService.ts — read path
const names = await this.logConnectionService.listNewest(LIST_LIMIT);
if (names.length) {
  for (const name of names) rows.push(await this.get<ILogRow>(name));
} else {
  for await (const value of this.values("", LIST_LIMIT)) { /* fallback + re-warm */ }
}
Copy

Adapters whose data influences decisions (Risk, Partial, Breakeven, Recent, State, Session, Memory, Interval) store when — the simulation timestamp in ms — alongside the payload, so backtest-kit can verify no read returns data written at a future simulation time. Measure is exempt because it caches LLM / external-API responses, where look-ahead bias is not meaningful.

Public surface — functions/setup.ts (setup/install/setLogger), config/params.ts (setConfig/getConfig), index.ts re-exports + getMinio/getRedis/waitForInit/BaseStorage/BaseMap.

Adapter classes (classes/Persist*Instance.ts) — each implements one backtest-kit IPersist*Instance contract and delegates to its domain DataService: PersistCandleInstance, PersistSignalInstance, PersistScheduleInstance, PersistRiskInstance, PersistPartialInstance, PersistBreakevenInstance, PersistStorageInstance, PersistNotificationInstance, PersistLogInstance, PersistMeasureInstance, PersistIntervalInstance, PersistMemoryInstance, PersistRecentInstance, PersistStateInstance, PersistSessionInstance.

Service layer (lib/services/):

• base/ — MinioService (lazy MinIO client, ensures the bucket on first touch), RedisService (lazy ioredis), LoggerService.
• data/ — one *DataService per domain: the object-key builders and upsert/find/list logic on top of BaseStorage.
• connection/ — LogConnectionService, NotificationConnectionService, StorageConnectionService: the per-minute Redis time index (register/listNewest).

Shared primitives (lib/common/) — BaseStorage (the S3 object-store pattern every DataService reuses: bucket/parent-folder name parsing, JSON set/get/has/delete, batched clear, streaming keys/values/iterate) and BaseMap (the Redis key-mapping pattern every ConnectionService reuses).

DI & config — lib/core/{di,provide,types}.ts (IoC container wiring data/connection/base services), lib/index.ts (container bootstrap), config/{minio,redis,params}.ts (connection builders + merged params), schema/*.schema.ts (stored row shapes), interfaces/Logger.interface.ts.