TSB — Backtest an AI/LLM Trading Strategy | Free, Open Source
$0.00
Description
Free · Open source (MIT) · Python CLI · Bring your own LLM
Your strategy is a paragraph, not a formula. Backtest it anyway.
DTSB backtests discretionary strategies — the kind that are judgment written in English rather than fixed rules — by asking the model bar by bar what it would have done, then scoring the trades against real price. No-lookahead point-in-time walking, anti-contamination controls, deterministic outcome scoring. Not an MCP server, not a live-trading bot: it generates and scores historical signals only.
What you actually write
“Buy pullbacks in an uptrend.”
a persona, in plain English“Fade RSI extremes in a range.”
needs: [rsi14, adx14]“Don't even convene unless ADX is above 25.”
triggers: adx_min“One persona, always convenes — keep it simple.”
strategies/solo-swing“Three personas, each gated by its own triggers, weighted consensus at 0.6.”
strategies/mini-council“Only take it when conviction clears my floor.”
min_convictionA strategy is a directory: one
strategy.yaml and one markdown file per persona. The persona's front matter
names the indicators it needs and the scout triggers that decide when it convenes; the body
is your method, written the way you'd explain it to a trader. That body becomes the prompt
the model reasons from — there is no Python to write, and no equation to reduce your
read to. Two working examples ship with the repo.
Why this is more than looping an LLM over some bars
| Failure mode | What goes wrong | DTSB's control |
|---|---|---|
| Training-data contamination | The model may have seen the test period during training and “predicts” the future by remembering it | Packets are anonymized (symbol → alias, prices scaled, dates shifted) before the model ever sees them, plus an explicit contamination probe |
| Lookahead | Using bar data that wouldn't have existed yet at decision time | A point-in-time walker builds each packet from only bars closed by that instant — unit-tested |
| Self-grading | The LLM decides both the trade AND whether it won | The LLM only generates ballots; a deterministic walker scores outcomes against real bars |
| Non-determinism | The same prompt can return a different verdict on repeat | A variance probe measures the flip rate and reports it as a noise band |
| Overfitting / small samples | Months of LLM-scale data ≠ statistical power | Chronological train/test split; bootstrap confidence intervals on the test split only |
These are the ways an LLM backtest goes wrong that a mechanical one never fails on — a Strategy Tester can't be contaminated by its own training data, because it doesn't have any. Full rationale in docs/LLM-VS-MECHANICAL.md; architecture in docs/DESIGN.md.
A run, end to end
- Sanity check —
driver --config dtsb.yaml --check-depsconfirms the data provider and the model endpoint are both actually reachable before you spend anything. - A small first run —
driver --config dtsb.yaml --limit 20walks 20 decision points, so you find out your prompt returns valid ballots on twenty calls, not two thousand. - Export the signals —
bridge <run_id>writes a signal CSV, if you want the raw trades elsewhere. Optional. - Score it —
score <run_id>computes expectancy, profit factor and bootstrap confidence intervals. - Read the verdict —
report <run_id>writesdata/runs/<run_id>/report.md.
Every report carries an honesty certificate
Alongside the expectancy and profit-factor numbers, report.md states how much
the run itself can be trusted: scout recall (what the triggers filtered
out), gate miss rate, ballot validity, and — once
you run the probes — vote variance and
contamination.
Data in, model in
Price data providers
| Provider | What it needs |
|---|---|
csv | Your own OHLCV files — header time,open,high,low,close,volume. No key, no account, no network. |
tradingview | A TVR_API_KEY in .env, via tvremix.xyz |
mt5 | A MetaTrader 5 terminal — pip install MetaTrader5, Windows only |
LLM engines
| Engine | What it talks to |
|---|---|
openai_compat | Any OpenAI-compatible endpoint — Ollama, LM Studio, vLLM, OpenAI itself, a paid model. Your key, your bill, your choice. |
openrouter | A pool of free OpenRouter models, health-checked automatically |
The openrouter engine
probes candidate models with a real ballot-shaped prompt rather than a trivial ping —
some models answer a hello fine and still fail to follow a “respond with only
JSON” instruction. It remembers which ones came back healthy, and learns your actual
daily free-request cap from OpenRouter's own 429 response instead of assuming a number.
Setting it up
Quickstart A — zero API keys
Your own CSV files, a model running locally on your own machine. Nothing to sign up for, nothing to pay, no rate limit but your GPU:
pip install -r requirements.txt →
cp .env.example .env (nothing to fill in for this path) →
cp config.example.yaml dtsb.yaml
Drop your OHLCV data in ./data/csv/, point the engine at your local
Ollama base_url, and run the driver. That's the whole setup.
Quickstart B — TradingView data + a free hosted model: set
provider: tradingview with a TVR_API_KEY, and
kind: openrouter with an OPENROUTER_API_KEY, both in
.env. Still no paid model required.
Requirements
- Python, and one dependency —
pyyaml - Price data: your own CSVs, a tvremix key, or MetaTrader 5
- An LLM endpoint — local, free-tier or paid
MetaTrader5only if you use themt5provider (Windows)
Source & support
MIT licensed, fully open on GitHub — nothing compiled, nothing to trust blindly. The test suite runs without network access, with all HTTP mocked. DTSB is signals-only: it never places a trade and never touches a live account.
View source, issues & updates on GitHubHonest warnings
- Results are pinned to one model version — a model upgrade invalidates the estimate until you re-run.
- Confidence intervals from a few months of history are wide. If the CI spans zero R, the report says so plainly rather than overclaiming an edge.
- Anonymization reduces contamination risk. It does not prove zero.
- A backtest — mechanical or LLM — is still a backtest. Regime change and execution reality aren't in the numbers.
If your strategy is mechanical instead
MBT — MT5 Backtest Toolkit
Free · Open sourceDTSB exists for strategies with no equation to test — judgment, written out. If yours is already an indicator or an EA, it has one, and MBT is the better tool: it backtests through Claude by replaying the indicator's own logged decisions against real broker price data. Same principle, opposite kind of strategy.
Get MBT
Reviews
There are no reviews yet.