Most trading ideas fail. The dangerous ones fail while their first backtest looks strong.
A good equity curve is a reason to test more. It isn't a reason to trade live.
I use this seven-gate sequence in my own research system to decide when a candidate can move from an idea to a larger test and, eventually, to capital. The first gate comes before a full backtest. A candidate must pass each gate in order.
The sequence is my synthesis, not a standard published protocol. It combines agentic research patterns, statistical controls for backtest overfitting, and my operating rules for sealed tests, shadow execution, and capital risk.
- Static validityStructure
- DiscoveryControlled search
- Rolling validationGeneralization
- Multiple-testing adjustmentSelection bias
- Sealed testFinal holdout
- Prospective shadow executionOperations
- Constrained capitalRisk
The first four gates cover research. The fifth protects the final test. The last two cover operations and risk. Coding agents can do much of the work, but they do not decide what counts as evidence or when a strategy receives capital.
1 Static validity
The first question is not whether the strategy made money. It is whether the result is structurally possible.
Before a full backtest, I check that:
- every signal uses only information available at that timestamp;
- higher-timeframe values appear only after their bars are complete;
- an order is filled at a price that could have been traded after the decision;
- commission, slippage, sizing, session rules, and contract rolls are explicit;
- the unchanged baseline reproduces on the same data path;
- the new rule matches its written specification;
- the trade log can reconstruct the reported result.
This gate exists because I've been burned without it. One strategy reported $2.23 million over two years because it decided at a bar's close and filled at that same bar's open; correcting the timing cut the result to $265,000. In another case, two pandas resampling arguments built every 15-second bar incorrectly, and the backtest ran normally on bad data for two years. I describe that failure in How a 15-Second Bar-Labeling Bug Changed Two Years of Backtests.
I also apply cheap smell tests here, before anything expensive runs. If a 15-second strategy reports thousands of trades in two weeks, it's trading noise. If a Sharpe ratio looks spectacular, the first suspect is lookahead, not genius.
A strategy passes Gate 1 only when its data, timing, implementation, and accounting checks all pass.
2 Discovery
During discovery, agents can propose variants, run short tests, inspect trades, and suggest another experiment. I set a limit on the search before it starts.
Each experiment gets an ID. I record its parent hypothesis, parameters, data window, costs, baseline, result, and verdict. I also fix the number of variants and the ranking metrics before a sweep.
A discovery result should answer a narrow question. For example: does blocking new entries after repeated trend flips reduce whipsaw without removing the large trends that pay for the strategy?
I also inspect representative trades with my backtest chart tool. The chart shows whether the rule behaved as intended. For example, it can reveal a filter that changed too often or a signal that fired repeatedly instead of once.
Gate 2 passes when the rule behaves as specified and the net result justifies a larger test. It does not yet show that the result will generalize.
3 Rolling validation
Markets change, so one train/test split can give a misleading result. My default is chronological walk-forward validation: three months of training followed by one untouched out-of-sample month, repeated through the available history.
The training window selects the parameters. The next month measures them. Within that out-of-sample month, the code and parameters do not move.
Every window includes the unchanged baseline on the identical bars, warm-up period, costs, and trade cutoff. I care about the paired comparison:
- Did the candidate improve or merely benefit from an easier month?
- How much in-sample performance survived out of sample?
- Did drawdown improve because the signal became better, or only because exposure fell?
- Were gains distributed across windows and regimes?
- Did a small parameter change reverse the conclusion?
If I change the strategy after seeing an out-of-sample month, that month has joined the research history. It can't remain evidence for the revised strategy. The new version must earn evidence from later windows.
This has been the most brutal gate in my own research, and the most valuable. One entry pattern won 78% of its trades in the training sample and 29% out of sample, and the filter with the best in-sample Sharpe I ever tested produced the worst out-of-sample result of any variant. In my data so far, in-sample beauty and out-of-sample survival have been close to unrelated.
For candidates that pass, I also reproduce the strategy in another engine or data path, such as Python, LEAN, QuantConnect, or Pine Script. Cross-engine agreement checks portability and execution semantics. It does not establish profitability on its own.
4 Multiple-testing adjustment
An agent can test many ideas in an afternoon, and as the trial count grows, the best result improves by chance alone. That is where false positives come from, so this gate evaluates the full search process rather than the winning candidate.
The experiment registry provides the trial count and keeps related variants visible. I check parameter sensitivity, non-normal returns, and correlation with earlier candidates. Deflated Sharpe Ratio and Probability of Backtest Overfitting are useful diagnostics. Purging, embargo, and combinatorial validation can help when observations overlap.
The total number of attempts must always travel with the best result. Reporting a great Sharpe without its trial count is like a casino advertising only its jackpot winners.
One strategy family went through about 250 configurations without a robust full-history result. Its best month looked strong. The full record showed the formulation wasn't worth more research time, and that conclusion exists only because the failures were recorded.
Gate 4 passes only when the candidate remains interesting after accounting for the process that selected it.
5 The sealed test
Walk-forward results influence later research because I see them and use them to form new hypotheses. A final holdout must remain separate.
Before a sealed test, I freeze:
- source code and dependencies;
- parameters and signal definitions;
- sizing and portfolio rules;
- transaction costs and fill assumptions;
- data and contract-selection rules;
- the exact pass/fail criteria.
An independent evaluator runs the frozen candidate once on data that was unavailable during research. The agent does not receive the result during the research process.
If the sealed result is disappointing, I can revise the strategy, but I can't retest the revision on the same holdout and keep calling it sealed. That dataset has become another validation set.
This is especially important with language models because their training data may contain historical market information. A holdout helps. New data collected after the research is stronger evidence because neither the researcher nor the model could have seen it.
6 Prospective shadow execution
The strategy now runs on incoming data without permission to place a live order.
Shadow execution tests questions a historical backtest cannot answer:
- When did the data actually arrive?
- How long did the computation take?
- Was the expected price obtainable?
- What spread and slippage appeared around the signal?
- Did contract rolls, missing bars, or late data change the decision?
- Did the process recover cleanly after an outage or restart?
- Would monitoring have caught a stale or duplicated signal?
The research implementation and the shadow system should make the same decisions. Any difference must be traced to an operational issue or a modeling assumption.
A backtest replays logic on historical data. Shadow execution tests the live data feed, computation time, stored state, and monitoring without placing orders.
7 Constrained capital
Only after prospective evidence does a strategy become eligible for capital, and eligibility does not mean full size.
The initial allocation should be deliberately small, with controls outside the strategy:
- hard position and loss limits;
- stale-data and impossible-price rejection;
- independent exposure monitoring;
- alerts for missed, duplicated, or delayed orders;
- a tested rollback path;
- an immediate kill switch;
- clear human authority to stop the system.
The research agent may propose an adjustment. It cannot change risk limits, increase its allocation, or override a shutdown. A separate risk process controls capital.
Scaling should follow prospective evidence rather than conviction. If live behavior diverges from the shadow model, or the assumptions behind the strategy stop holding, the correct action may be to reduce exposure or return to research.
Most ideas should stop early
The gates aren't a pipeline for moving every idea toward deployment. They're a filter, and a filter that passes everything is broken. Most candidates should fail early, and a clear negative result saves time and caps risk.
In my current project, many ideas have passed the code and behavior checks. Some have reached walk-forward validation or independent-engine testing. None should move further until sealed and prospective evidence supports it. Historical backtests are research results, not live performance.
AI makes implementation faster. The seven gates keep validation, reproducibility, and risk decisions separate from that speed.
Related work
These sources informed specific parts of the framework:
- RD-Agent(Q) (2025) describes an iterative multi-agent research and development loop for quantitative finance.
- AlphaAgent (2025) uses originality, hypothesis-alignment, and complexity controls during alpha generation.
- The Probability of Backtest Overfitting (2015) and The Deflated Sharpe Ratio (2014) address selection bias from repeated backtests.
- The Memorization Problem (2025) explains why historical evaluations involving language models need stronger isolation and prospective evidence.
This describes an independent research and risk-control framework. It is not investment advice or a claim of live trading performance.