My research runs on raw NQ futures trades and L1/L2 order-book updates, about 38 GB compressed. I had spent real effort making that pipeline fast, and much less validating what it produced.
Part 1: A fast cache, assumed correct
The pipeline is straightforward: unpack the raw archives once, build OHLCV bars at several resolutions, and store each derived dataset in Parquet. Backtests read the compact bar files, never the tick archives. Every cache can be rebuilt from immutable raw data.
Rebuildability matters because a fast cache can still be wrong. I describe the pipeline and its performance in the data-engineering post. This post focuses on a correctness bug in that cache.
Part 2: The $31K discrepancy
I validate everything against a second, independent source: TradingView exports of the same instrument. The same market should produce the same bars.
The first comparison looked reasonable. The median close difference was about 2 points on a 20,000-point index, and bar-to-bar return correlation was 0.36. I told myself the feeds just differed slightly. That explanation was convenient, and I never tested it.
Then the same strategy, on the same 10-day window, made +$28.9K on TradingView data and lost $2.3K on mine. Feeds differ. They don't differ by $31K.
Lag correlation found the shift
I compared returns at several time offsets:
a = mt_bars["close"].diff() # my tick-derived 15s bars
b = tv_bars["close"].diff() # TradingView 15s bars, same timestamps
for lag in (-2, -1, 0, 1, 2):
print(lag, a.corr(b.shift(lag)))Output: correlation at lag 0 was 0.36. At lag +1 it was 0.60.
A jump from 0.36 to 0.60 at lag +1 meant my bars were shifted by exactly one bar: the bar labeled 10:00:15 held the trades TradingView put at 10:00:00.
The cause was a single resample argument. My 15-second bars had been built from 5-second bars with pandas' label='right', meaning each bar was stamped with its close time. The entire downstream pipeline (and TradingView) assumes bars are stamped with their open time. Every 15-second bar, and every higher-timeframe bar built on top of them, was mis-aligned by one bar.
I shifted the labels back 15 seconds and reran the comparison. The result changed from -$2.3K to +$28.7K, close to TradingView's +$28.9K. A smaller mismatch remained.
A second resampling error
After the label fix, the feeds still differed by about 1.5 points at the median. I rebuilt the bars from the 5-second source with each binning convention and compared every result with TradingView:
bars = ticks_5s.resample("15s", label="left", closed="left").agg(
{"open": "first", "high": "max", "low": "min", "close": "last", "volume": "sum"}
)With label='left', closed='left', the result against TradingView over a 44,000-bar window was:
median |close difference| = 0.00 points. 95th percentile = 0.00 points. Return correlation = 0.999.
The rebuilt bars matched the reference. The original cache had a second bug: closed='right' assigned each boundary 5-second bar to the previous 15-second interval. Changing the label fixed the timestamps but not the interval contents.
Two arguments were wrong. label= controls the timestamp, while closed= controls which observations belong to the interval. Both must match the downstream convention.
Part 3: What two years of results were worth
After rebuilding the cache correctly, I re-ran the full two-year history for several strategy configurations:
- A high-frequency trend config (thousands of trades on 15s bars): its 25-month P/L moved by ~$80K with the fix. Another moved ~$175K. Both got worse on correct data. Some of what looked like edge was bar-construction artifact.
- A low-frequency structural config (~1.3 trades/day, entries at range-breakout levels): its P/L changed by only $1.2K, and its drawdown improved.
The market, engine, and costs were unchanged. Only bar construction changed. I now read the size of the move as a stress test: if a strategy's result depends on which side of a bar boundary a timestamp lands, its edge may be below the noise floor. The structural strategy barely noticed the fix, while the high-frequency variants moved by two orders of magnitude more.
The audit also answered two other questions:
- Contract rolls had little effect. Of about 16,000 trades, only 57 crossed a roll boundary. Their combined P/L was about $2K.
- General data quality was stable. Missing bars, flat bars, and price spikes did not increase in the weaker periods. The performance difference was more likely related to market conditions.
The checklist I now run on every bar cache
- Check lag correlation against an independent source. Run
corr(diff(a), diff(b).shift(k))for k ∈ [-2, 2]. The peak should occur at k=0. - Compare exact values after alignment. Measure the median and 95th percentile of the absolute close difference.
- Define the label convention once. My loaders assert
label='left', which stamps bars by open time. - Rebuild caches from raw data. Patching the old timestamps did not fix the interval contents.
- Rerun the main backtests after a data fix. Large changes identify strategies that depend on small construction details.
The two bugs cost days of rework and invalidated conclusions I had already written down. The diagnostic that finally caught them was a handful of lines of code. I should have run it two years earlier.