Overfitting: how a thousand variants fool you without noticing

Overfitting isn't an exotic bug: it's the inevitable consequence of trying many variants and keeping the best. With enough attempts, chance hands you a spectacular curve that won't repeat.

So the “winner's” result isn't enough: you must penalize how many tries you ran. Tools like the deflated Sharpe and the probability of backtest overfitting (PBO) estimate how much of your result is search luck.

A real case from our work: a variant that passed in one regime failed out of sample before a certain period. We discarded it. That's overfitting caught in time.