An interesting paper was published recently that drew the conclusion that most traders produce poor backtest optimizations compared to suitably trained machine learning classifiers (based on out of sample performance).

I'm sure I'm not alone in developing too good to be true algorithms that turn out to be just that in production trading. I notice there is an AWS ML classifier service and I'm wondering whether it would be suitably adapted as a backtest classifier/optimizer? In general, does anyone with experience in this area have an estimate on the kind of time investment required to produce an ML classifier model suitable for backtesting? What level of re-use across an entire portfolio could be acheived? Would it require a customized model for each algorithm for instance?