Reply to thread

One trap about out-of-sample testing that's easy to fall into is repeatedly using the same data sets. You test your system on in-sample data, looks great. You test it on out-of-sample data, it falls apart. What do you do? You go back to the system and make some adjustments. Test it again on the same data and it looks much better this time. But really, you've just made your out-of-sample data part of the in-sample data set. You can't keep making adjustments until the same out-of-sample test comes good.