This looks ok and has a high accuracy, but falls apart on walk forward. DAX/daily/5 contracts/$5comm per contract. Not sure what to do - any suggestions appreciated.
View attachment 69419
Greetings --
The short answer is that the model overfit the in-sample data during learning and does not fit the out-of-sample data.
Some explanation might be helpful.
The system development process is classical learning using the scientific method. I do not understand why, but the trading community has been super slow to recognize that systems to predict stock direction (trades) are similar in almost all respects to systems such as those that predict loan default. It is critical that the data processed for prediction has the same distribution, with respect to the signal being identified, as the data processed for model fitting.
To develop a system that will predict whether a borrower will repay a loan, the lender gathers data that hopefully has some predictive value from a large group of customers, some of whom repaid and others whom have defaulted. If conditions for the period the data represents are relatively constant, the distribution of any randomly chosen subgroups will be the same as the distribution of the entire group and, importantly, of future customers. With respect to loan repayment, the data is stationary. The future resembles the past.
The data scientists develop the model that goes with the data by selecting a random subsample of the data and fitting the rules to it. This is the "training" data and fitting is the learning process. This is "supervised" learning -- each data point has values for the predictor variables (income, job history, etc) and the loan repayment -- the target -- is known. The fitting process is a mathematical process of finding the best solution to a set of simultaneous equations. aX = Y. Where X is a large array of values for the indicator variables and y is a column array of values for the target. The model is the array "a" -- the coefficients of the solution. There will be a solution -- there will be an "a" array -- whether the model has predictive capability or not.
The model -- the rules -- may have identified one or more important features in the data that are consistently associated with probability of repayment. But the developer cannot tell by looking at the learning process. He or she has reserved a separate subsample that is not used at all in learning -- call it the "test" data. As a one-time test, the model is applied to the test data. The value of the target is known, but is only used for later reference. A predicted value for each test data point is computed using the a array. Comparison between the known target values and predicted target values lets the person building the model know whether the model learned general features or just fit the particular data it was given -- including all the randomness.
The scientific method insists on two phases -- learning and testing. Without independent testing, nothing can be said about the model or its predictions.
The trading system development profession has largely ignored the scientific method. Independent, one-time, testing using data that follows the training data and has not been used in development is seldom done. Trading system developers see the equity curve from the training portion, and assume completely without justification, that future results will be similar.
That may be true, but in order for it to be true, two conditions must hold.
1. The future must resemble the past. That is, the data used for learning and the data used for testing / trading must have the same distribution (with respect to the signal). This is stationarity.
2. The model must learn real, predictive signals rather than simply fit to the randomness of the data. This is learning.
When building models for stationary data such as loan repayment, some model algorithms produce in-sample results that can be used to give estimates of out-of-sample performance, while other model algorithms always overfit in-sample and have no value for estimating out-of-sample performance. Out-of-sample testing is always required before estimating future performance.
Back to trading.
A system that works in-sample has fit a set of equations to a set of data. Whether there is true learning or not, there is always a solution -- a set of trades and an associated equity curve. Most are bad and are discarded. We test so many combinations that eventually one fits and the result looks good. It might be simply an overfit solution or it might be a representation of a truly predictive system. We cannot tell anything about the future performance without testing never-seen-before future data. When results of this out-of-sample test are poor, it is because one or both of the two conditions do not hold. Either the data is not stationary beyond the time period of the learning data; or the model has not learned.
I hope this helps.
Thanks for listening, Howard