- Joined
- 28 May 2020
- Posts
- 127
- Reactions
- 229
Hi,
As I understand it...
In Sample (IS) training:
IS training is when you are optimizing your (hopefully!) frozen code. The object of the IS training is to solve for the best outcome of some objective function, say Net Profit, CAR/MDD, % System Drawdown, etc. Let's further say you might decide on a combination of objective functions, for example the combination of Net Profit, CAR/MDD, % System Drawdown, or whatever other combination makes you most "comfortable". So in this respect, the selection of "best" outcome is subjective - not very data science-y but perhaps acceptable? I know data science likes cold hard mathematics - perhaps someone smarter than me could codify some objective function (say Bandy's CAR25?) in a custom back test (CBT) that would, objectively and without any doubt, determine which set of optimization parameters are the best outcome.
As I understand it, in data science terms the sum total of your rules (code), optimization parameters, universe of instruments traded (your training/test/production data), analysis settings (eg. daily vs. weekly), and whatever else I'm forgetting comprises your model. (For the purposes of this discussion "model" means your trading system). ANY change to any of this collection of rules and parameters comprises a new model. In other words, if I have a system and change one parameter or one line of code, I've got a new model. An exception might be the universe of instruments traded, since that is data for the model and not the model itself. Still, if I changed the universe I would likely want to re-optimize.
When training your model, if you use all your data for training, you run the risk of overfitting. You can optimize your code, parameters, settings, or whatever else to the N-th degree to the point that your model is absolutely perfect for your training data, but it may then be "fragile" and not robust when given data with different characteristics to its test data. For this reason, you should not use the entire market data for your optimization. You need some out of sample data to test your model.
For the sake of argument let's say your optimization and therefore test data was from 2000 - 2014.
Out of Sample (OOS) testing:
OOS testing is when you run your model against data that was not used for training, with the goal of predicting how your model will solve an objective function, say Net Profit or CAR/MDD. The extent that your model predicted a favourable outcome (say max Net Profit) determines the "success" of your model. Otherwise you continue development until you get favourable results (or discard your model).
Amibroker's Walk Forward process also does OOS testing, in that it tests a set of data that was not used for training by "walking forward". However, it then re-optimizes, adding that OOS data to the previous IS data. Re-train, re-test, rinse and repeat, and display the aggregated results of the OOS testing.
To me, this implies that, if you conducted a Walk Forward test, got favourable end results, and expected your live system to mimic the Walk Forward, then you would need to re-optimize your model (system) at the Walk Forward time period, for example at the beginning of each year.
This would bring forward your training data across your entire set of production data, i.e. the market data up to "now". I wonder if this could result in overfitting? I suppose you could configure the Walk Forward to stop before consuming the entire data set, say end of 2018 if we're now in 2021. If that gave satisfactory results, you would then "hold back" three years when doing your annual re-optimization.
BTW, I know it's not as cut-and-dried as this. If your model isn't performing well in the current market (new data that doesn't fit the training data), you might re-optimize then instead of waiting for the beginning of the year.
For the sake of argument let's say your test data was from 2015 - today. As I understand it, that is OK - it is still just testing, as long as you are not using that data to optimize your model. If you get poor results from that more recent data, you continue development of your model, and re-optimize against the same test data as before. The minute you start including all your data in the training, you run the risk of overfitting, and have no data with which to test for that. But backtesting is fine.
Question: when do you re-optimize your model? On a regular basis, or only when it is broken ("if it ain't broke don't fix it"?)
Paper Trading:
I've read that, once you develop a system, you should paper trade it for some time before live trading. I guess I'd call this the "gold standard" of OOS testing, in that you are testing with the most current data, and of course it was OOS. And, of course, you're not losing money if current market conditions prove your model is out of sync with the current market data. I've read that you should allow a fair bit of time for that paper trading, say 3 months or more. But what if you're an impatient person like me?
Here's my main question...
Assuming historic market data is frozen, i.e. Norgate won't change data once it's been received, then wouldn't retrospective paper trading be the equivalent of live market paper trading?
For example, say I finish my system on 01Jan and consider it ready for trading and I want to trade NOW.
Scenario #1: In market paper trading
Each week I run an explore, record the results in some paper trading application (say Excel), and start live trading on 01Mar if the paper trading was successful.
Scenario #2: Retrospective paper trading
I run a weekly explore in Amibroker starting 01Sep, record the results in some paper trading application (say Excel), and start live trading on 01Jan if the paper trading was successful.
What's the difference between the two approaches?
As I understand it...
In Sample (IS) training:
IS training is when you are optimizing your (hopefully!) frozen code. The object of the IS training is to solve for the best outcome of some objective function, say Net Profit, CAR/MDD, % System Drawdown, etc. Let's further say you might decide on a combination of objective functions, for example the combination of Net Profit, CAR/MDD, % System Drawdown, or whatever other combination makes you most "comfortable". So in this respect, the selection of "best" outcome is subjective - not very data science-y but perhaps acceptable? I know data science likes cold hard mathematics - perhaps someone smarter than me could codify some objective function (say Bandy's CAR25?) in a custom back test (CBT) that would, objectively and without any doubt, determine which set of optimization parameters are the best outcome.
As I understand it, in data science terms the sum total of your rules (code), optimization parameters, universe of instruments traded (your training/test/production data), analysis settings (eg. daily vs. weekly), and whatever else I'm forgetting comprises your model. (For the purposes of this discussion "model" means your trading system). ANY change to any of this collection of rules and parameters comprises a new model. In other words, if I have a system and change one parameter or one line of code, I've got a new model. An exception might be the universe of instruments traded, since that is data for the model and not the model itself. Still, if I changed the universe I would likely want to re-optimize.
When training your model, if you use all your data for training, you run the risk of overfitting. You can optimize your code, parameters, settings, or whatever else to the N-th degree to the point that your model is absolutely perfect for your training data, but it may then be "fragile" and not robust when given data with different characteristics to its test data. For this reason, you should not use the entire market data for your optimization. You need some out of sample data to test your model.
For the sake of argument let's say your optimization and therefore test data was from 2000 - 2014.
Out of Sample (OOS) testing:
OOS testing is when you run your model against data that was not used for training, with the goal of predicting how your model will solve an objective function, say Net Profit or CAR/MDD. The extent that your model predicted a favourable outcome (say max Net Profit) determines the "success" of your model. Otherwise you continue development until you get favourable results (or discard your model).
Amibroker's Walk Forward process also does OOS testing, in that it tests a set of data that was not used for training by "walking forward". However, it then re-optimizes, adding that OOS data to the previous IS data. Re-train, re-test, rinse and repeat, and display the aggregated results of the OOS testing.
To me, this implies that, if you conducted a Walk Forward test, got favourable end results, and expected your live system to mimic the Walk Forward, then you would need to re-optimize your model (system) at the Walk Forward time period, for example at the beginning of each year.
This would bring forward your training data across your entire set of production data, i.e. the market data up to "now". I wonder if this could result in overfitting? I suppose you could configure the Walk Forward to stop before consuming the entire data set, say end of 2018 if we're now in 2021. If that gave satisfactory results, you would then "hold back" three years when doing your annual re-optimization.
BTW, I know it's not as cut-and-dried as this. If your model isn't performing well in the current market (new data that doesn't fit the training data), you might re-optimize then instead of waiting for the beginning of the year.
For the sake of argument let's say your test data was from 2015 - today. As I understand it, that is OK - it is still just testing, as long as you are not using that data to optimize your model. If you get poor results from that more recent data, you continue development of your model, and re-optimize against the same test data as before. The minute you start including all your data in the training, you run the risk of overfitting, and have no data with which to test for that. But backtesting is fine.
Question: when do you re-optimize your model? On a regular basis, or only when it is broken ("if it ain't broke don't fix it"?)
Paper Trading:
I've read that, once you develop a system, you should paper trade it for some time before live trading. I guess I'd call this the "gold standard" of OOS testing, in that you are testing with the most current data, and of course it was OOS. And, of course, you're not losing money if current market conditions prove your model is out of sync with the current market data. I've read that you should allow a fair bit of time for that paper trading, say 3 months or more. But what if you're an impatient person like me?
Here's my main question...
Assuming historic market data is frozen, i.e. Norgate won't change data once it's been received, then wouldn't retrospective paper trading be the equivalent of live market paper trading?
For example, say I finish my system on 01Jan and consider it ready for trading and I want to trade NOW.
Scenario #1: In market paper trading
Each week I run an explore, record the results in some paper trading application (say Excel), and start live trading on 01Mar if the paper trading was successful.
Scenario #2: Retrospective paper trading
I run a weekly explore in Amibroker starting 01Sep, record the results in some paper trading application (say Excel), and start live trading on 01Jan if the paper trading was successful.
What's the difference between the two approaches?
