# My current systems - walking forward



## CanOz (4 April 2012)

I thought i would make use of my newly reacquired access to ASF and post some results from three or four of my systems that I'm currently testing on the simulator. 

I developed these systems with a Genetic Program called Adaptrade Builder. I run them on MultiCharts.
I also have an NT system or two that i may post as well.

There is much discussion on these types of genetic programs and i thought it may be interesting to get some system discussion going again, there is much to learn from many on ASF on this topic. 

Since i just returned from a short break i thought the timing seemed right to start the thread.

Fingers crossed that i can continue this thread, i certainly have the time available now...i just hope the access continues.

Cheers,


CanOz


----------



## CanOz (4 April 2012)

The first results are from a DAX 15 minute system. 

This is OOS data from July 2011 to the last contract expiry.

I'm a little worried for the Profit factor, and the W/L ratio. Its testing forward better than the first and second OOS runs.

It has not yet been optimized.

CanOz


----------



## Trembling Hand (4 April 2012)

Got a scatter chart of trades?


----------



## CanOz (4 April 2012)

Scatter gram...


----------



## Trembling Hand (4 April 2012)

I liked it better before I looked at that


----------



## CanOz (4 April 2012)

Trembling Hand said:


> I liked it better before I looked at that




Yeah, for having such a wide MM stop, it doesn't produce enough big winners. My worry about the PF and the W/L Ratio.

I can try and optimise the MM stop, or i could try to build in a trailing stop.

OR i could try and filter...but i think the GP should have found the optimum filter already.

CanOz


----------



## waza1960 (4 April 2012)

Thanks for the Thread CanOz 
    Questions: What time period are you conducting in sample testing ?
                   What kind of Simulator are you using ?
                   Do you have any concerns that the Adaptrade approach is isolating you  
                   from the Strategy development at all and if so are you confortable with
                     this?
         Thoughts :Win/Loss ratio and PF is a problem IMO .The metric that I have been giving most weight to for Optimisation and observing in Backtests is avg win/avg loss ratio and then % profitable trades .I have actually written  my own metric within Ninja Trader to optimise on which links these two functions.
 A more sophisicated stop (Trailing/ATR)would be beneficial I don't use normal stops anymore.


----------



## CanOz (4 April 2012)

waza1960 said:


> Thanks for the Thread CanOz
> Questions: What time period are you conducting in sample testing ?
> What kind of Simulator are you using ?
> Do you have any concerns that the Adaptrade approach is isolating you
> ...




Hiya Waza, glad to hear from you...

I also used 15 minutes to develop the strategy over 1 year of data from 2009-2010. Then i ran the top 100 over the next year and chose a strategy that had a consistent metric profile.

I'm comfortable with using the Builder (its allot of work to find a decent algorithm), but still going through the learning curve with Easy Language. I have more ideas than i could possibly test in a lifetime but I'll do my best to get through them!

Love to exchange some ideas sometime.

Cheers,


CanOz


----------



## Punta (5 April 2012)

Interesting thread. What is a simulator? Does this mean you've been paper trading it for a year? Or is this a way of taking historical data, and "trading" it, so as to try and factor slippage into your testing?


----------



## CanOz (5 April 2012)

For these systems i am using Interactive Brokers simulator (paper trader) which is almost a direct copy of their TWS. I can even copy the settings from the Sim TWS to the live account once I'm confident all of the settings are working as hoped.

There is no need to paper trade the system for a year, but it needs to be 'walked forward' to ensure it performs similar on new data as it did on past data.

The paper trading just ensures that the automation works ok, the orders get executed properly etc. before going live.

Cheers,


CanOz


----------



## Punta (5 April 2012)

Yeah, I didn't mean to specify a year, but have you been paper trading that since July 2011?  Seems like a long time.  If my paper trading is matching up with the backtesting system, I would go live after a much shorter "walking forward" period.


----------



## CanOz (5 April 2012)

Punta said:


> Yeah, I didn't mean to specify a year, but have you been paper trading that since July 2011?  Seems like a long time.  If my paper trading is matching up with the backtesting system, I would go live after a much shorter "walking forward" period.




These systems were developed in early March, so i've got a few months of testing left to go. I would think i'll be going through the summer, hopefully ready for last couple of quarters of the year.

For me to be confident to trade them i think i'll need at least two months of WF. If they can't convince me after that then i'll start over.

These are my first systems to WF, so its new to me.

Cheers,


CanOz


----------



## CanOz (5 April 2012)

This my current HSI system. I have optimized the MM Stops, and the ATR stops.

It doesn't win as often, but it wins bigger. 

An interesting metric that Sunny Harris (author trader etc.) uses if the PF x  W/L Ratio x %profitable and it must be above 1.2. None of my systems exhibit this on WF testing. This one is .993

What i would like is .5 x 1.6 x 1.6 = 1.28 up to .55 x 1.7 x 1.7 = 1.58

CanOz


----------



## CanOz (5 April 2012)

Lots of issues left to resolve...like this sell stop on the DAX...the algo put up the order well before price even got near it, but for whatever reason it didn't fill, on the simulator

CanOz


----------



## tech/a (5 April 2012)

CanOz said:


> Lots of issues left to resolve...like this sell stop on the DAX...the algo put up the order well before price even got near it, but for whatever reason it didn't fill, on the simulator
> 
> CanOz




Had that happen on the odd occasion R/T.
If a trade doesn't occur at the exact same price your stop is at you won't get done.
That can and does happen in a quickly moving market.
The DAX is often really quick gapping over prices.

I also have commoners on your systems I'll have more time as the weekend goes on.


----------



## waza1960 (5 April 2012)

> Yeah, I didn't mean to specify a year, but have you been paper trading that since July 2011? Seems like a long time. If my paper trading is matching up with the backtesting system, I would go live after a much shorter "walking forward" period.




  I tend to agree with this my testing is indicating that shorter time frames (less than 30 min) need to have considerably shorter testing periods or rather testing periods closer to current price action at least with regards to FX anyway .


----------



## CanOz (6 April 2012)

waza1960 said:


> I tend to agree with this my testing is indicating that shorter time frames (less than 30 min) need to have considerably shorter testing periods or rather testing periods closer to current price action at least with regards to FX anyway .




There is a whole debate around this and i also tend to agree that more recent data tend to reflect the current market behavior...think participants too.

I use 1 year to construct and 1 year as my first OOS period at the moment, but today i will try and construct over shorter time frames with many more generations (using a GP to reverse engineer) and see if i get something that performs closer to the construct period on OOS1 and OOS2.

Mike Bryant from Adaptrade (literally a rocket scientist) is also in favor of frequent optimizations. 

Again ,there is a huge raging debate among quants on how much to optimize...i stay out of it.....no place for me to jump in on a bunch of math and physic Phd's duking it out on a forum:headshake

Great topic.


CanOz


----------



## CanOz (6 April 2012)

tech/a said:


> Had that happen on the odd occasion R/T.
> If a trade doesn't occur at the exact same price your stop is at you won't get done.
> That can and does happen in a quickly moving market.
> The DAX is often really quick gapping over prices.
> ...




Would you believe it was my fault. With MultiCharts you need to have the chart loaded with IB data in order for the orders to execute...makes sense...with NinjaTrader you can use eSignal data and still execute through IB, which i am used to....my bad...doh!

Too many platforms...

CanOz


----------



## sinner (7 April 2012)

> Mike Bryant from Adaptrade (literally a rocket scientist) is also in favor of frequent optimizations.
> 
> Again ,there is a huge raging debate among quants on how much to optimize...i stay out of it.....no place for me to jump in on a bunch of math and physic Phd's duking it out on a forum




Hi Canoz,

I think this website is an important read for you

http://www.adaptivetradingsystems.com/blog/

The main thing I took away from the blog is that it's much much better to 'swarm' rather than have a single adaptive bot and take their aggregate signal as the actual signal.

The blog finds that trading a single adaptive bot is not a very good way to trade, the most common problem is adapting to any optimal solution too late. However there are other issues which can't be simply overcome.  

My testing concurs with these findings, although I admittedly don't use 'regular' methods which most use like genetic algorithms or ANN or similar. 

As for your shorter time period for testing/training, my advice is to just stay the F* out of the forex markets for this type of trading, stick to the index futs.


----------



## CanOz (7 April 2012)

sinner said:


> Hi Canoz,
> 
> I think this website is an important read for you
> 
> ...




Thanks Sinner, i can't access Blogs. Can you give me an idea of what the discussion is?

There is allot of discussion on Mike Bryant's google group as well here...

Agree on Forex, but I've got other ideas for that, they don't involve using the GP to develop though....Actually the time of day/session is one of the most critical factors in FX trading according to my testing so far...not allot of research done on it yet though...save it for a rainy day.

Again, appreciate the feedback.

CanOz


----------



## CanOz (7 April 2012)

Sinner,

Knowing only a very little about Neural Networks and assuming without a Google that ANN is Adaptive Neural Network i can tell you that i have no experience with these. 

I am currently using a genetic program called Adaptrade Builder to construct an algorithm in Easy Language code that i then test on more OOS data, then optimize and walk forward.

Were you thinking i was using some kind of neural network software?

Cheers,


CanOz


----------



## tech/a (8 April 2012)

I agree with Index futs.

What I'm seeing here is a system *WITHOUT* an edge.

Low win ratio over a shorter time frame with no opportunity to catch any decent trends.
with such a low win ratio you need much higher winning trades.
Or a lot lower losses.

_Its only churning numbers._

The second system the HSI is similar if you take out the largest winning trade.


----------



## CanOz (8 April 2012)

Yes, i agree on the poor metrics....thats why i like that formula %Wins*PF*RW/L

Its a volitility system, not trend following. Non are trend following systems.

My SPI system seems the best so far in short OOS data, but i have a strange price anomoly in the data that i'm trying to work out...once i get that sorted i'll test it on two years of data and post some results.

In the meantime i think its back to the drawing board until i get something that meets my metrics. Let see what we can find...hopefully something like this one or better.

Thanks for the feedback...

Cheers,


CanOz


----------



## tech/a (8 April 2012)

With slippage and commissions your probably looksas good.


----------



## CanOz (8 April 2012)

Yeah it really Irks me that most vendors don't put in S&C. At the very least you can include the IB commissions, and a point of slippage each way.

CanOz


----------



## sinner (8 April 2012)

CanOz said:


> Sinner,
> 
> Knowing only a very little about Neural Networks and assuming without a Google that ANN is Adaptive Neural Network i can tell you that i have no experience with these.
> 
> ...




Hi canoz, was just pointing out that although my adaptive model uses much more 'advanced' principles of machine learning than genetic algorithms or ANN (without a very solid understanding of the ins and outs, these are really like the moving averages of machine learning), my own research confirmed what is coming out in the financial science (and that blog) recently about requiring an 'aggregate' rather than individual decision for machine learning trading signals. 

For example, my machine learning model which has pretty good prediction rates for weekly moves in an equity index (incl magnitude) is actually a highly abstracted hierarchy of smaller models operating under a larger one.


----------



## CanOz (8 April 2012)

sinner said:


> my machine learning model which has pretty good prediction rates for weekly moves in an equity index (incl magnitude) is actually a *highly abstracted hierarchy of smaller models operating under a larger one*.




Thats the 'aggregated decision' then...?

You know it wouldn't be difficult for me to aggregate decisions of several similar systems. The Builder has a habit of constructing very similar algorithms (makes sense on the same data) if you run it over the same data with only slight variations in the indicators or settings...I wonder if it would make sense to use several of these systems and wait until they all generated the same signal before taking a position...then test this on OOS data and see what the results are like....just thinking out load...



CanOz


----------



## tech/a (8 April 2012)

Sinner
What software are you using?


----------



## sinner (8 April 2012)

tech/a said:


> Sinner
> What software are you using?




tech, I write all the vectorisers, training wrappers and modellers in Python (numPy and sciPy usually can do the maths but if not I plug the script into R) myself and use SVM-light (a free, very robust Support Vector Machine implementation for most Operating Systems) as my learning and classifying mechanism.

http://svmlight.joachims.org/

This is all an 'adaptation' of stuff I learned in CompSci at uni (now years ago) to do stuff like ranking http links for usefulness (think Google PageRank, our prof was a goog PhD) and spam filtering. 

Here is a 2004 example from the scientific literature using SVM in a very very (I can't stress how very) simple model to predict (not incl magnitude) weekly direction in the Nikkei futs. It's a really good, inspiring paper though, recommended reading.

http://www.sciencedirect.com/science/article/pii/S0305054804000681

I notice this guys most successful techniques use SVM too

http://themarketpredictor.com/blog/

Hope you don't mind the hijack there canoz :/



CanOz said:


> Thats the 'aggregated decision' then...?
> 
> You know it wouldn't be difficult for me to aggregate decisions of several similar systems. The Builder has a habit of constructing very similar algorithms (makes sense on the same data) if you run it over the same data with only slight variations in the indicators or settings...I wonder if it would make sense to use several of these systems and wait until they all generated the same signal before taking a position...then test this on OOS data and see what the results are like....just thinking out load...
> 
> ...





Check out that adaptivetrading blog link I posted earlier, go back through the posts, it gives very useful hints as to 

* Adaptive techniques
* How to value individual bots in a swarm
* Different techniques to trade the aggregate decision (hint: waiting for everyone to agree is not it, think more 'democratic' or 'representative', or net long/net short)


----------



## CanOz (9 April 2012)

No worries Sinner, i welcome any systematic discussion.....machine generated or otherwise...

I can't access Blogs/You-tube/FB etc...

CanOz


----------



## sinner (9 April 2012)

CanOz said:


> No worries Sinner, i welcome any systematic discussion.....machine generated or otherwise...
> 
> I can't access Blogs/You-tube/FB etc...
> 
> CanOz




Is that the China Firewall thing?

I will try and PM you some excerpts if I get a chance then.


----------



## CanOz (9 April 2012)

sinner said:


> Is that the China Firewall thing?
> 
> I will try and PM you some excerpts if I get a chance then.




We call it the Great Fire Wall.....

Appreciate that Sinner.

I've scrapped all of my trial work from  Builder and have started all over again. The first month was a learning period anyway. Now that i have my objective function sorted and a standard protocal for develop, testing, Optimization and Walk Forward I'll give all my markets another go.

I may change to my old thread "Developing a Trading system from Scratch"  and show the progress, gives me a bit of a journal as well.

Cheers,


CanOz


----------



## McLovin (9 April 2012)

CanOz said:


> We call it the Great Fire Wall.....




A VPN will get around it. I have used VyprVPN in China and have had no issues with it. Whenver I travel I use Vypr, it's great if you're constantly using unsecure WiFi.

(sorry the post sounds a bit like an advert)


----------



## CanOz (9 April 2012)

McLovin said:


> A VPN will get around it. I have used VyprVPN in China and have had no issues with it. Whenver I travel I use Vypr, it's great if you're constantly using unsecure WiFi.
> 
> (sorry the post sounds a bit like an advert)




Yes and i used one for a while, they shut that down or somehow made it so slow it was useless. 

I have great internet speed now and i can access what i need to conduct my business, so I'm happy for now. My wife wants a VPN for FB though, so we may get one that she can use on the iPad. Thanks McLovin.

Cheers,


CanOz


----------

