Getting Started in Machine Learning for Trading

[entropy]

I haven't seen or heard from Howard in a long time. Last time i spoke to him he had stopped trading.

Personally i use RSI().
I also use c/ma(c,X) a lot ( in almost everything ), usually as a series to add a momentum input.

The 2 examples are auto normalising formulas, hence they will work on a variety of different instruments and classes. So to answer your question fully, use any indicator that self normalises, it makes life easier.

The timeframes you need to use are a LOT faster than what you would as a system trader. RSI() i would use 2 -> 5 on a daily. But there is no reason why you can't throw ALL of them in. The model should work out which is relevant and which isn't.

In that vein, any indicator that you overlay on a price chart ( bollinger bands, moving averages etc ) are out OR if you really want to use them you have to normalise them to the underlying price somehow.

You also need to input the underlying market as an input. This becomes another complex can of worms.

In terms of models I like Neural networks better. Most quants prefer trees because they can create a human readable system. THIS IS NOT THE POINT OF ML. More edges can be gained from ML models by finding patterns that humans can't and NN's are more adept at finding patterns. Trees are just repeating a process that quants have been working on for years.

Thanks Dave, brilliant post!

And plenty for me to go on with. I have only traded since last year so missed the input from Dr Bandy of four to five years ago.

Your comments re tree-based methods and the need for quants to use these so their clients can understand the output are apt and also encouraging.

When you say to input the underlying market do you mean something like the All Ordinaries Index?
If I was trying to predict, say, CBA movement, include some sort of Bank Index, to predict FMG, some sort of Mining Index etc?

 

[DaveDaGr8]

Yes the index or indexes.

The inputs should be whatever is relative to what output you're trying to achieve.

If you are looking for specific patterns within the stock itself, then you don't need to know about the market. However if you're looking for trading signals, or price prediction then the market as a minimum is crucial.

I must admit, i've never seen xgboost before ... it looks interesting so i might have a play with it.

 

[markrmau]

Blow me down, I didn't expect people to be talking about using machine learning on this forum! I was on this path myself and was just putting some tools in place to start investigating. I have no doubt it will be challenging to get useful algorithms - machine learning won't be a magic bullet due to the inherrent randomness of the stock market.

I did the following course a year or so ago and thouroughly enjoyed it. I think it gives a lot of the basic concepts which are important to understand before launching into the more complex details. https://www.coursera.org/learn/machine-learning

Useful comments in the above thread!

As for implementation, I am in the process of putting tools together for the training and testing of algorithms. If anyone is interested, I am happy to share. The big proviso is that I am not an expert coder. However the hard work is already done in python libraries so I don't think this will be a barrier.

The tools I have put together are from the old unix ethos - lots of little tools to do specific tasks:

1. Download stock data from yahoo finance and save
2. Search stock data to provide a series of buy / sell signals (currently for testing I have implemented techtrader, but the code can use use the buy/sell signals from a machine learning algorithm).
3. Undertake monte carlo analysis based on taking a selection of the above buy/sell signals (not finished yet though). Arguably this could be done using amibroker or other programs, but I prefer my own tools so I can verify I am not using "future signals". It is not a complex program.

So to reiterate, I am not an expert coder, but here is my simple code to download and save asx stock data (based on a download of stocks listed in a csv file). Obviously make a directory somewhere you want to hold the project in, with a subdirectory 'Data'.

import yfinance as yf
import pandas as pd
import time, os, pickle

asx_df = pd.read_csv(r'Data/ASX_Listed_Companies_31-10-2021_06-25-30_AEDT.csv')
asx_sym = asx_df['ASX code'].tolist()
for sym in asx_sym:
if not os.path.exists('Data/'+sym+'_hist.pkl'):
print ('Looking at '+sym)
stock = yf.Ticker(sym+'.AX')
stock_inf = stock.info
if len(stock_inf)>5: # Weed out stocks you can't download
stock_hist = stock.history(period='max',auto_adjust=False) # Note, OHLC data is adjusted for splits, but not dividends
with open('Data/'+sym+'_inf.pkl','wb') as f:
pickle.dump(stock_inf, f, protocol=pickle.HIGHEST_PROTOCOL)
stock_hist.to_pickle('Data/'+sym+'_hist.pkl')
print ('Saved '+sym)

This is python 3. Edit - python won't like that - the indentation seems to have disappeared! I'll attach - need to change the extension of course

This code simply downloads buckets of data from yahoo and saves it as pickled dataframe. Next steps would be to add code to be able to update the data as time goes on, and also for machine learning we need to create separate training and validation sets

If there is interest, I can share more of this sort of basic code - or someone might point out that this framework already exists in python!

 

[qldfrog]

Blow me down, I didn't expect people to be talking about using machine learning on this forum! I was on this path myself and was just putting some tools in place to start investigating. I have no doubt it will be challenging to get useful algorithms - machine learning won't be a magic bullet due to the inherrent randomness of the stock market.

I did the following course a year or so ago and thouroughly enjoyed it. I think it gives a lot of the basic concepts which are important to understand before launching into the more complex details. https://www.coursera.org/learn/machine-learning

Useful comments in the above thread!

As for implementation, I am in the process of putting tools together for the training and testing of algorithms. If anyone is interested, I am happy to share. The big proviso is that I am not an expert coder. However the hard work is already done in python libraries so I don't think this will be a barrier.

The tools I have put together are from the old unix ethos - lots of little tools to do specific tasks:

1. Download stock data from yahoo finance and save
2. Search stock data to provide a series of buy / sell signals (currently for testing I have implemented techtrader, but the code can use use the buy/sell signals from a machine learning algorithm).
3. Undertake monte carlo analysis based on taking a selection of the above buy/sell signals (not finished yet though). Arguably this could be done using amibroker or other programs, but I prefer my own tools so I can verify I am not using "future signals". It is not a complex program.

So to reiterate, I am not an expert coder, but here is my simple code to download and save asx stock data (based on a download of stocks listed in a csv file). Obviously make a directory somewhere you want to hold the project in, with a subdirectory 'Data'.


This is python 3. Edit - python won't like that - the indentation seems to have disappeared! I'll attach - need to change the extension of course

This code simply downloads buckets of data from yahoo and saves it as pickled dataframe. Next steps would be to add code to be able to update the data as time goes on, and also for machine learning we need to create separate training and validation sets

If there is interest, I can share more of this sort of basic code - or someone might point out that this framework already exists in python!

@markrmau ,
following with interest.
Spent last 2y using AB for basic system building and trading, but with background in AI, obviously ML teaining has never been far of my mind.
I believe the experiences I gained so far in standard system building is not wasted if aiming the ML way and would not be surprised to see myself in that area in 4 or 5 years.
Good luck in your progress : all thes posts are really encouraging

 

[KevinBB]

@markrmau well ... I haven't been following this thread, as I don't do machine learning ... but my latest project is something similar. Your Python reference has sparked my interest. I am like you, a novice Python programmer.

I currently trade a futures system, very similar to the Carver system, which I am trying to adapt to Australian stocks. So far, unsuccessfully. Those who know the Carver system will know that lately he has been playing with portfolio optimisation. So, this is what I've been working on in Python.

The process I am going through is to select stocks based on their minimal correlation to an "uncorrelated portfolio". I add stocks one by one to this portfolio based on their low correlated returns. Breakthrough last night ... got the code working.

Now for the fun bit ... with this "uncorrelated portfolio", apply the Carver system and see how it turns out.

KH

 

[markrmau]

Here is a modified form of techtrader - applys all buy/sell signals. The techtrader buy/sell signals can be replaced with ML signals.

This produces a list of all possible trades (asssume sell on last day of data series). Then the next step would be to do montecarlo analysis on these trades to fit with available capital, commissions etc. I have this 50% complete (excludes the montecarlo analysis - just does one run).

This code is a bit messy. I originally coded using pandas which produced beautiful clean code but was too slow. I moved to numpy for speed but it is little less readable - I know I could use dictionaries for the column numbers but I wanted to keep the code fast.

@qldfrog & KH - I haven't heard of Carver - I'll have to look it up. Thanks for the interest!

 

[DaveDaGr8]

Rob Carvers book systematic trading was a real eye opener and the depth he goes into is not for the faint hearted. It made me realise just how important it is to trade uncorrelated systems.

Andreas Clenow's Trading evolved is also a MUST READ for anyone who wants to get started in using Python for any sort of trading application.

 

[markrmau]

Thanks Dave! I had a quick flip through the book - I see that a lot of what I have been doing is already implemented here:

GitHub - quantopian/zipline: Zipline, a Pythonic Algorithmic Trading Library

Zipline, a Pythonic Algorithmic Trading Library. Contribute to quantopian/zipline development by creating an account on GitHub.

github.com

I think I'll ditch what I have been doing and use zipline.

 

[KevinBB]

Rob Carvers book systematic trading was a real eye opener and the depth he goes into is not for the faint hearted. It made me realise just how important it is to trade uncorrelated systems.

Andreas Clenow's Trading evolved is also a MUST READ for anyone who wants to get started in using Python for any sort of trading application.

True. I found his books about one year ago, implemented Leveraged Trading first, and then Systematic Trading, both with CFDs, before progressing on to a small futures portfolio. I'm now trying to implement ST into Australian stocks.

Sometimes, though, his Python coding and his recent writing is far beyond my understanding. Its taken me a couple of months, and much discussion with another person (afaik not on this forum) to get this "optimised portfolio" to a stage where it can be tested.

We're off the topic of machine learning .. so not too much more.

KH

 

[markrmau]

Ick, zipline isn't what I want to use. This looks a lot better:

Backtesting.py - Backtest trading strategies in Python

Fast Python framework for backtesting trading and investment strategies on historical candlestick data.

kernc.github.io

It seems well suited to integration into ML and even has a ML specific tutorial:

https://kernc.github.io/backtesting.py/doc/examples/Trading%20with%20Machine%20Learning.html

I can assist with python if anyone needs help.

 

[entropy]

Blow me down, I didn't expect people to be talking about using machine learning on this forum! I was on this path myself and was just putting some tools in place to start investigating. I have no doubt it will be challenging to get useful algorithms - machine learning won't be a magic bullet due to the inherrent randomness of the stock market.

I did the following course a year or so ago and thouroughly enjoyed it. I think it gives a lot of the basic concepts which are important to understand before launching into the more complex details. https://www.coursera.org/learn/machine-learning

Useful comments in the above thread!

As for implementation, I am in the process of putting tools together for the training and testing of algorithms. If anyone is interested, I am happy to share. The big proviso is that I am not an expert coder. However the hard work is already done in python libraries so I don't think this will be a barrier.

The tools I have put together are from the old unix ethos - lots of little tools to do specific tasks:

1. Download stock data from yahoo finance and save
2. Search stock data to provide a series of buy / sell signals (currently for testing I have implemented techtrader, but the code can use use the buy/sell signals from a machine learning algorithm).
3. Undertake monte carlo analysis based on taking a selection of the above buy/sell signals (not finished yet though). Arguably this could be done using amibroker or other programs, but I prefer my own tools so I can verify I am not using "future signals". It is not a complex program.

So to reiterate, I am not an expert coder, but here is my simple code to download and save asx stock data (based on a download of stocks listed in a csv file). Obviously make a directory somewhere you want to hold the project in, with a subdirectory 'Data'.


This is python 3. Edit - python won't like that - the indentation seems to have disappeared! I'll attach - need to change the extension of course

This code simply downloads buckets of data from yahoo and saves it as pickled dataframe. Next steps would be to add code to be able to update the data as time goes on, and also for machine learning we need to create separate training and validation sets

If there is interest, I can share more of this sort of basic code - or someone might point out that this framework already exists in python!

Excellent post mark, and your generous offer to share is appreciated!

I have been learning Python and some of the ML and DeepNN models.

Currently using Python 3, Anaconda Navigator, Jupyter Notebooks.

The Kaggle site is interesting: working through the revealed code and the thought processes of prizewinners is instructive.
Looking forward to all discussion and posts re ML and the markets.