Tools being used by your trading competition

[Joe Blow]

I'd like to have my two cents worth here.

Howard has contributed a lot of valuable content here at ASF and he has answered a lot of questions posed by a lot of people, especially on the topic of Amibroker. He is an acknowledged expert in his field who is trying to engage in a serious discussion on a topic of real importance to traders. So I think it is incumbent upon everyone to participate in the discussion in a constructive and respectful way. There is nothing wrong with questions or genuine disagreement but questions shouldn't seem like accusations. They should be framed in such a way as to advance and expand the discussion, not to tear it down or sabotage it.

If a post doesn't add something positive or constructive to the debate then perhaps it's something that is better left unsaid.

So, having said that, can we please get back to discussing the topic at hand? I will tidy up this thread a little later and remove as many off topic posts as is practicable.

 

[McLovin]

The metaphors in this thread are next level ridiculous. The poor buying Cessnas because they can't afford jets.

Cessna make jets. I guess that would make you middle class if you're flying around in one of theirs.

Its like listening to a researcher who is looking for
a cure to cancer but rejecting their credibility because
they don't have evidence of curing it!

This thread looks more like a researcher claiming to have cured cancer without any evidence of curing it. IMO, of course.

 

[tech/a]

Oh come on !!!! They are classic

 

[howardbandy]

Greetings --

I have not gone away. But I am discouraged.

When someone -- anyone -- asks how to learn trading system development, and specifically what books to read or courses to take, my first recommendation is to read Daniel Kahneman's "Thinking, Fast and Slow." I have previously mentioned it on ASF and it is at the top of the bibliography I have posted. Kahneman's thesis is that humans have two thinking systems. System 1 is fast and reacts in a way that reflects situations as WYSIATI -- What You See Is All There Is. System 2 is conscious thought. He explains in detail, supported by his own Nobel Prize winning research and experience as well as that of other psychologists and economists, the very many biases we all have. When we allow our thinking to stop with System 1, we all are limited by our biases. We are naturally expert at fooling ourselves.

He points out the ease with which we assign causality when there is none, inflate probability of events based on recentness, modify our estimates toward arbitrary and often unnoticed anchors. Without using mathematics, or even giving the names to the techniques, he describes the scientific method and Bayesian analysis and insists that all forecasting should follow them.

Some of his examples are of professional money managers.

I am a member of a nonfiction book discussion club and Kahneman is the book for the July meeting. One of our members is a clinical psychologist. He said that Kahneman is among the required readings for the periodic re-certifications that his profession requires. It is not an easy read, but it is an important one. At very least, it will explain a lot of our thought processes; perhaps, it will change your life.

---------

About specific results associated with machine learning and artificial intelligence -- the topic I intended this thread to follow when I started it.

Every trader will make his or her own determination of personal risk tolerance, what to trade, how to trade, how to develop techniques, and how to build confidence in them. I have posted some specific recommendations based on both academic-style research and actual trading results from my own experiences and those of the hedge fund where I was a senior analyst. I have described the techniques in two books and a video posted to YouTube.

The conclusions and results are crystal clear. Begin with a statement of personal risk tolerance. Let the Data Prospector evaluate the risk and profit potential inherent in the issue you are considering trading. The data -- daily bars for most people, but the technique works with any bar length -- must satisfy three criteria: enough volatility to offer profit from trading it; not so much volatility that risk exceeds the trader's tolerance for accuracy and holding period that is attainable; and persistent and discoverable signals that precede profitable trades. The first two can be determined from the Data Prospector. If he, the Data Prospector, passes the data series, then the trading system developer can try to find a set of rules that identify trades that offer profit potential adequate to compensate for the risk.

All of the above is independent of trading system development platform and the model that will signal the trades. It applies in all trading circumstances.

Much as traders wish it to be so, long holding periods, infrequent trading, portfolios of issues drawn from a large pool of issues, and extensive reliance on either judgment or backtesting alone cannot produce systems that pass validation that are sufficiently profitable for reasonable risk.

Trading system development can be seen from two points of view -- 1. Begin by identifying desirable trades, then see what patterns occurred earlier -- 2. Begin with indicators, then see what trades occurred after. These correspond to 1. Machine learning / pattern recognition -- and 2. Traditional chart analysis / decision tree rules.

There is a steep learning curve to work with machine learning. But there is evidence that some of the largest and most profitable trading houses and hedge funds are increasingly moving toward machine learning. I have given some examples -- such as James Simons.

Is it important that I am personally and currently an active trader in order for my advice to be valuable? Professionals in the field of data analysis, modeling, simulation, prediction, or whatever you wish to call it require three areas of expertise: domain knowledge; programming skills; analysis skills. I have all three. More than that, I am expert in all three. Sports coaches are no long playing in league matches, but they have valuable knowledge and experience and can pass that on. They can provide the coaching for the frequent repetitions with prompt feedback that players require to fully develop. A male data analyst does not need to personally use lipstick and skin conditioners in order to develop accurate forecasts for development and distribution of those products.

So it is disappointing to me that the discussion left science-based development and went into full-throated criticism without justification. One of my colleagues regularly reminds us that saying "just saying" does not automatically convert an intractable condition into one at the top of the list and worth consideration.

Stay with traditional development if you wish. It is familiar. It may work for you. There are two risks. One, without understanding risk-normalized profit potential and applying it trade-by-trade, it will be difficult to manage trading if or when economic conditions change and performance declines. Two, machine learning develops better systems. Hands down better. Always better. No single decision tree model is ever at the top of a run where more capable models were included as alternatives. No Kaggle competition is ever won by a decision tree. In machine learning, simpler is not always better -- in fact, it is seldom better. The winner of the Netflix million dollar prize was a very complex ensemble. Read about the competition and some of the models that did well here: http://blog.echen.me/2011/10/24/winning-the-netflix-prize-a-summary/

Back to my first piece of advice -- begin by reading Kahneman.
https://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555/

Thanks for listening.

Best, Howard

 

[Trembling Hand]

Howard I think it would go 1000 steps in the right direction if you could post a simple decision tree model result and then an example of an improvement or better approach with machine learning. It's hard for someone to see new technology as better when you cannot see the new tech or how it is applied.

I believe if you started this thread with an example we wouldn't have 384 posts of bitching and endless speculation about possibilities. We would have moved far further down the road but we are stuck at the denial stage....... still.

 

[Skate]

Howard I think it would go 1000 steps in the right direction if you could post a simple decision tree model result and then an example of an improvement or better approach with machine learning. It's hard for someone to see new technology as better when you cannot see the new tech or how it is applied.

I believe if you started this thread with an example we wouldn't have 384 posts of bitching and endless speculation about possibilities. We would have moved far further down the road but we are stuck at the denial stage....... still.

Trembling Hand you nailed it.

"if you could post a simple decision tree model result and then an example of an improvement or better approach with machine learning"

Exactly what I was asking in post #357 & #359
"Dr Howard Bandy uses languages like Python, ML and AI while at the same time having a deep understanding of Amibroker Formula Language and as a fence sitter I’m not fully convinced that either software has an advantage over the other but if there is I would like to see evidence"

 

[Lone Wolf]

Two, machine learning develops better systems. Hands down better. Always better.

It may be true that a machine learning system written by an expert will outperform a standard decision tree model. However, most of us are not experts in the field of machine learning. A more important question for a public trading forum might be whether the average Joe would be more profitable using a decision tree model or developing their own ML system.

Most people work full-time jobs in unrelated fields, they first need to learn a programming language, then learn basic ML techniques. What are the chances of them using that basic level of competence to produce a trading system that significantly outperforms what they could have achieved through the decision tree model?

It could also be part of the reason for resistance to the idea that machine learning is better. ML is beyond most people, or so they believe. People want to believe that the path they can take is just as good as the path they can't take.

 

[tech/a]

Perfect
Can't agree more
"like"

 

[howardbandy]

Greetings --

It is difficult to make a direct comparison between a system using a decision tree model developed using a traditional platform such as TradeStation or AmiBroker and a system using machine learning even if that model is decision tree. Difficult because of the focus on indicator and rules versus focus on target.

-------------------

Using a traditional platform and a common mean reversion entry using the RSI indicator and impulse signals, the model looks like this in an AmiBroker or TradeStation-like pseudocode:
RSILookback = 2
RSIValue = RSI(C,RSILookback)
EntryLevel = 30
ExitLevel = 40
Buy = Cross(EntryLevel,RSIValue)
Sell = Cross(RSIValue,ExitLevel)

The result is a series of trades identified by the impulse signals Buy and Sell.
There will be about 50 trades each year, each holding between 1 and 6 days.
When these trades are analyzed, there will be three points at which risk can be measured: Closed trade, Mark-to-market daily, Intra-day.
Since most people think in terms of daily data and daily accounting, Mark-to-market daily risk is the most appropriate metric. But that is relatively difficult to use when trades are held more than one day.

We can use state signals -- beLong and beFlat -- without loss of accounting generality. That gives a signal for every day so an n-day trade can be manipulated as if it is n 1-day trades. This helps a lot and I recommend everyone use state signals even if they decide to stay with traditional platform.

The fitting process consists of varying the values of the three parameters: RSILookback, EntryLevel, ExitLevel.
Find the best fit in-sample, test out-of-sample. Then analyze the OOS trades as the best estimate of future performance. Safe-f and CAR25 can be computed for these trades.

-----------------------

Using a machine learning program, the process looks like this:
RSILookback = 2
RSIValue = RSI(C,RSILookback)
Target = ref(C,1) > C
RSIValueY1 = ref(RSIValue,-1)
Model = DecisionTree

Pause for explanation before proceeding.

The target is True if tomorrow's close is higher than today's close. That is the beLong state signal.
RSIValueY1 is the value of the RSIValue yesterday. It is given its own variable because the data points will not be processed in sequence. During fitting, there is no memory of yesterday's value. If it is important, it must have its own variable. Each data point must be self contained and independent of the others.
Note there is no 30 or 40. If the fitting process identifies that entering when the RSIValue drops through 30, the data point for those bars will have RSIValue <30 and RSIValueY1 >30. We could force the comparison to 30, but that would remove much of the advantage of using the machine learning. We want the model to identify the pattern. To pick the signal out of the noise.

The data that is important for the discussion is, at this point, two predictor columns -- RSIValue and RSIValueY1, and one column of Target. Say there are 252 daily data points and the system behaves as if it is stationary for at least one year.

Split the data. Store the predictors in a two dimensional array named X. It has 252 rows, 2 columns.
Store the target in a one dimensional array named y. It has 252 rows, 1 column.

For a given index -- the index of any row is taken from the date -- row i of X and row i of y correspond. y is the known target.

Split both X and y through a randomizing process on the indexes. Separate about 60% of X and y to use to train the model. Call the X portion XTrain and the y portion yTrain. Let the machine learning function that implements decision tree find the best fit of XTrain predicting yTrain.

Save the model in an array named A. A is essentially the coefficients of the solution to the linear algebra that fits XTrain to yTrain.

Call the portion of the X array that was left over when XTrain was removed XTest. Call the y portion yTest.

Validate the model out-of-sample by applying the A matrix to XTest and have it predict what y would be according to the model. Call those y values yPredict. There is a state for every bar, day, row. If the target is 1, the state is beLong, otherwise the state is beFlat.

The metric for goodness of fit is comparison between known yTest and predicted yPredict. There are many metrics that can be used. Confusion matrix is one. Area under the ROC curve is another. Or the distribution of the rows of yPredict that have the value 1 -- beLong -- can be used directly to calculate risk (safe-f) and profit potential (CAR25).

----------------------

The difficulty in comparison is that the trading system disappeared when it was translated into a machine learning application. We can still test several values for RSILookback, but not for 30 and 40.

Let me say that again. When a trading system is developed using machine learning, the rules have disappeared. We are looking for patterns the precede desirable trades. There is no direct comparison.

-------------------

The "Quantitative Technical Analysis" book goes through all this in much more detail with charts, diagrams, and more detailed explanatins. It also provides Python code that does what I have described using a logisic regression model. The chart on page 382 shows some results -- XLV daily 2010 through 2014 has a TWR of about 2.4 traded at full fraction.

Additionally, read the chart on page 403 that compares OOS trades with buy and hold. And the charts on pages 416 and 417 that show the effect of using safe-f to manage trading day-by-day.

------------------

I hope this helps rather than clouds the discussion. It should illustrate the process and also illustrate the difficulty of making an apples to apples comparison. The Python code in the QTA book can be downloaded and run.

Best regards, Howard

 

[howardbandy]

It could also be part of the reason for resistance to the idea that machine learning is better. ML is beyond most people, or so they believe. People want to believe that the path they can take is just as good as the path they can't take.

These are some of the points Kahneman makes as well. We look for confirmation of what we already believe. When faced with a difficult question we substitute an easier question.

Best, Howard

 

[howardbandy]

It may be true that a machine learning system written by an expert will outperform a standard decision tree model. However, most of us are not experts in the field of machine learning. A more important question for a public trading forum might be whether the average Joe would be more profitable using a decision tree model or developing their own ML system.

Most people work full-time jobs in unrelated fields, they first need to learn a programming language, then learn basic ML techniques. What are the chances of them using that basic level of competence to produce a trading system that significantly outperforms what they could have achieved through the decision tree model?

We are all trading in the same arena. James Simons and Goldman Sachs take their side of the trade without asking whether we are experts or novices.

It is hard to compare directly, but subjectively, if a machine learning trading system scores 100 arbitrary points, traditional platform decision trees seldom score above 70 or 80.

Stay with what you know and like. But consider implementing daily mark-to-market accounting, use state signals in your trading model code, analyze the recent trade result about once a week, adjust safe-f based on recent trade results, compute CAR25 regularly, and take any system whose CAR25 drops (say to single digits) offline.

Best, Howard

 

[traderxxx]

id just like to say that you dont need powerful fast programes to trade,
and like SKC i wouldnt be without my calculator,
some of my little algo/formulars are as simple as abc,
or i should say a+b+c
having said that i would still like to be working with a programmer to
help keep my too many systems sorted out.

 

[Value Hunter]

Frankly I don't know how fundies make a buck today and I've not seen any evidence on this forum that any one of the many here ----are.

tech/a are you simply not believing that people like Craft, Value Collector, ROE, etc are making huge returns when they post return figures or talk about their positions? Do you expect them to send snapshots of bank statements/brokerage statements to a stranger on the internet? Why do successful fundamental investors on the forum have to prove anything to you or anybody else? The forum is not a dick swinging contest.

If somebody seems credible based on their post count and length of time on ASF, the quality of their writing/analysis and the stock picks/comments they publicly made in the past over a long period of time can't you just give them the benefit of the doubt that they are being truthful about the returns they are making?

 

[Triathlete]

are making huge returns when they post

Just so we can make our own comparison what are you calling Huge returns , and in that case what do you call a small and medium return and over what timeframe.

 

[Value Hunter]

Well its of course subjective but a good example is in the SMSF thread https://www.aussiestockforums.com/threads/smsf-returns.25070/page-2

Craft posted that his 9 year CAGR for his super fund was 34.4%. Now admittedly the post was in 2013 but I am sure if you asked Craft for an update on his SMSF it would still be powering along.

Or what about Value Collector who bought a large number of shares in Capilano Honey at under $3.00 per share and still owns them today (now $16+ per share)? Or the big win he had on Fortescue metals? Or ROE buying shares in Dominos Pizza under $4 and holding it until it was a 10+ bagger? And ROE buying Credit Corp at under $1.00 and still owning it today when its at $17+

If you follow the posts of the top fundamental investors over time you can get a sense of how well they are doing. You know stuff like that is likely to be true because the posting occurs in real time, for example ROE buying Credit Corp at under $1.00 and posting about how he bought some and why it is a good buy and then later coming in thread and saying he still owns it.

 

[Value Hunter]

Of course looking at a random assortment of an investors positions does not give you the full story but it certainly gives you a good glimpse.

 

[minwa]

Cool, now we have examples of both "dinosaur" methods (P2's a nice representation of a technical method) - still waiting on example from the supposed superior ML/AI method. And please don't use Renaissance/Goldman, 99.9% of us here can not relate to an institution with billions dollar worth of resources.

 

[howardbandy]

I give up.

 

[tech/a]

Howard

I would too
Clearly some don't take the time to read
Your posts-----from one who does a huge thanks.

Value Hunter.
We've all had multi bagged wins.ive had many over the years.
I'd be very surprised if any Fundamental trader here is clearing 25%
Or more----consistantly year in year out.
Particularly if they have a portfolio of over 10 stocks.

 

[Trembling Hand]

Can we cut and move the sh!te out of this thread?