# Tools being used by your trading competition



## howardbandy

Greetings --

I have long encouraged individual traders and small trading companies to improve their analytical skills, particularly in areas of mathematics, programming, modeling, and simulation.  The link at the bottom of this posting is a presentation by Stephen Simmons at one of the professional conferences held in 2016 related to use of the Python language.  Stephen is a developer working in the commodities trading area of J. P. Morgan in London.   

His talk is about Pandas, a Python library developed by Wes McKinney, formerly with Cliff Asness' hedge fund, AQR Capital, about five years ago.  Pandas is central to analysis of time series data, and is a popular topic at these conferences and seminars.  If you are planning to learn to develop systems using Python, and perhaps machine learning, you will need to learn about Pandas -- but begin with basic Python first.  

The first two minutes of the video are the part I recommend everyone hear.  Stephen describes that there are several thousand developers working with Python in J. P. Morgan.  Add Chase, Citi, ANZ, Goldman Sachs, Shaw, Renaissance, Simons, and hundreds more we seldom hear about.

As I write in my "Foundations" book, the days of chart reading, long term holding, and simple trading algorithms are over.  The business of trading is changing with astonishing speed.  It is now about applied mathematics, machine learning, Bayesian statistics.  Traders without skills in math, programming, statistical analysis, and scientifically developed trading techniques are at a severe disadvantage.  Stephen and his colleagues will "eat the lunch" of unprepared traders. 

https://www.youtube.com/watch?v=CowlcrtSyME

Best regards,  Howard


----------



## minwa

howardbandy said:


> As I write in my "Foundations" book, the days of chart reading, long term holding, and simple trading algorithms are over.  The business of trading is changing with astonishing speed.  It is now about applied mathematics, machine learning, Bayesian statistics.  Traders without skills in math, programming, statistical analysis, and scientifically developed trading techniques are at a severe disadvantage.  Stephen and his colleagues will "eat the lunch" of unprepared traders.




Haven't watched the video so can't comment on that.

Can't see how chart reading, long term holding, and simple trading algorithms are over though. Perhaps in the large capital funds world, but certainly not in the retail capacity. Statistics from fundseeder of aspiring retail traders shows discretionary being the highest return group with systematic being the lowest style. This is average from the top 10% percentile of funds so it is not dragged down by wannabe coders testing systems.

I would say your conclusion is very biased. That being said - if one could replicate discretionary trading with maths and science and extract the same results, I would definitely jump on the wagon.


----------



## Wysiwyg

minwa said:


> Can't see how chart reading, long term holding, and simple trading algorithms are over though.



Agree. Patterns are ever present and they, as usual, either play out or not.


----------



## tech/a

Minwa / Wysiwyg 

Your replies are tantamount to telling Captain Cook the worlds Flat.


----------



## DaveDaGr8

Isn't that Columbus in the 1500's ? and he turned out to be right !!!

Be that as it may, i don't disagree with Howard, but it hasn't happened yet. I have several simple systems that have worked just as well this year as when backtested 10 years ago, with the exception of US Swing trading, which has shown a gradual reduction in profits, but it is still profitable. 

I guess the old adage, show me a system that has failed or is showing signs of failing so we have more context to go on.


----------



## Wysiwyg

tech/a said:


> Minwa / Wysiwyg
> 
> Your replies are tantamount to telling Captain Cook the worlds Flat.



On the contrary. Capt. Hook won't need a weather man to know which way the wind blows.


----------



## ThingyMajiggy

I doubt me or any systems I were to make with my 300ms latency is going to be able to compete with thousands of Python developers at JP Morgan(plus the rest of the banks/funds) that are no doubt directly next to the exchanges. Plus I suck at Maths  

Oh and I'm yet to see a systems trader OR discretionary trader last, from the time I've joined ASF to now. They've either blown up, lost their edge, stopped because of the stress/lifestyle. Seems to be just a matter of time. I've heard about a few that have lasted for years, or heard interviews etc. but never actually seen or know of one personally, or from here etc.


----------



## Porper

ThingyMajiggy said:


> Oh and I'm yet to see a systems trader OR discretionary trader last, from the time I've joined ASF to now. They've either blown up, lost their edge, stopped because of the stress/lifestyle. Seems to be just a matter of time.




I know a few on here that are consistently profitable...both systems and discretionary. That doesn't mean to say they are profitable every month, or even every year. Depends what time frame you use.


----------



## ThingyMajiggy

Porper said:


> I know a few on here that are consistently profitable...both systems and discretionary. That doesn't mean to say they are profitable every month, or even every year. Depends what time frame you use.




Yeah I've seen some profitable trades and profitable statements as well, but they never seem to last. But if you have then great 

I can only speak for myself and what I've seen or not seen 

Anyway this is heading in the off-topic direction so I'll shoosh


----------



## Boggo

ThingyMajiggy said:


> ....
> 
> Oh and I'm yet to see a systems trader OR discretionary trader last, from the time I've joined ASF to now. They've either blown up, lost their edge, stopped because of the stress/lifestyle. Seems to be just a matter of time. I've heard about a few that have lasted for years, or heard interviews etc. but never actually seen or know of one personally, or from here etc.




That's a big statement.

Just because they are not on here getting involved in the production of pages of reasons why Enron style data doesn't fit in with the current stock price doesn't mean they don't exist, they are quietly getting on with their own processes.

They do exist ThingyMajiggy


----------



## ThingyMajiggy

Boggo said:


> That's a big statement.
> 
> Just because they are not on here getting involved in the production of pages of reasons why Enron style data doesn't fit in with the current stock price doesn't mean they don't exist, they are quietly getting on with their own processes.
> 
> They do exist ThingyMajiggy




I hope you're right, believe me. I'm not saying its never been done or impossible, seen plenty of profitable stints and moments of wow, I've just never seen it sustained over long periods or know of a very successful trader personally. I know a few traders(and plenty who have attempted it!) and had a stint at a prop shop, seems they're all just getting by at best. But I guess that's all part of it, only the strong survive and getting by is good enough  Time will tell I guess, if Howard's suggestion that maybe all this discretionary stuff isn't good enough anymore, we'll see. 

Machine Learning/AI certainly seems to be the hot topic in the trading world of recent times.


----------



## Boggo

This book is worth a read for anyone interested in the topic.

Some articles by the author...
http://www.futuresmag.com/author/kevin-j-davey


----------



## Wysiwyg

howardbandy said:


> Traders without skills in math, programming, statistical analysis, and scientifically developed trading techniques are at a severe disadvantage.



That to me is saying there will no longer be trends and range trading will be the only game in town. Like supply and demand of commodities or widgets will no longer influence the market price.


----------



## minwa

ThingyMajiggy said:


> Machine Learning/AI certainly seems to be the hot topic in the trading world of recent times.




Manipulation of how the big guys keep their edge. You spread distraction to misdirect. To keep the retails chasing the wrong stuff fed by the media of the newest colorful trendy algos/quants and all that stuff. Feed the hope that a struggling trader can buy a few courses and buy some softwares for a few thousand and compete with the funds at their own game against their billions of dollars invested into engineers/hardware/technology. Follow the herd, stay in the 90% losers.

There definitely are successful retail guys doing it out there - just like there are chart readers/fundamentalists long term holders. Doesn't change anything. Price is price, human psychology is the same. There are 90% losers in trading 50 years ago, there are still 90% losers today with all the improvement in technology as said by some market wizard (forgot who).


----------



## ThingyMajiggy

minwa said:


> Manipulation of how the big guys keep their edge. You spread distraction to misdirect. To keep the retails chasing the wrong stuff fed by the media of the newest colorful trendy algos/quants and all that stuff. Feed the hope that a struggling trader can buy a few courses and buy some softwares for a few thousand and compete with the funds at their own game against their billions of dollars invested into engineers/hardware/technology. Follow the herd, stay in the 90% losers.
> 
> There definitely are successful retail guys doing it out there - just like there are chart readers/fundamentalists long term holders. Doesn't change anything. Price is price, human psychology is the same. There are 90% losers in trading 50 years ago, there are still 90% losers today with all the improvement in technology as said by some market wizard (forgot who).




Is that you or your mentor talking though? It was mentioned earlier in this thread about biases.


----------



## minwa

ThingyMajiggy said:


> Is that you or your mentor talking though? It was mentioned earlier in this thread about biases.




Don't see how that matters ? Still same point being made. My bias is towards the contrarian yes. I didn't say science/maths that Howard proposes can't make money. It has similar odds like chart-reading/long term holding. The common denominator is not actually which method..which is why saying those method's days are over and will get eaten for lunch is really just nonsense. My guess is Howard worded it a bit too over passionate.


----------



## VSntchr

ThingyMajiggy said:


> I hope you're right, believe me. I'm not saying its never been done or impossible, seen plenty of profitable stints and moments of wow, I've just never seen it sustained over long periods or know of a very successful trader personally. I know a few traders(and plenty who have attempted it!) and had a stint at a prop shop, seems they're all just getting by at best. But I guess that's all part of it, only the strong survive and getting by is good enough  Time will tell I guess, if Howard's suggestion that maybe all this discretionary stuff isn't good enough anymore, we'll see.



I know quite a few successful traders. How long is long enough to classify as a sustained long period?


----------



## ThingyMajiggy

VSntchr said:


> I know quite a few successful traders. How long is long enough to classify as a sustained long period?




You know them personally and seen the results/rewards of their successful trading? and that's all they do? Or know some people on the internet who say that's what they do? Hmm I would say 5 - 10 years, more like 10 though.


----------



## Gringotts Bank

There's another aspect to 'your trading competition'.

I recently spoke to a project manager for IT security and he said nothing can truly be made secure in the online world.  Triple firewalls can be penetrated in seconds.  The best and brightest in the US cannot stop some of the attacks coming out of Russia & Eastern Europe.

Consider this - Zuckerberg puts tape over his laptop microphone and webcam, because it's _that _easy for hackers to get access and listen/watch his every move.

Also this - Lenovo laptops concealed special chips designed to collect and transmit data to China.  http://www.darkreading.com/risk-man...pcs-after-chinese-acquisition/d/d-id/1110950?

So your charts, systems, trades, keystrokes, voice, phone calls, etc etc.  ... all hack-able.  Not really a problem for me, but if you have a big account and fancy security on your PC, they see that as a target.  In fact this guy said the more fancy your security systems, the more interested they are.  They know there's something of value there.

I've mentioned this before, but talented hackers could enter big brokerages, find the best equity curves and front run every move.


----------



## ThingyMajiggy

Gringotts Bank said:


> There's another aspect to 'your trading competition'.
> 
> I recently spoke to a project manager for IT security and he said nothing can truly be made secure in the online world.  Triple firewalls can be penetrated in seconds.  The best and brightest in the US cannot stop some of the attacks coming out of Russia & Eastern Europe.
> 
> Consider this - Zuckerberg puts tape over his laptop microphone and webcam, because it's _that _easy for hackers to get access and listen/watch his every move.
> 
> Also this - Lenovo laptops concealed special chips designed to collect and transmit data to China.  http://www.darkreading.com/risk-man...pcs-after-chinese-acquisition/d/d-id/1110950?
> 
> So your charts, systems, trades, keystrokes, voice, phone calls, etc etc.  ... all hack-able.  Not really a problem for me, but if you have a big account and fancy security on your PC, they see that as a target.  In fact this guy said the more fancy your security systems, the more interested they are.  They know there's something of value there.
> 
> I've mentioned this before, but talented hackers could enter big brokerages, find the best equity curves and front run every move.





"Hackers" being your own Government using it's compulsory backdoors in devices/software, in the case of Zuckerberg  plus he's kind of a big target.  

Hackers just expose a flaw that is already there, stop listening to the mainstream media scare tactics about hacking. It's actually not THAT easy, and there are a lot of GOOD hackers, it's not all NCIS Russia and China are the baddies nonsense. In 99% of the cases you, the human, is the weakest link.(bad passwords, phishing scams etc.)

As far as hackers frontrunning the nice equity curves, again sounds more like something out of Hollywood, you'd still need the faster connection to frontrun, to somehow know HOW they're trading, when they're executing, all of which is very hard to get from an equity curve.


----------



## Gringotts Bank

ThingyMajiggy said:


> As far as hackers frontrunning the nice equity curves, again sounds more like something out of Hollywood, you'd still need the faster connection to frontrun, to somehow know HOW they're trading, when they're executing, all of which is very hard to get from an equity curve.




I don't think it's fantasy at all, especially after talking to this expert.  You'd identify the individuals/companies who own the equity curves, and just intercept orders at an earlier point, like at the ISP.  Or, hack the PC that houses the source code for the system.

Anywhere there's big money, there's an attempt to collect sensitive data illegally.  I never said it was an issue for us small fish.  Imagine knowing that Soros had made a decision to aggressively short the pound?  Or that GS had decided to dump Apple?


----------



## OmegaTrader

Related to the whole Quants are winning debate:


It has also fascinated me that one of the richest men in the world is a buy and hold investor with no stock quote machine!

All he does is conservatively value a company and its prospects and buy it at bargain prices.

No algorithms, no charts, no fancy computers

Just buying bargains......


He has been around for a long time and get special deals, 

but shouldn't the richest man in the world be a Quant???

That never made sense to me.


----------



## Wysiwyg

Be interesting to see an example of these tools but that is most likely top secret as not many reveal their trading success method.


----------



## tech/a

Oh my have you really thought about this?



OmegaTrader said:


> Related to the whole Quants are winning debate:
> 
> It has also fascinated me that one of the richest men in the world is a buy and hold investor with no stock quote machine!
> 
> All he does is conservatively value a company and its prospects and buy it at bargain prices.
> 
> No algorithms, no charts, no fancy computers
> 
> Just buying bargains......




Buffett BUYS COMPANIES---*all of the company*.
Then transforms them and if you think he does this* WITHOUT *the input of a team of Quants then think again!



> He has been around for a long time and get special deals,
> 
> but shouldn't the richest man in the world be a Quant???
> 
> That never made sense to me.




The world is run by Quants.

Aerospace
Aeronautics
Medicine
Technology
Engineering
Climate control and the lack of it
Mining
Energy production
Military

*FINANCE*

The point Howard (Captain Cook) is making is that all of those looking out at the vast sea in front of them
can rest assured that the world isn't flat and he knows as unlike those staring out at the sea in front of them
he has been there --- that's where he lives---he sails it.




Wysiwyg said:


> Be interesting to see an example of these tools but that is most likely top secret as not many reveal their trading success method.




Examples are all around you.
*The problem is most are so blind they cannot see.*
Just because you cant see it doesn't mean its not there doing exactly what its supposed to do for exactly the reason its supposed to and for exactly the people its meant to serve.

Ever been on a plane?
Seen a Bank make 10 Billion?
Driven over a bridge?
Heard of the name Zuckerberg?
Seen the growth of google?
Watched an invasion? 

Yeh Quants rule the world.
The richest men in the world are surrounded by Quants
They employ 1000s of them.
----and pretty soon you'll be able to join them---even you Thingamiggy
There is a vast wave of Howard Bandies on the horizon.


----------



## ThingyMajiggy

Gringotts Bank said:


> I don't think it's fantasy at all, especially after talking to this expert.  You'd identify the individuals/companies who own the equity curves, and just intercept orders at an earlier point, like at the ISP.  Or, hack the PC that houses the source code for the system.
> 
> Anywhere there's big money, there's an attempt to collect sensitive data illegally.  I never said it was an issue for us small fish.  Imagine knowing that Soros had made a decision to aggressively short the pound?  Or that GS had decided to dump Apple?




Haha you've seen too many movies GB. Yes, technically if it's connected to the internet it can be hacked, but it's highly unlikely, for something to be hacked it needs an actual vulnerability to begin with, a buffer overflow or remote code execution or some kind of weakness in the code, you can't just "hack" things like they do in movies, there needs to be an actual weak point that can be exploited, all these hacks you hear about in the news are 90% social engineering, someones account got hacked, or some website got hacked and data was released, pretty much always social engineering, tricking someone into thinking you're someone who should have access, or building a replica fake site that looks exactly like the one they have to login to, where you then intercept their login details, a MITM attack. 

In a up to date system, which if a brokerage and ESPECIALLY an ISP DIDN'T have would be incredibly stupid, there isn't much chance of it being hacked. It's just not that easy, nearly everything is encrypted these days, everywhere you login, everything you transfer, so the different points at which you would need to gain access and/or intercept would be insane. 

Then you have honeypots and firewalls and tons of other stuff to make it even harder, then once they're through all of that, to be able to find Soro's position on the pound is just not possible, however juicy in our minds/imagination. It would probably be easier to hack Soros himself than hack the brokerage he uses and the ISP. 

Us small fish are actually the ones that are probably more vulnerable because the big fish and especially ISP's would make sure they have decent security simply because of the funds/knowledge they have. People and pathetically easy passwords are the weak part, not the systems themselves connected to the internet.

Plus we kind of can tell when big positions are put on, it's called volume isn't it?


----------



## So_Cynical

howardbandy said:


> G
> a presentation by Stephen Simmons at one of the professional conferences held in 2016 related to use of the Python language.  Stephen is a developer working in the commodities trading area of J. P. Morgan in London.
> 
> His talk is about Pandas, a Python library developed by Wes McKinney, formerly with Cliff Asness' hedge fund, AQR Capital, about five years ago.  Pandas is central to analysis of time series data, and is a popular topic at these conferences and seminars.  If you are planning to learn to develop systems using Python, and perhaps machine learning, you will need to learn about Pandas -- but begin with basic Python first.




Python is a programming language that has been around for a while, a good language but just a language.

numpy is an extension of that language that allows for greater number crunching and a few other things.

They are not magic formulas, they simply allow for very complex programs to run, computers to do stuff.


----------



## Wysiwyg

tech/a said:


> The world is run by Quants.
> 
> Aerospace
> Aeronautics
> Medicine
> Technology
> Engineering
> Climate control and the lack of it
> Mining
> Energy production
> Military



Engineers have been around for thousands of years. Can you be specific with a quantitative analysis example?


----------



## tech/a

Wysiwyg said:


> Engineers have been around for thousands of years. Can you be specific with a quantitative analysis example?




Yes but we aren't talking about engineering.
We are talking about analysis of data to be able to quantify a decision or group of decisions.

Not available in the Roman era.

Lets say I want to build a bridge.

I have 

Geotech data
Product data
Costing data
Weather data
Capability data
Past data
Traffic and load data

I want to analyse all of that data to give me the best possible information relative to the type of bridge I'm going to design and build.

EG
Beam/Arch or perhaps a Cantilever type bridge

Hell I might even look at a Tunnel alternative!

Maybe the Pyramids would have been better designed as a underground sealed tomb!
Rome may have been built sooner.

Quats will/can give me that information.


----------



## Wysiwyg

tech/a said:


> There is a vast wave of Howard Bandies on the horizon.



Yes I understand Howard is running  sentiment analysis on ASF posters now.


----------



## tech/a

Wysiwyg said:


> Yes I understand Howard is running  sentiment analysis on ASF posters now.




They're not all going by the name Howard Bandy.
And I think they are all more sophisticated than that---


----------



## Wysiwyg

tech/a said:


> I want to analyse all of that data to give me the best possible information relative to the type of bridge I'm going to design and build.
> 
> 
> Quats will/can give me that information.



I believe it is simply data building on data. Nothing new, just more people have access to that information than ever before.


----------



## tech/a

Wysiwyg said:


> I believe it is simply data building on data. Nothing new, just more people have access to that information than ever before.




Data by itself is meaningless.

Firstly what data is required to determine
What result
What result is required to be meaningful and useful.

If you don't know how to travel the road you won't know what it is that is at it's end.

So it's my belief that the masses looking out to see have no idea what it is they are looking at and looking for.

So grabbing a boat is pointless.
Stick Captain Cook in the boat and you might actually find something worth while.
Better still embrace Cooks vision ( That the world may not be flat---even though he knows it isn't ) and set about proving it!


----------



## minwa

So tech back to the original statement that started the debate - how come your analysis is full of charts instead of numbers/stats/program language if charting days are over as deemed by Captain Cook


----------



## tech/a

minwa said:


> So tech back to the original statement that started the debate - how come your analysis is full of charts instead of numbers/stats/program language if charting days are over as deemed by Captain Cook




Took a while to rotate here!

I still trade discretionary and still do ok.

But I have for the last 2 yrs been the venture capital partner in a company 
Which is at the forefront of jumping off the edge.
I've had the opportunity of watching these guys on a daily basis 
I've watched in aw as they write code ( python ) as quick as my
P/A types reports and letters.

I've learnt and am still learning at an exponential rate.

I've posted nothing here as I have nothing to post.
But all will become clear-er.


----------



## minwa

tech/a said:


> Took a while to rotate here!
> 
> I still trade discretionary and still do ok.
> 
> But I have for the last 2 yrs been the venture capital partner in a company
> Which is at the forefront of jumping off the edge.
> I've had the opportunity of watching these guys on a daily basis
> I've watched in aw as they write code ( python ) as quick as my
> P/A types reports and letters.
> 
> I've learnt and am still learning at an exponential rate.
> 
> I've posted nothing here as I have nothing to post.
> But all will become clear-er.




OK thanks that puts things into perspective. No problem that you think that way's the future, an opinion.

My only problem with the first post was that Howard stating that was the only way to go and the other ways don't work.


----------



## tech/a

minwa said:


> OK thanks that puts things into perspective. No problem that you think that way's the future, an opinion.
> 
> My only problem with the first post was that Howard stating that was the only way to go and the other ways don't work.




My personal opinion is that it's no harder than it has been
But would I trade 500k or more in a discretionary way--my way--no-- not yet knowing what I know.

There is a lot more to this topic than long term trading / investing.

I think that a discretionary trader can/should benefit from the information they can gain from quant analysis of the data and methodologies they employ or are thinking of employing.

What you can add and subtract with confidence from your discretionary trading.
It's helped me already but I'm still on the beach!


----------



## Gringotts Bank

This video will appeal to the die-hard discretionary traders.  May their edge never be eroded by AI.  From *9:27 *especially.  

TLW  Turning off your higher level brain processes may confer an advantage in trading.


----------



## Gringotts Bank

That should read TL;DW 

So how to switch off the analytical mind?  Play computer games until your alert gets triggered, switch screens and decide in an instant?  That might be one way.  I know personally that when I've attempted discretionary trading that the longer I take to make my decision, and the more factors I take into consideration, the less likely I will make a winning trade.


----------



## tech/a

Gringotts Bank said:


> That should read TL;DW
> 
> So how to switch off the analytical mind?  Play computer games until your alert gets triggered, switch screens and decide in an instant?  That might be one way.  I know personally that when I've attempted discretionary trading that the longer I take to make my decision, and the more factors I take into consideration, the less likely I will make a winning trade.




I agree

How do you trade G/B?

I've asked before
What do you do for work?


----------



## Gringotts Bank

tech/a said:


> I agree
> 
> How do you trade G/B?
> 
> I've asked before
> What do you do for work?




I trade with systems, and every now and then I try my hand at discretionary.  I'm not very good at disc. though.

Prefer not to say re work.


----------



## VSntchr

ThingyMajiggy said:


> You know them personally and seen the results/rewards of their successful trading? and that's all they do? Or know some people on the internet who say that's what they do? Hmm I would say 5 - 10 years, more like 10 though.



Yes I know a couple of traders personally that have been successful over a 10yr+ period. There is also at least one ASF poster that made the transition from trader to investor after enough success. My point being that there are plenty of traders out there that are doing better than 'just getting by', at least in my own opinion. 

I also know a lot more that have been successful over much shorter time frames (<3yrs) - so the indication that success over short periods is far more common has merit. I would, however, expect this to hold true in many ventures which require ongoing self-motivation and growth mindset. I also know many other business owners that were successful for a year or two before giving up (cafes, swimwear, imports, graphic design, etc etc...).

Part of retaining an edge in the market involves constantly adapting. Floor traders from the past had to adapt to electronic trading..the ones who were motivated to succeed did the work and made the transition, others probably didn't and moved on to something else. Perhaps electronic discretionary traders will have to adapt to incorporate more quantitative work into their process in order to retain edge going forward.


----------



## howardbandy

Greetings --

My posting was intended to show the extent to which artificial intelligence and machine learning is being used by large trading houses.  I am surprised by the messages posted in response.  They cover a wide range, and many imply denial.  Let me make a few comments.  

None of my comments are intended to be critical of individuals or their trading and investing techniques.  If something works for you, ignore me and keep it up.

1.  Some comments suggest that fundamental analysis is more important than quantitative / technical / statistical analysis.

In the 1990s I was hired by a company based in a southern US state.  They wanted a stock selection and signal generation program that could be sold to individual investors / traders that used fundamental data for the stock selection portion.  I was vice-president for research and in charge of the project.  Years earlier -- late 1960s and early 1970s -- one of my projects as a graduate student had been a stock selection computer program, based on artificial intelligence, that worked.  So I thought we might succeed.  We tried everything we could think of, including analysis of a variety of corporate fillings.  It was during this period that my analysis of the use of fundamental data crystallized.  A recent version of that has been published several times, and is an appendix to the "Foundations" book.  The paper can be downloaded from the book's website:
http://www.blueowlpress.com/wp-content/uploads/2016/10/FT-Fundamental-Analysis-Appendix-A.pdf

During that period, a professional colleague was working for a highly respected research organization that published (at a very high subscription rate) buy / hold/ sell recommendations.  His mother lived in a retirement home in my community.  On his regular visits to her, he stopped in my office and we talked about my project.  His summary when I asked him "What works?" was "Nothing we have found.  That is, nothing outside the board room."  

My group could not find anything either, and the stock selection portion of that project was dropped.

2.  The world's richest person uses buy and hold.  I am assuming the reference is to Warren Buffett.  

Mr. Buffett is as much the president of a conglomerate corporation as he is an investor.  By his own assessment, he is not a trader.  The price of the Buffett-led Berkshire Hathaway stocks (BRK-A and BRK-B) dropped over 50% from early 2008 to the lows in 2009.  In my opinion, holding resources through losses that large is more a characteristic of a corporation than of an investor or a trader.    

The 50% drawdown was recovered.  In my opinion, the recovery in equities, bonds, and real estate from that "crisis" has been the result of low interest rates set by central banks.  There is a very high risk that as interest rates rise, the majority of those gains in all three areas will be lost.  We are already seeing some of the effect of rising interest rates on bonds.  

3.  Productivity gains will improve company profits and raise all boats.  

Several political economists have pointed out that the "developed" countries benefited from recovery following world war II, which provided strong growth and rising values of all investments beginning about 1945.  That increasing globalization has recently been spreading the effects throughout the world, and that we (the US, at least) should not expect strong growth in the 21st century.

Some of the gains in productivity are due to automation.  Those profits accrue to stock holders, but not to workers.  Investors may benefit, inequality may increase. 

4.  Some comments suggest that AI/ML is unnecessary -- that simple techniques work.  

I was a senior research analyst at a commodity trading company (a hedge fund) in the 1990s.  The company had been one of the pioneers in development and use of trend-following quantitative systems.  Their specialty was breakouts -- buy new highs and other "turtle-like" techniques.  We watched, literally day by day, as those techniques stopped working.  In my opinion, they will never work again.  Trends are shorter in duration and price movement.  Systems that take counter-trend positions are quicker to fade the trends, breaking them into smaller pieces which are less easily identified and less profitable.  That company returned all investor money and ceased operation.

But assume that some simple indicator does work today.  Take Stephen Simmons comment about J. P. Morgan employing thousands of individuals to develop trading systems as literal and think through the implications.  We know from many articles that research personnel are highly educated -- one-third or more have PhD degrees in scientific disciplines -- and highly compensated -- say $100,000 base salary on average.  1000 people at $100,000 per year is an annual salary cost of $100,000,000.  Add office space, support staff, benefits, etc and the sum grows.  Morgan will not build those departments and pay those costs unless the trading results are adequate to fund them.  

A trading company does not need 1000 people and a department budget of several hundred million dollars to tell them to buy when the 5 day moving average rises up through the 20 day moving average.

Anyway -- thanks for listening,

Best regards,  Howard


----------



## CanOz

Thanks Howard, great topic and fascinating discussion!


----------



## jjbinks

I think what howard is saying is true and inevitably becomes more true everyday.

I think we continue to trade because we feel that there a gaps left by the bigger players (maybe trades which are "too small" for them or for some other reason). My feeling is that the reason us small fries can trade is because the wins we make is different to the wins big guys trading hundreds of millions of dollars are try to achieve. 

THe other thing which i think wil change the landscape is if neural networks gets implemented into trading. For those of you who follow artificial intelligence, you may be aware of how google developed a AI to beat the worlds greatest "go" player.  My simple understanding was was that the computer was programmed to recognise patterns in game play and develop successful strategies. With this kind of technology i think discretionary trading could be simulated better. If any of you have played go and understand how complex it can be you will understand why I say this.


Anyway that's my 2c

Also if people are interested in learning pandas for python:
https://courses.edx.org/courses/course-v1:Microsoft+DAT210x+6T2016/info
Its a good starter course. (assumes basic python knowledge though)


----------



## Gringotts Bank

howardbandy said:


> 4.  Some comments suggest that AI/ML is unnecessary -- that simple techniques work.
> 
> I was a senior research analyst at a commodity trading company (a hedge fund) in the 1990s.  The company had been one of the pioneers in development and use of trend-following quantitative systems.  Their specialty was breakouts -- buy new highs and other "turtle-like" techniques.  *We watched, literally day by day, as those techniques stopped working.*  In my opinion, they will never work again.  Trends are shorter in duration and price movement.  Systems that take counter-trend positions are quicker to fade the trends, breaking them into smaller pieces which are less easily identified and less profitable.  That company returned all investor money and ceased operation.
> 
> But assume that some simple indicator does work today.  Take Stephen Simmons comment about J. P. Morgan employing thousands of individuals to develop trading systems as literal and think through the implications.  We know from many articles that research personnel are highly educated -- one-third or more have PhD degrees in scientific disciplines -- and highly compensated -- say $100,000 base salary on average.  1000 people at $100,000 per year is an annual salary cost of $100,000,000.  Add office space, support staff, benefits, etc and the sum grows.  Morgan will not build those departments and pay those costs unless the trading results are adequate to fund them.




I believe what you say about many edges eroding, but that doesn't mean you can't trade using a highly intuitive approach.  You can bet there would have been some people at that company who would have had a sense of "I feel like this might not last, but I don't know why" well before the statistics showed up losses.  They would have been well ahead of the curve.

In current day terms, highly intuitive traders may just 'feel' like they need to broaden their discretionary time frame, because it 'feels' safer.  If you asked them to quantify that, they might give you some sentences that describe a general trend, but not necessarily be able to turn that into an equation.

So I think both approaches are viable.  If I had to match a highly intuitive trader versus a machine I'd back human intuition every time.  Such geniuses are rare, but they exist - we may even have a few here.


----------



## Gringotts Bank

To put it another way, the human brain is by far the greatest machine on earth.  I believe a seasoned chartist (say 5-10 yrs experience) can look at a chart and know in a split second what's going to happen next.  What gets in the way of accessing that knowledge is fear and greed.  Everyone knows how easy it is to perform spectacularly well on a demo account.

Not to take anything away from those who can program machines at a high level.  That's spectacular in its own way.  I won't ever be able to compete with those guys in the future... hence the interest in improving my discretionary skills.


----------



## minwa

howardbandy said:


> My posting was intended to show the extent to which artificial intelligence and machine learning is being used by large trading houses.  I am surprised by the messages posted in response.  They cover a wide range, and many imply denial.  Let me make a few comments.




I don't see anyone denied that they are being used by LARGE TRADING HOUSES. Denials are of your claim that charts/longterm don't work for individuals anymore and will get eaten for lunch. That's the impression that I and others got from your first post. 



howardbandy said:


> 4. Some comments suggest that AI/ML is unnecessary -- that simple techniques work.
> 
> I was a senior research analyst at a commodity trading company (a hedge fund) in the 1990s. The company had been one of the pioneers in development and use of trend-following quantitative systems. Their specialty was breakouts -- buy new highs and other "turtle-like" techniques. We watched, literally day by day, as those techniques stopped working. In my opinion, they will never work again. Trends are shorter in duration and price movement. Systems that take counter-trend positions are quicker to fade the trends, breaking them into smaller pieces which are less easily identified and less profitable. That company returned all investor money and ceased operation.




That system must've been short term fitted to rigid rules that worked well in that period. Ed Seykota says all mechanical systems expire, and he's probably right.

Dunn Capital is the most famous fund for break out trend following. His fund is 30 years+ and today is still alive and well. 

Larry Williams won the trading contest in late 80's. 10 years later in late 90's his daughter won it again. Another 10 years later another of his student won again. Today I still use his principals. We read charts and some hold for long term. Core trading principals don't change.  



howardbandy said:


> We know from many articles that research personnel are highly educated -- one-third or more have PhD degrees in scientific disciplines -- and highly compensated -- say $100,000 base salary on average.  1000 people at $100,000 per year is an annual salary cost of $100,000,000.  Add office space, support staff, benefits, etc and the sum grows.  Morgan will not build those departments and pay those costs unless the trading results are adequate to fund them.
> 
> A trading company does not need 1000 people and a department budget of several hundred million dollars to tell them to buy when the 5 day moving average rises up through the 20 day moving average.




Completely agree - for a large trading firm. But most of individuals trading from our ADSL at home in our tiny accounts do not have to play the the same game as the large guys. They have to average in massive amounts of funds/orders. We get filled in an instant. We can go cash in an instant. We take a position into anything in an instant. It is the BEST advantage of the small guys. 

Just because large funds are using all that doesn't necessarily mean we have to. Do you like the odds of going against a team of thousands of high IQ people with hardwares & softwares most of us can't even dream of to beat/mirror them in their game ? A few pure genius can probably do it..most of us can't.

Thanks for your response.


----------



## minwa

ThingyMajiggy said:


> hot topic in the trading world of recent times.







HFT - an example of a trading fad that's seen its glory days gone.


----------



## OmegaTrader

tech/a said:


> Oh my have you really thought about this?
> 
> 
> 
> Buffett BUYS COMPANIES---*all of the company*.
> Then transforms them and if you think he does this* WITHOUT *the input of a team of Quants then think again!
> 
> 
> 
> The world is run by Quants.
> 
> Aerospace
> Aeronautics
> Medicine
> Technology
> Engineering
> Climate control and the lack of it
> Mining
> Energy production
> Military
> 
> *FINANCE*
> 
> The point Howard (Captain Cook) is making is that all of those looking out at the vast sea in front of them
> can rest assured that the world isn't flat and he knows as unlike those staring out at the sea in front of them
> he has been there --- that's where he lives---he sails it.
> 
> 
> 
> 
> Examples are all around you.
> *The problem is most are so blind they cannot see.*
> Just because you cant see it doesn't mean its not there doing exactly what its supposed to do for exactly the reason its supposed to and for exactly the people its meant to serve.
> 
> Ever been on a plane?
> Seen a Bank make 10 Billion?
> Driven over a bridge?
> Heard of the name Zuckerberg?
> Seen the growth of google?
> Watched an invasion?
> 
> Yeh Quants rule the world.
> The richest men in the world are surrounded by Quants
> They employ 1000s of them.
> ----and pretty soon you'll be able to join them---even you Thingamiggy
> There is a vast wave of Howard Bandies on the horizon.





I will be clear.

I am *not *saying Quant analysis is not useful or profitable by itself or with other indicators

I am *not* saying Quants will not eventually win or have not won.

I am *not* saying that Quant analysis is meshed with everyday life.

But what I am saying is that* Buffet is not a Quant.*

He does not use sophisticated algorithms or advanced computers.

Sure he uses data and figures, but I would not consider this to be a Quant.

First year university students learn DCF models and how to read balance sheets,

That is not being a Quant that is basic finance.

Right back at you!!! 

*Oh my have you really thought about this?*


Investing in a Quant field does not make your investment strategy quantitative.

No disrespect to you


----------



## tech/a

> Oh my have you really thought about this?




Yes.


----------



## Wysiwyg

Supply/demand and sentiment drive prices, not computer programs.


----------



## tech/a

Wysiwyg said:


> Supply/demand and sentiment drive prices, not computer programs.




Wysi

How is it your missing the whole point here.
Its about data analysis
Machine learning of changes and anomalies in data.

Would you rather hand fly a plane all night over the Atlantic or set on Auto Pilot?
Jet engines propel planes not computers!

This is the next stage in financial analysis?

According to you then I should only need my Abacus?
Throw away my calculator and forget a PC---not needed.

You sound like a pigeon who's heard that the telephone is the next big thing in communication. 

Its going to help everyone not be their demise!!


----------



## Wysiwyg

tech/a said:


> Wysi
> 
> How is it your missing the whole point here.



To the contrary again. I understand the I.T. era (in trading - data mining for patterns if you like) is here but the same conditions that drive prices have not changed. Trading a million times per day on a nano second turn around doesn't appeal to me.


----------



## tech/a

Wysiwyg said:


> To the contrary again. I understand the I.T. era (in trading - data mining for patterns if you like) is here but the same conditions that drive prices have not changed. *Trading a million times per day on a nano second turn around doesn't appeal to me.*




Ok.
Its not about that either.

Ill leave it there.


----------



## Wysiwyg

tech/a said:


> Ok.
> Its not about that either.
> 
> Ill leave it there.



No please elaborate. An EOD trade example maybe.


----------



## tech/a

Just a snippet of what's out there.

Not associated in anyway and not endorsing either.

https://www.quantopian.com/


----------



## ThingyMajiggy

minwa said:


> View attachment 69083
> 
> 
> HFT - an example of a trading fad that's seen its glory days gone.




Over-saturation of everyone trying to do the same thing when there's only a certain amount of milliseconds available to be exploited, and probably why the "herd" are all onto it now in forums etc. always lagging behind, just like in trading? I don't doubt for a second that HFT profits have dropped, I'm not really on either side in this discretionary vs HFT/algo/AI debate, but you go to any trading forum and you'll find an AI/Algo/HFT thread in a matter of seconds, like it's some eye opening holy grail. 

It would be good to see this machine learning in action in trading, seems to be plenty of tutorials and the like on it all over the web, yet those people are still making tutorials so it obviously didn't do anything special for them. 

I'm just yet to see a successful trader maintain that success over a long period without turning to investing, I think investing is a different story because some people can get on a good thing as far as a company goes etc. Maybe the trader doesn't need to be successful over a long time because he made so much in the short term


----------



## tech/a

Sam

Petes proven it can be done
What do you need a room full of people
Trading for 10 yrs plus all with audited
Results in profit.

You need capital more than most think if you
Want to trade for a living.
You need application
You need to be business orientated---not for everyone.

You need to be different.

I've met you 
Why don't you trade for a second income as we discussed?
I showed you how to do it and offered help.----?


----------



## ThingyMajiggy

tech/a said:


> Sam
> 
> Petes proven it can be done
> What do you need a room full of people
> Trading for 10 yrs plus all with audited
> Results in profit.
> 
> You need capital more than most think if you
> Want to trade for a living.
> You need application
> You need to be business orientated---not for everyone.
> 
> You need to be different.
> 
> I've met you
> Why don't you trade for a second income as we discussed?
> I showed you how to do it and offered help.----?




Pete? Where's he and what's he doing? 

I have traded as a second income and still do occasionally and trading has bought me some toys and holidays, doesn't mean I know what I'm doing or feel confident in how I go about it to be certain that yep, I can do this for the rest of my working life or until I'm floating in my yacht around Santorini.  

But no, I don't need a room full of audited people, even you don't trade for a living, you have a successful business outside of trading that dominates your time, hence why the help/advice faded(along with the desire to not hound you constantly because I'm an idiot) and you're developing a quant platform/system/whatever that I assume you're going to sell and have even less of a reason to trade? Not having a dig, you've done extremely well but I doubt trading is your main focus with your business doing so well, which is fair enough! 

As I said previously, I'm not saying it can't be done, quite the opposite, I hope like hell it can be as I would still love to do it, I just haven't seen anyone sustain it, even what I thought were some of the best traders I've seen, which then begs the question what chance has stupid old me got?  I'm all for trading and wish I was the modern day Livermore(minus the two blowups and eventual suicide), but just going from what I've seen, it seems to have plateaued.  

As others have mentioned they have seen guys doing well apparently, so that's awesome. Traders that are actually profitable are hard to meet/know so you're priviledged to know of these guys 

But this is all off topic in here, PM or email me if you want to keep the discussion going tech, especially if it's about me, people here wouldn't care for that.


----------



## skyQuake

ThingyMajiggy said:


> I'm just yet to see a successful trader maintain that success over a long period without turning to investing, I think investing is a different story because some people can get on a good thing as far as a company goes etc. Maybe the trader doesn't need to be successful over a long time because he made so much in the short term




If a trader is truly successful, they run into liquidity issues. They can then choose to go wide eg. employ same strats using algos across more stocks/more markets (maybe even layer on diff strats) or go deep eg. investing. I think most people mellow out with age and kids etc so they go the investing route


----------



## barney

ThingyMajiggy said:


> PM or email me if you want to keep the discussion going, especially if it's about me, *people here wouldn't care for that*.





You may be surprised Sam .... you have been one of the most interesting people around ASF in recent years ...I would be very interested in your personal trading journey and trading strategies on many levels so please don't think the rest of the punters aren't interested in your story etc

Personally i don't fit your desire to see a long term successful trader, although I am now into my third successful positive year in succession but that is another story, and one I'm obviously happy about, but it is rightly tempered with great humility and respect of the markets! Will it continue?  Who knows; I'm no Quant!!

I know you were doing some great stuff on the Dax, so I'd be the first to welcome another "Sam thread" on any or all of your musings in relation to anything trade related.

Cheers M8.


----------



## ThingyMajiggy

skyQuake said:


> If a trader is truly successful, they run into liquidity issues. They can then choose to go wide eg. employ same strats using algos across more stocks/more markets (maybe even layer on diff strats) or go deep eg. investing. I think most people mellow out with age and kids etc so they go the investing route




Yeah good point 



barney said:


> You may be surprised Sam .... you have been one of the most interesting people around ASF in recent years ...I would be very interested in your personal trading journey and trading strategies on many levels so please don't think the rest of the punters aren't interested in your story etc
> 
> Personally i don't fit your desire to see a long term successful trader, although I am now into my third successful positive year in succession but that is another story, and one I'm obviously happy about, but it is rightly tempered with great humility and respect of the markets! Will it continue?  Who knows; I'm no Quant!!
> 
> I know you were doing some great stuff on the Dax, so I'd be the first to welcome another "Sam thread" on any or all of your musings in relation to anything trade related.
> 
> Cheers M8.




Thanks barney! Well done on the 3rd positive year!  Would definitely be keen to hear more! 

Yeah I've still got some ideas to further build on the work I was doing in the Pull up ya DAX thread, whether it will come of anything or not is another matter, but interesting none the less  

Thanks for the input!


----------



## howardbandy

Greetings --

A few of the comments posted cry out to me for response:

-------------------

Supply/demand and sentiment drive prices, not computer programs.

We -- all people -- regularly try to assign causality.  Correlation is fairly easy to detect, but causality is much more difficult.  The best we often can do is react to the evidence.  In this case, react to whatever prices appear in the history as we are developing trading systems and in the tick-by-tick transactions as we are trading.  Computer program may or may not drive prices, but computer programs are the tools of choice for analysis.   

-------------------

If a trader is truly successful, they run into liquidity issues. 

There are hundreds of equity related tradable issues that have daily liquidity on excess of 10 million dollars per day.  SPY has liquidity in excess of 20 billion dollars per day.  Many highly liquid issues have bid-ask spreads of one cent.  

Following the advice of "the data prospector" outlined in both my "Quantitative Technical Analysis" book and  "Foundations" book, select one tradable issue that has the characteristics that allow it to be traded profitably and work with it.  To recap, those characteristics are:
1.  enough volatility to offer some profit.
2.  not so much volatility that risk is excessive.
Given that the issue passes both filters 1 and 2:
3.  patterns in the data that precede profitable trading opportunities.

If liquidity is an issue, then either:
1.  the trader is trying to trade issues that are intrinsically illiquid -- penny stocks with wide bid ask spreads, for example.  Try sector-oriented exchange traded funds as an alternative -- XLB, XLE, XLF,  ...
2.  the trader is already a big player trading millions of dollars per day.

--------------------

Variations of "some people make better judgements than do computer programs"

Both research and actual performance consistently show that well developed computer algorithms are much more accurate and consistent than the subjective decisions made by people. 

Granted, some people have good results over some periods using subjective judgement -- and we hear about them.  And many people do not -- and we seldom hear about them.  The single book I recommend to everyone who asks "what should I read to learn about trading?" is Daniel Kahneman's "Thinking, Fast and Slow."  Dr. Kahneman explains how, and in some cases why, we are so good at fooling ourselves -- the biases in our thinking that we ignore.  

Those traders who are good at identifying global trends and selecting stocks that will benefit should continue to do that.  Those who are successful due to luck in some form or another should plan for a time when performance suffers and decide how to recover.  If the recovery will be an algorithm, the techniques I recommend will help.  If the recovery will be subjective, I cannot offer helpful advice.

-----------------

Python is just a language

Correct.  It is the opportunities available for development and management of trading systems through use of Python that are exceptional.

For people who want to simply translate a decision tree trading model from TradeStation or some other development platform, I have published a line by line example.  The result is the same trading system, with the added capability of follow-on analysis of risk and position sizing not available in a traditional platform.

For people who want to expand to any of twenty or more models, scikit-learn opens an extensive library of machine learning techniques.  One that works well is an ensemble of thousands of individual decision trees combined as a "random forest" model.   

------------------

Thanks for listening,  Howard


----------



## OmegaTrader

tech/a said:


> Yes.




Warren Buffet is a Quant.


You learn something  new everyday.......


----------



## minwa

OmegaTrader said:


> Warren Buffet is a Quant.
> 
> 
> You learn something  new everyday.......




Unlikely.

_The geeky glamour of the quant took a big hit with the financial downturn, beginning with the so-called quant crisis in the summer of 2007, when quant funds took a nosedive. *“All I can say is, beware of geeks bearing formulas,” Warren Buffett* memorably told Charlie Rose in October 2008. _

http://www.nytimes.com/2010/05/16/magazine/16FOB-OnLanguage-t.html

_In my opinion, investment success 
will not be produced by arcane formulae, computer programs or 
signals flashed by the price behavior of stocks and markets.  
Rather an investor will succeed by coupling good business 
judgment with an ability to insulate his thoughts and behavior 
from the super-contagious emotions that swirl about the 
marketplace._

http://www.berkshirehathaway.com/letters/1987.html


----------



## Roller_1

Does Howard (or do you Howard) have some kind of track or performance record that is publicly available?


----------



## Gringotts Bank

minwa said:


> "...an ability to insulate his thoughts and behavior
> from the super-contagious emotions that swirl about the
> marketplace".




That's successful trading right there.  That one sentence.  How to understand the potential for pain (losing) and pleasure (winning), and be relatively unaffected by the possibility of both... all the time retaining an intention to win.  

It requires a complete re-wiring of the 'usual' mind.  The usual mind is afraid of loss and desperately seeks to win.  

Some people cope with that threat mathematically ("if I do x and y, then z is likely"), others by mental discipline.


----------



## Wysiwyg

howardbandy said:


> Supply/demand and sentiment drive prices, not computer programs. (My post)
> 
> We -- all people -- regularly try to assign causality.  *Correlation is fairly easy to detect, but causality is much more difficult.*  The best we often can do is react to the evidence.  In this case, react to whatever prices appear in the history as we are developing trading systems and in the tick-by-tick transactions as we are trading.  Computer program may or may not drive prices, but computer programs are the tools of choice for analysis.



Respectfully disagree. Another (direct) cause that drives price is news events. One less obviously causal is inside information which is not immediately obvious until the outsiders get the news.


----------



## CanOz

Roller_1 said:


> Does Howard (or do you Howard) have some kind of track or performance record that is publicly available?




Interesting request. Howard is not here to sell you python courses....he's a recognized author and retired professor....why would he need to have a track record?


----------



## Gringotts Bank

CanOz said:


> Interesting request. Howard is not here to sell you python courses....he's a recognized author and retired professor....why would he need to have a track record?




He means trading track record, not academic or career record.  I'd like to know too.  After all, Howard is right up there when it comes to systems trading, so it would give us all a good benchmark of what's possible.  Even if Howard was using a bank of quantum computers to generate signals, it wouldn't mean anything without return$.


----------



## Boggo

minwa said:


> Unlikely.
> 
> _The geeky glamour of the quant took a big hit with the financial downturn, beginning with the so-called quant crisis in the summer of 2007, when quant funds took a nosedive. *“All I can say is, beware of geeks bearing formulas,” Warren Buffett* memorably told Charlie Rose in October 2008. _




That was similiar to what Kerry Packer and Ashok Jacob told Mark Silbermann when he told them that his formula on Onetel's finances indicated that Onetel was headed for failure.
They then had Silbermann physically removed from the meeting because they thought they knew better.

Three weeks later Onetel went into receivership.

(Kerry also commented on ch9 that Merv Lincoln (StockDoctor) was just another idiot and didn't know what he was talking about when he reported that by his StockDoctor formula Onetel was doomed to failure)


----------



## Wysiwyg

An excerpt from an interesting article posted on November 3rd 2016. History never repeats? 



> *"Someone Is Wrong" - Why The Quant War Means Vol Is Set To Soar*
> 
> An example of just this "war of the robot (traders)", is that as Credit Suisse observes, a "quant war" is quietly taking place, and as Bloomberg eloquently adds, "computer is fighting computer in a hedge fund tilt that could get messy regardless of the winner."
> 
> -------------------
> 
> The problem facing the quant world is learning how to respond to the same signal inputs. For the most part, the responses have not led to a wide divergence, however something appears to have changed this time around: "as the quant population exploded, analysts have spent more time trying to pin down how they will react to volatility and other inputs. And with good reason, as long-short funds are sitting on $215 billion in assets, while CTAs oversee some $330 billion, according to BarclayHedge, a database that tracks hedge fund performance."
> 
> The major divergence observed above is troubling to Connors, because it is reminiscent of the periods before the two most recent corrections, when funds took similar opposing bets. Trend-following commodity trading advisers and equity long/short funds are currently 47 percentage points apart in their U.S. equity exposure, the largest spread since just after the S&P 500 Index bottomed in February. Before that, the last time the spread grew to more than 40 percentage points was in August and September of 2015, the worst two-month stretch for the S&P 500 since 2011.
> 
> The causes for the variance in quant sentiment have been isolated: CTAs bearishness is being spurred by an event occurring after the election: the Federal Reserve’s December meeting, when rates may rise, Connors said. On the other side, equity long-short managers have been buying neglected companies like banks in anticipation of higher rates, he said due to relative valuation mispricing.



http://www.zerohedge.com/news/2016-11-03/someone-wrong-why-quant-war-means-vol-set-soar


----------



## tech/a

The Fundamental champion is Buffet
The Quant champions Massive hedge funds

But it's about being able to use fundamentals and increasingly data analysis in YOUR trading
That doesn't remotely resemble the above.

I just can't see how people can't grasp this opportunity.

Oh well.


----------



## ThingyMajiggy

tech/a said:


> The Fundamental champion is Buffet
> The Quant champions Massive hedge funds
> 
> But it's about being able to use fundamentals and increasingly data analysis in YOUR trading
> That doesn't remotely resemble the above.
> 
> I just can't see how people can't grasp this opportunity.
> 
> Oh well.




I think some are missing the point, that it's not about algos/HFT vs discretionary, but using data analysis in/with your own trading. I guess a lot of it is statistics work and helping find/validate an edge or just improving your chances of finding profitable trades. From what I've seen of machine learning, guys are just using it in a directional sense, if it can give you a strong hint about whether tomororw, or next week will be up or down, that's a fairly big hint that you could potentially position for. 

I'd like to see this stuff in action, but to do that I'd have to learn/do it all myself no doubt, which I'm in the process of, seems to be lots of discussion on it but not much evidence...yet. I'm all ears/eyes, if there's something that can improve your trading, I'm not sure why you'd want to shut it down just because it's an algo/ML and you're adamant about the algo vs discretionary war.


----------



## Wysiwyg

I'm interested in some substance to this thread. Is there any substance?


----------



## Trendnomics

Wysiwyg said:


> I'm interested in some substance to this thread. Is there any substance?




Nope.


----------



## CanOz

Good grief....there are so many different time frames and markets that are traded...seems a broad brush to tar everyone with the same feathers here. 

Regarding intra-day trading. You can clearly see algo activity at certain times of the day/session. Anyone worth thier salt has always known that there are times of day to trade, Algoes still need baby sitters. If you trade intra-day know *when* to trade, applies to discretionary and algorithmic-ally....


----------



## CanOz

As long as the futures market has an order book with 100% transactions hitting it, i think the opportunities will be there for alert, trained eyes....if transactions somehow migrate away from the book, to some kind of 'dark pools' all bets are off for the little guy.


----------



## howardbandy

Wysiwyg said:


> I'm interested in some substance to this thread. Is there any substance?




What would substance be?


----------



## Wysiwyg

howardbandy said:


> What would substance be?





howardbandy said:


> As I write in my "Foundations" book, the days of chart reading, long term holding, and simple trading algorithms are over.  The business of trading is changing with astonishing speed.  It is now about applied mathematics, machine learning, Bayesian statistics.  Traders without skills in math, programming, statistical analysis, and scientifically developed trading techniques are at a severe disadvantage.
> 
> Best regards,  Howard



Hello Howard. Substance? Evidence in the form of results or more to the point, evidence in the form of trading algorithms or methods.


----------



## howardbandy

Wysiwyg said:


> Hello Howard. Substance? Evidence in the form of results or more to the point, evidence in the form of trading algorithms or methods.




The techniques for developing trading systems using machine learning, and the techniques for managing trading using sequential Bayesian learning, including fully disclosed ready to run code that can be used as a template for additional development, have been published in the "Quantitative Technical Analysis" book.  

Best, Howard


----------



## DaveDaGr8

Howard,

you mentioned "Random forest" earlier. Presumably what will happen is each decision tree will become a classifier for a particular pattern.

1 - Is a decision tree the best choice for classifying data patterns ?
2 - how do you determine if a tree is worth putting in the forest ?
3 - Do you look at purely raw data, or do you filter it, or use indicators ?
4 - what variables would you use for the nodes ? Standard indicators.

Over the last 3-4 years i have built an adaptive trading model. In one of my stages, the classification stage i use clusters of NN's, which act more like agents or bots to determine a chart pattern. I might try plugging in a random forest ( decision trees ) instead. I think my conclusions will be fairly identical, but who knows ???


----------



## howardbandy

Hi David --

There is a "theorem" of machine learning that is called "no free lunch."  It states, loosely, that it is not possible to know which model is best for a given problem in advance of running the fit and test processes.

Random forest is one of several "ensemble" techniques that use many (often thousands) of relatively simple but, importantly, dissimilar models, each of which is a "weak learner" to combine into a more powerful learner.  Ensemble techniques do well in many applications.   

Each individual tree is predicting the same target.  (More below)  Individually, they may be identifying different patterns, but then together they are predicting the same target. 

The logic used to decide how individual trees will be included depends on the ensemble algorithm used -- the individual trees may be equally weighted or the weight may depend on the accuracy.

The modeler (that is the person building the model) chooses what the target to be modeled is.  A common target is the relation between today's close and tomorrow's close.  There are two broad model categories -- classification and regression.  

For a classification problem, the classification categories are binary -- for example "long" or "flat."  For a regression problem, the target is a more-or-less continuous numeric value -- for example the percentage change from today's close to tomorrow's close.  In the vocabulary of traditional analysis, this is the dependent variable.  

The predictor variables (independent variables) can be anything.  To be useful, they should change in some identifiable way at about the same frequency as the target changes.  Technical indicators with short lookback periods work -- RSI(2), for example.  Pick one -- multiple fast oscillating technical indicators are redundant and usually reduce the accuracy.  Raw price is almost never used.  Formulation of predictor variables is one of the major tasks of the modeler.

Nodes?  What are you thinking of when you say node?  

Ensemble models are made up of "simpler" models.  Pruned decision trees are often used because they are fast to create and train, and they have high variance.  Conceivably, neural networks could be used.  Calculation time could be a problem.  Try it to see.  Think through when, relative to placing the trading order, the models will be trained and how they will be recalled for prediction.

Best regards,  Howard


----------



## OmegaTrader

howardbandy said:


> Hi David --
> 
> There is a "theorem" of machine learning that is called "no free lunch."  It states, loosely, that it is not possible to know which model is best for a given problem in advance of running the fit and test processes.
> 
> Random forest is one of several "ensemble" techniques that use many (often thousands) of relatively simple but, importantly, dissimilar models, each of which is a "weak learner" to combine into a more powerful learner.  Ensemble techniques do well in many applications.
> 
> Each individual tree is predicting the same target.  (More below)  Individually, they may be identifying different patterns, but then together they are predicting the same target.
> 
> The logic used to decide how individual trees will be included depends on the ensemble algorithm used -- the individual trees may be equally weighted or the weight may depend on the accuracy.
> 
> The modeler (that is the person building the model) chooses what the target to be modeled is.  A common target is the relation between today's close and tomorrow's close.  There are two broad model categories -- classification and regression.
> 
> For a classification problem, the classification categories are binary -- for example "long" or "flat."  For a regression problem, the target is a more-or-less continuous numeric value -- for example the percentage change from today's close to tomorrow's close.  In the vocabulary of traditional analysis, this is the dependent variable.
> 
> The predictor variables (independent variables) can be anything.  To be useful, they should change in some identifiable way at about the same frequency as the target changes.  Technical indicators with short lookback periods work -- RSI(2), for example.  Pick one -- multiple fast oscillating technical indicators are redundant and usually reduce the accuracy.  Raw price is almost never used.  Formulation of predictor variables is one of the major tasks of the modeler.
> 
> Nodes?  What are you thinking of when you say node?
> 
> Ensemble models are made up of "simpler" models.  Pruned decision trees are often used because they are fast to create and train, and they have high variance.  Conceivably, neural networks could be used.  Calculation time could be a problem.  Try it to see.  Think through when, relative to placing the trading order, the models will be trained and how they will be recalled for prediction.
> 
> Best regards,  Howard




Hi Howard,

What would you recommend as a starting guide/ actions/resources for someone beginning their Quant journey?

cheers


----------



## DaveDaGr8

Howard,

when i mentioned node, i meant the branch on a tree, or decision point. Hence if a formula such as c<ma(c,5) is true then go left otherwise go right. Building such a tree would be relatively simple, is that what you should use, or is there something better.

I could actually see this working a lot better on fundamental data rather than price action.

Thanks,


----------



## howardbandy

DaveDaGr8 said:


> Howard,
> 
> when i mentioned node, i meant the branch on a tree, or decision point. Hence if a formula such as c<ma(c,5) is true then go left otherwise go right. Building such a tree would be relatively simple, is that what you should use, or is there something better.
> 
> I could actually see this working a lot better on fundamental data rather than price action.
> 
> Thanks,




Hi David --

You are describing creation of the rules that define the decision tree.  There are several techniques for deciding how to divide the data points -- entropy and information gain are the two major ones.  This webpage has a reasonable explanation:
http://www.saedsayad.com/decision_tree.htm

When developing trading systems using a traditional development platform, the modeler (that is the person guiding the development) creates a set of rules and their associated parameters.  The example you use -- c<ma(c,5) -- is a good one.  After a sequence of these rules, the system arrives at a terminal node and issues Buy or Sell signals.  The metric used is neither entropy or information gain.  Rather, it is whatever objective function was provided by the platform or designed by the developer, and is usually based on some variation of profit and risk.

Traditional development is an example of "compute an indicator, then see what happens after."

When developing using machine learning, the modeler provides a target, such as gain for the next evaluation period (regression) or a beLong / beFlat signal (classification), and some predictor variables that represent whatever indicators the developer thinks might contain information.  Think of each data point being a row in a spreadsheet for a single day with columns for each of the predictor variables and one column for the target.  The machine learning process decides how to divide the set of data so that the rows are divided or sorted accurately according to the values of the target.  The machine learning library function builds the rules from the data, remembering the relationships in a matrix of coefficients for later use.  

Machine learning is an example of "identify some desirable outcome, then see what preceded it."

In both cases, as with all modeling, in-sample results are (almost) always good.  The value of the system for trading can only be determined by testing data not used during development -- out-of-sample data.  If the model is overfit to the data, or if the data is not stationary beyond the in-sample period, OOS results will be poor and trading would not be profitable.

Decision trees are among the most basic of the models machine learning can develop.  They are fast and easy to build, easily stored for later retrieval and prediction, and readily interpreted.  They are the model type used by traditional trading system development platforms.

Decision trees are one of many types of models that can be used to develop trading systems using machine learning.  To name a few, others include:  linear regression, logistic regression, discriminant analysis, neural network, nearest neighbor, support vector, Bayesian, random forest, ...  They vary in many aspects -- accuracy, ease of training, robustness, speed of prediction, interpretability, ...

------------------

Related to your comment about using fundamental data --

The data used to predict the target must itself change state at about the same frequency as the target changes state.  Predictor variables that have the same state for extended periods of time are not helpful.  An example that is commonly used is a long-lookback trend filter such as a 200 day moving average:  c>MA(c,200).  If this condition is imposed in a traditional system, trades are blocked for long periods of time.  That might be what the developer intended.

Including a predictor variable that is related to that same filter -- perhaps a binary variable that indicates whether the close is above or below the moving average -- or the ratio of the closing price to the moving average value -- as an input to a machine learning function will result in that variable being ignored.

With regard to use of fundamental data, a paper I published several years ago might be helpful.  It can be read and/or downloaded from this page:
http://www.blueowlpress.com/wp-content/uploads/2016/10/FT-Fundamental-Analysis-Appendix-A.pdf

Best regards,  Howard


----------



## howardbandy

OmegaTrader said:


> Hi Howard,
> 
> What would you recommend as a starting guide/ actions/resources for someone beginning their Quant journey?
> 
> cheers




Hi Omega --

For free, begin by watching these videos:
http://www.blueowlpress.com/video-presentations

The very first book to read is Daniel Kahneman's "Thinking, Fast and Slow."  
https://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555/

Of my books, "Quantitative Technical Analysis." 
http://www.blueowlpress.com/123-2/quantitative-technical-analysis

Best regards,  Howard


----------



## OmegaTrader

howardbandy said:


> Hi Omega --
> 
> For free, begin by watching these videos:
> http://www.blueowlpress.com/video-presentations
> 
> The very first book to read is Daniel Kahneman's "Thinking, Fast and Slow."
> https://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555/
> 
> Of my books, "Quantitative Technical Analysis."
> http://www.blueowlpress.com/123-2/quantitative-technical-analysis
> 
> Best regards,  Howard




Thank you very much and for fast reply.

Time to read, will get back to annoy you later

ahaha


----------



## AlterEgo

howardbandy said:


> Morgan will not build those departments and pay those costs unless the trading results are adequate to fund them.




Hi Howard,

So what returns are these companies making, on average, vs the market index? Because I have often read that the vast majority of fund managers fail to beat the market index. Is that correct, or am I way off the mark there?


----------



## Gringotts Bank

AlterEgo said:


> Hi Howard,
> 
> So what returns are these companies making, on average, vs the market index? Because I have often read that the vast majority of fund managers fail to beat the market index. Is that correct, or am I way off the mark there?




I've read the top ones are consistently in the 20-30%pa range.  Pretty good for their size I guess.

http://www.barrons.com/articles/best-100-hedge-funds-1466223924?tesla=y


----------



## AlterEgo

Gringotts Bank said:


> I've read the top ones are consistently in the 20-30%pa range.  Pretty good for their size I guess.
> 
> http://www.barrons.com/articles/best-100-hedge-funds-1466223924?tesla=y




It's only a 3 year annual return shown, so not really representative of their typical returns, but even so it shows that these very top 100 hedge funds returned an average of 16.98% vs 15.13% for the S&P500. I'd say that if you looked at their 10 year performance the difference would be far less. I just came across an article stating "The 20-Year Performance Of Hedge Funds And The S&P 500 Are Almost Identical". So if that is right, you would have to wonder why these companies would spend vast amounts of money employing all these quants, when a strategy of just buying and holding the index would produce more or less the same returns.


----------



## minwa

AlterEgo said:


> when a strategy of just buying and holding the index would produce more or less the same returns.




The index has the advantage of no commissions/spread and assumed unlimited liquidity in its stocks. So to match the index in the real world after all the costs is actually outperforming it.

Also of course everyone in the game thinks they can outperform.


----------



## tech/a

AlterEgo said:


> It's only a 3 year annual return shown, so not really representative of their typical returns, but even so it shows that these very top 100 hedge funds returned an average of 16.98% vs 15.13% for the S&P500. I'd say that if you looked at their 10 year performance the difference would be far less. I just came across an article stating "The 20-Year Performance Of Hedge Funds And The S&P 500 Are Almost Identical". So if that is right, you would have to wonder why these companies would spend vast amounts of money employing all these quants, when a strategy of just buying and holding the index would produce more or less the same returns.




Managing Millions or Billions year on year to a profit is a vastly different and more difficult task than doing the same as a single retail investor.


----------



## AlterEgo

tech/a said:


> Managing Millions or Billions year on year to a profit is a vastly different and more difficult task than doing the same as a single retail investor.




Exactly! However Howard is painting this picture that these companies are going to totally destroy us retail traders, due to all their resources and talented people that they employ. But I don’t see any evidence of that in these results. These funds are at a severe disadvantage to us retail traders due to their size. Perhaps Howard can show us some evidence to support his claim.

I don’t see these big trading companies as being our competition anyway. Due to their size, they are playing at a completely different ballgame to what we are playing.

And as for simple systems not working anymore – well you can’t get any simpler than buy-and-hold a low cost index fund, but this approach will, apparently, beat the majority of professional fund managers.


----------



## Gringotts Bank

AlterEgo said:


> Exactly! However Howard is painting this picture that these companies are going to totally destroy us retail traders, due to all their resources and talented people that they employ. But I don’t see any evidence of that in these results. These funds are at a severe disadvantage to us retail traders due to their size. Perhaps Howard can show us some evidence to support his claim.
> 
> I don’t see these big trading companies as being our competition anyway. Due to their size, they are playing at a completely different ballgame to what we are playing.
> 
> And as for simple systems not working anymore – well you can’t get any simpler than buy-and-hold a low cost index fund, but this approach will, apparently, beat the majority of professional fund managers.




I consider myself an expert in another field (not trading or financial markets), and I can tell you the "top" people in this field - the 'names', the professors, the lecturers, the academics, the 'published', have very little idea about the real world or what's really possible on the leading edge.  Those who are really ahead of the game - no one recognizes them, or they look like lunatics.  So if the hedge funds with their team of Ivy League quants are achieving 20%pa, you can bet there are individuals who are absolutely smashing that by many times (using computing or discr.), even with big accounts.


----------



## tech/a

AlterEgo said:


> Exactly! However Howard is painting this picture that these companies are going to totally destroy us retail traders, due to all their resources and talented people that they employ. But I don’t see any evidence of that in these results. These funds are at a severe disadvantage to us retail traders due to their size. Perhaps Howard can show us some evidence to support his claim.




I don't think he is making that claim at all. What he is saying is that the masses are able to access the sort of quant analysis which was the realm of the bigger companies. So our/your competition is now very likely to be a growing throng of very skilled (Or learning to be very skilled) people with growing computer capability and advanced analytic ideas and techniques. This new breed will become the norm in trading and investing.



> I don’t see these big trading companies as being our competition anyway. Due to their size, they are playing at a completely different ballgame to what we are playing.




Yes they are but the skills and ideas which are forever growing can now be accessed by Joe Normal.
Pretty soon Joe Normal will become Joe Abnormal!
Personally I traded my horse in as soon as I saw the first T Ford!



> And as for simple systems not working anymore – well you can’t get any simpler than buy-and-hold a low cost index fund, but this approach will, apparently, beat the majority of professional fund managers.




If you want to just get so so returns---fine. There is also bank interest.


----------



## Habakkuk

tech/a said:


> I don't think he is making that claim at all.





He *is* making that claim. The last line of post #1 :

Stephen (Simmons) and his colleagues will "eat the lunch" of unprepared traders.


----------



## Gringotts Bank

If you're really bright and can program at a high level, why on earth would you work for a hedge fund where other people are getting wealthy on the back of your hard work... and in the process eroding your edge?  You'd maybe work for them for a while, keep your best work to yourself until you save up and then go solo.  

The only time it would make sense to use someone else's money is if you're a good discretionary trader, and even then, the moment you succeed, everyone is going to be watching your every move.

What I'm saying is, the big funds aren't set up to be able to find the best ideas.  People know what ideas are worth and they don't want to have their IP used unless they are paid extremely well.


----------



## tech/a

If He means This Stephen Simmons

https://github.com/personal

Exactly the type of person You'll/We'll come up against ---More and More.

I think its in line with my interpretation.
Happy to be educated though


----------



## tech/a

Gringotts Bank said:


> If you're really bright and can program at a high level, why on earth would you work for a hedge fund where other people are getting wealthy on the back of your hard work... and in the process eroding your edge?  You'd maybe work for them for a while, keep your best work to yourself until you save up and then go solo.
> 
> The only time it would make sense to use someone else's money is if you're a good discretionary trader, and even then, the moment you succeed, everyone is going to be watching your every move.
> 
> What I'm saying is, the big funds aren't set up to be able to find the best ideas.  People know what ideas are worth and they don't want to have their IP used unless they are paid extremely well.




What you seriously think they wouldn't be paid well??


----------



## Gringotts Bank

tech/a said:


> What you seriously think they wouldn't be paid well??




Compared to what they can make doing their own thing from home with their own ideas.  

What's to say the really clever ones don't use such positions to gain an insight to insto trading, then make their own algos to exploit what they've learned?  

Think about it.  You're a quant and you get this awesome idea, and you run it at home and it's a big winner... are you going to take it to work and hand it over to Mr Morgan?  Too many vested and competing interests to ever allow major outperformance.


----------



## AlterEgo

tech/a said:


> If you want to just get so so returns---fine. There is also bank interest.




Well, for me personally, I'm not happy with those returns, which is why I'm actively trading instead. I was just adressing Howard's claim that simple methods don't work. Well here is a simple method that works. And what about your mentor, Nick Radge? His methods also appear to be very simple, like the methods in his Holy Grails book. In fact, wasn't the whole premise of this book about showing that simple methods work? So do these methods not work any more?



tech/a said:


> If He means This Stephen Simmons
> 
> https://github.com/personal
> 
> Exactly the type of person You'll/We'll come up against ---More and More.




You could well be right, and maybe these simple methods that work now may stop working at some point in the future as a result. But while my current method continues to work, I'll keep trading it.


----------



## CanOz

tech/a said:


> What you seriously think they wouldn't be paid well??




Exactly....great salary...risk free!


----------



## AlterEgo

Gringotts Bank said:


> Compared to what they can make doing their own thing from home with their own ideas.




Well they may not be able to make more on their own, as they may not have a large enough capital to work with, so I can see the attraction of trading other people's money, like say trading with a prop firm.

This is my problem at the moment. Even though I am making a large return in percentage terms, my household expenses and taxes that are drawn from the account are preventing me being able to compound the profits in to a serious amount of money. I just wish I had 2-3x the funds to trade with.


----------



## DaveDaGr8

Gringotts Bank said:


> Compared to what they can make doing their own thing from home with their own ideas.
> 
> What's to say the really clever ones don't use such positions to gain an insight to insto trading, then make their own algos to exploit what they've learned?
> 
> Think about it.  You're a quant and you get this awesome idea, and you run it at home and it's a big winner... are you going to take it to work and hand it over to Mr Morgan?  Too many vested and competing interests to ever allow major outperformance.




That's why you sign your life away when you join these companies !!!. Something along the lines of "Any and all IP derived by you whilst in our employment remains the companies property".


----------



## Gringotts Bank

AlterEgo said:


> Well they may not be able to make more on their own, as they may not have a large enough capital to work with, so I can see the attraction of trading other people's money, like say trading with a prop firm.
> 
> This is my problem at the moment. Even though I am making a large return in percentage terms, my household expenses and taxes that are drawn from the account are preventing me being able to compound the profits in to a serious amount of money. I just wish I had 2-3x the funds to trade with.




Leverage.  

Solves that problem so long as your DD can handle it.


----------



## AlterEgo

Gringotts Bank said:


> Leverage.
> 
> Solves that problem so long as your DD can handle it.




Already doing that.


----------



## tech/a

This



tech/a said:


> Managing Millions or Billions year on year to a profit is a vastly different and more difficult task than doing the same as a single retail investor.




Answers



Gringotts Bank said:


> Compared to what they can make doing their own thing from home with their own ideas.
> 
> Think about it.  You're a quant and you get this awesome idea, and you run it at home and it's a big winner... are you going to take it to work and hand it over to Mr Morgan?  Too many vested and competing interests to ever allow major outperformance.




This.


----------



## Wysiwyg

Gringotts Bank said:


> If you're really bright and can program at a high level, why on earth would you work for a hedge fund where other people are getting wealthy on the back of your hard work...



Primarily they are computer language skilled and market savvy newbies. Syntax does not maketh the profit. Predicting the direction could be 51% right. Anyway I think it is more encouragement to keep people looking for something better.


----------



## ROE

FA vs TA now it move on to this method vs that method vs this quantz vs data mining

whatever works for you is the right tool, all this stuff are just like fitness and fashion industry
always something new and better, it may be for some people but it may not be for you
what works for you is your edge.

My tools of choice, year 10 algebra maths, reading and simple calculator
it ain't changing any time soon or ever 

I am sure it wont blow the light out but it meet my objectives of compounding at least 10% a year, that way my money always command the same purchasing power or better.

I been doing it for 10-12 years, for first 5-6 years I track nothing dont have any tools, then I bought topstock and start punching in the number from 12/13 onward this is what I get. Another 6 months till F16/17 end


----------



## DaveDaGr8

I think the intent of this thread is to think about the changing face of trading.

Will computers dominate the world of trading and cut out the little man ?
What effect will the increase in computers affect our trading strategies ?
Will the tools that we currently use continue to work ?

Howard proposed that our tools will stop working.

I don't fully subscribe to that theory, but only because i haven't seen a change on the landscape. All my tools and systems continue to work and haven't shown any degradation either on the ASX or the US markets.

Having said that, i have tinkered with Adaptive algorithms for a long time so i am getting prepared for the next evolution.

AI itself relies on the trends that we follow continue to behave in a predictable manner. Without this they lose their own predictive ability. AI also needs our tools to continue to work predictably, because again it will use this knowledge to gain an edge by knowing in advance how the humans ( or even other AI ) will react. AI will try and learn market behaviour, not change market behaviour.

So will our tools stop working ? NO.
Will an AI system beat a human system ? YES.

But that's just my opinion


----------



## ThingyMajiggy

ROE said:


> it ain't changing any time soon or ever




You make some good points, but I don't think this sounds very healthy either, I've read too many times about stubborn traders who refused to change their ways and it all ends in tears. What always works may not always work. I think you've got to have some open-mindedness about things, but going by my parents, it's hard to budge the "we've always done it this way" mentality


----------



## ROE

ThingyMajiggy said:


> You make some good points, but I don't think this sounds very healthy either, I've read too many times about stubborn traders who refused to change their ways and it all ends in tears. What always works may not always work. I think you've got to have some open-mindedness about things, but going by my parents, it's hard to budge the "we've always done it this way" mentality




It all depends on who you are as a market participant, investors, traders or gambler.

I am not a day trader so I dont have to put up with these things, these tools and techniques only temporary affect price in hours, days, weeks or months not years.

No amount of data analysis, HTF programs, and algorithm affect the underlying value of the business over a long period of time, it affect the day to day trading price but it can not and will never change the underlying business value

If you are a retail player and dont have the processing power and capability you may think twice or catch up
or there is a much easier way, these machine wins at processing information quickly and in micro second but they can not process information slowly, 

Become a long term value investor with plenty of reading and slowly process and acquire the knowledge over time and you will have an edge.


----------



## howardbandy

Hi ROE --

No offense intended,  but the points I am making are precisely the opposite.

My view is that stock selection cannot be better than random without insider information.  Assuming insider information is lacking, the best a trader can do is swing trading with high accuracy and short holding period.

In my opinion, and I understand that there may be some disagreement, we (at least in the US) are entering a period with the highest risk in many aspects of our lives that most of us have ever experienced.  

The rise in standard of living among the "winners" following World War II is unlikely to be repeated.  Quantitative Easing is at or near its end.  The Trump Bump may be the last gasp of a bubble in stocks, bonds, and real estate.  Globalization and automation were already affecting us, and the political change we are facing is unprecedented.  If global climate change is not addressed and effective procedures to stop and reverse it put in place very quickly, the world (people, as well as other creatures) will suffer irreversible harm sooner than we imagined could happen.  

The solution for traders is to pay very close attention -- day by day -- and manage trades carefully.  Drawdowns are indications of future crises, not opportunities to double up.   Please, be very careful.  

Best regards,  Howard


----------



## Gringotts Bank

howardbandy said:


> Hi ROE --
> 
> *we ....are entering a period with the highest risk* in many aspects of our lives that most of us have ever experienced.
> 
> The rise in standard of living among the "winners" following World War II *is unlikely to be repeated*.  Quantitative Easing is at or near its end.  The Trump Bump may be the *last gasp of a bubble in stocks, bonds, and real estate*.  Globalization and automation were already affecting us, and the political change we are facing is *unprecedented.*  If global climate change is not addressed and effective procedures to stop and reverse it put in place very quickly, *the world (people, as well as other creatures) will suffer irreversible harm sooner than we imagined could happen.*




Howard, this is all opinion!  Sentiment!  Fear!  ie. the *opposite *of a statistical/mathematical approach!  Now I'm unsure of your stance.

Why wouldn't you just go about trading as usual, adapting as usual?  Surely that's what you're advocating in the OP?  

If on the other hand you intuit a coming disaster, and that's a very strong feeling, why not trust that intuition and start scaling in to a big short position?


----------



## Gringotts Bank

uhhh...

ok I'm going to read between the lines.  Thanks Howard.  I get it.


----------



## howardbandy

Hi GB --

My opinion is that we are entering a period of high risk.  You are correct that it is an opinion.  I might be wrong.  I hope everything works out smoothly.  

If I were to just follow my opinion, my inclination would be to take a short position.  If I did that without having a set of rules to exit the trade, I could be wrong on timing and, even if eventually right on direction, suffer an intra-trade drawdown causing me to exit the trade at a loss.  

Whether my opinion is correct or not, I really do follow my own advice to be mechanical.  I estimate risk of drawdown based on recent performance, then trade in such a way that I can manage drawdown.  That is -- holding a few days at most whether long or short.  Again and again.


Best,  Howard


----------



## Gringotts Bank

howardbandy said:


> Hi GB --
> 
> My opinion is that we are entering a period of high risk.  You are correct that it is an opinion.  I might be wrong.  I hope everything works out smoothly.
> 
> If I were to just follow my opinion, my inclination would be to take a short position.  If I did that without having a set of rules to exit the trade, I could be wrong on timing and, even if eventually right on direction, suffer an intra-trade drawdown causing me to exit the trade at a loss.
> 
> Whether my opinion is correct or not, I really do follow my own advice to be mechanical.  I estimate risk of drawdown based on recent performance, then trade in such a way that I can manage drawdown.  That is -- holding a few days at most whether long or short.  Again and again.
> 
> 
> Best,  Howard




Got it.

I'm going to assume you've noticed some changes in the performance of your own systems - prompting the warning of things to come.  I certainly have noticed changes with my systems - more so since mid Nov.  
I don't have any good short systems, but I can trade the index short using trendlines, and may start doing more of this.  Thanks for the heads up anyway.


----------



## Wysiwyg

Here is an example from Quant Connect of a strategy. Fairly basic stuff for beginners.


----------



## howardbandy

Greetings --

In response to questions related to "How do I get started with machine learning?" --

The outline provided by Matthew Mayo of KDnuggets is a good start:
http://www.kdnuggets.com/2015/11/seven-steps-machine-learning-python.html

The seven steps are:
1.  Basic Python skills
2.  Foundational machine learning skills
3.  Scientific python packages overview
4.  Getting started with machine learning in Python
5.  Machine learning topics with Python
6.  Advanced machine learning topics with Python
7.  Deep learning in Python

Everything listed is free.  Matthew's article gives links to many resources.

For applications in machine learning in general, I agree with the complete list. 

For applications specific to trading, postpone Mayo's step 7, deep learning.  In its place, substitute additional work with the Pandas library and applications to time series.  Listen to presentations by Wes McKinney and Jake VanderPlas; avoid GARCH, wavelets, Fourier series.  Unfortunately, there is very little high quality material published specific to modeling financial time series.

-----------------  

This article by James Le is an excellent introduction to some of the concepts and algorithms of machine learning:
https://gab41.lab41.org/the-10-algo...ngineers-need-to-know-f4bb63f5b2fa#.c63aei127

------------------

Someone is certain to ask about the choice of Python for the base language -- compared to and rather than R.    

Python is a more general language than R; R is more complete in support of frequentist statistics.  

Eventually, if / when you are successful in developing machine learning-based trading systems and moving to trading, you will want easy access to the stack of procedures needed to generate trading signals:
1.  Retrieve and update data.
2.  Retrieve previously trained trading model in preparation for generating new trading signal.
3.  Retrieve and update trading history. 
4.  Retrieve previously trained trading management model in preparation for computation of system health -- position size, estimation of risk, and estimation of profit potential.
5.  Compute new signal.
6.  Compute estimate of system health and position size.
7.  Place trade.

All of these steps can be done within a single Python program.  With the exception of step 7, automatically placing the trade, the programs published in my "Quantitative Technical Analysis" book demonstrate.  Note that there is an API (the IBPy library) that connects Python to Interactive Brokers.  

Note that you can translate and implement your traditional decision tree-based trading models into Python for use in step 2, if you wish.  (Decision tree is one of the machine learning algorithms supported by Python.)  The QTA book gives examples.  Everything else is the same.

Whichever you choose, pick one and become an expert in it.  In my opinion, Python is the clear choice.  I recommend the Anaconda implementation of Python.  

Best,  Howard


----------



## jjbinks

Hey Howard,

Why do you recommend python for beginners instead of something like AFL or the other languages with other software?

Thanks


----------



## ThingyMajiggy

jjbinks said:


> Hey Howard,
> 
> Why do you recommend python for beginners instead of something like AFL or the other languages with other software?
> 
> Thanks




Did you miss his last post just now?


----------



## howardbandy

jjbinks said:


> Hey Howard,
> 
> Why do you recommend python for beginners instead of something like AFL or the other languages with other software?
> 
> Thanks




Hi JJ --

If you will be staying with a traditional trading system development platform, then my recommendation has long been and continues to be AmiBroker and the afl language.

If you will be developing trading systems using machine learning, then my recommendation is Python with the scikit-learn library.  

The models that can be created using traditional platforms are primarily decision tree.  Due to the limitations of the platform, there is seldom any other model algorithm that can be created.  Scikit-learn hosts about 20 model algorithms, including decision tree and several others that often perform better than decision tree.  Additionally, the entire stack of trading operations can be coded into a single Python program -- development, validation, risk assessment, trading management.

Best regards,  Howard


----------



## Quant

Great thread and good to see Howard instigating a meaningful discourse . Id love to dig in and say my bit but its xmas and there is plenty of time for that  . Watching with interest .


statistical based empirically proven rules based systems can have defined expectancy and by their very nature can be exploited to maximize returns exponentially J curve style  . This is the realm of quants , dismiss at your peril   ...  

rock on

Ohh and MERRY XMAS to all , make 2017 matter


----------



## OmegaTrader

Quant said:


> Great thread and good to see Howard instigating a meaningful discourse . Id love to dig in and say my bit but its xmas and there is plenty of time for that  . Watching with interest .
> 
> 
> statistical based empirically proven rules based systems can have defined expectancy and by their very nature can be exploited to maximize returns exponentially J curve style  . This is the realm of quants , dismiss at your peril   ...
> 
> rock on
> 
> Ohh and MERRY XMAS to all , make 2017 matter




Meaningful?

I don't even know what you just said ahaha

Only joking 

Why do you Quants always pull this jargon!!
What the hell does that even mean

Your not exploiting anything, just optimising your position sizing
???
I think haha

and why bring in the j-curve???

to confuse me more and show your smarter than me
aahaha



http://www.investopedia.com/terms/j/j-curve-effect.asp


> J-Curves in Equity Funds
> In private equity funds, the J-curve effect occurs when funds experience negative returns for the first several years. This is a common experience, as the early years of the fund include capital drawdowns and an investment portfolio that has yet to mature. If the fund is well managed, it will eventually recover from its initial losses and the returns will form a J-curve. Losses in the beginning dip down below the initial value, and later returns show profits above the initial level.







Can't you just say

Given my *ASSUMPTIONS* and statistical/ historical back testing I  know what risk and return will be for my system.

Therefore I can increase of decrease position sizing within my system to *optimise returns*.


----------



## Wysiwyg

Got a question for Howard regarding "data". Data availability, data reliability, data latency and cost of data are some factors that limit data analysis. I notice on Quantopian they claim 4 million data sets are available through Quandl and Yahoo Finance. What is a data set, please?


----------



## howardbandy

Wysiwyg said:


> Got a question for Howard regarding "data". Data availability, data reliability, data latency and cost of data are some factors that limit data analysis. I notice on Quantopian they claim 4 million data sets are available through Quandl and Yahoo Finance. What is a data set, please?




A data set is a series of values associated with a tradable issue, such as a stock, fund, future; or with an economic, econometric, or demographic indicator, such as gross domestic product or population.  Being a series implies that there are dates associated with the data values, such as daily closing price for a stock, quarterly GDP, or annual population.  

Quandl is a data aggregator.  They do not produce data.  Rather, they gather data from a variety of sources and redistribute it.  Some of their data sets, such as those gathered from Yahoo or government agencies, are free; others have a subscription charge.  

For Quandl, the data is stored on the Quandl servers.  Quandl maintains a single master copy.  Customers retrieve the latest copy as it is needed.  In this respect, Quandl data is similar to Yahoo data, but with many more data series available.

Compare with Norgate where the data is stored on the customer's computer and updated automatically by Norgate.  Norgate's data is excellent, but limited to financial trade prices.

My experience with Quandl is good.  What you get from a single Quandl data series is essentially a comma separated file similar to a spreadsheet.  A customer, free or subscription, can download a data series as a flat file in csv format for processing on a local computer.  Quandl also provides APIs for many programming languages that access the Quandl data and pass it to an application program on the subscriber's computer.  For example, I use the Quandl Python API.  When I execute an analysis program written in Python, a call to the Quandl API specifies the Quandl identifier (such as a ticker symbol or string) and returns the data as a Python Pandas dataframe.  If I want to perform analysis using some trading data and some econometric data, I make several calls to Quandl, storing each data series in the Pandas dataframe.  The dataframe automatically takes care of date and time alignment, filling in missing values, etc.

Quandl's web page:
https://www.quandl.com/
One of their pages for US stocks:
https://www.quandl.com/data/WIKI-Wiki-EOD-Stock-Prices/documentation/documentation?modal=null
To my limited knowledge, they do not cover Australian stocks.  

Best,  Howard


----------



## ThingyMajiggy

howardbandy said:


> A data set is a series of values associated with a tradable issue, such as a stock, fund, future; or with an economic, econometric, or demographic indicator, such as gross domestic product or population.  Being a series implies that there are dates associated with the data values, such as daily closing price for a stock, quarterly GDP, or annual population.
> 
> Best,  Howard




So Howard I've been meaning to ask, with all this talk of machine learning etc. do you actually have/run any ML systems yourself? I imagine that you must have a few systems that earn you an income. what success have you had with ML and trading?


----------



## howardbandy

ThingyMajiggy said:


> So Howard I've been meaning to ask, with all this talk of machine learning etc. do you actually have/run any ML systems yourself? I imagine that you must have a few systems that earn you an income. what success have you had with ML and trading?



Greetings --

The short answer is yes, I do.

I do follow my own advice, most of which is outlined in the presentations I have made and writings I have published.  I'll break my response into a few parts. 

1.  Most of the development work I am doing now is with machine learning.  The flexibility and power of the Python base, coupled with the variety of models available through scikit-learn and other libraries, the ability to determine hyperparamters, fit models, test whether learning took place, and calculate the risk and profit potential all in a single program run gives a great deal of control over the development process.  I continue to work in AmiBroker -- particularly with some of my clients.

2.  The sweet spot in terms of risk-normalized profit potential really is high accuracy and short holding periods.  Using one-day lookahead targets fits naturally into machine learning.  Anticipating market-to-market daily management of trades after the system has moved on from development, using day-to-day state signals rather than impulse signals works well.

3.  I am making fewer trades these days.  One of the reasons to stop trading is when you have enough.  At my stage of life, I do not know what I would do with more money.

I highly recommend following the scientific method of learning then validating.  And becoming competent in machine learning for trading.  Pay particular attention to learning, validation, and aspects of modeling and simulation unique to time series.  There is a lengthy learning curve and it includes becoming competent in both mathematics and programming.  Do not be fooled by suggestions that these skills are unnecessary.  

The best traders have already moved to machine learning.  They would not have done that unless it gave an advantage over all other trading techniques -- including subjective trading and quantitative trading using only the decision tree model available in traditional development platforms.  Large trading organizations are already employing thousands of highly educated, highly skilled system developers.  There is only one trading arena.  Trading is zero-sum at best.  In order to be profitable, every trader must be able to compete profitably along side David Shaw, James Simons, and Goldman Sachs. 

Best,  Howard


----------



## ThingyMajiggy

howardbandy said:


> Greetings --
> 
> The short answer is yes, I do.
> 
> I do follow my own advice, most of which is outlined in the presentations I have made and writings I have published.  I'll break my response into a few parts.
> 
> 1.  Most of the development work I am doing now is with machine learning.  The flexibility and power of the Python base, coupled with the variety of models available through scikit-learn and other libraries, the ability to determine hyperparamters, fit models, test whether learning took place, and calculate the risk and profit potential all in a single program run gives a great deal of control over the development process.  I continue to work in AmiBroker -- particularly with some of my clients.
> 
> 2.  The sweet spot in terms of risk-normalized profit potential really is high accuracy and short holding periods.  Using one-day lookahead targets fits naturally into machine learning.  Anticipating market-to-market daily management of trades after the system has moved on from development, using day-to-day state signals rather than impulse signals works well.
> 
> 3.  I am making fewer trades these days.  One of the reasons to stop trading is when you have enough.  At my stage of life, I do not know what I would do with more money.
> 
> I highly recommend following the scientific method of learning then validating.  And becoming competent in machine learning for trading.  Pay particular attention to learning, validation, and aspects of modeling and simulation unique to time series.  There is a lengthy learning curve and it includes becoming competent in both mathematics and programming.  Do not be fooled by suggestions that these skills are unnecessary.
> 
> The best traders have already moved to machine learning.  They would not have done that unless it gave an advantage over all other trading techniques -- including subjective trading and quantitative trading using only the decision tree model available in traditional development platforms.  Large trading organizations are already employing thousands of highly educated, highly skilled system developers.  There is only one trading arena.  Trading is zero-sum at best.  In order to be profitable, every trader must be able to compete profitably along side David Shaw, James Simons, and Goldman Sachs.
> 
> Best,  Howard




Thanks very much Howard  

Yeah the Mathematics is the hard part for me, I know some programming already, but have been out of school for years and I am terrible with my Maths unfortunately, so I'm going to have to raid the likes of Khan Academy or something as well if I am to learn ML.


----------



## OmegaTrader

Hi again, 

Some more questions this time, more on the philosophical side..



howardbandy said:


> 3.  I am making fewer trades these days.  One of the reasons to stop trading is when you have enough.  At my stage of life, I do not know what I would do with more money.




You can always forward some money to me. haha

Pm me for details  




howardbandy said:


> The best traders have already moved to machine learning.  They would not have done that unless it gave an advantage over all other trading techniques -- including subjective trading and quantitative trading using only the decision tree model available in traditional development platforms. In order to be profitable, every trader must be able to compete profitably along side David Shaw, James Simons, and Goldman Sachs.





This raises  interesting questions.

*Is it better to just passively invest, if there is no possibility of alpha given the strong competition?*

If one cannot trade effectively in general, the the next best thing is to passively invest.

Why should I read books or learn if I am fighting bruce lee.

Better to acquiesce

It also raises another point.

*How do you explain the success of buffet type strategies who generally are not Quants?
*
Also

I saw before someone put the 20-30% return range.

In your experience, in general, I know a bit hard to generalise...

*What is the risk  and reward profile of these quant type strategies?

Return, STD/Volatility etc
*
Finally a philosophical reflection on quants, assumptions and human nature.

A big theme of the GFC was counter party failure and Quants

The banks made risky loans, on sold those loans in collateralised derivative black magic instruments created by quants.

That hardly anyone understood.

The quants then valued these given *ASSUMPTIONS.
*
The models were completely off.

Some parties took on the risk of the loan and then hedged the risk with an insurer.

The problem was that the insurer went bankrupt and *COUNTERPARTY FAILURE* ensued.

Until the people ended up taking the cost through government assistance.

Of course people who took out the loans did not have clean hands  either.

Aren't these the smart quants at goldman sachs. or did they just ignore the warning signs?

So the question is whether the  quants got it wrong of if *HUMAN NATURE,*corruption and under regulation led the data being put into these models to be incorrect, assumptions incorrect and then  led to an mistake in the valuation of risk.

Or whether there really is a chance for the human creativity that we cannot encompass yet into a computer.

Until we reach/if we reach   the singularity.

Part of me doesn't want to accept that yet.

cheers

my twocents


----------



## howardbandy

OmegaTrader said:


> Hi again,
> 
> I'll number the parts I can address:
> 
> *1.  Is it better to just passively invest, if there is no possibility of alpha given the strong competition?*
> 
> *2.  How do you explain the success of buffet type strategies who generally are not Quants?
> 
> 3.  What is the risk  and reward profile of these quant type strategies?
> 
> 4.  *The problem was that the insurer went bankrupt and *COUNTERPARTY FAILURE* ensued.
> 
> 5.  Until we reach/if we reach   the singularity.
> 
> my twocents




My thoughts -- some that can be justified using the principles of mathematics, modeling, simulation, and statistics.  Others not so easily.

1.  Review my thoughts on investing versus trading versus expenses.  If I make an investment, I have my lawyer write the details and disposition into my will.  If there is a reasonable chance I will close out that transaction sometime before I die, it is a trade.  Trades need rules.  I need to know what conditions will cause the exit.  If there are so few examples that I cannot use the scientific method of evaluating many in-sample examples followed by testing previously unused out-of-sample examples, then that transaction is subjective and no analysis is possible.

Passive investing, to me, means buying a stock or fund, putting it in my will, and forgetting about it.  Not even another GFC would cause a sale.  Passive trading, to me, means buying a stock or fund, expecting to sell it sometime, but not having clear rules.  I am terrible at that.  The first book I recommend to everyone is Daniel Kahneman's "Thinking, Fast and Slow."  Dr. Kahneman explains how we fool ourselves.

Passive holding of any asset implies relatively long holding periods.  Long enough that there will be substantial drawdown.  An assumption of passive holding is that the drawdown will be short enough and shallow enough that it can be tolerated -- held through.  Some people even think that is an opportunity to increase position size.  In some cases, that works.  The drawdown in 2009 was over 50%.  Easy money policies by central banks bought a quick recovery.  We will learn the true consequences of those policies in the next year or two.  I expect another very steep drawdown.  The drawdown in 1932 was about 80% and took over 25 years to recover.  

Passive holding assumes that returns will compensate for risk and inflation.  Much of the period those of us alive and trading today remember is that following WWII.  There are many signs that we cannot expect the conditions that brought those gains to continue.  For example,  increasing globalization, increasing unrest, and serious climate changes.  

2.  Warren Buffett acts more like the president of a corporate conglomerate than a trader.  Berkshire Hathaway buys large positions in companies expecting to hold them long periods.  They experienced drawdowns of more than 50% in 2009.  My risk tolerance is much lower.  To my thinking, a trader holding through steep drawdowns is making a mistake.  A better technique is exiting and returning later.

3.  By quant strategies, I'll assume those similar to James Simons and David Shaw.  When I was working in a hedge fund, we, and most funds, charged "2 and 20" -- 2% annually of notional funds under management and 20% of profits -- and our customers were profitable.  Some of the best funds charged higher fees -- as high as 5 and 45 -- and still made customers wealthy.  But funds that profitable are rare.  And the 2 and 20 model is disappearing.

4.  I recommend not trading anything that is not cleared by an independent central clearing agent.  In over-the-counter trades, when I buy from you, you are my counterparty.  If you fail to meet your obligations, I must deal with you directly.  When trades are cleared by an independent central clearing agent, the clearing house is the counterparty to both traders.  Be very wary of trading anything OTC.

5.  I began working with artificial intelligence and machine learning in the 1960s.  Some of the research I did in graduate school contributed to early work with neural networks and nearest neighbor analysis.  Alan Turing posed the Turing Test in 1950, anticipating increasing capabilities of computer-based applications.  The Turing Test has been convincingly passed; computers are now champions of every board and card game; artificial intelligent applications are better than human experts in almost all professions, including medicine, law, and logic.

Some people estimate that the singularity -- computer abilities surpassing human abilities in almost everything -- will happen soon.  I agree with them.  I believe the singularity is near.  

I also believe the singularity will be harmful to the world, including to people.  I imagine the similarities with being discovered by a superior society.  Consider the Incas being discovered by Pizarro, or the US plains Indians by European settlers, or ...

--------------

I know -- this is more than you expected to hear.  I hope it helps in convincing readers of the importance of following the scientific method in trading system development and in trading.

Thanks for listening.

Best,   Howard.


----------



## captain black

Great post Howard. Plenty of great minds (Elon Musk, Bill Gates, Stephen Hawking etc) warning about the singularity. Sadly, like a few of the other issues you mentioned, most of the warnings are being ignored.


----------



## OmegaTrader

howardbandy said:


> My thoughts -- some that can be justified using the principles of mathematics, modeling, simulation, and statistics.  Others not so easily.
> 
> 1.  Review my thoughts on investing versus trading versus expenses.  If I make an investment, I have my lawyer write the details and disposition into my will.  If there is a reasonable chance I will close out that transaction sometime before I die, it is a trade.  Trades need rules.  I need to know what conditions will cause the exit.  If there are so few examples that I cannot use the scientific method of evaluating many in-sample examples followed by testing previously unused out-of-sample examples, then that transaction is subjective and no analysis is possible.
> 
> Passive investing, to me, means buying a stock or fund, putting it in my will, and forgetting about it.  Not even another GFC would cause a sale.  Passive trading, to me, means buying a stock or fund, expecting to sell it sometime, but not having clear rules.  I am terrible at that.  The first book I recommend to everyone is Daniel Kahneman's "Thinking, Fast and Slow."  Dr. Kahneman explains how we fool ourselves.
> 
> Passive holding of any asset implies relatively long holding periods.  Long enough that there will be substantial drawdown.  An assumption of passive holding is that the drawdown will be short enough and shallow enough that it can be tolerated -- held through.  Some people even think that is an opportunity to increase position size.  In some cases, that works.  The drawdown in 2009 was over 50%.  Easy money policies by central banks bought a quick recovery.  We will learn the true consequences of those policies in the next year or two.  I expect another very steep drawdown.  The drawdown in 1932 was about 80% and took over 25 years to recover.
> 
> Passive holding assumes that returns will compensate for risk and inflation.  Much of the period those of us alive and trading today remember is that following WWII.  There are many signs that we cannot expect the conditions that brought those gains to continue.  For example,  increasing globalization, increasing unrest, and serious climate changes.
> 
> 2.  Warren Buffett acts more like the president of a corporate conglomerate than a trader.  Berkshire Hathaway buys large positions in companies expecting to hold them long periods.  They experienced drawdowns of more than 50% in 2009.  My risk tolerance is much lower.  To my thinking, a trader holding through steep drawdowns is making a mistake.  A better technique is exiting and returning later.
> 
> 3.  By quant strategies, I'll assume those similar to James Simons and David Shaw.  When I was working in a hedge fund, we, and most funds, charged "2 and 20" -- 2% annually of notional funds under management and 20% of profits -- and our customers were profitable.  Some of the best funds charged higher fees -- as high as 5 and 45 -- and still made customers wealthy.  But funds that profitable are rare.  And the 2 and 20 model is disappearing.
> 
> 4.  I recommend not trading anything that is not cleared by an independent central clearing agent.  In over-the-counter trades, when I buy from you, you are my counterparty.  If you fail to meet your obligations, I must deal with you directly.  When trades are cleared by an independent central clearing agent, the clearing house is the counterparty to both traders.  Be very wary of trading anything OTC.
> 
> 5.  I began working with artificial intelligence and machine learning in the 1960s.  Some of the research I did in graduate school contributed to early work with neural networks and nearest neighbor analysis.  Alan Turing posed the Turing Test in 1950, anticipating increasing capabilities of computer-based applications.  The Turing Test has been convincingly passed; computers are now champions of every board and card game; artificial intelligent applications are better than human experts in almost all professions, including medicine, law, and logic.
> 
> Some people estimate that the singularity -- computer abilities surpassing human abilities in almost everything -- will happen soon.  I agree with them.  I believe the singularity is near.
> 
> I also believe the singularity will be harmful to the world, including to people.  I imagine the similarities with being discovered by a superior society.  Consider the Incas being discovered by Pizarro, or the US plains Indians by European settlers, or ...
> 
> --------------
> 
> I know -- this is more than you expected to hear.  I hope it helps in convincing readers of the importance of following the scientific method in trading system development and in trading.
> 
> Thanks for listening.
> 
> Best,   Howard.




Great post again.

Where do I start? haha

Point 1


howardbandy said:


> My thoughts -- some that can be justified using the principles of mathematics, modeling, simulation, and statistics.  Others not so easily.
> 
> 1.  Review my thoughts on investing versus trading versus expenses.  If I make an investment, I have my lawyer write the details and disposition into my will.  If there is a reasonable chance I will close out that transaction sometime before I die, it is a trade.  Trades need rules.  I need to know what conditions will cause the exit.  If there are so few examples that I cannot use the scientific method of evaluating many in-sample examples followed by testing previously unused out-of-sample examples, then that transaction is subjective and no analysis is possible.
> 
> Passive investing, to me, means buying a stock or fund, putting it in my will, and forgetting about it.  Not even another GFC would cause a sale.  Passive trading, to me, means buying a stock or fund, expecting to sell it sometime, but not having clear rules.  I am terrible at that.  The first book I recommend to everyone is Daniel Kahneman's "Thinking, Fast and Slow."  Dr. Kahneman explains how we fool ourselves.
> 
> Passive holding of any asset implies relatively long holding periods.  Long enough that there will be substantial drawdown.  An assumption of passive holding is that the drawdown will be short enough and shallow enough that it can be tolerated -- held through.  Some people even think that is an opportunity to increase position size.  In some cases, that works.  The drawdown in 2009 was over 50%.  Easy money policies by central banks bought a quick recovery.  We will learn the true consequences of those policies in the next year or two.  I expect another very steep drawdown.  The drawdown in 1932 was about 80% and took over 25 years to recover.
> 
> Passive holding assumes that returns will compensate for risk and inflation.  Much of the period those of us alive and trading today remember is that following WWII.  There are many signs that we cannot expect the conditions that brought those gains to continue.  For example,  increasing globalization, increasing unrest, and serious climate changes.




Sorry to disagree with you but
It is very hard to argue against passive investing as a strategy in the long term.

*I haven't seen a lot of evidence to show that passive doesn't work in the long term.
*
1932 was the most extreme of the extreme.

I can't remember the book/publication but they looked across countries and over 100 of years.
In most countries given the long term it worked.
But there is always exceptions. Japan etc

I suppose it is a bet on the future of the country.
Growth by population, efficiency from technology, better regulations etc etc

To me passive trading just means spending less time on researching and implementing the trading decisions. Not being an idiot and giving up on being strategic or rational. hahaha


*If passive investing is not a suitable strategy and one cannot beat the big boys trading....

and 

If Profitable funds are also rare

Then what do you do?

Leave the money in the bank and get destroyed by inflation and taxes?*

I can't agree or disagree with you on your pessimism.

My opinion :The fog of war is too noisy for anyone to predict what will happen.

Point 2

The point I was making was that Buffett is successful and doesn't use advance quant strategies..
That goes against your ethos of quantifying as much as possible.
Doesn't that prove that qualitative strategies can work??

Point 3

So 20+%. No one is making money anymore???
What?



Point 4

Yes OTC is a minefield.

I am not saying quant strategies don't work

and following scientific evaluation can really really help.

But the point I was making is that all of the aspects were quantified but the counter parties failed.
and these were big counter parties and the smartest of the smartest quants.

AIG Goldman Sachs JP etc etc



The numbers on the sheet did not tell the wholes story, unless someone was fabricating the numbers...

I think sometimes quants forget that life is not like a poker, blackjack,chess game, with the cards/board always being the same and the optimum strategy never changing once you have found it

We can't even work out the weather in one weeks time...

Going back to your previous point, a will can be challenged, so even that is not set in stone.

*Assumptions are the key and are still mainly in the realm on qualitative thought*

Qualitative analysis still must have some value at least at the moment.



Point 5

I used to think that the   singularity was close and fear it..

But now  think it will be more augmentation, even some biological engineered/evolutionary aspects thrown in there too.

No wolverines or skynet in my lifetime,



 I hope

cheers


----------



## howardbandy

OmegaTrader said:


> Great post again.
> 
> *I haven't seen a lot of evidence to show that passive doesn't work in the long term.
> *
> I suppose it is a bet on the future of the country.
> Growth by population, efficiency from technology, better regulations etc etc
> 
> cheers




I'll comment just on passive versus active.

By active, I mean all purchases and sales are the result of signals given by rules.  If development was done following the scientific method, trade-by-trade position size adjusted to keep anticipated risk within personal tolerance, then there is a chance results will be profitable.  

By passive, I mean picking a stock or fund that follows an index, then buying and holding that for a long time -- years -- without regard for rises or falls in the value.  At some point, the funds will be withdrawn, but that is because they are needed for some other purpose, rather than as a result of performance.

-----------

That said, passive is essentially subjective selection of a country, sector, or stock.  It is subjective even if there are pseudo-rules based on historical data because there is not an opportunity to test on out-of-sample data that follows the historical data. 

My concern is that, in my opinion, the future is unlikely to resemble the past.  The US economy grew strongly following WWII, up through about 2000.  The period since 2000 has been much weaker.  The recovery from the 2009 crisis was due almost entirely to central banks buying a recovery with borrowed money -- yet to be repaid.  Some of the factors that are different now compared with 1945 to 2000 are development in previously under-developed countries, globalization in manufacturing and intellectual activities, nationalism, terrorism, automation in manufacturing, artificial intelligence in intellectual activities, climate change, and the debt that must be repaid to unwind the recovery from 2009.

Which country to pick?  

Not the US.  We are in a bubble as extreme as we were in 2000.  Even without Trump, the US is unlikely to experience any growth at all over the next decade or two.  If Trump succeeds in implementing his stated plans, I expect a short, temporary spurt in apparent growth -- maybe we are seeing the start of it already in anticipation -- followed by a crisis worse than 2009 that will go on for decades and cripple the US, and the world, forever.  I think the risk of being a passive investor / trader in US equities, bonds, and real estate is the highest it has ever been.  

Japan has very high debt and low productivity.  Europe is in trouble.  China is restricting financial activity.  Russia is a poor country with only two export products -- oil and fear.  The mid-east is a disaster that, at best, we can hope will not spread.  Africa is unruly.    

I do not see a good choice for buy and hold decades.  As you say, these include my assumptions and subjective estimates.  

I do see opportunities to trade using scientific techniques with day-to-day management, reducing position size when recent results are poor and increasing when they are good.  Quantitative.  Scientific.  Day-by-day.  Using the best model development techniques available.  The tools -- the mathematics -- will tell us what and when to buy and sell, and when to stop trading.

Thanks for listening,

Best,  Howard


----------



## tech/a

None are as blind as those who cannot see.
None are as deaf as those who cannot hear.

We all march to the beat of our own drum.

The best we can hope for is to manage our own
affairs. If we can do that you'd be surprised
what can happen.

Thanks to everyone sharing.


----------



## OmegaTrader

howardbandy said:


> I'll comment just on passive versus active.
> 
> 
> 
> My concern is that, in my opinion, the future is unlikely to resemble the past.  The US economy grew strongly following WWII, up through about 2000.  The period since 2000 has been much weaker.  .....




BY this thought process almost all historical back-testing is invalid.

As the assumption of back-testing is that the future will follow similar rules.

Isn't that what a lot of systems use as part of the analysis?

Passive investing is simply buying and holding and selling in 20-30 years when you need the money to retire.

That is it. No time, no effort, no complicated formulas.

That is the rule:
*
 buy every year of work using savings and hold it until retirement.
*
I would be suprised if your could show me a period of 20-30 years were buying each year would not work in Australia or the us

Save on tax, save on commission, save on fees, save on grey hairs!!!

No timing the market.

It has worked for the history of the Australian stock market

and the US.

It is not perfect but it saves the opportunity cost of spending time and mental capital on trading.



I was reading an an article based in the 1980/90s about how the US was going to default on Gov debt now it is 120% + f GDP and still going....

Of course logical scientific analysis is useful.

But don't forget we are not playing blackjack.....

No one knows what cards are in the deck.

The GFC should have aught alot of quants that.

cheers


----------



## howardbandy

OmegaTrader said:


> 1.  BY this thought process almost all historical back-testing is invalid.
> 
> 2.  As the assumption of back-testing is that the future will follow similar rules.
> 
> 3.  But don't forget we are not playing blackjack.....No one knows what cards are in the deck.
> 
> cheers




Greetings --

1.  Correct.  Almost all historical backtesting is without value in estimating future performance.  The phrase I used was:  profitable backtesting is necessary, but not sufficient.

2.  Let me make a slight change in the terminology:  Successful trading requires that the future resemble the past.  That is one of the tenets of technical analysis, as well as machine learning.  

In order for a trading system to be profitable in live trading, the system must be profitable for the out-of-sample validation period and into the live trading period.  (It will have been profitable for the in-sample period too, but that is always true of a system that enters validation testing.)  We rely on the conditions being stationary.  That is, the future must resemble the past with respect to the signals and trade generated by the system.  Watch my YouTube presentation on "The Importance of Being Stationary."
http://www.blueowlpress.com/video-presentations

3.  Trading is like some gambling games.  

Trading is similar to blackjack.  Blackjack has a memory.  The cards already played change the distribution of the cards yet to be played, and change the players odds of winning.  An expert blackjack player does know the cards that remain in the deck, can compute the probability of winning and losing given those yet-to-be-dealt cards, can identify when the player has an advantage over the house, and can bet accordingly.  My "Modeling" book has an analysis of blackjack as a business and compares trading as a business.  Both require capital adequate to stay solvent through drawdowns.    

But trading is not similar to roulette.  Roulette has no memory.  No history or sequence of recent winning pockets changes the probability of winning a bet on any pocket on the next play.  The odds of a player winning is never in his or her favor.  And no money management scheme can turn that negative-expectation game into a winning system.  The "Modeling" book also describes roulette.

Best,  Howard


----------



## OmegaTrader

howardbandy said:


> Greetings --
> 
> 1.  Correct.  Almost all historical backtesting is without value in estimating future performance.  The phrase I used was:  profitable backtesting is necessary, but not sufficient.
> 
> 2.  Let me make a slight change in the terminology:  Successful trading requires that the future resemble the past.  That is one of the tenets of technical analysis, as well as machine learning.
> 
> In order for a trading system to be profitable in live trading, the system must be profitable for the out-of-sample validation period and into the live trading period.  (It will have been profitable for the in-sample period too, but that is always true of a system that enters validation testing.)  We rely on the conditions being stationary.  That is, the future must resemble the past with respect to the signals and trade generated by the system.  Watch my YouTube presentation on "The Importance of Being Stationary."
> http://www.blueowlpress.com/video-presentations
> 
> 3.  Trading is like some gambling games.
> 
> Trading is similar to blackjack.  Blackjack has a memory.  The cards already played change the distribution of the cards yet to be played, and change the players odds of winning.  An expert blackjack player does know the cards that remain in the deck, can compute the probability of winning and losing given those yet-to-be-dealt cards, can identify when the player has an advantage over the house, and can bet accordingly.  My "Modeling" book has an analysis of blackjack as a business and compares trading as a business.  Both require capital adequate to stay solvent through drawdowns.
> 
> But trading is not similar to roulette.  Roulette has no memory.  No history or sequence of recent winning pockets changes the probability of winning a bet on any pocket on the next play.  The odds of a player winning is never in his or her favor.  And no money management scheme can turn that negative-expectation game into a winning system.  The "Modeling" book also describes roulette.
> 
> Best,  Howard




The deck in poker does not change.
The chess pieces do not change.

The rules of the game do not change. Once the optimum/game theory optimum/ basic strategy is found.
That is the end of the game.
It has been solved now any computer or idiot on basic strategy using rule of thumb can win it.



As you stated yourself.  



> My concern is that, in my opinion, the future is unlikely to resemble the past.




*Assumptions again....*

Incorrect assumptions lead to incorrect statements !!!

Roulette is beatable and *not always* a negative ev game.

The man who broke montecarlo.

The first historical reference of a biased wheel in roulette.

Edward O Thorp  one of the main contributors to basic blackjack strategy

beat roulette as well...

In sense he constant turning of the wheel over time biases the numbers.
Colloqually a memory of sorts...


In O Thorps case, calculating where the ball will likely land given it's current position speed etc etc 

Once again a memory of sorts...


I am merely trying to prevent my point of view that qualitative though still has value because of assumptions.

Pure Quants often forget this.


You are clearly an extremely intelligent and successful individual  being well respected also.

So please do not be offended.


cheers


----------



## howardbandy

OmegaTrader said:


> The deck in poker does not change.
> 
> Roulette is beatable and *not always* a negative ev game.
> 
> Edward O Thorp  one of the main contributors to basic blackjack strategy
> 
> beat roulette as well...
> 
> cheers




Greetings --

I have snipped out some of your reply to allow the focus on points where we disagree:

The deck in poker and blackjack definitely does change.  Every card that is exposed while the game is in progress changes the distribution of the remaining yet-to-be-dealt cards.  That change can be detected.  It is precisely that change in distribution that players use to estimate their advantage or disadvantage, and play accordingly.  With no change in distribution, as when the blackjack deck is shuffled before each hand, or continuously shuffled, the player always has a negative expectation due to rules that favor the house. 

Roulette is not beatable.  A true wheel has no memory.  Between the table limits (which prevent full use of Martingale betting techniques) and the house advantage (green pockets) that creates a negative expectation, no player can win consistently.

Edward Thorpe was a major contributor to analysis of blackjack.  He also successfully managed money.  His success with roulette was due to recognizing and taking advantage of an unfair (out of balance) roulette wheel.  Neither he, nor anyone else, could or can predict which pocket the ball will land in on a fair roulette wheel.

Best, Howard


----------



## DaveDaGr8

I would love to see AI beat a professional poker player and to be honest I'm not sure that it could. In poker, maths has very little to do with anything. Bluffing is everything. Good poker players can change the way the game is played to their advantage ie, get people to change their thinking. They could probably get an AI to modify it's thinking too right before they punish it.

Blackjack now has 8 deck shoes, so you have to count 412 cards. A fresh shoe is cut by a player and this is discarded, so now you have to guess the number of discarded cards and assume an even distribution of the cut cards to guess the remainder. The edge of calculating a probability shift has been eroded by new casino strategies.


----------



## OmegaTrader

howardbandy said:


> Roulette is not beatable.  A true wheel has no memory.  Between the table limits (which prevent full use of Martingale betting techniques) and the house advantage (green pockets) that creates a negative expectation, no player can win consistently.
> 
> His success with roulette was due to recognizing and taking advantage of an unfair (out of balance) roulette wheel.  Neither he, nor anyone else, could or can predict which pocket the ball will land in on a fair roulette wheel.
> 
> Best, Howard




That does not make sense.

He had success but it is not beatable?????


That is like saying no-one has read hair. Its rare but I have seen it... 



*See assumptions*

Fair wheel. You made an *assumption.*

That is the assumption that broke montecarlo.

and lost casinos around the world money

sounds a bit like efficient market doesn't it.

Of course the casino will change the game to prevent increased bet sizing and add extra zeros of course then will stop people from winning, that is their bread and butter.

Of course the odds are in their favour. Basic Ev depicts that.

Checking the games are fair, so that they can win by giving worse odds.

*But the point is the games were not fair all of the time*

Just like the market. People think that it will follow maths and it will,

But only if the assumptions are correct. 

That is the point.

*We don't know the deck. *

What I meant in blackjack was that the rules don't change. They state the rules, they follow the rules.
The deck is this, the draw is that, the rule are this.
Same in poker, a flush is flush every game.

I didn't mean cards weren't taken out of the deck.

But that is the reason it is beatable is because the game is the same.
*
The market is not always the same.
*
That is why a lot the banks failed in the GFC because the Quants assumptions were wrong..

Then the people bailed them out.

The smartest quants too, forget about assumptions.


----------



## howardbandy

DaveDaGr8 said:


> I would love to see AI beat a professional poker player and to be honest I'm not sure that it could. In poker, maths has very little to do with anything. Bluffing is everything. Good poker players can change the way the game is played to their advantage ie, get people to change their thinking. They could probably get an AI to modify it's thinking too right before they punish it.
> 
> Blackjack now has 8 deck shoes, so you have to count 412 cards. A fresh shoe is cut by a player and this is discarded, so now you have to guess the number of discarded cards and assume an even distribution of the cut cards to guess the remainder. The edge of calculating a probability shift has been eroded by new casino strategies.




Greetings --

Just as with games like checkers, chess, and go, the machine plays many thousands of games, learning strategy, tactics, psychology, traits of individuals, etc.  AI applications are winning against human card players, including bridge and poker.  

In blackjack there are not only a lot of decks, but continuous shuffling.  As soon as the dealer picks up discards from a hand, those cards are put back into the dealing shoe, which is continuously shuffling.  A discard from one hand can be dealt in the next hand.  The count is never far from neutral and never in the player's favor.  

Some dealing shoes are automatic.  Several decks are shuffled and placed in the shoe.  Whenever a card is requested, the shoe releases one. The human dealer does not touch the cards as they are dealt.  This is explained as removing an opportunity for a dishonest dealer to favor a compatriot.  While illegal to do so in most locations, automatic dealing also allows the dealing shoe to recognize what cards have been dealt, compute the count, and reshuffle whenever it becomes favorable for players.

There are still a few casinos where single or double decks are dealt by hand and dealt nearly completely.  They attract card counters, some of whom are very good.  But the casino uses a different payout schedule -- particularly the odds for blackjack -- so it remains very difficult for the player to realize an advantage.

All of that said, casinos have little to fear from customers.  Having a few big winners is seen as good publicity.  And even when a blackjack deck can sometimes have a count in the player's favor, the mean gain per hand is low, the variance is high, it takes a large bankroll to withstand the drawdowns, and the casino must allow players to make big changes in change bet size -- at least 10 to 1.  Those teams that won large amounts did so by having a team member bet the minimum while counting, signaling another member when the deck was favorable, and having that new player make a large bet.  The casinos caught on pretty quickly and that tacit no longer works.

Additionally, facial recognition software alerts casino security as soon as a known winner comes through the door.

But it is still fun to be a spectator and watch people who do not understand the math.

Best,  Howard


----------



## Gringotts Bank

*https://www.pokernews.com/news/2015/05/man-vs-machine-pro-ahead-450k-21434.htm*

*Conclusion*
With three out of four of the professional players ahead of the poker bot so far for a combined $458,902, it appears that man is still greater than machine when it comes to no-limit Texas hold'em. Even if "Claudico" is unable to mount a comeback, it proves that AI is stronger now than ever, as it did give some problems to Kim and is dominating Les.


----------



## ThingyMajiggy

I wish I was smart enough and had the Mathematics knowledge to do this stuff, but I fear I'd need to spend the next 10 years becoming an expert at Maths, then another 10 learning about systems and machine learning, then beyond that you need the know-how of actually having a strategy that works, still need the ideas. Hard to know what to concentrate on in life, especially when I'm stronger in other parts of my life, like creative things, music & photography, then you have 10 million people all fighting for those positions and thinking they're amazing at it. 

Decisions decisions  Interesting discussion in here anyhow, it kind of points to having programming knowledge as being an essential skill in the future, as so many things are being automated, they all need code and maintaining, may be the only jobs in the future with the way things are going. 

It's hard to imagine that us little guys can compete in this world of trading, surely anything we think of has already been thought of and tested etc. Much smarter minds out there working on this stuff than me so what chance have I got?


----------



## Gringotts Bank

ThingyMajiggy said:


> ...especially when I'm stronger in other parts of my life, like creative things, music & photography, .....Much smarter minds out there working on this stuff than me so what chance have I got?




None, but it will be a long time before a robot can imbue a creative piece of work with soul.  Perhaps never.


----------



## OmegaTrader

howardbandy said:


> Greetings --
> 
> Just as with games like checkers, chess, and go, the machine plays many thousands of games, learning strategy, tactics, psychology, traits of individuals, etc.  AI applications are winning against human card players, including bridge and poker.
> 
> In blackjack there are not only a lot of decks, but continuous shuffling.  As soon as the dealer picks up discards from a hand, those cards are put back into the dealing shoe, which is continuously shuffling.  A discard from one hand can be dealt in the next hand.  The count is never far from neutral and never in the player's favor.
> 
> Some dealing shoes are automatic.  Several decks are shuffled and placed in the shoe.  Whenever a card is requested, the shoe releases one. The human dealer does not touch the cards as they are dealt.  This is explained as removing an opportunity for a dishonest dealer to favor a compatriot.  While illegal to do so in most locations, automatic dealing also allows the dealing shoe to recognize what cards have been dealt, compute the count, and reshuffle whenever it becomes favorable for players.
> 
> There are still a few casinos where single or double decks are dealt by hand and dealt nearly completely.  They attract card counters, some of whom are very good.  But the casino uses a different payout schedule -- particularly the odds for blackjack -- so it remains very difficult for the player to realize an advantage.
> 
> All of that said, casinos have little to fear from customers.  Having a few big winners is seen as good publicity.  And even when a blackjack deck can sometimes have a count in the player's favor, the mean gain per hand is low, the variance is high, it takes a large bankroll to withstand the drawdowns, and the casino must allow players to make big changes in change bet size -- at least 10 to 1.  Those teams that won large amounts did so by having a team member bet the minimum while counting, signaling another member when the deck was favorable, and having that new player make a large bet.  The casinos caught on pretty quickly and that tacit no longer works.
> 
> Additionally, facial recognition software alerts casino security as soon as a known winner comes through the door.
> 
> But it is still fun to be a spectator and watch people who do not understand the math.
> 
> Best,  Howard





oh yeah the casinos stop people from winning.

But the rules are set and known.

Computers will win all of the games with set rules eventually

In life we do not always know all of the rules.

Markets are a representation of this chaos.

That is why assumptions and rule of thumb have allowed our species to survive so long.

Until computers  can philosophise these assumptions we are still ok for the moment


----------



## tech/a

Why would you not improve what your doing with the inclusion 
Of quant analysis/programming/data analysis/systems testing.

You can still post a letter or use an email.

If you can use it I personally think you should.


----------



## OmegaTrader

tech/a said:


> Why would you not improve what your doing with the inclusion
> Of quant analysis/programming/data analysis/systems testing.
> 
> You can still post a letter or use an email.
> 
> If you can use it I personally think you should.




Yeah agree.

If can use it do it.

But remember thee humble assumptions


----------



## ThingyMajiggy

Howard, what sort of % accuracy are you personally getting with your ML systems with Python? Looking around I've seen anywhere from 58% up to 95%, I know it would depend on what algorithm you use, but curious as to what you're getting from your efforts


----------



## howardbandy

ThingyMajiggy said:


> Howard, what sort of % accuracy are you personally getting with your ML systems with Python? Looking around I've seen anywhere from 58% up to 95%, I know it would depend on what algorithm you use, but curious as to what you're getting from your efforts




Yes, the algorithm (model) makes a difference.  And, for many models, accuracy is tunable.  Often very high accuracy -- 90% or higher -- comes with many small gains.  That is OK as long as the losses are about the same magnitude.  One difficulty with taking small gains is that the losses can be excessively large.

The sweet spot is high accuracy and short holding period.  Holding longer than about three days increases risk considerably, as does accuracy below about 65%. 

The metric to maximize is risk-normalized compound annual rate of return -- CAR25.

That said, using close-to-close change one day ahead and state signals, you will be able to develop systems with accuracy in the 75 to 80% range, trades that last 1 or 2 days, and gains and losses about the same distribution.    

The key is to identify conditions unlikely to continue and take the position that gives a profit when the condition changes -- mean reversion entry, then trend following for the duration of the trade.  Any fast oscillating function will work.  Work with some that do not require integer lookback periods.  RSI works.  Detrended price oscillator works.  Simple moving average is restricted to integer lookback, so is less likely to work.  In any given period of time, you want as many signals -- zero crossings or equivalent -- as trades.  So long lookback indicators [such as RSI(14) or MACD(12,26,9)] are too slow and will not work.  Try to use only one -- but at most two.  Additional indicators increase overfitness at the cost of poor generalization.

Pay very close attention to good learning practice -- in-sample fitting, followed by out-of-sample validation, followed by trade-by-trade management. 

Best,  Howard


----------



## Gringotts Bank

This looks ok and has a high accuracy, but falls apart on walk forward.  DAX/daily/5 contracts/$5comm per contract.  Not sure what to do - any suggestions appreciated.


----------



## tech/a

What's the code


----------



## Gringotts Bank

tech/a said:


> What's the code



The code is based on what Howard was just saying.  Oversold DPO (small period), buying end of day and selling fairly soon after.  And I added a few little tweaks, obviously.


----------



## Wysiwyg

Really, there is no "competition". The trader is not competing against another trader but simply trading the price action as it unfolds. When a company/individual can provide a claimer stating "historical results are indicative of future outcomes" then there will be no more auction.


----------



## howardbandy

Gringotts Bank said:


> This looks ok and has a high accuracy, but falls apart on walk forward.  DAX/daily/5 contracts/$5comm per contract.  Not sure what to do - any suggestions appreciated.
> 
> View attachment 69419



Greetings --

The short answer is that the model overfit the in-sample data during learning and does not fit the out-of-sample data.  

Some explanation might be helpful.

The system development process is classical learning using the scientific method.  I do not understand why, but the trading community has been super slow to recognize that systems to predict stock direction (trades) are similar in almost all respects to systems such as those that predict loan default.  It is critical that the data processed for prediction has the same distribution, with respect to the signal being identified, as the data processed for model fitting.

To develop a system that will predict whether a borrower will repay a loan, the lender gathers data that hopefully has some predictive value from a large group of customers, some of whom repaid and others whom have defaulted.  If conditions for the period the data represents are relatively constant, the distribution of any randomly chosen subgroups will be the same as the distribution of the entire group and, importantly, of future customers.  With respect to loan repayment, the data is stationary.  The future resembles the past.  

The data scientists develop the model that goes with the data by selecting a random subsample of the data and fitting the rules to it.  This is the "training" data and fitting is the learning process.  This is "supervised" learning -- each data point has values for the predictor variables (income, job history, etc) and the loan repayment -- the target -- is known.  The fitting process is a mathematical process of finding the best solution to a set of simultaneous equations.  aX = Y.  Where X is a large array of values for the indicator variables and y is a column array of values for the target.  The model is the array "a" -- the coefficients of the solution.  There will be a solution -- there will be an "a" array -- whether the model has predictive capability or not.  

The model -- the rules -- may have identified one or more important features in the data that are consistently associated with probability of repayment.  But the developer cannot tell by looking at the learning process.  He or she has reserved a separate subsample that is not used at all in learning -- call it the "test" data.  As a one-time test, the model is applied to the test data.  The value of the target is known, but is only used for later reference.  A predicted value for each test data point is computed using the a array.  Comparison between the known target values and predicted target values lets the person building the model know whether the model learned general features or just fit the particular data it was given -- including all the randomness.     

The scientific method insists on two phases -- learning and testing.  Without independent testing, nothing can be said about the model or its predictions.  

The trading system development profession has largely ignored the scientific method.  Independent, one-time, testing using data that follows the training data and has not been used in development is seldom done.  Trading system developers see the equity curve from the training portion, and assume completely without justification, that future results will be similar.  

That may be true, but in order for it to be true, two conditions must hold.
1.  The future must resemble the past.  That is, the data used for learning and the data used for testing / trading must have the same distribution (with respect to the signal).  This is stationarity.
2.  The model must learn real, predictive signals rather than simply fit to the randomness of the data.  This is learning.

When building models for stationary data such as loan repayment, some model algorithms produce in-sample results that can be used to give estimates of out-of-sample performance, while other model algorithms always overfit in-sample and have no value for estimating out-of-sample performance.  Out-of-sample testing is always required before estimating future performance.

Back to trading.

A system that works in-sample has fit a set of equations to a set of data.  Whether there is true learning or not, there is always a solution -- a set of trades and an associated equity curve.  Most are bad and are discarded.  We test so many combinations that eventually one fits and the result looks good.  It might be simply an overfit solution or it might be a representation of a truly predictive system.  We cannot tell anything about the future performance without testing never-seen-before future data.  When results of this out-of-sample test are poor, it is because one or both of the two conditions do not hold.  Either the data is not stationary beyond the time period of the learning data; or the model has not learned.

I hope this helps.

Thanks for listening,  Howard


----------



## howardbandy

Wysiwyg said:


> Really, there is no "competition". The trader is not competing against another trader but simply trading the price action as it unfolds.




I have a different opinion.  I believe there is a competition.  

Share prices change from one reasonably stable level to another for reasons that cannot be assigned causes.  There is a limited amount of profit available from a given price change.  That profit goes to the traders in reasonable proportion to the trader's ability.  The best traders get the most profit.  

As an analogy.  Imagine an airplane flies over a beach.  Everyone at the beach can see it as it drops several hundred $20 bills in an area where there are few people.  There is a dash toward the money -- a competitive event where the faster and more aggressive a person is, the more money he can gather.

I do not have to just be better than my naive brother-in-law.  There are other players in the game, and they vary in ability.  David Shaw, James Simons, and Goldman Sachs will always out-compete me.  I still have to be good enough to beat most of the players.

Best,  Howard


----------



## howardbandy

Gringotts Bank said:


> This looks ok and has a high accuracy, but falls apart on walk forward.  DAX/daily/5 contracts/$5comm per contract.  Not sure what to do - any suggestions appreciated.
> 
> View attachment 69419




Hi GB --

If this is in-sample, then my post of a few minutes ago applies -- only out-of-sample results have meaning in estimating future performance.  Show us the OOS.  

If it is out-of-sample, it looks pretty good.  Dynamic position sizing will probably handle the drops.

Best,  Howard


----------



## howardbandy

howardbandy said:


> Hi GB --
> 
> If this is in-sample, then my post of a few minutes ago applies -- only out-of-sample results have meaning in estimating future performance.  Show us the OOS.
> 
> If it is out-of-sample, it looks pretty good.  Dynamic position sizing will probably handle the drops.
> 
> Best,  Howard




Which brings up a useful point.  

There is no way to tell from published results alone whether those results are:
Real-time with real money.
Paper trade.
Out-of-sample from walk forward and good technique.
Pseudo out-of-sample after many passes at fitting and testing.
In-sample.
Hypothetical.
Intentionally misleading.

Best,  Howard


----------



## kid hustlr

Why is it the smartest people in the room are always the most bearish


----------



## tech/a

kid hustlr said:


> Why is it the smartest people in the room are always the most bearish




Id re phrase that.

Why do the smartest people need to keep re wrapping the same message?


----------



## minwa

howardbandy said:


> David Shaw, James Simons, and Goldman Sachs will always out-compete me.




With 75-80% accuracy and 1:1 and short term trades (high number of opportunities + compound every few days) I don't see how you can't easily destroy Shaw's and Simons's funds performance.

They have no hope of out competing against you at their size.


----------



## ThingyMajiggy

kid hustlr said:


> Why is it the smartest people in the room are always the most bearish




Usually because they see/understand things that most can't, or know the reality of the situation.


----------



## kid hustlr

I agree with every word Howard says. It's just depressing


----------



## Gringotts Bank

howardbandy said:


> Which brings up a useful point.
> 
> There is no way to tell from published results alone whether those results are:
> Real-time with real money.
> Paper trade.
> Out-of-sample from walk forward and good technique.
> Pseudo out-of-sample after many passes at fitting and testing.
> In-sample.
> Hypothetical.
> Intentionally misleading.
> 
> Best,  Howard




Hi Howard,

I did say that it falls apart on walk forward, which implies these are in-sample results.  It is simply an optimized backtest with a high (90+%) win rate, and I'm well aware that in itself this has no immediate value whatsoever.  It may _lead _somewhere, it may not.  Over the years I've had hundreds of similar curves which I have ended up discarding.  I'm well beyond the point of getting excited about a nice looking backtest.  OOS curve is all over the shop and not worth posting.


----------



## Gringotts Bank

kid hustlr said:


> I agree with every word Howard says. It's just depressing




As traders, I believe what we should be aiming for is a mix of Howard's intelligence, minwa's guile, skc's detachment, TH's nimbleness and Canoz's market knowledge.  That's your ultimate trader right there, if you could package it.

Hyper-intelligence _without an ability to switch it off_ will lead to negative mind states and poor trading performance.

Also, we have no idea of anyone else's performance.  Someone like Capblack or Skyquake could be quietly thrashing us with huge returns and no one would know.


----------



## Wysiwyg

tech/a said:


> Id re phrase that.
> 
> Why do the smartest people need to keep re wrapping the same message?



If the message is a huge Python is gonna eat us all then I'm outa here man.


----------



## minwa

Wysiwyg said:


> If the message is a huge Python is gonna eat us all then I'm outa here man.







Barclay's index of all funds/CTA they track, no indication that systematic has gained any edge over discretionary in recent years. Discretionary have half the draw down because they are not stuck to a rigid set of rules that takes every setup treating them the same regardless of other factors.


----------



## minwa

Systematic traders have also performed the same since 30 years ago despite all the advancements in computers and technology.


----------



## OmegaTrader

tech/a said:


> Id re phrase that.
> 
> Why do the smartest people need to keep re wrapping the same message?




What is the message?

That we will never win at trading because you need to have advanced computer knowledge or a lot of resources like the big boys?

If there is no chance of winning 

Might as well close down aussie stocks then.....

Invest in an etf and give up.


----------



## craft

OmegaTrader said:


> What is the message?
> 
> That we will never win at trading because you need to have advanced computer knowledge or a lot of resources like the big boys?




If you think your beat - you will be right.

If you think you'll win you could possibly be wrong - so what.

What a load of defeatist crap this thread is.  Be careful what you read - it eventually effects what you think.


----------



## OmegaTrader

craft said:


> If you think your beat - you will be right.
> 
> If you think you'll win you could possibly be wrong - so what.
> 
> What a load of defeatist crap this thread is.  Be careful what you read - it eventually effects what you think.




Can you please clarify what you are referring to.

I was being sarcastic to make a point..


----------



## craft

OmegaTrader said:


> Can you please clarify what you are referring to.
> 
> I was being sarcastic to make a point..



Sorry Omega - I was not referring to you rather the general content of this thread before I gave up reading and just quoted your post as the last post. I share the point you make.


----------



## tech/a

OmegaTrader said:


> What is the message?
> 
> That we will never win at trading because you need to have advanced computer knowledge or a lot of resources like the big boys?
> 
> If there is no chance of winning
> 
> Might as well close down aussie stocks then.....
> 
> Invest in an etf and give up.




Id be disappointed if that's how you or anyone else reads these posts.
They way I see it and understand it is this.

Everything changes and the financial markets are changing at a pace that is unprecedented.
The power of computers has change so dramatically that the Apollo 11 landing had the computing power of a current mobile phone infact one 3 yrs older than those today.

100,000s of people are more savvy today than yesterday. Computers are even learning from Computers. Yet there is in this thread an inherent negativity to the inevitable. Its here its available to you and I 1000s will make them selves available to it. More will learn and improve.

Why wouldn't anyone serious about making a few $$s in the financial markets not want to avail themselves to any tool that will not only help them shed light on a path they choose wether Fundamental OR Technical but answer the myriad of questions we all want answers to but are either afraid to ask, In capable of finding an answer remotely accurate or blissfully ignorant to the questions you should be asking and finding the answers to.

Its another very powerful tool
More than P&Ls or Balance sheets
VSA
Elliott
Candlesticks
and the list goes on and on.

Its there just like the computer was.
You had to learn how to use it. If you wanted to be better in certain areas you learnt Word and Excel and the 1000s of packages of software available. Or you could close your eyes lock yourself in a room with your adding machine and fountain pen and live miserably ever after.

You don't have to accept the challenge of exploring knowledge and implementation, but ignoring it will be at your peril. You'll be a complete fossil in 5 yrs.



craft said:


> *If you think your beat - you will be right.
> 
> If you think you'll win you could possibly be wrong - so what.*
> 
> What a load of defeatist crap this thread is.  *Be careful what you read - it eventually effects what you think*.




Monty Pythons

"*Its only a flesh wound " comes to mind.
Our Friend the knight faired well-------that's what!*

In* RED *never a truer word spoken ---- not only reading but learning!

*Its not a defeatist thread its a thread of awakening!*

Everything worth while takes effort. To some the effort will be minor
to others it will be mammoth and like me they may need to employ those who are 
far more capable in areas to help.


----------



## minwa

tech/a said:


> Everything changes and the financial markets are changing at a pace that is unprecedented.




For those us (99.99%) who can't trade in the microseconds/arbitrage scope, we can only trade price on a time frame where it hasn't changed. More efficient only, maybe. Price is driven by human nature, not machines. Human nature has not evolved.



tech/a said:


> 100,000s of people are more savvy today than yesterday. Computers are even learning from Computers. Yet there is in this thread an inherent negativity to the inevitable. Its here its available to you and I 1000s will make them selves available to it. More will learn and improve.




So do you think more people are more profitable today than 20 years ago as computers are more widely available ? You would be wrong.



tech/a said:


> Its another very powerful tool
> More than P&Ls or Balance sheets
> VSA
> Elliott
> Candlesticks
> and the list goes on and on.




That's odd, why would Howard add the lesser powerful RSI into it then ?



tech/a said:


> You don't have to accept the challenge of exploring knowledge and implementation, but ignoring it will be at your peril. You'll be a complete fossil in 5 yrs.




Heard that 5 years ago. 10 years ago too probably. Still not a fossil. Still see plenty (majority) of people profitable using "outdated" methods without advanced machine implementation.



tech/a said:


> thread of awakening!




Most would require some evidence of "outdated" methods failing/declining of performance or machines based systems with dominating performance to be awakened. I haven't seen (I basically spend 6-12 hours+ everyday in market related stuff) any recurring evidence of that. Maybe they are all silent with smashing performances - in that case I will miss out as I can't invest the time to delve into such a deep subject without any signs that it's the "future". Sorry but one Captain Cook who says he doesn't personally trade and that traditional methods are "outdated" and over when I see too much anecdotal evidence that those methods are performing the same as decades ago just won't cut it for me.

You have found it to be stronger than your TA personally though so kudos on you and maybe others will do the same too. That is encouraging to me..to see a wide diversity of methods working which dispels any fears of over saturation and dying out of a particular, especially outdated method.


----------



## OmegaTrader

tech/a said:


> Id be disappointed if that's how you or anyone else reads these posts.
> They way I see it and understand it is this.
> 
> Everything changes and the financial markets are changing at a pace that is unprecedented.
> The power of computers has change so dramatically that the Apollo 11 landing had the computing power of a current mobile phone infact one 3 yrs older than those today.
> 
> 100,000s of people are more savvy today than yesterday. Computers are even learning from Computers. Yet there is in this thread an inherent negativity to the inevitable. Its here its available to you and I 1000s will make them selves available to it. More will learn and improve.
> 
> Why wouldn't anyone serious about making a few $$s in the financial markets not want to avail themselves to any tool that will not only help them shed light on a path they choose wether Fundamental OR Technical but answer the myriad of questions we all want answers to but are either afraid to ask, In capable of finding an answer remotely accurate or blissfully ignorant to the questions you should be asking and finding the answers to.
> 
> Its another very powerful tool
> More than P&Ls or Balance sheets
> VSA
> Elliott
> Candlesticks
> and the list goes on and on.
> 
> Its there just like the computer was.
> You had to learn how to use it. If you wanted to be better in certain areas you learnt Word and Excel and the 1000s of packages of software available. Or you could close your eyes lock yourself in a room with your adding machine and fountain pen and live miserably ever after.
> 
> You don't have to accept the challenge of exploring knowledge and implementation, but ignoring it will be at your peril. You'll be a complete fossil in 5 yrs.
> 
> 
> 
> Monty Pythons
> 
> "*Its only a flesh wound " comes to mind.
> Our Friend the knight faired well-------that's what!*
> 
> In* RED *never a truer word spoken ---- not only reading but learning!
> 
> *Its not a defeatist thread its a thread of awakening!*
> 
> Everything worth while takes effort. To some the effort will be minor
> to others it will be mammoth and like me they may need to employ those who are
> far more capable in areas to help.





Yes understanding quantitative analysis is a useful tool to have at your disposal.
Yes computers are becoming more and more advanced.
Yes it can help with the $$

It is a mammoth task to learn...

Steady steady,  hedge my bets

I just disagree with the current established view that something has to be quantified by numbers.
That only by quantifying it with advanced qunatitiave analysis is it then acceptable.
That being a quant is the only way and that without numbers you will be a mammoth

The people who caused the GFC quantified insurance and housing loans....

Then the models failed and they asked the people for a bailout... 

I am pretty sure a rational person would have seen the loans were unreasonable if they just went and interviewed a couple of borrowers.

Human nature caused the crash and PHD's in suits legitimised it with numbers.

But herd all followed the quants numbers and crashed the world economy...

People trusted the numbers and quant analysis

I am just being the devils advocate to prove a point again!


----------



## Wysiwyg

tech/a said:


> Everything changes and the financial markets are changing at a pace that is unprecedented.



Still waiting for actual evidence where price action is different. Proof. The truth is price action has not changed and the only thing "unprecedented" is the hype.


----------



## tech/a

minwa said:


> For those us (99.99%) who can't trade in the microseconds/arbitrage scope, we can only trade price on a time frame where it hasn't changed. More efficient only, maybe. Price is driven by human nature, not machines. Human nature has not evolved.




Howard suggests the sweet spot is 3 days from his research. Clearly this may differ but longer timeframes are becoming harder to trade and "Predict ' Going forward than they were in years past.
Long periods of trending EG 2001-2008 are now not happening. Its not saying that micro robotic machine learning trading is going to be the only player in years to come. mind you if you have one---




> So do you think more people are more profitable today than 20 years ago as computers are more widely available ? You would be wrong.




I don't know do you? *But also see below.*



> That's odd, why would Howard add the lesser powerful RSI into it then ?




Not odd at all. Everything I mentioned above and everything I use and Howards RSI give us indicators or setups which generate signals to buy/sell/hold/place stops/position size/blah. By adding the now available skills and computing power to our trading we can answer many questions we couldn't answer before.
Including but not limited to.
(1) Does it/they work.
(2) If I add or sub tract this or that how does that impact my trading?
(3) Is there an optimum---sweet spot.
(4) Are my ideas profitable.

I'm sure Howard could add way more to this list.




> Heard that 5 years ago. 10 years ago too probably. Still not a fossil. Still see plenty (majority) of people profitable using "outdated" methods without advanced machine implementation.




*See below*
Accessibility to really advanced computer software and data analysis education to the general public is only now becoming available. The packages and education which went before us in the blooming computer age was at best rudimentary and at worst money making quackery. It really is only in recent years that quality people have become involved in making the sort of knowledge available to you and I. And the computer software and power is now at a point where if used properly can be very helpful.
In my opinion making it clearer *what doesn't work more so than what does.*





> Most would require some evidence of "outdated" methods failing/declining of performance or machines based systems with dominating performance to be awakened. I haven't seen (I basically spend 6-12 hours+ everyday in market related stuff) any recurring evidence of that. Maybe they are all silent with smashing performances - in that case I will miss out as I can't invest the time to delve into such a deep subject without any signs that it's the "future". Sorry but one Captain Cook who says he doesn't personally trade and that traditional methods are "outdated" and over when I see too much anecdotal evidence that those methods are performing the same as decades ago just won't cut it for me.




I don't see them (Out-dated as you put it) methods vanishing. To the contrary I see them being enabled and empowered. This Duck Trades---but he doesn't have to trade. He doesn't* have to do* a great deal actually. I love the challenge of business and Trading even if only when I feel like it is one.
This is another very enlightening challenge.



> You have found it to be stronger than your TA personally though so kudos on you and maybe others will do the same too. That is encouraging to me..to see a wide diversity of methods working which dispels any fears of over saturation and dying out of a particular, especially outdated method.




It doesn't replace T/A it adds immensely to it. Not just in possibilities through testing but in answering many of the accepted Truisms and questions that I didn't even know I should be asking.


----------



## minwa

tech/a said:


> Howard suggests the sweet spot is 3 days from his research. Clearly this may differ but longer timeframes are becoming harder to trade and "Predict ' Going forward than they were in years past.
> Long periods of trending EG 2001-2008 are now not happening.




Have you looked at a chart yourself or simply going on what Cap Cook is saying ? Chart of the S&P..most would not agree with you that long periods of trending are now not happening..quite the opposite actually the trends seem better than 01-08 do you not think ?








tech/a said:


> Not odd at all. Everything I mentioned above and everything I use and Howards RSI give us indicators or setups which generate signals to buy/sell/hold/place stops/position size/blah. By adding the now available skills and computing power to our trading we can answer many questions we couldn't answer before.
> Including but not limited to.
> (1) Does it/they work.
> (2) If I add or sub tract this or that how does that impact my trading?
> (3) Is there an optimum---sweet spot.
> (4) Are my ideas profitable.
> 
> I'm sure Howard could add way more to this list.




So statistics and optimization ? They've been around long before modern computing powers. And to point (1) Does it/they work. - if you need a computer to answer that you really shouldn't be trading.





tech/a said:


> Accessibility to really advanced computer software and data analysis education to the general public is only now becoming available. The packages and education which went before us in the blooming computer age was at best rudimentary and at worst money making quackery. It really is only in recent years that quality people have become involved in making the sort of knowledge available to you and I. And the computer software and power is now at a point where if used properly can be very helpful.
> In my opinion making it clearer *what doesn't work more so than what does.
> *



*
*
Amibroker ? Been around since 1995. I'm sure other alternatives too. Yes now data churning are faster but so is pulling up charts for a discretionary trader/filtering criteria on balance sheets for a fundamentalist. Things have gotten more efficient - they have NOT EVOLVED to the point where having the latest computing power and packages gives you an edge for the retail trader. 





tech/a said:


> I don't see them (Out-dated as you put it) methods vanishing. To the contrary I see them being enabled and empowered. This Duck Trades---but he doesn't have to trade. He doesn't* have to do* a great deal actually. I love the challenge of business and Trading even if only when I feel like it is one.
> This is another very enlightening challenge.




Howard deems them outdated and over in his first post. I commend you that you love the challenge and the mental stimulation. Keeps your mind sharp from the mental workout. Personally bottom line I'm here to make a living and if there was a unchallenging way to do it I would.


----------



## barney

Gringotts Bank said:


> As traders, I believe what *we should be aiming for is a mix of Howard's intelligence, minwa's guile, skc's detachment, TH's nimbleness and Canoz's market knowledge*.  That's your ultimate trader right there, if you could package it.




Quote of the week for mine Gringo!!


----------



## DeepState

The ultimate idea to is to find an edge that works well enough to make money on average through time...or, at least, be entertaining enough to keep your interest.

To this end, "Quant", with this broad meaning encapsulating:
+ data sciences which allow the gathering and analysis of vast amounts of information;
+ statistics which allows an investor to utilise another means to test their thinking;
+ quantitative finance which allows for increasingly rigorous and sophisticated concepts to be applied to pricing, portfolio optimisation and trading (for example); and
+ technology to harness information for investment/trading, organise this information rapidly and acccurately (as coded) and place ideas into the market quickly and efficiently...

can bring tools which are helpful to some investors.  So it is with the power of speech and the ability to write, for example.  It is possibly helpful but can be misapplied or abused accidently or purposefully.

Not all who can read do so with equal value.  Not all who can speak have much with is worth saying. Not all who can codify a random forest can see the wood for the trees. Running a neural net over a single time series for prediction of a share price may not very smart. Still, I'd rather be able to speak and write than not.

Anyone can have an opinion.  Very few opinions in investment withstand even the slightest quant analysis.  Quant analysis helps with finding the truth.  It will sometimes fail in this endeavour or give unreasonable confidence in its assertions.

"Individual results may vary."

I feel reasonably confident that, on the whole, speech and reading have added to human capacity.  So it is with quant.


----------



## tech/a

OmegaTrader said:


> Yes understanding quantitative analysis is a useful tool to have at your disposal.
> Yes computers are becoming more and more advanced.
> Yes it can help with the $$
> 
> It is a mammoth task to learn...




I agree if your like myself. I don't think I need a PHD to know what I need to know. I can use the skills of a quant to answer questions and guide me in what I want/need to know and I can be guided to understand what I need to know---*I don't have to know it ALL*---that's exciting.



> Steady steady,  hedge my bets




Money/trade management part of the equation.



> I just disagree with the current established view that something has to be quantified by numbers.
> That only by quantifying it with advanced qunatitiave analysis is it then acceptable.
> That being a quant is the only way and that without numbers you will be a mammoth




That's fine you can certainly hold that view, but every time I hop on a plane, drive over a bridge, take a
prescription drug, listen to my Linn Sondec speakers, or drive my fPace, Im glad quants were involved.
Personally if I'm placing large wads for $$s in the market then I (Personally again) want every opportunity available to me involved and explored.



> The people who caused the GFC quantified insurance and housing loans....
> 
> Then the models failed and they asked the people for a bailout...




Grab a hold of the book
"The Crisis of Crowding."
The quants were warning the market way before the Crap hit the fan.
Even Buffett the champion of "every man' very nearly became a catastrophic statistic having made a bid for LTCM (Long Term Capital Management) in 1998---fortunately the bid failed.



> I am pretty sure a rational person would have seen the loans were unreasonable if they just went and interviewed a couple of borrowers.
> 
> Human nature caused the crash and PHD's in suits legitimised it with numbers.




There was a sad lack of rational people leading up to the crash. Loans given with no collateral to people with no jobs like free chips at a casino. Human nature *played a part* in the crash, over leveraging of worthless paper caused the catastrophic domino effect which if you think is never capable of happening again---think again---its still being in wound!



> But herd all followed the quants numbers and crashed the world economy...
> 
> People trusted the numbers and quant analysis
> 
> I am just being the devils advocate to prove a point again!




Your not correct in this sweeping statement. Far more involved than what your alluding to.
*Quants were and are on both sides of the equation. Read the book if you can get it.*



Wysiwyg said:


> Still waiting for actual evidence where price action is different. Proof. The truth is price action has not changed and the only thing "unprecedented" is the hype.




Price action changes daily and weekly and monthly.
Today more than ever we have to tools to help us regardless of timeframe.
Would you rather dig a hole for a swimming pool with a shovel or an excavator.
Both will do the job!



minwa said:


> Have you looked at a chart yourself or simply going on what Cap Cook is saying ? Chart of the S&P..most would not agree with you that long periods of trending are now not happening..quite the opposite actually the trends seem better than 01-08 do you not think ?
> 
> View attachment 69459




Another sweeping statement, there are just as many charts supporting alternate views.
The trick of course is being able to place yourself with some confidence in front of a possible trend before it happens. Howards point *I think* is that he can do that far more often over a 3 day period than longer. You maybe able to do way better.



> So statistics and optimization ? They've been around long before modern computing powers.




So were fingers and toes and the abacas. Have advancements in computers made life easier or harder?



> And to point (1) Does it/they work. - if you need a computer to answer that you really shouldn't be trading.




Complacency and arrogance have bought down many in business. Those who fail to grow with technology can be eventually taken over by it. *Sure there are those who will survive*. Youll survive---but could you be better? Those that do tend to be in niche areas. Nothing wrong with that. But if I want to have multiple ways of profiting from the same pool would I rather have the power of computing behind me -- or if I wanted to see how I could improve what I'm doing would I welcome some quant in put?
For me yes---for others maybe you--- no.



> Amibroker ? Been around since 1995. I'm sure other alternatives too. Yes now data churning are faster but so is pulling up charts for a discretionary trader/filtering criteria on balance sheets for a fundamentalist. Things have gotten more efficient - they have NOT EVOLVED to the point where having the latest computing power and packages gives you an edge for the retail trader.




Its very good on a retail level but on a quant level it is very restrictive. You cant do with it what you can do with python. (I'm not a quant but I'm working with (Paying). them and that's what they all say).



> Howard deems them outdated and over in his first post. I commend you that you love the challenge and the mental stimulation. Keeps your mind sharp from the mental workout. Personally bottom line I'm here to make a living and if there was a unchallenging way to do it I would.



Wysi
	

		
			
		

		
	

View attachment 69447


I too--- work for myself. I'm for ever on the look out to find ways of being way way better than my competitors. One of the biggest ways I've found is being surrounded by people they don't have.
If I'm not an expert in a field I need to be (If I'm making a living) Then Ill find the best I can.
If I can have some equipment they haven't got------ then if it puts me even further out of their reach as a competitor Ill either buy it or better still lease it!.

I search out challenge. *Challenge is the engine room*. Without it you'll learn very little of worth.

Just saw Deep states Reply which crossed mine.
Very well said. Your writing a speaking are beyond my own skill set.


----------



## minwa

tech/a said:


> Another sweeping statement, there are just as many charts supporting alternate views.




Please post some, I'd really like to see how you concluded that markets have changed as a result of quants now on major markets that most people trade. Howard made that conclusion based on the system that he trades. Does he system cover every system ? I doubt it..Also his fear that a meltdown is coming to reduce time frame held. That's an opinion that hasn't happened yet.


----------



## tech/a

minwa said:


> Please post some, I'd really like to see how you concluded that markets have changed as a result of quants now on major markets that most people trade. Howard made that conclusion based on the system that he trades. Does he system cover every system ? I doubt it..Also his fear that a meltdown is coming to reduce time frame held. That's an opinion that hasn't happened yet.




I'll grab some when I get a little more time.
But I don't know where you think the markets have changed because of quants.
My only point is that in my view what they have to offer should be included in 
Market research.


----------



## minwa

tech/a said:


> Clearly this may differ but longer timeframes are becoming harder to trade and "Predict ' Going forward than they were in years past.
> Long periods of trending EG 2001-2008 are now not happening.




OK I was under the impression you think that quants have changed the market, if not quants then what do you propose is changing the longer timeframes and that they are harder to predict than before ?


----------



## tech/a

minwa said:


> OK I was under the impression you think that quants have changed the market, if not quants then what do you propose is changing the longer timeframes and that they are harder to predict than before ?




Economic world events.
Eg the end of the China boom
GFC not only in the US but also through Europe.
This has had massive impact on growth.

Just a couple of examples.
I'm sure you and others can think of many more Macro economic 
Events which have caused un certainty


----------



## minwa

tech/a said:


> Economic world events.
> Eg the end of the China boom
> GFC not only in the US but also through Europe.
> This has had massive impact on growth.
> 
> Just a couple of examples.
> I'm sure you and others can think of many more Macro economic
> Events which have caused un certainty




OK, that's the reason you think that market has changed it's "trendiness". I just fail to see that they have actually changed looking at the 2 major markets stocks and bonds.


----------



## tech/a

If you've been trading them and as a trend follower using longer holding times 
I can understand your point of view.


----------



## Trendnomics

Wow this thread is terrible!


----------



## OmegaTrader

Trendnomics said:


> Wow this thread is terrible!
> 
> 
> Funny pic
> lol
> But was that sarcasm???


----------



## craft

Edge is identification of risk which on a probabilistic basis is mispriced In conjunction with a system to exploit the mispricing.

Using data to sift out the possible existence of an edge and know it by no more than a posative expectancy calculation on historical data is fraught with danger. Most are mirages conjured by people snooping around in deserts of data destined to disappear as soon as you walk towards them.

Those oasis that may actually be real and stationary enough to be found can never sustain the throngs trying to survive off them for long. So why play such a stupid game unless you are indeed the fastest draw Quant in the game?

If you can't beat 'em, join them would have to be the dumbest advice in trading that I have ever heard.yet that is what the gurus on this thread seem to be saying. You have to get Quant to have a chance. I say rubish. I reckon they are just trying to baffle with bull**** either to hide the fact they are still stuck in the beginners cycle looking for new tech to give them the elusive answer to what eludes them or they are trying/preparing to sell something, even though of course they don't need to because  this new thing is the holy grail and money can be made at will if only they desired to do so.

When I see people who  I suspect are trying to fudge it with complexity totally misconstrue Buffett as part of their bedazzlement they wander into territory Where I know they have no idea what they are talking about and confirm my suspicions.

If you want a true Buffett direction in relation to this thread try this quote.
"I don't look to jump over 7-foot bars: I look around for 1-foot bars that I can step over."

The 7 foot bar is finding your edge by reverse engineering data. It's not robust and the guy with the most money can buy the most toys and the most toys wins – at least until nobody wants to play with you any more. Even if you win you lose and you have to go find another group who's mummies have unwittingly made you lunch.

The 1 foot hurdles is understanding you edge qualitatively and putting in the elbow grease to effectively and safely exploit it on a probabilistic basis. Maybe you might even find that in an arena of physical economic business production – maybe a business that makes lunches,  mmmmmm lunches forever without the hassle of who you going to take the next one off.

 Data analyses can be a tool to gain  knowledge and judgement. However plenty who can analyse data efficiently  will never see the forest for the trees. Whilst the better you qualatativelly  understand the cause of mispriced risk you seek to exploit  the less reliant on historical  data you will be.


----------



## Trendnomics

^^^^^^^^^^


----------



## tech/a

There will always be diversity in opinion and view.
That's the key to innovation.

Exciting!


----------



## barney

craft said:


> Data analyses can be a tool to gain  knowledge and judgement. However plenty who can analyse data efficiently  will never see the forest for the trees. Whilst the better you qualatativelly  understand the cause of mispriced risk you seek to exploit  the less reliant on historical  data you will be.




Great post Craft.

Been an interesting thread  ... my 2 bobs worth.

The failure of "Long Term Capital Management" in the late 1990's is a good example of quantitative analysis going off the rails and why perhaps this new breed of super Quants who are supposedly going to take all us little trading guys to the cleaners might be just a bit exaggerated.

LTCM had some of the biggest swinging Q's of the era and killed the market for 3 years before blowing $4.5 Billion (plus change) in less than 6 months!!

I think tech/a used the analogy of digging a hole with an excavator or a shovel which is probably a totally reasonable analogy if you are a big honcho in the trading world ..... no doubt the excavator will do thousands of times more work in the same period, however the cost of either buying the excavator or paying the dude who knows how to drive it is still relatively priced compared to using my Bunnings shovel, plus I'm a lot more likely to see that sewer pipe I'm about to crunch if I'm up close with my shovel

If you run a hedge fund and have your wages paid by multitudes of punters who don't want the hassle of trading for themselves I'm sure you need Quants to compete with the other companies who are doing the same thing.  At this stage I just can't see a lot of relative value or disadvantage for us little guys.


----------



## jjbinks

I think important thing is to be open to using/trying the different tools. 

If you are on this site you are probably reading/practising and learning to improve your trading. Statistics/programming are just another tool or area to learn about which will help you improve or even find an edge.


----------



## minwa

barney said:


> If you run a hedge fund and have your wages paid by multitudes of punters who don't want the hassle of trading for themselves I'm sure you need Quants to compete with the other companies who are doing the same thing.  At this stage I just can't see a lot of relative value or disadvantage for us little guys.




Even with all the resources and brainpower at the hedge fund level, it's massive competition that has hit a brick wall years ago. http://www.businessinsider.com.au/top-goldman-quant-quant-trading-is-dead-2009-12

If Goldman is not confident I don't see how any of us tiny flies can be with our resources. Only select few have thrived like Renaissance - they have resources and technology we probably don't even have the vocabulary for.


----------



## Gringotts Bank

minwa said:


> ... they have resources and technology we probably don't even have the vocabulary for.




I'd be investigating how the evening news bulletins, social media and the major finance dailys can be manipulated to drive the overnight or next day's moves.  Such things are already monitored, but afaik, no one has been caught profiting from the manipulation of sentiment.  It's absolutely possible to do such things.  The Russians tried to influence the US elections this way but as far as I can see their efforts were brute force, unsophisticated and ineffective.

Here's someone who knows how to do it...


----------



## tech/a

jjbinks said:


> I think important thing is to be open to using/trying the different tools.
> 
> If you are on this site you are probably reading/practising and learning to improve your trading. Statistics/programming are just another tool or area to learn about which will help you improve or even find an edge.




Exactly
Another tool
Not Armageddon by Quant!!

Jjgets it.


----------



## ThingyMajiggy

Right, so let's see something in action, Howard, tech, anyone? I think it's clear where everyone stands so we might as well start having something useful in this thread. Show us some Quant tool, or ML predictions, don't have to show any code, just the output if you like.


----------



## howardbandy

Greetings --

This thread has more diversity in responses than I expected it would.  If I may, let me refocus on what I believe are some of the critical issues we all face, and how we might prepare for them.

Some of the underlying assumptions and definitions:

1.  Trading is buying and selling financial instruments with the intent of selling at a higher price than buying.  Investing is buying and never selling.  My comments apply to trading.

2.  We are trading in order to increase our net worth.  

3.  Trading is a zero sum game, slightly biased in alignment with changes in GDP, workforce, and productivity.  

4.  Each trade has a buyer and seller.  For that trade, the net is zero sum -- less commissions that reduce the profit on both sides.

5.  Repeating a well known management tenet -- "It is not possible to manage a process without being able to measure that process."  

6.  Trading is competitive.      

7.  Traders stop trading when the drawdown of their trading account exceeds their personal risk tolerance.  

Given those, and probably some others that list members will add, there are rational techniques that traders can use to help them be among the winners rather than the losers.

1.  Every set of trades can be decomposed into a set of one-day holdings.  When a trade is held multiple days, there are no transactions between days, but there is accounting and -- importantly -- opportunity for management each day.  

2.  Each equity curve is a single data point representing one ordering of the trades that make it up.  The risk of drawdown we are exposed to when we trade is determined by the distribution of that set of trades.  The very best we can hope for in terms of future performance is that the distribution does not change.  We cannot expect the order of trades to be repeated.  

3.  Given a set of trades and a forecast horizon, we can create many equally likely trade sequences, measure the drawdown associated with each, and compute the distribution of drawdown.

4.  Given the distribution of drawdown and the trader's personal risk tolerance, we can compute the maximum safe position size associated with that set of trades.  I call it "safe-f" in the presentations I make.  Money management is critically important in maximizing profit.  Safe-f is the position size for those trades that maximizes account wealth while holding drawdown to the trader's tolerance.  

5.  After adjusting the position size, the final equity after a period of trading where trades come from that same distribution can be computed.  Pick a point on the distribution as the metric by which alternatives will be measured.  I recommend the 25th percentile, and call the metric "CAR25" -- the Compound Annual Rate of Return at the 25th percentile of the estimated profit from risk-normalized trades.

6.  Every set of trades has its own CAR25.  A rational trader will develop several trading systems, each with its own set of trades, and compute CAR25 for each of them.  The one that should be traded is the one with the highest CAR25.  It is the Dominant system.  Given a specified level of risk, it is the best system -- the one that will provide the highest return.

Like it or not, everything I have stated about the distribution of trades and risk up to this point exists.  It is not a theory.  Wishing it were not so is as futile as wishing the value of pi was 3.  Computations would be so much simpler than the unwieldy 3.14159...  

Whatever technique you use, the system has a CAR25.  Ignore that at risk to your trading account.  

We can use our knowledge to give ourselves an advantage over at least some of our competition.  We can manage our trading to keep ourselves maximally profitable while avoiding account-destroying drawdowns.

7.  CAR25 computed for the set of trades was static.  As the system is traded, new trades will be added to the distribution.  Risk will change.  CAR25 will change.  If we ignore these changes we will either lose opportunity for profit or realize drawdown beyond out tolerance.  A trading management system based on sequential learning and Bayesian analysis monitors trade-by-trade results and adjusts position size as necessary to keep risk normalized and maximize CAR25.

So ---- we need a technique for doing all of this.  For developing a system, gathering the set of trades that are the best estimate of future performance, estimating risk, computing safe-f, estimating CAR25, and managing the trading of the result.  

Our alternatives for trading system development are:

A.  Subjective trade selection.  In which I include all techniques that do not have explicit rules that signal trade entry and trade exit.  A set of subjectively chosen trades can be analyzed using the techniques I describe.  Trader's who are successful with subjective techniques should continue to use them and ignore me.

B.  Objective trade selection.  A set of rules coded into the language of a computer-based trading system development platform that specify all entries and all exits.  I recommend using state signals which are evaluated every day, giving management opportunities that coincide with accounting measurement.

All of the above material is preliminary to the original posting in this thread.  All of it is explained in much more detail in my books, including downloadable computer code, and in the videos I have posted to YouTube.  

We have two broad techniques for objective trading system development:

B1.  Traditional platforms such as AmiBroker, TradeStation, NinjaTrader, etc

B2.  Machine learning.

(Trading system management, including computation of trade-by-trade position size, is done in a separate computer program and should be a separate discussion topic.  See the brief note below.)

------------------------------

My original post suggested that some of the biggest and best of our trading counterparties -- our competition -- is using machine learning.  And we might consider doing so ourselves if we want to be among the winners.  

To briefly reiterate the differences between B1 and B2 --

B1.  Traditional platforms build models that are primarily decision trees.  These are sets of if-then rules that play "20 questions" with the data, following branches of logic and eventually arriving at a leaf of the tree with a label Buy or Sell.  In terms of state signals, the state for the next day-to-day holding advice at the leaf would be beLong, beShort, or beFlat.  

Decision trees have advantages of being easy to build, fast to build, tolerant of outliers, and easy to interpret.

They suffer from the disadvantages of using decision rules that are parallel to axes, being sensitive to branching, and too easily overfit.  

Traditional platforms use the paradigm of "compute an indicator, then see what happens after."

B2.  Machine learning expands model types.  Decision trees are still available.  Others include neural networks, support vector machines, genetic algorithms, ensembles, and dozens more. 

Machine learning uses the paradigm of "identify a profitable trading opportunity, then see what happened earlier."

-----------------------------

Note that the trading management system is separate from trading system development.  Since it uses only the sequence of recent trades, it is itself independent of the method used to issue the signals.  It does require a computer program to carry out its calculations.  The example of such a program I recommend is written in Python and published in my materials.

-----------------------------

I understand that some readers have different points of view.  Some are on topic and appreciated.  Others considerably off topic, some picking at nits that are not relevant to the main discussion, some denying facts, some wishful thinking.  

The intention of my original post was, and with this post continues to be, merely to let the readers of Aussie Stock Forum know what tools are being used by some of the best traders and how our members might use similar tools to their own advantage.  

Thanks for listening,
Howard


----------



## howardbandy

minwa said:


> Even with all the resources and brainpower at the hedge fund level, it's massive competition that has hit a brick wall years ago. http://www.businessinsider.com.au/top-goldman-quant-quant-trading-is-dead-2009-12
> 
> If Goldman is not confident I don't see how any of us tiny flies can be with our resources. Only select few have thrived like Renaissance - they have resources and technology we probably don't even have the vocabulary for.




Greetings --

That article is from 2009.  

This field is changing at an astonishingly rapid rate.  Consider anything older than two or three years as dated -- perhaps still valid, but needing review.  

Best,  Howard


----------



## howardbandy

ThingyMajiggy said:


> Right, so let's see something in action, Howard, tech, anyone? I think it's clear where everyone stands so we might as well start having something useful in this thread. Show us some Quant tool, or ML predictions, don't have to show any code, just the output if you like.




Greetings --

There will not be any four line examples, such as are possible in AmiBroker or TradeStation.

Start with the material, including examples, I have published.  See pages 15 through 447 of "Quantitative Technical Analysis."

Best, Howard


----------



## ThingyMajiggy

Why not? So we have to buy your book to see any proof or evidence? Surely one of the pro Quant and ML guys can show us the end result of what you're ramping, no need to show the code behind it. Just the predictions or patterns it finds that makes it silly to not try, what you've been arguing about all thread so far.


----------



## Gringotts Bank

ThingyMajiggy said:


> Right, so let's see something in action, Howard, tech, anyone? I think it's clear where everyone stands so we might as well start having something useful in this thread. Show us some Quant tool, or ML predictions, don't have to show any code, just the output if you like.




Howard's work is as solid as it gets, from the scientific perspective.  Remember, the scientific perspective is not the only perspective.  Those with great feel for the markets can do just as well, perhaps better, imo.

This is the one I worked on earlier, showing in- and out-of-sample testing.  DAX, daily, includes brokerage, 5 contracts.  You can see how it breaks down and becomes untradable when it runs on unseen data (after the yellow line).  Still working on it.

Unfortunately, the sudden big drops that ruin the OOS curve come without warning, so dynamic position sizing may not help me here.  Howard, if you're reading, is that correct?  They seem unheralded.


----------



## minwa

howardbandy said:


> Greetings --
> 
> That article is from 2009.
> 
> This field is changing at an astonishingly rapid rate.  Consider anything older than two or three years as dated -- perhaps still valid, but needing review.
> 
> Best,  Howard




It makes it more valid that even more competition has gotten their hands on same resources and thus even more crowding out ? 

"_Quantitative Trading Systems_ was originally published in 2007, with the revised Second Edition published in 2011." 

Time for a review ?


----------



## ThingyMajiggy

Gringotts Bank said:


> Howard's work is as solid as it gets, from the scientific perspective.  Remember, the scientific perspective is not the only perspective.  Those with great feel for the markets can do just as well, perhaps better, imo.
> 
> This is the one I worked on earlier, showing in- and out-of-sample testing.  DAX, daily, includes brokerage, 5 contracts.  You can see how it breaks down and becomes untradable when it runs on unseen data (after the yellow line).  Still working on it.
> 
> Unfortunately, the sudden big drops that ruin the OOS curve come without warning, so dynamic position sizing may not help me here.  Howard, if you're reading, is that correct?  They seem unheralded.




I'm not doubting his work(nor am I doubting tech's trading ability), but I'm yet to see any of his actual trading or systems that he's using live(if any at all), or if he just writes books and does webinars. The point is that he and tech have been going on for 11 pages now arguing back and forth stating how good the Quant skillset is and how helpful it can be to your trading, so how about enough of the talk and show us, without us having to buy anything, what are they doing related to quantitative stuff that is helping their trading so much so that they're recommending everyone at least give it a go, otherwise it's no different than someone coming on ASF and pumping up his "Make 3000% in 1 month" strategy that when someone asks for the "Let's see then" all goes quiet or he diverts their attention to something they have to buy first.

It's really quite simple, as I mentioned earlier, they don't need to disclose any special little secrets or formulas, but being pro-quant guys I'm sure they have stats on how their strategies are going, if they're actually running any, which they could easily show us, or how it's helped them to find an edge or whatever. Surely there's something, we're up to 11 pages of back and forth arguments with no evidence or stats backing it up.

I personally think machine learning looks promising myself, but I don't know enough about it or haven't coded anything up myself yet so I can't show you how good or bad it is, whereas Howard clearly seems to know what he's talking about and has being doing systems for years and tech is working on some mystery Quant assisting tool, so they're both more advanced than I am, and one way to prove your point in an argument is to show evidence/proof of what you're sprouting is legit. Then we can all be amazed and our eyes opened and can start talking about the useful things


----------



## tech/a

Sam

I cant agree more.
Hopefully our private mails have helped you with what I'm involved in.


----------



## howardbandy

minwa said:


> It makes it more valid that even more competition has gotten their hands on same resources and thus even more crowding out ?
> 
> "_Quantitative Trading Systems_ was originally published in 2007, with the revised Second Edition published in 2011."
> 
> Time for a review ?




Greetings --

Yes.  And there is one.

"Quantitative Technical Analysis" is the update.  It omits quite a lot of material that is based on "traditional" technical analysis, as well as material that I no longer recommend people spend any time with.  It also focuses more on the scientific method that essentially all research and development (somehow except the trading system community) embraces.  

There are several original techniques in the QTA book; and incorporation of several that were introduced in earlier books such as Modeling Trading System Performance.  The result is a clear description of the "stack" of processes required to develop and validate trading systems, and to manage trading systems.

The QTA book contains clear and detailed explanation of the process of translation from traditional trading system development platform to a Python-based trading system platform and then to machine learning.  

A lot of the material is available at no cost -- on the book's website, in forum discussions such as this one, and in the presentations I have made and posted to YouTube.     

For trading system developers who want to stay with AmiBroker, Quantitative Trading Systems continues to be a useful book.  It is a gentler introduction than QTA.  I was planning to take the QTS book out of publication, but several people said they are using it in their classes and they want it to be available.  It continues to steadily sell a few copies every month.

-------------

Yes, the markets are becoming more efficient.  There is less profit available.  To any traders, but to retail traders using end-of-day data in particular.  Competition is definitely increasing.  There is definitely crowding out.  Testing trading systems over periods of time show decreasing profit and increasing efficiency.  All to my point to be very careful to analyze any trading system you are using or planning to use with respect to its risk, and be quick to reduce position size when any system enters a drawdown.

I hear the people who think and say that people are better at identifying profitable trades than machines.  The evidence is otherwise.  In every judgement-based task, well designed algorithms are better than human judgement.  Perhaps I need to say nearly every.  But that is changing rapidly, as algorithms are demonstrably better today than they were yesterday.  Algorithm superiority is clear in board games such as chess and recently go.  Also in medical diagnosis, identification of irregularities in medical charts and images, legal research, engineering design, etc.  

And in identification of entry and exit points for trades.  Some members of this group may be exceptions.  But most are not.  I recommend that everyone read Dr. Daniel Kahneman's book, "Thinking, Fast and Slow" for a thorough discussion of the many ways we fool ourselves.  Dr. Kahneman was awarded a Nobel Prize for his work in this field.

Best regards,  Howard


----------



## minwa

howardbandy said:


> Yes, the markets are becoming more efficient.  There is less profit available.  To any traders, but to retail traders using end-of-day data in particular.  Competition is definitely increasing.  There is definitely crowding out.  *Testing trading systems over periods of time show decreasing profit and increasing efficiency.*  All to my point to be very careful to analyze any trading system you are using or planning to use with respect to its risk, and be quick to reduce position size when any system enters a drawdown.
> 
> I hear the people who think and say that people are better at identifying profitable trades than machines.  The evidence is otherwise.  In every judgement-based task, well designed algorithms are better than human judgement.  Perhaps I need to say nearly every.  But that is changing rapidly, as algorithms are demonstrably better today than they were yesterday.  *Algorithm superiority is clear in board games such as chess and recently go.*  Also in medical diagnosis, identification of irregularities in medical charts and images, legal research, engineering design, etc.




Yes I agree with trading systems showing decreasing profit over time. Set rules are optimized on the past and the market will always seek to break them. 

I don't see the parallel between chess/go & trading. In board games you have a LIMITED number of outcomes. If your opponent moves here, there is only an X number of spots to move and outcomes to be played out. The machine will calculate those and every single possible outcomes that can be played out and beat the human. This is not true in price - there is no limit to the number of outcomes of price, it is not bound by rules of the board game that you have to put the pieces on designated spots. Poker would be a closer parallel to the markets and as Omega has posted before the human were beating the machines.


----------



## howardbandy

ThingyMajiggy said:


> Why not? So we have to buy your book to see any proof or evidence? Surely one of the pro Quant and ML guys can show us the end result of what you're ramping, no need to show the code behind it. Just the predictions or patterns it finds that makes it silly to not try, what you've been arguing about all thread so far.




Greetings --

Everything should be free?  Authors and researchers should not be compensated?  Trade secrets should be openly discussed?  

Pardon me while I resist a rant about the inequality of compensation.

Much of the material in my "for sale" books is available free -- on each book's website, in the forum threads, in the presentations I have made and posted to YouTube.

I buy, read, and study more than one hundred books every year.  I also read articles posted to website, watch YouTube videos, subscribe to Coursera courses, correspond with colleagues.  Some is free, some has a monetary cost.  Some is worthwhile, some not.  Part of being competent in a professional is continuing education.  Even if the materials are free, the time is not.  Books are inexpensive.

---------------

To your question about the patterns being found.  One of my points is that machine learning models often do not have the ease of interpretation of decision tree models produced by AmiBroker or TradeStation.  The model is, in general, the coefficient matrix of the solution to a large set of simultaneous equations.  

------------------

At the risk of making things muddier rather than clearer, here are some of the graphs that represent a trading system that were produced during development. 

Trades produced by a trading system form a distribution.  The image below is an illustration of the distribution of maximum drawdown of a machine learning system when the position size has been set so that the drawdown remains within the trader's tolerance.   That position size is "safe-f."  The terminology is that position size is such that the results are "risk-normalized." 




The image below is the distribution of terminal equity when that system is traded at safe-f.  


Any single equity curve posted would be just one data point from the distribution.  One possible equity curve that might occur if the system is traded in the future.   Several equally likely equity curves are shown in the chart that follows.  We have no idea which of them it will be, if any.




The decision whether this system is tradable or not depends on the alternatives available to the trader.  CAR25 of this system is 11.1%.  If he or she has an alternative that has higher risk-normalized CAR25, then that alternative should be traded.

Best regards,  Howard


----------



## Wysiwyg

ThingyMajiggy said:


> Surely there's something, we're up to 11 pages of back and forth arguments with no evidence or stats backing it up.



Another one of those donkey chases the carrot games. Realistically most people are not going to learn computer language because of the time and brains required. For those that do or have, maybe they will show up one day with a positive OOS equity curve and some guidance on what data they used to get it. The meat in the sandwich so to speak otherwise it will be this persistent dangling carrot theme.


----------



## minwa

ThingyMajiggy said:


> The point is that he and tech have been going on for 11 pages now arguing back and forth stating how good the Quant skillset is and how helpful it can be to your trading, so how about enough of the talk and show us, without us having to buy anything, what are they doing related to quantitative stuff that is helping their trading so much so that they're recommending everyone at least give it a go, otherwise it's no different than someone coming on ASF and pumping up his "Make 3000% in 1 month" strategy that when someone asks for the "Let's see then" all goes quiet or he diverts their attention to something they have to buy first.




To make this a fair request I offer my eyes and brains to be pitted against the machine. Will be available in a few weeks to trade against any retail machine system in a live forward going setting. I don't like my odds against the 80% win rate 1:1 RR mentioned earlier but I am happy to be humbled for fun and education. PM me if interested and we can set something up.


----------



## ThingyMajiggy

howardbandy said:


> Greetings --
> 
> Everything should be free?  Authors and researchers should not be compensated?  Trade secrets should be openly discussed?
> 
> Pardon me while I resist a rant about the inequality of compensation.
> 
> Much of the material in my "for sale" books is available free -- on each book's website, in the forum threads, in the presentations I have made and posted to YouTube.
> 
> I buy, read, and study more than one hundred books every year.  I also read articles posted to website, watch YouTube videos, subscribe to Coursera courses, correspond with colleagues.  Some is free, some has a monetary cost.  Some is worthwhile, some not.  Part of being competent in a professional is continuing education.  Even if the materials are free, the time is not.  Books are inexpensive.




No everything should not be free and the authors and researchers should be compensated, I've already mentioned no trade secrets need to be released. But what you're saying is you can't test drive the car until you pay for it.

Everyone here involved knows you don't need to reveal anything to do with the actual strategy or trading method you use, all you need is an equity curve, a % return over X period, drawdown etc. stats, you know the deal so saying all the above is just a bit silly as nothing needs to be revealed that ACTUALLY makes the profit. Just the results of it.

To me, I'm more than happy to pay for someone's service/books/software/guidance, but first I like to see evidence that what I'm entering into is legit. That just makes sense to me, proof, then you can have my money, simple. Not the other way round.

Oh and it shuts everyone else up and we don't have to have this 11 page debate about what is better 

Oh and it's kind as you've already mentioned you don't need more money.....but that's no incentive is it lol, that's just the resolutions everyone makes each new year.


----------



## DeepState

Enjoying this exchange.

For stationary systems, quant will eventually win hands down.  Stationary means that the system itself basically remains unchanged.  Laws of physics don't change and so mechanical structures outside of the quantum realm are basically stable.  If you are training an algo to play chess, the rules never change.  The algo will eventually win (and has). Medical diagnosis is mostly like this.  The learning rate of an algo designed to spot anomalies in a scan can learn from more scans that could be viewed in a lifetime.  Eventually, sort of like chess, the computer becomes better than the radiologist on their own, and possibly better outright.  This day will come.  The target never changes, the competition never evolves.  The computer just gets better and eventually, if the problem is capable of codification, the algo is the best thing at it.

When you put yourself into situations where the competition evolves...it gets trickier.  Unless that evolution is sufficiently predictable (ie. I need to know how you will react to something I have not shown you yet), it gets a bit shaky.  Situations involving humans who adapt and algos which are also seeking to adapt moves things into a primarily non-stationary situation.  Generally, both machines and algos suck although the efforts can be valiant.

Sometimes, systems are chaotic.  Like weather.  So it is not really stationary, but enough to allow adequate prediction.  Weather forecasting is a great example of applying quant in difficult and unstable situations.  But weather is not attempting to beat the forecasters.

Whilst driving involves siutations where objects react to one another, they do so in a way that allows driverless cars to navigate around more safely than human ones.  On average, I would hands down rather be driven by a Google car than by just about anyone on the planet.  So this is a non-stationary system which displays enough order for quant to definitively exceed human capability.

---

Now, a market is not a uniform mass.  There are pockets in it which are all of the above.  So, whatever your view, you can be right if you qualify it enough.

---

Quant in equities generally applies a very different approach to outperformance than via traditional fundamental analysis.  The edges are not talked about in terms of stocks, but by factors.  If you think FMG has a 60% chance of break-out or NVT is super-cheap for a five year play, the equivalent for quant is to say something like my low-vol factor will produce a favourable outcome.  Any individual stock with a decent exposure to that factor would have a very low edge compared to the FMG/NVT examples.  But much more diverse portfolios are built to offset this.  The edges look very different.  We can talk about satellite imagery across crop fields etc... but let's not.

In a situation like the market, which is constantly evolving, there is an opportunity for someone who learns well to assess situations faster than an algo could.  This can happen in situations which look kind of similar across different companies, but where data may not be readily comparable from a data analysis sense.  With training, it is perfectly plausible for a person of talent to see situations which rhyme at sporadic intervals. You might spot similar kinds of market developments in different industries, for example, and successfully template one on another. An algo has a very hard time knowing what to look for in these situations. If this is your gig, and you are actually good at it, all power to you.  If you are mining price series with nothing else to add to that, it is a very valid question to ask why the vast tracts of algos run by highly profit motivated people haven't found it already.  They might have, but do not have sufficient capacity to fully exploit it or suffer other structural impediments.

There has been a revolution in data science in recent years.  It is truly amazing. Tools from all sorts of fields are applied to others, including the markets.  It might well be that this will seem more like applying astrology to markets.  I doubt it.  I think it will be more like the benefits I gain from using a computer today whereas it as an HP 12C when I first started work.  It was an impressive kit and the maths is the same.  I can just get a heck of a lot more done now, as can anyone else who does the same types of calculations.  It's still me writing this stuff.

---

Who are you taking money from? A machine? Another human? A cyborg? If you don't know, buy an ETF or understand that this is a speculative or otherwise entertainment related activity.  No problem with that either.



Have a good day.


----------



## howardbandy

Gringotts Bank said:


> This is the one I worked on earlier, showing in- and out-of-sample testing.  DAX, daily, includes brokerage, 5 contracts.  You can see how it breaks down and becomes untradable when it runs on unseen data (after the yellow line).  Still working on it.
> 
> Unfortunately, the sudden big drops that ruin the OOS curve come without warning, so dynamic position sizing may not help me here.  Howard, if you're reading, is that correct?  They seem unheralded.




Hi GB, and all --

As we try multiple approaches to a trading system, we see many test results.  Typically, it is an exceptionally nice equity curve that catches our eye.  A single equity curve is a single data point drawn from the distribution of trades.  The one we stop to look at is invariably better than most that could be drawn from that distribution.  So be prepared for risk-normalized results to be lower than the single equity curve would suggest.  

The analysis goes as follows:
1.  Any set of trades can be analyzed.  
2.  If you are using a rule-based model and want to estimate future risk and profit potential, use only those trades that are out-of-sample and selected by an objective-function-directed search.
3.  If it is present, remove the effect of position sizing.  At this point, we want the percentage (or points or dollars for futures or FOREX) gained for a constant trade size.  Sort this list.
4.  State your personal risk tolerance.  For example, "I am trading a $100,000 account, forecasting two years into the future.  I want to hold the risk of drawdowns greater than 20% to a chance of 5% or less."
5.  Work through the procedure of computing the cumulative distribution function of drawdown for two year trade sequences drawn from that list.  This procedure is explained in detail in several of my presentations, posted to YouTube, and free.  It is also in two of my books -- one using Excel code and the other Python code.
6.  Adjust the position size so that the risk at the 95th percentile is 20%.  This is safe-f -- the position size that will maximize profit growth while holding drawdown to within your tolerance.
7.  Using safe-f, compute the cumulative distribution of terminal equity at the end of two years.  Look at the 25th percentile.  Convert the percentage to compound annual rate of return.  This is CAR25.  
8.  Compare CAR25 from this set of trades with CAR25 from whatever other alternative use of the funds you have available.  Pick the best one.

To maximize equity growth over a period of time, we want:
1.  A high expectation.  High percentage gained per trade.  Express this as geometric gain per trade, where a gain of 2% is expressed as 1.02.  Call it "g"
2.  Many trades.  Say there are "n" trades in the period.  Terminal equity for any system is determined by only two numbers: n and g.  TWR = g ^ n.  Final equity as a ratio to initial equity is expectation raised to the power of the number of trades.
The TWR calculation holds for all sets of trades.  50 1% trades are always better than 1 50% trade, due to the compounding effect.  1.50^1 == 1.50  1.01^50==1.64  But that says nothing about drawdown.   
3.  Limit the number of losing trades and limit large losing trades.  Either increases the likelihood that there will be a drawdown that exceeds your tolerance.

Together, these form the trading mantra that maximizes return while minimizing risk:
Trade frequently
Trade profitably
Hold a short period
Avoid serious losses

The first three are easy.  What to do about losses?  
If the problem is many small losses, look for a model that is more accurate.  Again, this is pretty easy.  Any fast oscillating indicator based on price will show oversold accurately.  Try RSI(n), where n is in the range of 1.5 to 3.0.  Or DPO with a fast adaptive moving average.  
If the problem is a few large losses, look for a model that avoids them.  This is much trickier and often requires adding rules to the model.  If the model is being built using machine learning, try a two-step model.
Step A -- Avoid toxic trades.  Train the model to identify those conditions that result in the large losses.
Step B -- Proceed as usual after first blocking -- forcing to be flat -- all those days predicted to be toxic.

If the CAR25 of the resulting model is adequate, the system is passed to trading.  There, dynamic position sizing will control the trade-by-trade position size so avoid serious loss.  

--------------------

Specifically to the equity curve posted.  It appears that there are several large losing trades.  You are correct that dynamic position sizing will have difficulty adjusting quickly enough to keep a system like this out of trouble.  The model needs to be improved.

I hope this helps,  Best,  Howard


----------



## howardbandy

minwa said:


> I don't see the parallel between chess/go & trading. In board games you have a LIMITED number of outcomes. If your opponent moves here, there is only an X number of spots to move and outcomes to be played out. The machine will calculate those and every single possible outcomes that can be played out and beat the human. This is not true in price - there is no limit to the number of outcomes of price, it is not bound by rules of the board game that you have to put the pieces on designated spots. Poker would be a closer parallel to the markets and as Omega has posted before the human were beating the machines.




I am not drawing an exact comparison between chess and trading.  The examples are to indicate the skills that are available using algorithms.  Skills that were thought to be unique to humans until very recently.

In simple games, the computer can compute all alternatives and choose the best.  In complex games, such as chess and go, the number of paths is too large to be enumerated and exhaustively analyzed.  Rather, algorithms assess probabilities based on current conditions -- just as do people. 

You are correct that there is a great deal of variability in tomorrow's price.  Your approach to trading is probably different than mine.  I do not need to know tomorrow's specific price.  Up or down is sufficient for many models.  Greater than or less than is sufficient for others.

Depending on which game, which research, and which tournament is reported, humans do not do well against algorithms in poker.  But that aside, humans are not the primary opponents in trading -- algorithms are.  It is not me against a person at another terminal.  It is my system against all other systems, including those of Goldman Sachs and James Simons.  The gains are distributed among the best systems, taking the amount gained from the worst systems. 

Best,  Howard


----------



## howardbandy

minwa said:


> To make this a fair request I offer my eyes and brains to be pitted against the machine. Will be available in a few weeks to trade against any retail machine system in a live forward going setting. I don't like my odds against the 80% win rate 1:1 RR mentioned earlier but I am happy to be humbled for fun and education. PM me if interested and we can set something up.




Hi Minwa --

Let me understand.  You are willing to be the counterparty for any trades I want to make for two weeks?

If I Buy ES or SPY at today's close, you will Sell.  My gain (or loss) is your loss (or gain)?

My trades are typically:
A.  Market-on-close issued within a few minutes of the close.  They may hold for a day, or they may be closed out in the after hours session at profit limits.  They may be for futures, ETFs, or options on either.
B.  Limit orders to be filled intra-day at what my algorithms estimate to be extreme prices, with exits MOC or limit.    

I will pass your offer.  But thank you very much.


Best,  Howard


----------



## howardbandy

Greetings --

Some of this thread reminds me of some students I have had in my classes.  Every semester or so a young man (they are always male) approaches me with a scowl on his face, shows me the C, D, or F he received on a recent exam or assignment, and reprimands me for failure to recognize his ability.  I point out that he is on a path to failing the course.  He makes it very clear that, no, he did not read the book, and no, he did not do the homework, but he knows the material and deserves a gentleman's C so he can check off this box and get on with his life. 

There is an implicit reading assignment that goes along with my comments.  People who want to improve their skills in developing profitable trading systems need to be aware of the foundations of modeling and simulation, computer science, machine learning, statistical analysis, and everything else involved. 

There is no gentlemen's C in trading. 

I think there is a gross misunderstanding of the purpose I had in mind when I began this thread.  I intended it to be strictly educational.  To explain what is happening among firms that are using algorithms, and what individual traders should take into consideration.

The amount of denial in the postings is much higher than I expected.  Do the math.  

Use it, or leave it, as you wish. 

Perhaps the thread has reached its limits.

Best regards,  Howard


----------



## ThingyMajiggy

howardbandy said:


> Greetings --
> 
> Some of this thread reminds me of some students I have had in my classes.  Every semester or so a young man (they are always male) approaches me with a scowl on his face, shows me the C, D, or F he received on a recent exam or assignment, and reprimands me for failure to recognize his ability.  I point out that he is on a path to failing the course.  He makes it very clear that, no, he did not read the book, and no, he did not do the homework, but he knows the material and deserves a gentleman's C so he can check off this box and get on with his life.
> 
> There is an implicit reading assignment that goes along with my comments.  People who want to improve their skills in developing profitable trading systems need to be aware of the foundations of modeling and simulation, computer science, machine learning, statistical analysis, and everything else involved.
> 
> There is no gentlemen's C in trading.
> 
> I think there is a gross misunderstanding of the purpose I had in mind when I began this thread.  I intended it to be strictly educational.  To explain what is happening among firms that are using algorithms, and what individual traders should take into consideration.
> 
> The amount of denial in the postings is much higher than I expected.  Do the math.
> 
> Use it, or leave it, as you wish.
> 
> Perhaps the thread has reached its limits.
> 
> Best regards,  Howard




Why is it professors always think they're right and theres no other way, when actually challeneged backs away and uses big words instead in big posts full of words. Wasnt it people like this that made the world financial state like it is today? "My ways are right and I refuse to listen to any outside sources. In fact I'll claim them to be in denial or jealous. Oh and what I say goes, dont even think of checking any hard facts to back me up. I'm a professor you know."

A piece of paper saying you're qualified in something doesnt mean what you say is 100% and thats that. If a student tried to do something differently a professor is likely to shut him down because its not "meant" to be done that way. But the way we've "always done it" hasn't landed us in a very good position has it. A student isnt lacking ability because you think he should have done it your way. 

People seem to really struggle in accepting that there are multiple ways, and that what they're complaining about and saying about others is precisely what they're doing themselves with their own bias.


----------



## tech/a

Sam/Klogg (like??)

Really?

The amount of help given to people in this community by Howard has been enormous. To treat an expert in a field such as Howard with such un restrained contempt when your not in a position to do so is childish.
Denigration to personal attack demonstrates a lack of comprehension and understanding. 

I suppose Charles Teo shouldn't be listened to when he advocates using hand held mobiles are your biggest risk for Brain Cancer. Experts earn their expertise. In which area is yours?

Grow up.



howardbandy said:


> Perhaps the thread has reached its limits.




It has.


----------



## Trembling Hand

tech/a said:


> I suppose Charles Teo shouldn't be listened to when he advocates using hand held mobiles are your biggest risk for Brain Cancer. Experts earn their expertise. In which area is yours?
> 
> Grow up.




Well there is a perfect example of an appeal to authority fallacy which has very robust data showing that the expert is wrong. Very bad example to make your case Tech.


----------



## tech/a

Trembling Hand said:


> Well there is a perfect example of an appeal to authority fallacy which has very robust data showing that the expert is wrong. Very bad example to make your case Tech.




Way off topic ---But I'm not sure its that bad.

Source-http://www.radiationresearch.org/pdfs/reasons_us.pdf



And so it was with Cigarette smoking!!


----------



## Klogg

tech/a said:


> Sam/Klogg (like??)



I didn't mean to like anything. I was just reading through the thread earlier, must have accidentally clicked it.


----------



## craft

tech/a said:


> Sam/Klogg (like??)
> 
> Really?
> Blah blah blah....
> 
> Grow up.




If growing up means conforming and not questioning, then better you don't grow up I reckon.

Disrespect is in the eye of the beholder. Its the "my way, nothing else will work" attitude expressed in the first post and follow up posts that I find disrespectful. And the calling the thread to an end when challenged that I find childish.

There's nothing wrong with the comprehension and understanding of those that have a different view.

I'll like thingy's post because I can't give tech's a dislike and I really dislike it.


----------



## Trembling Hand

tech/a said:


> Way off topic ---But I'm not sure its that bad.
> 
> Source-http://www.radiationresearch.org/pdfs/reasons_us.pdf




Dude that looks like cherry picked nut job site from the fringes of the net. You cannot talk about the 'scientific processes' then quote that. But anyway..... ML and the cloud of doom coming to the keyboard warriors.....


I have spent the last year and a half learning Python and ML applications towards trading (its been cool as an old dog learning new tricks and I really nerded out  ). Initially my interest was the use of ML predictions from another very traditional and what I saw as a non scientific approach, training response from prescribed loads set by coaches in endurance sports.... long story but very interesting things coming in that field, like many others with ML applications. Some coaches have already taken the dive and the outcome has been evolutionary rather than revolutionary so far but very early days yet. That of course led to doing different tuts from all over the net on many different subjects and of course a good look at how it was being used in the trading world. What is available out there on the net from what I could find was truly woeful. Stuff like feeding in EOD OHLC data into scikit learn algos and hoping to predict next day being up or down. Just sh1te!!

The deeper I went looking for applications the more and more disappointed I became with ML likelihood at being a revolution for the single guy on a keyboard. Once I had experience with what was possible with simplistic applications of ML it became clear that it was no different from any other angles at cracking the market. You are going to need may hours of dedicated learning, a very thorough background in market fundamentals other wise you'll will waste a heap of time trying to make simplistic and unrealistic applications/ideas work and mostly, as always in the market, a unique approach. 

I suspect the pessimism from system traders about the future is the result of them now playing in a field that has had so little barrier to entry for so long. The EOD systematic trader looking for an edge with low risk over a few days by running decision tree algos is now 40 or so years old as a field. At the start you were a first mover with access to a lot of low hanging fruit. Try and imagine how many backtest have been performed on EOD and 1 min bar data since the start of the PC era!!! Talk about a carcass being picked over. 

Python is a very cool language to learn and play around with but its no 'insider knowledge' tech. I have had a basic look at what a few Hedgies are doing in Melb with ML too and from what I was able to guess its more about risk and metrics than uncovering a new edge but when you are trading many many mil maybe that where the edge is?? Its another tool added to what has come before it. I'll continue to investigate what I can do with it. I have now moved far enough away from the beginners stage that I can apply my own unique approaches to collecting, assembling and searching data. From my discretionary trading results that is where the profits are. 

What is disappointing in ML & the trading field is how little there is to find in true applications to trading. Useful ones. Unfortunately this thread adds to that.


----------



## ThingyMajiggy

tech/a said:


> Sam/Klogg (like??)
> 
> Really?
> 
> The amount of help given to people in this community by Howard has been enormous. To treat an expert in a field such as Howard with such un restrained contempt when your not in a position to do so is childish.
> Denigration to personal attack demonstrates a lack of comprehension and understanding.
> 
> I suppose Charles Teo shouldn't be listened to when he advocates using hand held mobiles are your biggest risk for Brain Cancer. Experts earn their expertise. In which area is yours?
> 
> Grow up.
> 
> 
> 
> It has.





Yeah really.

Must be people of your generation that blindly believe and take on board what someone says as gospel simply because of the word before their name and the language they use, professor. Forgive me for questioning the unquestioned.

How am I not in a position to do so, he is recommending I buy his books, wanting me to buy his products when I've never seen any evidence that he even trades, sure he knows a lot about Maths and Machine Learning, not doubting that for a second.

You lot need to learn to take some criticism, not just hand it out. Open your minds a little, think outside the square, really, it's okay. Your way isn't the only way. Where was the personal attack by the way? How do we advance if no one asks questions or probes for the real deal and blindly follows.

But of course when questioned and told the above, you get snooty and start calling me childish. There's a reason the likes of Khan Academy/Coursera/Udacity(online education) were thought up in the first place, there are better ways to learn than the traditional listening to a professor drone on with theory for 2 hours with no practical advice/effort/evidence in the real world.

Maybe you feel you have to defend him as you've bought all his books, I dunno.

But yeah, clearly it has reached its limits.


----------



## tech/a

Trembling Hand said:


> Dude that looks like cherry picked nut job site from the fringes of the net. You cannot talk about the 'scientific processes' then quote that. But anyway..... ML and the cloud of doom coming to the keyboard warriors.....
> 
> 
> I have spent the last year and a half learning Python and ML applications towards trading (its been cool as an old dog learning new tricks and I really nerded out  ). Initially my interest was the use of ML predictions from another very traditional and what I saw as a non scientific approach, training response from prescribed loads set by coaches in endurance sports.... long story but very interesting things coming in that field, like many others with ML applications. Some coaches have already taken the dive and the outcome has been evolutionary rather than revolutionary so far but very early days yet. That of course led to doing different tuts from all over the net on many different subjects and of course a good look at how it was being used in the trading world. What is available out there on the net from what I could find was truly woeful. Stuff like feeding in EOD OHLC data into scikit learn algos and hoping to predict next day being up or down. Just sh1te!!
> 
> The deeper I went looking for applications the more and more disappointed I became with ML likelihood at being a revolution for the single guy on a keyboard. Once I had experience with what was possible with simplistic applications of ML it became clear that it was no different from any other angles at cracking the market. You are going to need may hours of dedicated learning, a very thorough background in market fundamentals other wise you'll will waste a heap of time trying to make simplistic and unrealistic applications/ideas work and mostly, as always in the market, a unique approach.
> 
> I suspect the pessimism from system traders about the future is the result of them now playing in a field that has had so little barrier to entry for so long. The EOD systematic trader looking for an edge with low risk over a few days by running decision tree algos is now 40 or so years old as a field. At the start you were a first mover with access to a lot of low hanging fruit. Try and imagine how many backtest have been performed on EOD and 1 min bar data since the start of the PC era!!! Talk about a carcass being picked over.
> 
> Python is a very cool language to learn and play around with but its no 'insider knowledge' tech. I have had a basic look at what a few Hedgies are doing in Melb with ML too and from what I was able to guess its more about risk and metrics than uncovering a new edge but when you are trading many many mil maybe that where the edge is?? Its another tool added to what has come before it. I'll continue to investigate what I can do with it. I have now moved far enough away from the beginners stage that I can apply my own unique approaches to collecting, assembling and searching data. From my discretionary trading results that is where the profits are.
> 
> *What is disappointing in ML & the trading field is how little there is to find in true applications to trading. Useful ones. Unfortunately this thread adds to that*.




From my experience over the last 18 mths, it appears that there is a misunderstanding or more to the point misconception with regard to data analysis.

There seems to be an expectation that Howards work or any other quant for that matter should provide the user with a fully operative profitable trading method.

The use of Python as a data analysis tool leads the user to in put their criteria to use in conjunction with the algos designed by Howard and others to then determine if the methods inputs (Found and added by the user) are more likely to be profitable than another or indeed random.

I agree its not insider trading knowledge far from it. But it is a diagnostic tool recognised and accepted as critical in almost every field but the smaller end of town finance.

I still trade discretionary. But as I've said before if and when I trade with 500K or more it wont be discretionary and not without my own Quant that I can trust. Fortunately I have one.

We agree with you that what is available out there is very disjointed, Barely practical unless you have weeks to make it applicable.  Change is coming from a great number of areas. I think this is evolving and may take less years than most think. Interesting and exciting.


----------



## tech/a

craft said:


> There's nothing wrong with the comprehension and understanding of those that have a different view.




Yeh but become acquainted with knowledge before sledging something that is clearly beyond Sams Comprehension and understanding.
His frustration is clear!


----------



## howardbandy

ThingyMajiggy said:


> Why is it professors always think they're right and theres no other way, when actually challeneged backs away and uses big words instead in big posts full of words. Wasnt it people like this that made the world financial state like it is today? "My ways are right and I refuse to listen to any outside sources. In fact I'll claim them to be in denial or jealous. Oh and what I say goes, dont even think of checking any hard facts to back me up. I'm a professor you know."
> 
> A piece of paper saying you're qualified in something doesnt mean what you say is 100% and thats that. If a student tried to do something differently a professor is likely to shut him down because its not "meant" to be done that way. But the way we've "always done it" hasn't landed us in a very good position has it. A student isnt lacking ability because you think he should have done it your way.
> 
> People seem to really struggle in accepting that there are multiple ways, and that what they're complaining about and saying about others is precisely what they're doing themselves with their own bias.



Greetings --

Don't be that person.  Just do the math.

Best,  Howard


----------



## McLovin

tech/a said:


> I still trade discretionary. But as I've said before if and when I trade with 500K or more it wont be discretionary and not without my own Quant that I can trust. Fortunately I have one.




What sort of "quant" is working for someone with $500k and how could you trust their analysis? Monkeys, peanuts etc...


----------



## craft

McLovin said:


> What sort of "quant" is working for someone with $500k and how could you trust their analysis? Monkeys, peanuts etc...




His son???  how could you not trust family???

That start up investment - probably in reality translates to buying a family member a job and trying to get something back from the uni expenses.

But the marketing job to lay the foundations to sell some rabbit hole crap to the bunnies and recoup the investment is entrepreneurial.

Or am I just too sceptical?


----------



## craft

howardbandy said:


> Greetings --
> 
> Don't be that person.  Just do the math.
> 
> Best,  Howard





Just do the math!


So will the math tell you if the regularity you have mined out with your math and computing power exists because of causation or coincidence?  I’m guising with the focus on monitoring things staying stationary; you acknowledge the difficulty in determining which it is.


If it is causation – does the math help with the explanation?


If you can make it to an explanation then at best you are drawing explanations from the field of social sciences and those explanations are subject to change – we are not talking laws of nature here.


If you have an objective explanation from the social sciences to explain your historical regularity – the second you apply it to the future it becomes subjective anyway – nobody knows the future.


I have more a math oriented brain then a creative brain and perhaps the hardest thing I had to do in my journey is accept that math is just a small part of the puzzle. The biggest part is story – accepting no matter what I did to prepare a profitable plan for the future the most I could ever go forward with was a “subjective story” of how the trade/system of trades might turn out and then manage my story against what ever reality felt like it damn well wanted to do.


No amount of math or data analysis is going to change trading/investing away from being a subjective story game unless it can predict the future with 100% and I’ll bet you that will never happen.


The math can help formulate a decent story – but it’s not the only way to formulate a decent profitable story.


----------



## ThingyMajiggy

tech/a said:


> Yeh but become acquainted with knowledge before sledging something that is clearly beyond Sams Comprehension and understanding.
> His frustration is clear!






howardbandy said:


> Greetings --
> 
> Don't be that person.  Just do the math.
> 
> Best,  Howard




Haha right. The refusal and zero real-world proof continues from the walking textbook and the guy with the smart son.

Can only ask simple questions that would be simple to answer so many times. Heaven forbid you question the Sensei. I tried to get something useful out of them but alas, there obviously isn't anything, at least yet, on this topic. All the best folks


----------



## Klogg

McLovin said:


> What sort of "quant" is working for someone with $500k and how could you trust their analysis? Monkeys, peanuts etc...




This post I liked intentionally.

Tech/a - go nuts.


----------



## tech/a

craft said:


> His son???  how could you not trust family???




Correct



> That start up investment - probably in reality translates to buying a family member a job and trying to get something back from the uni expenses.




In correct
Has his own Company and spends approx. a day a week on this--Currently
for the last 18 mths.



> But the marketing job to lay the foundations to sell some rabbit hole crap to the bunnies and recoup the investment is entrepreneurial.




You could look at it in that light.



> Or am I just too sceptical?




No.


----------



## OmegaTrader

howardbandy said:


> Greetings --
> 
> Don't be that person.  Just do the math.
> 
> Best,  Howard




I think we need to show respect to Howard even if we disagree with him.

The is not like other members which claim to make exorbitant risk adjusted returns without proof.

Those charlatans deserve disrespect...

I disagree with some points he has made.

But the posts are really in-depth and of value.

He is clearly stating that he has products for sale related to his area of research/business activities.

Go for the ball not the man.

The debate is centering around Quantitative analysis not if Howard is a good guy or a bad guy.

My twocents


----------



## McLovin

tech/a said:


> In correct
> Has his own Company and spends approx. a day a week on this--Currently
> for the last 18 mths.




Why not just give him your $500k, or however much, and let him trade it for you then? If being a quant is the bit that adds value what value do you (or anyone who isn't a quant) add?


----------



## tech/a

Could do.

There is a complexity with trusts etc.

But no problem with that as long as I understand it.
Unless I understand what risks I can quantify (to a degree)
I wont invest in anything regardless of who/what it is.
Just due diligence as best I can---most would be no different.


----------



## McLovin

tech/a said:


> Could do.
> 
> There is a complexity with trusts etc.
> 
> But no problem with that as long as I understand it.
> Unless I understand what risks I can quantify (to a degree)
> I wont invest in anything regardless of who/what it is.
> Just due diligence as best I can---most would be no different.




Presumably you don't understand what risk you can quantify and that's why you have a quant?

It seems a bit far-fetched to have a guy working one day a week as a "quant"/trader/investor who could provide a meaningful edge. Is your son's company the one in which you are a venture capital partner?

I don't know much about trading, but I'll tell you something about investing; the answer is rarely "more data".


----------



## tech/a

Private mailed you.


----------



## relval

With regard to ML/quantitative modelling, your output is only as good as your input.

1. Using simple inputs like technical indicators absolutely will not work.   Rest assured your data has been analysed by thousands of others and any obvious inefficiency exploited.  If you build enough models, some will pass validation purely by chance.  You will think you have something, but you don't.

2. Getting more creative and using inputs like intermarket relationships, you will find some correlations that enable you to build some models with an apparent decent edge.  Only problem is that these kinds of relationships aren't stationary.  Best case scenario is the model works for awhile in production before decaying, you switch it off then recalibrate/build a new one.  I believe this is how a lot of quant shops operate.

3. Finding inputs that have a stable predictive correlation over time is really difficult.  But here's the thing - if you find something like this you don't need a neural net or anything fancy to implement it.


----------



## Trendnomics

Trendnomics said:


> Wow this thread is terrible!




Still stand to be corrected. Very entertaining, but adding no value at all!



OmegaTrader said:


> Go for the ball not the man.




It has become quite apparent what real profitable traders/investors think of Howard's ball(s).


----------



## Modest

Quants and trading


----------



## OmegaTrader

Trendnomics said:


> Still stand to be corrected. Very entertaining, but adding no value at all!
> 
> 
> 
> It has become quite apparent what real profitable traders/investors think of Howard's ball(s).




The only reason you are saying that is because he commented on your thread.


When I asked you on your thread what your alpha was you still haven't replied.

That was dec 03-2016


Who are the real profitable traders?


----------



## DeepState

Thanks for your perspectives.



relval said:


> With regard to ML/quantitative modelling, your output is only as good as your input.
> 
> 
> 2. Getting more creative and using inputs like intermarket relationships, you will find some correlations that enable you to build some models with an apparent decent edge.  Only problem is that these kinds of relationships aren't stationary.  Best case scenario is the model works for awhile in production before decaying, you switch it off then recalibrate/build a new one.  I believe this is how a lot of quant shops operate.



Correct on how quant shops operate.  Utilisation of factors is not unlike picking stocks.  Stocks you like will not be cheap forever.  Factors you like are unlikely to be permanent unless of a type which is compensation for risk (eg. equities is expected to outperform cash because it is riskier to hold an equity than cash).




relval said:


> 3. Finding inputs that have a stable predictive correlation over time is really difficult.  But here's the thing - if you find something like this you don't need a neural net or anything fancy to implement it.




If your signal required a neural net to identify and construct, you are going to need a neural net to develop the signals.  A neural net is not readily approximated by simple linear relationships, for example.

---

Howard has an axe to grind.  If that's not welcome here, fair enough.  This community does not react well to sales motivated posts. I understand that this forum is meant to be more of an exchange of ideas rather than a means to generate sales.

I don't begrudge the man his desire to commercialise his knowledge or experience. Perhaps this isn't the place for that.

His tool set is solid. The arguments have rationale that you can concretely debate. It is useful for some - maybe not you. Nevertheless, you may not like what he does with the tools or the rationale used to jump from point to point. I acknowledge that the technical matter is beyond the maths paygrade of many here, but his stuff is ABC to finance students and not an effort to obfuscate.

This is not unlike a situation where someone thinks P/B is the ant's pants and another thinks the Montgomery formula is the ant's pants. Both read the same balance sheet and prices.  Perhaps treatment of debate participants should proceed accordingly...which is probably to heap scorn on the opposing idea relentlessly...haha...sigh


----------



## Trendnomics

OmegaTrader said:


> The only reason you are saying that is because he commented on your thread.
> 
> 
> When I asked you on your thread what your alpha was you still haven't replied.
> 
> That was dec 03-2016
> 
> 
> Who are the real profitable traders?




Howard did more than comment - he stated that all the back-testing my system was based on is flawed. He also added his usual doom-and-gloom spiel.

I'm no longer able to provide posts on my system - I'm potentially entering an arrangement which would apply a non-disclosure agreement.

Minwa is one of the profitable traders (many more lurking and posting).


----------



## OmegaTrader

Trendnomics said:


> Howard did more than comment - he stated that all the back-testing my system was based on is flawed. He also added his usual doom-and-gloom spiel.
> 
> I'm no longer able to provide posts on my system - I'm potentially entering an arrangement which would apply a non-disclosure agreement.
> 
> Minwa is one of the profitable traders (many more lurking and posting).



That is not about you it is about your strategy. don't attack the man.
Ok, so are you making risk adjusted returns?


----------



## minwa

OmegaTrader said:


> Ok, so are you making risk adjusted returns?




Well we know for sure Howard isn't making any - he stated he doesn't currently trade. And from the tone he hasn't for a while. Everything is theory and back tested stuff which just contradicts everything he is saying on using the latest buzz to level the playing field in the markets against your competition.

When you make big controversial claims in profiting in the markets, especially as an educator/author, people like results to back them up. All we really got was - "buy the book".


----------



## barney

Modest said:


> Quants and trading





LOL .... 
I was suspicious at the Unilateral phase detractors (23 sec), but automatically synchronizing the Cardinal Grand Meters at 28 seconds really had me super skeptical given that it was obviously a rip off of the much more reliable "turbo-encabulator" which has proven itself over time to be much more reliable  ... 

I kid you not!!  ...Created by Engineers at General Electric back in the 60's it was the most advanced piece of 'non-engineering' ever!  Superseded by the advanced model, the "Retro-Encabulator: in the late 80's

PS All the above meant in good fun and totally factual so no hate mail please


----------



## howardbandy

craft said:


> If growing up means conforming and not questioning, then better you don't grow up I reckon.
> 
> Disrespect is in the eye of the beholder. Its the "my way, nothing else will work" attitude expressed in the first post and follow up posts that I find disrespectful. And the calling the thread to an end when challenged that I find childish.
> 
> There's nothing wrong with the comprehension and understanding of those that have a different view.
> 
> I'll like thingy's post because I can't give tech's a dislike and I really dislike it.



Hi Craft --

I began to thread.  With the simple intent of pointing out techniques that some of our competition in trading is using.  The ensuing discussion went well for many postings.  Every civil question addressed to me, or answerable by me, was answered.  Then began both going considerably off-topic, and involving personal attacks.  The purpose I had was complete. 

I had no idea the first post would be interpreted as "my way, nothing else will work."  My apologies.  How should that material have been presented?    

I am open to additional civil on-topic discussion.  What more, related to the first post of the thread, should be discussed?

Best,  Howard


----------



## howardbandy

Greetings --

I have read through the thread again.  Please don't shoot the messenger. 

But please do allow for an analysis of techniques being used. 

On risk --

If you have not read through my postings on risk, or watched the presentation "The Four Faces of Risk" on YouTube, please do.  The math starts there. 

Any set of trades can be analyzed to determine the risk of drawdown in them, and if, as we all rely, the future resembles the past, the risk of future trading that same system.  Traders using algorithms have an easier time doing the analysis, but discretionary traders can apply exactly the same procedures. 

There are two fundamental facts about risk:
1.  Risk increases with holding period.
2.  Risk increases with trading inaccuracy. 

Each trader has his or her own risk tolerance.  It can be quantified.  The risk associated with any set of trades can be compared with the trader's risk tolerance.  If the risk of the trades is in excess of the trader's tolerance, position size is too high, there is a high probability of a drawdown that will cause either serious account loss or to stop trading.  Lower the position size until the risk of the trades is within the risk tolerance of the trader.  If the resulting position size is too low, do not trade that system.  

There is no "my way or the highway" from me here.  There is nothing here but the math of modeling, simulation, statistics, and the scientific method applied to a set of trades.  The mathematical analysis does, however, establish some boundaries for what works and what does not.  I hear you -- the boundaries are beyond what many of you are now doing.  Wishing it were different does not make the math wrong.  Neither does making personal criticisms or unreasonable demands of me.

On machine learning and artificial intelligence --

My original post quoted a statement made in a presentation that Morgan Stanley is employing several thousand developers working with Python.  This was Stephen Simmons statement, not mine.  My comments are intended to point out some ways us ordinary retail traders can still compete, even though it is becoming more difficult.

The responses in this thread include several that dismiss the competition from Morgan Stanley and those similar. 

I understand the personal psychological investment we have in our creations and faith in our abilities.  Some 90% of Americans believe they are among the top half of drivers in terms of safety.  An item we already own is more valuable to us than one available for sale.  One we have made ourselves is more valuable still. 

Ignore the messages about artificial intelligence projects and accomplishments we are hearing if you wish.  But realize this is not a single voice from science fiction, or from me.  This is a tidal wave coming from giant players -- Morgan Stanley, Goldman Sachs, James Simons, David Shaw, Apple, Google, Microsoft, Facebook, Carnegie Mellon University, Stanford, ...  The list goes on and on.  It should be evident without my saying so directly, but retail traders must be certain they are in the top half -- the half that gets a portion of the profit.  

All presented with the best intentions.  Please read them as such.

Best regards,  Howard


----------



## howardbandy

minwa said:


> Well we know for sure Howard isn't making any - he stated he doesn't currently trade. And from the tone he hasn't for a while. Everything is theory and back tested stuff which just contradicts everything he is saying on using the latest buzz to level the playing field in the markets against your competition.
> 
> When you make big controversial claims in profiting in the markets, especially as an educator/author, people like results to back them up. All we really got was - "buy the book".



Greetings --

Reread the reasons why traders stop trading.  One is they have enough and no longer need to put funds at risk.

Theory, yes.  And research.  But not backtested.  Backtested is necessary, but not sufficient.  Way beyond backtested.  

Where is the contradiction?

The product I am selling is professional advice.  The part you can see is books.  If you have not yet read them, please do.  If they are not valuable to you, send them back and ignore them.  

I offered and sent out hundreds of free copies of the latest book before the official publication date.  The current price is as close to free as Amazon will allow me to make it.  My profit is at most a few cents per copy, depending on where it is ordered and shipped.  Some copies are net negative profit.  
https://www.amazon.com/Foundations-Trading-Developing-Profitable-Scientific/dp/0979183863

Or go to the book's webpage and read almost all of it free:
http://www.blueowlpress.com/123-2/foundations-of-trading

Best,  Howard


----------



## howardbandy

Trendnomics said:


> Howard did more than comment - he stated that all the back-testing my system was based on is flawed. He also added his usual doom-and-gloom spiel.
> 
> I'm no longer able to provide posts on my system - I'm potentially entering an arrangement which would apply a non-disclosure agreement.
> 
> Minwa is one of the profitable traders (many more lurking and posting).



Hi Trendnomics --

I do not recall evaluating a trading system for you.  Please remind me of the details.

What I do say regularly is that backtesting is necessary, but not sufficient.  Backtested results always underestimate risk and always overestimate profit.  They have no value in estimating future performance.  

That's what I say to Everyone about All backtesting.  That is what every professional practitioner in the entire modeling and learning community says.  Except trading system developers, who, for some reason I do not understand, ignore many of the basic tenets of the scientific process.  And too often seem to be proud of it.  

Backtesting without independent validation is necessary for us to notice that we might have found a good model.  But it has no value beyond that.  Some models are guaranteed to be perfect in-sample and give absolutely no guidance about out-of-sample.  Others have reasonable agreement between in-sample and out-of-sample providing some criteria have been met.  It depends on the model, on the data, and on the technique the practitioner uses.

The decision trees model type used by traditional trading system development platforms are notorious for overfitting in-sample.  Very few pass independent rigorous validation.  Consequently, most are unsafe to trade.  Everyone who has seen a walk forward fail will understand.  

Best, Howard


----------



## howardbandy

DeepState said:


> Howard has an axe to grind.



Greetings --

What is my axe?

Best,  Howard


----------



## Ves

howardbandy said:


> I had no idea the first post would be interpreted as "my way, nothing else will work."  My apologies.  How should that material have been presented?



In my opinion, when dealing in the public sphere no matter what content you present,  and how you present this content, different people will always have different interpretations of both the content and your original intentions.  Once it has been published you have really lost most of your own control over it - people will do with it as they see fit.  You can disclaim it as much as you like,  "this doesn't apply to..."  "this is correct in cases where..." etc.  but that in itself can also be misinterpreted or missed.

I don't think there is really anything wrong with this,  it just _is.  _It is human nature.  Conflict around ideas and personalities is unavoidable. 

Howard,  you are not generally rude and I think you remain pretty civil and you certainly don't hide the fact that you are an academic with publications available for purchase. 

You have started an interesting conversation,  and there have been some fantastic responses _for_ and _against._ 

Cheers


----------



## DeepState

howardbandy said:


> Greetings --
> 
> What is my axe?
> 
> Best,  Howard




Here it is. And it's a beauty.




and also here:



howardbandy said:


> Greetings --
> 
> The product I am selling is professional advice.  The part you can see is books.




If you are readily identifiable, make posts which bring direct invitation to visit a portal through which you can generate revenue (profit is another thing) and where such activities increase your marketability (eg. Author of twenty books, web site visited by 200k...) then this looks like commercial activity.

If you offered your opinion in a way that is largely anonymous and could not reasonably be seen to benefit anyone other than the free exchange of information, I might not perceive an axe. Perhaps my perception of an axe is a Type II error.

In any case, respect to you.


----------



## minwa

howardbandy said:


> Hi Craft --
> 
> I began to thread.  With the simple intent of pointing out techniques that some of our competition in trading is using.  The ensuing discussion went well for many postings.  Every civil question addressed to me, or answerable by me, was answered.  Then began both going considerably off-topic, and involving personal attacks.  The purpose I had was complete.
> 
> I had no idea the first post would be interpreted as "my way, nothing else will work."  My apologies.  How should that material have been presented?
> 
> I am open to additional civil on-topic discussion.  What more, related to the first post of the thread, should be discussed?
> 
> Best,  Howard




_As I write in my "Foundations" book, the days of chart reading, long term holding, and simple trading algorithms *are over*. The business of trading is changing with astonishing speed. It is now about applied mathematics, machine learning, Bayesian statistics. Traders without skills in math, programming, statistical analysis, and scientifically developed trading techniques are at a severe disadvantage. Stephen and his colleagues will *"eat the lunch" of unprepared traders.*_

This is obviously the part from first post most of us had problems with. I've bolded the parts where the wording made it pretty clear to us you are trying to stat facts, not opinion. Facts with nothing backing them, I've already posted 2 indexes (1 is from Fundseeder other from Barclays) of accumulation chart of real world traders where discretionary outperformed.

Most of the fireworks then started when Tech started hammering away that it's gona take over the trading and use it or become obsolete (fossil was the exact world he used).

What some have a problem is - lack of real world results. Sorry but "I have reached my desired financial target and don't know what to do with more money", while commendable in other context, is a pretty lame excuse when you are a trading educator actively talking and not trading. People will be more receptive to someone that actually do, in the live markets - until then, it's all theory. Don't see how trading algos can't be run with minimal maintenance so time is not a problem. Money is obviously not a problem - in a fact it's a good solution for your extra money - trade to enhance education for your listeners/readers.  There are so many third party tracking methods this days no one will dispute it's authenticity. It will talk louder than anything anyone can write on this thread.






_*
*

_


----------



## ChaosHedge

I've found this thread really interesting and have learnt a lot from many of the people posting.

Not sure if my understanding is correct but its seems that short term discretionary traders have a set of rules regarding their strategy and then use instinct or gut-feel to confirm. Is the instinct part driven by subconscious pattern recognition, or feeling positive based on previous experience? Would a ML algo basically be doing the same thing? Just having some defined rules and matching patterns with historic ones, however complex they may be.

I believe there are many profitable discretionary traders here and it's seems logical that there should be a similar number of quant traders. The time required to become successful discretionary would be enough time to learn ML methods? Are quant traders less willing to share their systems because they can be more easily taken compared to the skills of discretionary trader?

Still learning, keen to hear thoughts. Cheers


----------



## minwa

ChaosHedge said:


> I've found this thread really interesting and have learnt a lot from many of the people posting.
> 
> Not sure if my understanding is correct but its seems that short term discretionary traders have a set of rules regarding their strategy and then use instinct or gut-feel to confirm. Is the instinct part driven by subconscious pattern recognition, or feeling positive based on previous experience? Would a ML algo basically be doing the same thing? Just having some defined rules and matching patterns with historic ones, however complex they may be.
> 
> I believe there are many profitable discretionary traders here and it's seems logical that there should be a similar number of quant traders. The time required to become successful discretionary would be enough time to learn ML methods? Are quant traders less willing to share their systems because they can be more easily taken compared to the skills of discretionary trader?
> 
> Still learning, keen to hear thoughts. Cheers




Discretionary is quite broad, I think a lot have general rules and others simply use gut. Having defined rules and matching patterns makes it more systematic than discretionary. The discretionary part is the freedom/judgement to take that trade or not/bend rules a bit or not..something the machine cannot do.

Time required to be successful in ANY discipline is different for everybody..probably never for a good amount of people. Yes quant traders system would be more easily replicated if shared, like Howard's book's RSI system - however no one is asking for any systems to be shared..just the results.


----------



## tech/a

minwa said:


> Most of the fireworks then started when Tech started hammering away that it's gona take over the trading and use it or become obsolete (fossil was the exact world he used).




If taken in the context it was meant----Not including sound data analysis in my view will mean in the future you will be regarded as a fossil when trading by those who have at their disposal the knowledge and the ability to apply quant methods and testing to their trading arsenal even if your a discretionary trader many questions can be answered with data analysis.(Well answered with more certainty than gut feel).



> What some have a problem is - lack of real world results. Sorry but "I have reached my desired financial target and don't know what to do with more money", while commendable in other context, is a pretty lame excuse when you are a trading educator actively talking and not trading. People will be more receptive to someone that actually do, in the live markets - until then, it's all theory. Don't see how trading algos can't be run with minimal maintenance so time is not a problem. Money is obviously not a problem - in a fact it's a good solution for your extra money - trade to enhance education for your listeners/readers. There are so many third party tracking methods this days no one will dispute it's authenticity. It will talk louder than anything anyone can write on this thread.




I'm no different I am one of those.
2 things
(1) Having worked with these types of people for so long I understand that its about study of data. Application is up to the user. 20 different users of the same data analysis using different inputs and or money risk inputs. There are certain things we want to know about the data, profitability is just one.

So a Quant could post up any data set with any number of inputs and show those with both profit and loss.
Just a slight digression and to *make a point.*

In 2000 I started trading live on Radges site Tech Trader. A very rudimentary average trading method designed by me---a duck.
We traded 30K with another 60k on margin using the BT margin stocks available to trade on margin at the time. Traded it for 7 yrs. Our 30k went to a high of $438k and I closed it all out at around $350K prior to the GFC.

At the time the system was picked to death over the whole period---fine that's what I wanted as I was trading it.
Then I got ---but you traded it in a screaming bull market---yup but I had no idea that it was going to be a bull market when I started.
Then ---but you traded on margin then compounded your profits---urr yes.

The end result was despite protests as loud as a jumbo jet it did well.

*To my point.
people can post systems traded/paper traded/ test results only and you'll have a great number of other people who wont be happy with one condition or another.
But I agree results are great to see---but they are the results of the user not you

See I've done it myself!!!*


----------



## craft

howardbandy said:


> Hi Craft --
> 
> I began to thread.  With the simple intent of pointing out techniques that some of our competition in trading is using.  The ensuing discussion went well for many postings.  Every civil question addressed to me, or answerable by me, was answered.  Then began both going considerably off-topic, and involving personal attacks.  The purpose I had was complete.
> 
> I had no idea the first post would be interpreted as "my way, nothing else will work."  My apologies.  How should that material have been presented?
> 
> I am open to additional civil on-topic discussion.  What more, related to the first post of the thread, should be discussed?
> 
> Best,  Howard





minwa said:


> _As I write in my "Foundations" book, the days of chart reading, long term holding, and simple trading algorithms *are over*. The business of trading is changing with astonishing speed. It is now about applied mathematics, machine learning, Bayesian statistics. Traders without skills in math, programming, statistical analysis, and scientifically developed trading techniques are at a severe disadvantage. Stephen and his colleagues will *"eat the lunch" of unprepared traders.*_
> 
> This is obviously the part from first post most of us had problems with. I've bolded the parts where the wording made it pretty clear to us you are trying to stat facts, not opinion. Facts with nothing backing them, I've already posted 2 indexes (1 is from Fundseeder other from Barclays) of accumulation chart of real world traders where discretionary outperformed.
> 
> Most of the fireworks then started when Tech started hammering away that it's gona take over the trading and use it or become obsolete (fossil was the exact world he used).




Hi Howard

My answer to you regards your first post would be basically as Minwa has posted.

Add to this your often cited definition of an investment as something you write into your will.



howardbandy said:


> 1. Trading is buying and selling financial instruments with the intent of selling at a higher price than buying. Investing is buying and never selling. My comments apply to trading.



 and what I do is clearly a trader by your definition (as is nearly everybody) so take some interest in this thread which otherwise I may have left alone.

It was Tech/a post that I replied too and objected too with his line on lack of comprehension and understanding.  Dismissing the objections and agreeing the thread had reached its limit.

There is plenty in this thread I disagree with and it’s not because of lack of understanding and comprehension.

Lots to debate if you want to hear the other side – lets start with:

You talk about the scientific method of learning then validating. The more standard definition would be hypothesizing and then attempting to invalidate.

How do your assertions in this thread fit with the traditional definition of scientific method.
_
_


----------



## minwa

tech/a said:


> *To my point.
> people can post systems traded/paper traded/ test results only and you'll have a great number of other people who wont be happy with one condition or another.
> But I agree results are great to see---but they are the results of the user not you
> 
> See I've done it myself!!!*




I wasn't here when your Tech Trader but if I was and looking to learn from someone, I certainly would've looked into your system as opposed to other with absolutely no results. I started out questioning Radge as well but then someone posted some of their results with his system/subscription and it was solid. Not outstanding but definitely solid for the average trader and I have changed my views about him. I wouldn't actively recommend him but when askedwould not warn others away from him like I do with almost all other educators. 

The results are others and not you is true, but no one has the time to look into everything so at least learning from someone who actually does what they teach in the real markets over theorists is increasing your chances, in my opinion.


----------



## DeepState

craft said:


> You talk about the scientific method of learning then validating. The more standard definition would be hypothesizing and then attempting to invalidate.
> 
> How do your assertions in this thread fit with the traditional definition of scientific method.




That's a goodie. 

Just some observations:

In the past, some one used their fingers to add 2+8 to arrive at ten.  Your PC can do around 10^9 times that now or more.

In the past, a theory was created when someone saw something interesting in their natural setting and put forward an idea.  Some could be tested, some were inherently unfalsifiable.

Now, your PC together with a decent financial engineering package and a decent data source, bolted together with some machine learning methods, will produce a large number of theories which fit the data to a level that you specify. Large being arbitrarily big....but big.

Computational power has increased the speed at which we can perform calculations.  Machine learning and datamining/datascience has increased the speed at which we generate theories which 'fit' the data or other supervision criterion.

Do you believe an edge in trading rests with finding some non-linear relationship on a combination of obvious and obtuse data?  That's what ML does best.  A normal fundie/trader has access to heaps of data so, just for this purpose, let's assume a level playing field except for ML algos which can find heaps of relationships that the other traders would not be able to generate quite as quickly.

Is there an edge to be gained for having a special calculator which trawls over the data you already have to find relationships you would not have been able to find in a reasonable time frame? If so, how significant might it be?


----------



## Trembling Hand

DeepState said:


> Do you believe an edge in trading rests with finding some non-linear relationship on a combination of obvious and obtuse data?  That's what ML does best.




DeepState the problem that Joe Average Trader is going to face is they are likely to apply ML to standard backtested systems or variations not that far from the standard decision tree approach on single instruments or at most a portfolio of EOD or min bar data. Then all ML is going to do is hyper optimise it. Therefor rendering the result useless in forward test.

The reason why JP Morgan has many thousand python developers is because of the breadth of data their systems run on.  Large data is what ML is all about. For example American futures exchanges generates over *100 billion order messages each day* and the stock markets billions more day in day out, thats just crazy numbers. With 1.7 trillion under management there is value for them in large amount of data. I cannot imagine a trader having the resources to data mine exchange wide tick and quote data to build a ML system from. Then at the other end I'm yet to see a 'simplish' system that has been improved by ML. It doesn't find hidden patters in small data sets. It over fits it.


----------



## DeepState

Trembling Hand said:


> DeepState the problem that Joe Average Trader is going to face is they are likely to apply ML to standard backtested systems or variations not that far from the standard decision tree approach on single instruments or at most a portfolio of EOD or min bar data. Then all ML is going to do is hyper optimise it. Therefor rendering the result useless in forward test.
> 
> The reason why JP Morgan has many thousand python developers is because of the breadth of data their systems run on.  Large data is what ML is all about. For example American futures exchanges generates over *100 billion order messages each day* and the stock markets billions more day in day out, thats just crazy numbers. With 1.7 trillion under management there is value for them in large amount of data. I cannot imagine a trader having the resources to data mine exchange wide tick and quote data to build a ML system from. Then at the other end I'm yet to see a 'simplish' system that has been improved by ML. It doesn't find hidden patters in small data sets. It over fits it.





Your response re: JP Morgan suggests that there is some belief that ML does have benefits, even though it might be restricted primarily to those with the true savvy??  

I think it comes down to the insight being used.  Does the idea produce an edge for reasons you can understand and which can be expected to prevail in the future? If so, I don't care much how you got it. Whether you came up with it in the bath or ran a CART over it, if it makes sense, it's valid to use (theory, empirical, sniff test etc etc).  If you pile garbage upon garbage hoping that it might fuse into diamond, that seems less likely to be a successful thing to attempt...but the computational cost is small and the outcomes might sustain hope for another day for Joe Average.


----------



## craft

DeepState said:


> That's a goodie.
> 
> Just some observations:
> 
> In the past, some one used their fingers to add 2+8 to arrive at ten.  Your PC can do around 10^9 times that now or more.
> 
> In the past, a theory was created when someone saw something interesting in their natural setting and put forward an idea.  Some could be tested, some were inherently unfalsifiable.
> 
> Now, your PC together with a decent financial engineering package and a decent data source, bolted together with some machine learning methods, will produce a large number of theories which fit the data to a level that you specify. Large being arbitrarily big....but big.
> 
> Computational power has increased the speed at which we can perform calculations.  Machine learning and datamining/datascience has increased the speed at which we generate theories which 'fit' the data or other supervision criterion.
> 
> Do you believe an edge in trading rests with finding some non-linear relationship on a combination of obvious and obtuse data?  That's what ML does best.  A normal fundie/trader has access to heaps of data so, just for this purpose, let's assume a level playing field except for ML algos which can find heaps of relationships that the other traders would not be able to generate quite as quickly.
> 
> Is there an edge to be gained for having a special calculator which trawls over the data you already have to find relationships you would not have been able to find in a reasonable time frame? If so, how significant might it be?




I was reading Howard Mark's latest memo today and a quote contained in it pretty much sums things up for me.



> No amount of sophistication is going to allay the fact that all of your knowledge is about the past and all your decisions are about the future.
> 
> 
> 
> – Ian Wilson (former GE executive)


----------



## craft

DeepState said:


> I think it comes down to the insight being used.  Does the idea produce an edge for reasons you can understand and which can be expected to prevail in the future? If so, I don't care much how you got it. Whether you came up with it in the bath or ran a CART over it, if it makes sense, it's valid to use (theory, empirical, sniff test etc etc).




This is my point exactly - doesn't matter how you come up with it, quantitatively or qualitatively - if you are applying it to the future its subjective. More art than science.

Don't get me wrong - I think statistical data analysis and computing power has huge benefit but more so in process control rather than predictive value.

As far back as twenty years ago as part of my trading I was using statistical control charts to monitor my trading system performance using western electric rules as decision points to turn off systems.  I don't know how wide spread that sort of thing was at the time but it was something I adapted from working in a processing industry and it worked well for me in trading. Getting more sophisticated in that area I can see as an advantage but when it comes to predicting the future you ain't ever going to turn subjective into objective no matter how smart or sophisticated you are and its a real danger to think you can.


----------



## odds-on

Trembling Hand said:


> DeepState the problem that Joe Average Trader is going to face is they are likely to apply ML to standard backtested systems or variations not that far from the standard decision tree approach on single instruments or at most a portfolio of EOD or min bar data. Then all ML is going to do is hyper optimise it. Therefor rendering the result useless in forward test.
> 
> The reason why JP Morgan has many thousand python developers is because of the breadth of data their systems run on.  Large data is what ML is all about. For example American futures exchanges generates over *100 billion order messages each day* and the stock markets billions more day in day out, thats just crazy numbers. With 1.7 trillion under management there is value for them in large amount of data. I cannot imagine a trader having the resources to data mine exchange wide tick and quote data to build a ML system from. Then at the other end I'm yet to see a 'simplish' system that has been improved by ML. It doesn't find hidden patters in small data sets. It over fits it.



I can believe this. I once watched an interview with James Simons from Renaissance Technologies and he mentioned how much data they process every day. Truly mind boggling.


----------



## tech/a

Craft
Why does data science have to be looking for predictive ability?

As Howard has said all he cares is there is movement
If you have a look at the way he trades and he spelled it out
It makes sense 

If you could profit from movement regardless of it being long or short
Wouldn't that be optimal?


----------



## Wysiwyg

I heard on CNBC (Josh Brown) that algorithms read speeches (speech to text I presume) and make trade decisions based on keywords. Justifying the market selloff during Trumps utterances this morning. I thought data mining social media sites could be advantageous. Charting the frequency of keywords or something and checking for market correlation.


----------



## craft

tech/a said:


> Craft
> Why does data science have to be looking for predictive ability?
> 
> As Howard has said all he cares is there is movement
> If you have a look at the way he trades and he spelled it out
> It makes sense
> 
> If you could profit from movement regardless of it being long or short
> Wouldn't that be optimal?




???

For a system to remain stationary it has to be, by definition, on balance predictive.

I'll refer you to Howards post because your post makes no sense in light of it.



howardbandy said:


> Greetings --
> 
> The short answer is that the model overfit the in-sample data during learning and does not fit the out-of-sample data.
> 
> Some explanation might be helpful.
> 
> The system development process is classical learning using the scientific method.  I do not understand why, but the trading community has been super slow to recognize that systems to predict stock direction (trades) are similar in almost all respects to systems such as those that predict loan default.  It is critical that the data processed for prediction has the same distribution, with respect to the signal being identified, as the data processed for model fitting.
> 
> To develop a system that will predict whether a borrower will repay a loan, the lender gathers data that hopefully has some predictive value from a large group of customers, some of whom repaid and others whom have defaulted.  If conditions for the period the data represents are relatively constant, the distribution of any randomly chosen subgroups will be the same as the distribution of the entire group and, importantly, of future customers.  With respect to loan repayment, the data is stationary.  The future resembles the past.
> 
> The data scientists develop the model that goes with the data by selecting a random subsample of the data and fitting the rules to it.  This is the "training" data and fitting is the learning process.  This is "supervised" learning -- each data point has values for the predictor variables (income, job history, etc) and the loan repayment -- the target -- is known.  The fitting process is a mathematical process of finding the best solution to a set of simultaneous equations.  aX = Y.  Where X is a large array of values for the indicator variables and y is a column array of values for the target.  The model is the array "a" -- the coefficients of the solution.  There will be a solution -- there will be an "a" array -- whether the model has predictive capability or not.
> 
> The model -- the rules -- may have identified one or more important features in the data that are consistently associated with probability of repayment.  But the developer cannot tell by looking at the learning process.  He or she has reserved a separate subsample that is not used at all in learning -- call it the "test" data.  As a one-time test, the model is applied to the test data.  The value of the target is known, but is only used for later reference.  A predicted value for each test data point is computed using the a array.  Comparison between the known target values and predicted target values lets the person building the model know whether the model learned general features or just fit the particular data it was given -- including all the randomness.
> 
> The scientific method insists on two phases -- learning and testing.  Without independent testing, nothing can be said about the model or its predictions.
> 
> The trading system development profession has largely ignored the scientific method.  Independent, one-time, testing using data that follows the training data and has not been used in development is seldom done.  Trading system developers see the equity curve from the training portion, and assume completely without justification, that future results will be similar.
> 
> That may be true, but in order for it to be true, two conditions must hold.
> 1.  The future must resemble the past.  That is, the data used for learning and the data used for testing / trading must have the same distribution (with respect to the signal).  This is stationarity.
> 2.  The model must learn real, predictive signals rather than simply fit to the randomness of the data.  This is learning.
> 
> When building models for stationary data such as loan repayment, some model algorithms produce in-sample results that can be used to give estimates of out-of-sample performance, while other model algorithms always overfit in-sample and have no value for estimating out-of-sample performance.  Out-of-sample testing is always required before estimating future performance.
> 
> Back to trading.
> 
> A system that works in-sample has fit a set of equations to a set of data.  Whether there is true learning or not, there is always a solution -- a set of trades and an associated equity curve.  Most are bad and are discarded.  We test so many combinations that eventually one fits and the result looks good.  It might be simply an overfit solution or it might be a representation of a truly predictive system.  We cannot tell anything about the future performance without testing never-seen-before future data.  When results of this out-of-sample test are poor, it is because one or both of the two conditions do not hold.  Either the data is not stationary beyond the time period of the learning data; or the model has not learned.
> 
> I hope this helps.
> 
> Thanks for listening,  Howard




I have absolutely no issue with how Howard advocates monitoring a system - I think its sound advice.  I do have an issue with .







> As I write in my "Foundations" book, the days of chart reading, *long term holding*, and simple trading algorithms are over. The business of trading is changing with astonishing speed. It is now about applied mathematics, machine learning, Bayesian statistics. Traders without skills in math, programming, statistical analysis, and scientifically developed trading techniques are at a severe disadvantage. Stephen and his colleagues will "eat the lunch" of unprepared traders.



I don't think he has proven his case on this paragraph.  Nor have I disproved his *opinion*, but I am mighty happy to continue using qualitatively judgment of fundamentals as a basis for investing/trading over longer term holding periods, in spite of his prediction and if I or anybody else succeed using alternative methods to what that paragraph advocates then on a traditional scientific basis - his hypothesis is disproved.

Again - my summary on this whole topic is the following quote.



> No amount of sophistication is going to allay the fact that all of your knowledge is about the past and all your decisions are about the future.
> 
> – Ian Wilson (former GE executive)


----------



## tech/a

howardbandy said:


> You are correct that there is a great deal of variability in tomorrow's price. Your approach to trading is probably different than mine. I do not need to know tomorrow's specific price. Up or down is sufficient for many models. Greater than or less than is sufficient for others.






howardbandy said:


> Hi Minwa --
> 
> Let me understand.  You are willing to be the counterparty for any trades I want to make for two weeks?
> 
> If I Buy ES or SPY at today's close, you will Sell.  My gain (or loss) is your loss (or gain)?
> 
> My trades are typically:
> A.  Market-on-close issued within a few minutes of the close.  They may hold for a day, or they may be closed out in the after hours session at profit limits.  They may be for futures, ETFs, or options on either.
> B.  Limit orders to be filled intra-day at what my algorithms estimate to be extreme prices, with exits MOC or limit.
> 
> I will pass your offer.  But thank you very much.
> 
> 
> Best,  Howard




I don't know made sense to me.


----------



## craft

tech/a said:


> I don't know made sense to me.





> Up *or* down is sufficient for many models. Greater than *or* less than is sufficient for others.




If you can not see the need for prediction here then I can not fathom how a duck comprehends things


----------



## skc

Great discussions guys. It gets me thinking about how I should evolve/revolve as a trader. I think there are lots of places for data analysis in financial markets overall.
- Things that Howard does with respect to position size, system monitoring etc.
- Alternate data analysis like satellite image analysis for crops, gas fields, industrial production etc.
- Behavioral analysis - insurance, credit, who panic sells during Brexit etc.

But what this thread seems to be focusing on is using price/volume/time data to find insights and edge... and potentially consistently "true" insights and edge. And I agree with Craft that it is to be proven. I can see big data providing some unique edge that would last varying durations. But the rest of the market and players will always be revolving. There is no doubt that one machine will inevitably be trumped by people and other machines. 

Another example I was thinking about is in sports. Big data collected using new technology (like motion capture and analysis) are ever so popular in elite sports. One trend in the NBA (basketball league) has been the the trend in shot distribution - and in particular the rise of the 3 pointers. Having 3 point shooters at all 5 positions draws the defense out to the perimeters which opens up the driving lanes for easy shots close to the ring. So big players (who are traditionally slower) with speed and 3 point range are very important to any team. Yet, in less than 2 seasons, this edge is beginning to be eroded - every team is copying the model and and the slower big players are getting less and less playing time... which means the faster shooting big men have no slower player to exploit. The competition doesn't stand still, and the sport quants will need to find another edge.

These few articles discuss the above much better than I have here.
https://theringer.com/under-the-influence-of-moreyball-1ea4ba34b85c#.95wllsvjm
https://theringer.com/inside-daryl-...basketball-laboratory-8f5ccb6c6302#.r7cujq66q
https://theringer.com/the-nba-might-have-too-many-big-men-1a654d0b1dec#.hhl94n9z2



craft said:


> I have absolutely no issue with how Howard advocates monitoring a system - I think its sound advice.  I do have an issue with .
> I don't think he has proven his case on this paragraph.  Nor have I disproved his *opinion*, but I am mighty happy to continue using qualitatively judgment of fundamentals in spite of his prediction and if I or anybody else succeed using alternative methods to what he advocates then on a traditional scientific basis - his hypothesis is disproved.




There's something unique about the market that is not the same as predicting weather or having self driving cars. A machine learned system may be able to make some smart predictions, but such predictions can only become correct when someone else is also correct. Say my magic system found that the SPI closing price next week will be 6,000, against today's close of 5,722. In order for my prediction to be correct, someone else have to bid the price up to 6,000. This fact tells me that there is always scope for alternate methods to succeed. Indeed, it is impossible for machines to eat everyone else for lunch... the market will die and by definition the predictions cannot come true no matter how smart the machine is.

So my summary on this topic is:
- Machine learning / big data is an advanced tool that could discover a temporary edge.
- Machines are unlikely to be universal winners. There will be winning machines, much like there will be winner system traders, discretionary traders, and winning dartboards.
- Renaissance is consistently profitable because it is continuously evolving its strategy. It also gains an edge by using data outside of the markets itself to make predictions.
- The chance of an aspiring trader discovering a magical edge using advanced computational techniques are perhaps no better than him/her discovering an edge in  any other alternate method.


----------



## tech/a

craft said:


> If you can not see the need for prediction here then I can not fathom how a duck comprehends things




That's a good thing


----------



## skyQuake

skc said:


> - The chance of an aspiring trader discovering a magical edge using advanced computational techniques are perhaps no better than him/her discovering an edge in  any other alternate method.




This! Be it charting/quantopia/stocktwits/Kosec report/amibroker its the same old story.

In fact I would go one step further and say any easily _*backtestable *_strategy that you can think of has been done to death by hedge funds already. The ease of access of global markets allows the best/fastest guy in the room to steal everyone's lunch around the globe.
P.S. wouldn't it be hilarious if all Ren Tech did all these years was to play simple breakout/breakdown trades across the entire stock universe... rather than go deep they just go super wide


----------



## skc

skyQuake said:


> In fact I would go one step further and say any easily _*backtestable *_strategy that you can think of has been done to death by hedge funds already. The ease of access of global markets allows the best/fastest guy in the room to steal everyone's lunch around the globe.
> P.S. wouldn't it be hilarious if all Ren Tech did all these years was to play simple breakout/breakdown trades across the entire stock universe... rather than go deep they just go super wide




I am pretty sure that have a moving average cross over strategy as well.


----------



## Trembling Hand

DeepState said:


> Your response re: JP Morgan suggests that there is some belief that ML does have benefits, even though it might be restricted primarily to those with the true savvy??




Yeah well I suspect it should . For two reasons. Firstly as already stated the size and spread of their trading, literally everything, everywhere. And two its an arms race and they are one of the few with the resources to be on the top of the pile.


----------



## tech/a

skyQuake said:


> This! Be it charting/quantopia/stocktwits/Kosec report/amibroker its the same old story.
> 
> In fact I would go one step further and say any easily _*backtestable *_strategy that you can think of has been done to death by hedge funds already. The ease of access of global markets allows the best/fastest guy in the room to steal everyone's lunch around the globe.
> P.S. wouldn't it be hilarious if all Ren Tech did all these years was to play simple breakout/breakdown trades across the entire stock universe... rather than go deep they just go super wide




I don't think this is an issue.
Retail Traders are using vastly smaller capital bases.
As such they can be very nimble with their entire account.

With the power and knowledge available to retail traders NOW and not before why wouldn't you incorporate it.

Skyquake

Your logic runs in the face of all Fundamental analysis as well.
If retail can spot an under valued stock so too can the bigger players and the were doing it before computers as was everyone else.

What has been highlighted is that edges are harder to define and find---and keeping any edge for any length of time harder again.

But they are there as many have pointed out as they are trading them.
Wether they can quantify them or not.

I see most of this argument like comparing the way Retail Traders should trade to BUFFETT.
Your just not in the same league. Principals may be similar but application isn't and cant be remotely close.


----------



## Klogg

tech/a said:


> Your logic runs in the face of all Fundamental analysis as well.
> *If retail can spot an under valued stock so too can the bigger players and the were doing it before computers as was everyone else.*
> 
> What has been highlighted is that edges are harder to define and find---and keeping any edge for any length of time harder again.




That's not necessarily true. There are other reasons why F.A. is not replicated to the same extent at an institutional level as T.A. Some of these include:
- The need for instant returns (due to different incentives). T.A. generally is on a shorter timeframe
- Lack of specialisation within institutions. F/A positions are usually held for years, even decades. Try holding onto your staff that long...
- Knowledge of the company/industry and how they interact - relating to the previous point, you cannot easily train someone to have this sort of working knowledge of all relevant information. An individual can train themselves to do it through years of work, however.

I can go on, but my point is I don't necessarily agree with your bolded statement.


----------



## craft

tech/a said:


> Skyquake
> 
> Your logic runs in the face of all Fundamental analysis as well.



How so?

Quantitative judgement is not easy, if possible at all, to backtest. Despite the lack of 'objectivity' I will take an understanding of competative advantage (for example) as an edge any day over the best looking back test equity curve from the most sophisticated computing power available.

Ps if you actually read some of the BH letters to share holders you might learn something rather than just misrepresenting Buffett and what he has taught.


----------



## skyQuake

tech/a said:


> I don't think this is an issue.
> Retail Traders are using vastly smaller capital bases.
> As such they can be very nimble with their entire account.
> 
> With the power and knowledge available to retail traders NOW and not before why wouldn't you incorporate it.




Point is any hedge fund can go wide and trade in $5k parcels across 50,000 stocks. They can be nimble too.



> Skyquake
> 
> Your logic runs in the face of all Fundamental analysis as well.
> If retail can spot an under valued stock so too can the bigger players and the were doing it before computers as was everyone else.



Buffet sized players will probably need to look at $1bil+ MC before they even begin. They can arguably use ML/algos to look at 10 x $100m co's or 100x $10m co's but still they'll need to review each one on a case by case scenario.
Certainly they can't have a robot talking to management and asking insightful questions


----------



## Ves

craft said:


> How so?
> 
> *Quantitative* judgement is not easy, if possible at all, to backtest. Despite the lack of 'objectivity' I will take an understanding of competative advantage (for example) as an edge any day over the best looking back test equity curve from the most sophisticated computing power available.



Just to clarify; do you mean qualitative? (see bold)


----------



## craft

Ves said:


> Just to clarify; do you mean qualitative? (see bold)




yep

Really should proof read hay - makes no sense with the typo - thanks.


----------



## tech/a

skyQuake said:


> Point is any hedge fund can go wide and trade in $5k parcels across 50,000 stocks. They can be nimble too.




Yes I get that-----you see bots plugging away all the time.
But 2 things come to the Ducks Potato head.

(1) With such a broad base aren't results *less likely* to out perform and more likely to conform with the returns seen in an/the index?

This is one of the reasons we don't see large funds out performing--there are a few yes but clearly they aren't doing what the others are.

(2) In the case of a sudden move they still have the same problem *getting out in a hurry*.

This can be seen when funds take large hits like everyone else. Some of the smaller retail guys get out before and I know a few here have when they have seen/suspected it coming.


----------



## DeepState

Been a few comments made which leave me wondering if this is a Man vs Machine discussion.

In my view it is a Man vs Man+Machine (Learning) discussion at this point, but it isn't my thread.  Ultimately it is an all-comers cage fight which requires that some of them stay fighting in order to actually have a secondary capital market.

Somewhere along the line, a person created an algo which then went on to mine data.  These algos are generally heavily supervised, which means their outputs and regression outcomes are heavily censored by some expert to see that it makes sense.  The skill of the expert remains paramount. Let these things run wild and you get comedy - albeit of a niche kind.  Some of these are self optimising, but the structure of the algo is heavily checked and monitored.

Some people like to do fundamental analysis of the traditional kind.  It can work.  If none of it does, the capital markets are garbage.

Others see capital markets differently and like having heaps of data to hand (which is very different to saying that they actually use much of data at all in the actual implementation) which is used to correlate against subsequent stock price movements (these types of models are generally not fair value models, but relative return models and have time horizons for implementation of somewhat less than a year, typically).  It too can work.

ML is just a tool, albeit a powerful one in the right environments (particularly sationary situations) and some situations in financial markets.

If you ran a simulation using a genetic algo to see how this all ends up, it's a seething mess that sees all sorts of aproaches having periods of success and failure. The market is organic. It's a journey.

If the markets were ruled by guys doing fundamental analysis with pocket protectors and pencils, and I was at good as they were, I'd bring a calculator to the gun fight and feel pretty good about it.  Is that controversial?  Naturally if a quant can't understand what it means to do those calculations in the first place, having that calculator allows you to pump out an amazing amount of total garbage.


----------



## DeepState

tech/a said:


> Yes I get that-----you see bots plugging away all the time.
> But 2 things come to the Ducks Potato head.
> 
> (1) With such a broad base aren't results *less likely* to out perform and more likely to conform with the returns seen in an/the index?




If you go long 1,000 stocks and short sell 1,000 other stocks, you get nothing like the index unless by an utter fluke of the highest order or purposefully doing it for a joke.

All else equal, there is nothing to indicate that the portfolio built this way has to perform better or worse than a concentrated one.  You can always lever to an arbitrary risk level.  What matters greatly is the risk/reward ratio.

Port con to a quant, to maximise this ratio, is yet another discussion.  It's a heck of a lot more than whacking a stop loss on and putting 1% of portfolio at risk...although that approach is quantitatively sound under certain conditions for reasons I'll wager very few who use it actually understand.



tech/a said:


> (2) In the case of a sudden move they still have the same problem *getting out in a hurry*.
> 
> This can be seen when funds take large hits like everyone else. Some of the smaller retail guys get out before and I know a few here have when they have seen/suspected it coming.




What makes the use of ML imply that they all do the same thing?  Do all T/A officiandos get hammered at the same time?


----------



## tech/a

Yes I see your point.

Back to not caring which direction only that there is movement.


----------



## Wysiwyg

Another thing I must point out is that you cannot prove a vague theory wrong. [...] Also, if the process of computing the consequences is indefinite, then with a little skill any experimental result can be made to look like the expected consequences.

—Richard Feynman [1964]

A backtest story.   https://poseidon01.ssrn.com/deliver...7103010121001108015117094086026000127&EXT=pdf


----------



## Valued

DaveDaGr8 said:


> I would love to see AI beat a professional poker player and to be honest I'm not sure that it could. In poker, maths has very little to do with anything. Bluffing is everything. Good poker players can change the way the game is played to their advantage ie, get people to change their thinking. They could probably get an AI to modify it's thinking too right before they punish it.
> .




This is incorrect. High level poker is all maths, down to what hands to bluff with. You can prove mathematically how many hands you should be bluffing with and the ratio of bluffs to value bets in any given situation and you can prove mathematically what hands you should be bluffing with as long as your initial ranges leading up to that point is correct. The trouble with AI is that the game is so large and so complex that no computer has been able to solve it and currently the best human players can beat AI in no limit games. 

The best players in the world like Ike Hakton, Doug Polk and OTB Red Barron play a highly mathematical game. OTB Red Baron is substantially better at no limit holdem then say Phil Ivy who is recognised as the best all round poker player in the world. However, he is certainly not the best no limit holdem player in the world. Phil Ivy plays the old school exploitative strategy where as OTB plays a highly mathematical game and crushes the competition.


----------



## ThingyMajiggy

Some interesting articles from Bloomberg: 

https://www.bloomberg.com/news/arti...sustain-record-losses-in-setback-to-fink-plan

https://www.bloomberg.com/news/arti...ning-models-often-fail-to-learn-quicktake-q-a


----------



## skc

ThingyMajiggy said:


> Some interesting articles from Bloomberg:
> 
> https://www.bloomberg.com/news/arti...sustain-record-losses-in-setback-to-fink-plan
> 
> https://www.bloomberg.com/news/arti...ning-models-often-fail-to-learn-quicktake-q-a




Some good articles there. Thanks.


----------



## Valued

ThingyMajiggy said:


> Some interesting articles from Bloomberg:
> 
> https://www.bloomberg.com/news/arti...sustain-record-losses-in-setback-to-fink-plan
> 
> https://www.bloomberg.com/news/arti...ning-models-often-fail-to-learn-quicktake-q-a




It seems like they are just trying to justify their management fees. Right now the average joe is probably thinking heh if I think the market is going up why don't I just stick my money in an index fund. If you put your money in an index fund after the GFC and just held it you would be laughing right to the bank right now. Hell, if you put your money in an index fund 20 years ago and just held through the tech crash and the GFC you would still be doing better than most people. I think that's exactly what the article is getting at, people just put their money in an ETF and be done with it. They pay very small management fee compared to hedge funds that probably won't even beat a market and they have little to think about. 

I think a minimum standard for any system is whether it just just beat buying and holding an index fund forever. If you can't beat that by a significant margin why waste your time.


----------



## minwa

Valued said:


> It seems like they are just trying to justify their management fees. Right now the average joe is probably thinking heh if I think the market is going up why don't I just stick my money in an index fund.




Majority of average joes will never think that - as by their very definition average joe think they can outperform and in reality don't hence they stay average.



Valued said:


> If you put your money in an index fund after the GFC and just held it you would be laughing right to the bank right now.




Hindsight always easy..not many can time the end of GFC and also have the balls to enter.



Valued said:


> I think a minimum standard for any system is whether it just just beat buying and holding an index fund forever. If you can't beat that by a significant margin why waste your time.




Diversification and low draw downs are more important than absolute returns for high net worth portfolios.


----------



## Valued

minwa said:


> Diversification and low draw downs are more important than absolute returns for high net worth portfolios.




Well that's true because high networth individuals have to be concerned about asset allocation etc moreso then someone with a full time job who has money set aside for trading but the point is the same but it would only apply to the portion of the portfolio that is set aside for equities. Obviously, I am not suggesting someone go and take their portfolio which is say only 20% equities and then go sell all their property and go put it in an index fund. It applies to the portion of your portfolio that is set aside for trading in the equity markets. It equates to simply a question of whether you're beating the market while being in the market since you're in that market with a certain amount of your portfolio in the first place.


----------



## minwa

Valued said:


> Well that's true because high networth individuals have to be concerned about asset allocation etc moreso then someone with a full time job who has money set aside for trading but the point is the same but it would only apply to the portion of the portfolio that is set aside for equities. Obviously, I am not suggesting someone go and take their portfolio which is say only 20% equities and then go sell all their property and go put it in an index fund. It applies to the portion of your portfolio that is set aside for trading in the equity markets. It equates to simply a question of whether you're beating the market while being in the market since you're in that market with a certain amount of your portfolio in the first place.




I wasn't really talking about other assets, just the part of equities portfolio. A high net worth individual will pick a fund that has achieved a 30% gain with a drawdown of 15% over a index fund that may have achieved 40% gain with drawdown of 35% eg. like GFC type of periods for the majority of his portfolio. It's not wasting time to under return the index like you have put it if you are doing so with much lower risk. You only mentioned absolute return vs index while risk and draw down is actually what is more important for a fund because investors will sell out and reallocate when there is a big dip without participating in the recovery.


----------



## Valued

minwa said:


> I wasn't really talking about other assets, just the part of equities portfolio. A high net worth individual will pick a fund that has achieved a 30% gain with a drawdown of 15% over a index fund that may have achieved 40% gain with drawdown of 35% eg. like GFC type of periods for the majority of his portfolio. It's not wasting time to under return the index like you have put it if you are doing so with much lower risk. You only mentioned absolute return vs index while risk and draw down is actually what is more important for a fund because investors will sell out and reallocate when there is a big dip without participating in the recovery.




I think I am looking at this from a poker orientated aspect where you would always take the most profitable decisions regardless of the possible drawdown in your bankroll because the optimal way to play is to always take the highest EV option (the decision that will provide the most profit over the long term). Obviously, it depends on your life situation. If you're about to retire you definitely can't handle any drawdowns, you need that money. However, if you're young I would argue it is always correct to take the most profitable decision at any point. It depends on whether you can handle the drawdowns mentally though.


----------



## DaveDaGr8

Valued said:


> This is incorrect. High level poker is all maths, down to what hands to bluff with. You can prove mathematically how many hands you should be bluffing with and the ratio of bluffs to value bets in any given situation and you can prove mathematically what hands you should be bluffing with as long as your initial ranges leading up to that point is correct. The trouble with AI is that the game is so large and so complex that no computer has been able to solve it and currently the best human players can beat AI in no limit games.
> 
> The best players in the world like Ike Hakton, Doug Polk and OTB Red Barron play a highly mathematical game. OTB Red Baron is substantially better at no limit holdem then say Phil Ivy who is recognised as the best all round poker player in the world. However, he is certainly not the best no limit holdem player in the world. Phil Ivy plays the old school exploitative strategy where as OTB plays a highly mathematical game and crushes the competition.


----------



## DaveDaGr8

http://www.gizmodo.com.au/2017/01/w...man-poker-pros-are-getting-trounced-by-an-ai/


----------



## ThingyMajiggy

DaveDaGr8 said:


> http://www.gizmodo.com.au/2017/01/w...man-poker-pros-are-getting-trounced-by-an-ai/




That's pretty amazing if that's legit. I want to see a Libratus vs Libratus! Maybe the world will cave in on itself?  

The way this stuff is advancing poses some questions about our future, they're talking there about how it's going to be used in fields like cybersecurity, medical diagnosis, in a couple years it looks like we won't be driving anymore(no delivery/truck drivers eventually?), factory jobs are out because of robots. I gather one day AI will be writing the actual code themselves too? What are we all going to be doing in the future? In what areas do humans have an advantage over AI? Creativity? On the spot reactions I guess, although in that article they're also talking about developing AGI, artificial general intelligence. Interesting times....especially to see how it develops in the trading world.


----------



## howardbandy

ThingyMajiggy said:


> ...  developing AGI, artificial general intelligence. Interesting times....especially to see how it develops in the trading world.




AGI == the singularity.  Some believe it will happen in the very near future, others a few years in the future.  There is pretty high agreement that it will be achieved by the end of the 21st century.  

The articles describing the increase in use of AI in trading were the motivation for beginning this thread.

Best,  Howard


----------



## Valued

ThingyMajiggy said:


> That's pretty amazing if that's legit. I want to see a Libratus vs Libratus! Maybe the world will cave in on itself?
> 
> The way this stuff is advancing poses some questions about our future, they're talking there about how it's going to be used in fields like cybersecurity, medical diagnosis, in a couple years it looks like we won't be driving anymore(no delivery/truck drivers eventually?), factory jobs are out because of robots. I gather one day AI will be writing the actual code themselves too? What are we all going to be doing in the future? In what areas do humans have an advantage over AI? Creativity? On the spot reactions I guess, although in that article they're also talking about developing AGI, artificial general intelligence. Interesting times....especially to see how it develops in the trading world.




The poker pros they have that computer playing arn't the best headsup players in the world though. The three best players in the world right now at headsup no limit holdem specifically (the game the computer is playing) are likely Doug Polk, Ike Haxton and Ben Sulsky and maybe Daniel Cates fits in the top 3 it's a bit hard to tell. I know Doug Polk played a computer for 8 hours and won. We arn't really there yet. The computers have a hard time too with 6 handed and 9 handed games because of how much more complex they are. The edge humans have is they adapt faster to changes in the rules. In poker if you change the rules it invalidates everything the AI has learned up to that point. For example, Omaha is just like texas holdem but you draw four cards and not two and this changes the game so much that a new bot has to be built to play Omaha. You can also change the blind structure etc. 

Maybe the same could be said with trading. If traditional correlations in fundamental data or even just between two different asset classes break down because humans change the rules (new inventions making some commodities obsolete or large structural changes or even changes in laws and how markets work) then a computer that learns based on the fundamentals may not be able to adapt very fast.


----------



## howardbandy

Valued said:


> Maybe the same could be said with trading. If traditional correlations in fundamental data or even just between two different asset classes break down because humans change the rules (new inventions making some commodities obsolete or large structural changes or even changes in laws and how markets work) then a computer that learns based on the fundamentals may not be able to adapt very fast.





The primary use of computers -- where they have an advantage over humans -- is trading rather than investing.  By trading I mean frequent changes in positions based on recent price history with short holding periods; and by investing I mean infrequent changes in position based on use of fundamental information and long holding periods.  

It is my opinion that market efficiency limits the use of fundamental information and that it is of near-zero value to individuals other than those with a seat in the boardroom and their best friends.  I have published my thoughts on use of fundamental information several times -- most recently here:
http://www.blueowlpress.com/wp-content/uploads/2016/10/FT-Fundamental-Analysis-Appendix-A.pdf

In trading, automated techniques such as model design, signal generation, trade selection, timing, order management, and risk management are superior to those of human traders.  It is those aspects where I recommend that humans who want to compete do so by broadening their skills -- as the video referenced in post number 1 of this thread suggests many large trading firms are already doing.

Thanks for listening, Howard


----------



## tech/a

Howard

Thank you for sharing your view.
Anyone having a good read and using nothing more than common sense.
can see that fundamental analysis has as many flaws as any other form of analysis.

To the point as you say of being meaning less unless you are on the inside looking out.

Happy to stick with price action.


----------



## Klogg

howardbandy said:


> It is my opinion that market efficiency limits the use of fundamental information and that it is of near-zero value to individuals other than those with a seat in the boardroom and their best friends.




It limits it yes, but stating it is of 'near-zero value' is very mis-leading.

I'm not a professional and this 'near-zero' value has lead to returns of more than double any Australian index of your choice. I have no friends with a seat in any boardroom, nor do I have any.

I accept that there are markets that are very efficient, but blanket statements like that are very misleading.

Even taking the first statement of your publication:
_"In order to be valuable, any data series or indicator, company data or economic series, must be:
• Timely
• Accurate
• Predictive"_

This is massively wrong. I don't need very accurate data to make money from an investment. As Buffett says, _"you don't need to know an man's exact weight to know he's obese"._
And backing that up is Keynes' famous quote _"it is better to be roughly right than precisely wrong"._

Likewise, it doesn't need to be 'timely' for it to be useful.

Perhaps you meant that the fundamental data is of near-zero value in your particular type of analysis. That I can agree with.

EDIT: I should mention that the idea that markets are completely efficient has clearly been killed. It works in theory and creates nice mathematical equations for all, but in practice it clearly doesn't exist.


----------



## Valued

@howard by your definition are swing traders who hold positions for a few weeks at a time traders or investors? What about someone who buys USD for the intention of holding it for the next 3 - 6 months because they think the US will rise interest rates? Are they traders or investors? What is infrequently?

I don't think it follows that if markets are efficient fundamental data is useless. Efficient markets should reflect the underlying fundamentals. If the price of oil is rising this should have a follow on effect eventually. You should see changes in the Baltic Dry Index (indicating shipping goods by sea is becoming more expensive) and then you should see a gradual rise in other commodities and from there inflation and potential interest rate rises. If you get on board when this starts and put your money in commodities then because you know the leading fundamental indicators that predict commodity price rises an efficient market will make you a lot of money. To make a lot of money you have to be with the trend and with everyone else anyway.

I am not convinced markets are that efficient in the short term but even if they are there is nothing magical about this that prevents you making money knowing the fundamentals support your position and everyone else agrees and you ride the trend upwards for 6 months or whatever it may be. 

@Tech for break out trading price action is most important but fundamentals are vital in other asset classes. Commodities without fundamentals is going in blind.


----------



## howardbandy

Klogg said:


> It limits it yes, but stating it is of 'near-zero value' is very mis-leading.




Hi Klogg --

I stand firmly with the material I published in that article.  

Between delays, revisions, granularity of data, unknowable biases, and general agreement that the efficient market hypothesis is not usable, what remains is broad market timing and subject sector selection.  I do not see how the scientific method can help.   

If you have found techniques that work for you, ignore me and continue as you were.  

What happens in market downturns will sort us all out.

Best regards,  Howard


----------



## Klogg

howardbandy said:


> I stand firmly with the material I published in that article.




I didn't expect you to change your stance from my one post. You've committed so much to publishing that piece (a few hours as a minimum), so I would expect some resistance before writing that off as a lost cause - and that's before considering the fact that you've opened it up to the public, so you'd want to save face and remain consistent (we all do this to a degree).
My post was to mitigate the impact on others who are new and to create discussion.

Another problem with this stance is it completely ignores the success of firms like Berkshire Hathaway, Oaktree Capital, Markel (I can go on and on...), as well as many who post here.

In my view, it's really a form of cognitive dissonance - you can see evidence of the EMH not holding true, yet you use EMH to form a view on fundamental analysis. It confuses me a little.
Perhaps you know a way in which these investors (listed above) are successful whilst EMH holds true - if that's so, please do tell me. I'm genuinely interested.


----------



## Valued

I don't understand why the EMH applying means fundamental analysis is any less valid. I disagree with the EMH but assume markets are efficient, say CBA shares are trading at 60 dollars and the market is efficient so they are actually worth 60 dollars, if you do some fundamental analysis of that stock and come to the view that CBA is a good company, it will keep growing, it has a competitive edge in the market, it is coming out with exciting products, and you want to invest and get some dividends and watch your capital grow with the company, and then over time the company does grow and becomes more profitable and dividends increase and this raises the share price, you have made money. I would say an efficient market rewards good stock selection since to make money you need to invest in a good company and in fact an efficient market means you're not overpaying too and you're likely to get the benefits of the growth. 

Richard Farleigh, an Australian investor, believes markets are much more efficient than people give them credit for but also notes you can just ride a trend as an investor and make money. If then the conclusion is that fundamental analysis is only valid for investments or longer term trading then obviously that is true. However, that has nothing to do with whether markets are efficient or not.

So how does market efficiency impact on the validity of using fundamental analysis? It does not. Presumably, we want to use fundamental analysis to pick the best performing stocks in the best performing sectors who have competitive edges and who are going to grow and generate more profits and the shareholders then benefit from that.


----------



## Klogg

Valued said:


> *So how does market efficiency impact on the validity of using fundamental analysis? It does not. *Presumably, we want to use fundamental analysis to pick the best performing stocks in the best performing sectors who have competitive edges and who are going to grow and generate more profits and the shareholders then benefit from that.




Perhaps you've misunderstood EMH. From investopedia:

_The efficient market hypothesis (EMH) is an investment theory that *states it is impossible to "beat the market" because stock market efficiency causes existing share prices to always incorporate and reflect all relevant information.*

EMH http://www.investopedia.com/terms/e/efficientmarkethypothesis.asp#ixzz4WvdMHJgY _

The bolded section says it all. Hence, if people are beating the market, it cannot be 100% efficient.


Yes, some markets are more efficient than others, but that doesn't mean they reflect ALL available information (that is, not all markets are 100% efficient, as EMH suggests) - hence, fundamental analysis should (and does) work.


----------



## Triathlete

Klogg said:


> Yes, some markets are more efficient than others, but that doesn't mean they reflect ALL available information (that is, not all markets are 100% efficient, as EMH suggests) - hence,* fundamental analysis should (and does) work*.




I have no doubt that Fundamental analysis works although in my case I will also use Technical analysis for my entries and exits.

One of the issues with Fundamentals is how does an investor stay on top of it since the information is usually in a quarterly,half yearly ,yearly report or a special announcement.

The reason I bring this up is what if in between these reports the company starts to experience difficulties and no information is forthcoming early from the company or is delayed in its announcements investors are left holding the bag.

I do use Fundamentals to choose financially strong companies but use Technicalls for entries and exits.

My belief is that a technical person may have and advantage because they may start to see something changing on their charts that alerts them to something that does not seem right and so will take steps to protect their positions.

I mean there could be times that information has gotten out to big players that has not got to the general public and the only way we can see this early is on a chart and not in a company report that are months apart.

The last two companies that come to mind are SGH and now Bellamys. 

Does anyone have any comments on how just being a Fundamental investor would have helped these investors..?????


----------



## Valued

Klogg said:


> Perhaps you've misunderstood EMH. From investopedia:
> 
> _The efficient market hypothesis (EMH) is an investment theory that *states it is impossible to "beat the market" because stock market efficiency causes existing share prices to always incorporate and reflect all relevant information.*
> 
> EMH http://www.investopedia.com/terms/e/efficientmarkethypothesis.asp#ixzz4WvdMHJgY _
> 
> The bolded section says it all. Hence, if people are beating the market, it cannot be 100% efficient.
> 
> 
> Yes, some markets are more efficient than others, but that doesn't mean they reflect ALL available information (that is, not all markets are 100% efficient, as EMH suggests) - hence, fundamental analysis should (and does) work.




I don't think it follows that if you beat the market it invalidates the EMH. Again, I don't think the EMH is right but the fact that someone beats the market (which I assume means the same as beating the index) does not in itself invalidate the EMH. The market is an average. Not every stock grows at the same rate as the index. Some stocks out perform the index and some stocks under perform the index. Some businesses are just better than others. The fact that an efficient market reflects all available information does not mean you can't beat the index.


----------



## Klogg

Valued said:


> The fact that an efficient market reflects all available information does not mean you can't beat the index.




My take on it is that if it were efficient, every stock would give the same return.
Let's agree to disagree, as I've derailed the thread more than enough... my apologies


----------



## Valued

Triathlete said:


> Does anyone have any comments on how just being a Fundamental investor would have helped these investors..?????




For SGH I believe Warren Buffet for a long time warned to be very cautious of a company who grows by acquisitions since acquisitions may on paper grow the company but in reality may negatively affect its value. Growing by acquiring is exactly what SGH is doing. I actually chose not to invest because of how they kept acquiring other firms, especially firms overseas that operated in a legal jurisdiction that most SGH management would have no experience in. I also knew a lot of the PI claims they were acquiring were very likely to be rubbish but that's because I have inside knowledge of the industry. That same knowledge to acquire for a retail trader would likely cost about 15k - 30k or so which isn't really realistic for most investors simply conducting some due diligence.


----------



## Valued

Klogg said:


> My take on it is that if it were efficient, every stock would give the same return.
> Let's agree to disagree, as I've derailed the thread more than enough... my apologies




That makes no sense.


----------



## skc

Klogg said:


> Perhaps you've misunderstood EMH. From investopedia:
> 
> _The efficient market hypothesis (EMH) is an investment theory that *states it is impossible to "beat the market" because stock market efficiency causes existing share prices to always incorporate and reflect all relevant information.*
> 
> EMH http://www.investopedia.com/terms/e/efficientmarkethypothesis.asp#ixzz4WvdMHJgY _



I can never quite understand the EMH. It's not like there's actually an independent being called "the market". The market is the collective result of traders / investors / computers etc etc.

If one player has correctly discounted all the relevant information (and is hence being efficient), then by definition the counterparty (there's always one) who took the opposite side of the trade has been inefficient. So clearly... some market players are more efficient than others.

So to say that market is efficient so no player should try seems a bit irrational.



Triathlete said:


> One of the issues with Fundamentals is how does an investor stay on top of it since the information is usually in a quarterly,half yearly ,yearly report or a special announcement.




I think one needs to define fundamental analysis as being much broader than just company financials and announcements. To me it includes all information other than price action. There are lots of information out there that impacts a company and are released periodically, or can be researched/collected by those who has the willingness and the ability to do so. For example, data like retail sales, housing starts, webpage impressions, news by peers in other industries, regulatory announcements or even just social media and regular news (look at TNE on Wednesday). Some of these are easily accessible by all market participants.

Then there are others that are harder to access for the general public. For example, the hedge fund who shorted BAL used supermarket scan data (which revealed supermarket sales were falling way before any announcement was made) as part of their trade rationale. Analysis of satellite images is also a good example, anything from new cars arriving/leaving the port, number of flares in a gas field, green-ness of certain crops etc can inform how commodities or companies are travelling.

This is a field where big data/machine learning could be very useful... e.g. what's the correlation between temperature and coke sale, or clothing sale, or salmon prices? Or between rainfall and almond production or wine glut? Nonetheless, I think this will most likely provide a temporary edge until everyone else copies it, especially if it proves profitable. Part of the reason for Renaissance Technologies enduring success is probably in no small part due to the secrecy behind its quantitative methods...

Now there's no doubt that, those who possess the fundamental information edge (note they are not necessarily insiders) will make their move before any price action tells them to do so. Indeed, their actions are probably what created the price actions in the first place, followed by the technical traders. That's why, in the absence of anyone being absolutely sure that they have an information edge (which probably applies to most retail traders), price action should to be respected.

Another interesting aspect worth mentioning relates to how price actions in one market feeds into another. For example, price action in US bond yields' feeds into the performance of yield stocks on the ASX. I am not sure whether that should be categorised as technical or fundamental information... but it is the collection of all these information that forms the "context" for an astute trader/investor. I do believe that a machine may be able to paint the context picture in a more quantitative manner than a trader, in a steady-state kind of situation. Though I remain to be convinced if the machine has a definite edge when the context changed.


----------



## Triathlete

Valued said:


> I also knew a lot of the PI claims they were acquiring were very likely to be rubbish but that's because *I have inside knowledge of the industry.* *That same knowledge to acquire for a retail trader would likely cost about 15k - 30k or so which isn't really realistic for most investors simply conducting some due diligence*.



Thanks for your comments Valued........Your comment is exactly my point,the only way I stand a chance of not getting caught is following my Fundamental information with a chart since the information is instant and  I would not have inside information.


Now there's no doubt that, those who possess the fundamental information edge (note they are not necessarily insiders) *will make their move before any price action tells them to do so. Indeed, their actions are probably what created the price actions in the first place, followed by the technical traders.* *That's why, in the absence of anyone being absolutely sure that they have an information edge (which probably applies to most retail traders), price action should to be respected.*

[/QUOTE]

SKC......I think your last comment is also what I mean and if your latest information is saying one thing and the price action is saying something different than you really need to take notice.........


----------



## howardbandy

Greetings --

Please understand that my perspective is development and use of trading systems that are based on the scientific method.  That is: observe data, postulate rules, validate the rules using data not previously used.  The metric I use is risk-normalized profit potential -- CAR25. 

The points I am making regarding fundamental analysis are that no information usable in creating scientifically-based trading models that have reasonable risk-normalized profit potential can pass through the filters of granularity, revision, and bias.

If a person or organization feels something in the fundamental data is helpful and increases their confidence in taking a position in the issue, they should continue to do that.  Similarly for Gann, Fibonacci, divergences, crossover of the 50 and 200 day moving averages, etc.  I have found no predictive value in any of those, but many people use them and defend their use.  I have no argument against doing that.

Generation of trades is separate from system health and position size.  That is, the system that generates trading signals is separate from the system that determines position size.  Any set of trades, no matter how they are generated, can be used in the trading management system.  It is there that risk-normalized profit potential is measured.  

I have found no way to use fundamental data to form rules that generate trades that have reasonable risk-normalized profit potential.  That is, the CAR25 value of every set of trades that I can generate using a set of rules derived from fundamental information is too low to be worth trading.

Prove me wrong.   

Post a system that uses fundamental data from, say 2000 through 2010 (or a period of your choice that you have found to be stationary), to form a discoverable set of rules to buy and sell some tradable issue.  Test those rules on previously unused data that follows the data mined to discover the rules.  Define a statement of risk tolerance and analyse the out-of-sample trades.  Determine the profit potential.  If you prefer a different metric, state it and defend it as preferable to CAR25.  Post a report describing the whole experiment.

-----------------------      

Regarding Warren Buffett and Berkshire Hathaway.  From my perspective, Mr. Buffett is the CEO of a multi-faceted conglomerate corporation.  BH buys very large stakes in companies, expecting to hold them a very long time.  They influence the management of those companies.  That is a corporate view, rather than a trader's view.  

My personal risk tolerance is to hold risk of drawdown to a small chance of a drawdown greater than 20%.  Many money managers view even that as too risky.  When BH held through drawdowns in excess of 50% in 2009, that was an indication to me that they were managing corporate subsidiaries, not trades.  

My comparison would include Mr. Buffett in the same group as Jack Welsh (General Electric) and Ken Lay (Enron) -- CEOs of conglomerate corporations.  A group of CEOs of trading organizations would include David Shaw and James Simons.  

Best regards,  Howard


----------



## Triathlete

howardbandy said:


> Greetings --
> 
> Please understand that my perspective is development and use of trading systems that are based on the scientific method.  That is: observe data, postulate rules, validate the rules using data not previously used.  The metric I use is risk-normalized profit potential -- CAR25.
> 
> The points I am making regarding fundamental analysis are that no information usable in creating scientifically-based trading models that have reasonable risk-normalized profit potential can pass through the filters of granularity, revision, and bias.
> 
> If a person or organization feels something in the fundamental data is helpful and increases their confidence in taking a position in the issue, they should continue to do that.  Similarly for Gann, Fibonacci, divergences, crossover of the 50 and 200 day moving averages, etc.  I have found no predictive value in any of those, but many people use them and defend their use.  I have no argument against doing that.
> 
> Generation of trades is separate from system health and position size.  That is, the system that generates trading signals is separate from the system that determines position size.  Any set of trades, no matter how they are generated, can be used in the trading management system.  It is there that risk-normalized profit potential is measured.
> 
> I have found no way to use fundamental data to form rules that generate trades that have reasonable risk-normalized profit potential.  That is, the CAR25 value of every set of trades that I can generate using a set of rules derived from fundamental information is too low to be worth trading.
> 
> Prove me wrong.
> 
> Post a system that uses fundamental data from, say 2000 through 2010 (or a period of your choice that you have found to be stationary), to form a discoverable set of rules to buy and sell some tradable issue.  Test those rules on previously unused data that follows the data mined to discover the rules.  Define a statement of risk tolerance and analyse the out-of-sample trades.  Determine the profit potential.  If you prefer a different metric, state it and defend it as preferable to CAR25.  Post a report describing the whole experiment.
> 
> -----------------------
> 
> Regarding Warren Buffett and Berkshire Hathaway.  From my perspective, Mr. Buffett is the CEO of a multi-faceted conglomerate corporation.  BH buys very large stakes in companies, expecting to hold them a very long time.  They influence the management of those companies.  That is a corporate view, rather than a trader's view.
> 
> My personal risk tolerance is to hold risk of drawdown to a small chance of a drawdown greater than 20%.  Many money managers view even that as too risky.  When BH held through drawdowns in excess of 50% in 2009, that was an indication to me that they were managing corporate subsidiaries, not trades.
> 
> My comparison would include Mr. Buffett in the same group as Jack Welsh (General Electric) and Ken Lay (Enron) -- CEOs of conglomerate corporations.  A group of CEOs of trading organizations would include David Shaw and James Simons.
> 
> Best regards,  Howard





 Howard with your systems is there a specific group of companies that you concentrate on eg top 50 or 300 to generate your trades or do you also use  maybe speculative companies that are included as well...?????


----------



## howardbandy

Triathlete said:


> Howard with your systems is there a specific group of companies that you concentrate on eg top 50 or 300 to generate your trades or do you also use  maybe speculative companies that are included as well...?????



Greetings --

Company selection is a system in itself.  

I recommend Not applying a set of trading rules to 3000 tickers, finding that results for 30 of them seem to be good, and trading those 30.  That is "optimizing / curve fitting the symbol space" and is a poor idea.

Rather, begin by looking for issues that have the potential of being tradable -- adequate profit with acceptable risk.  There is a technique described in both the "Foundations" book and the "Quantitative Technical Analysis" book called the "The Prospector."  It is possible to determine the risk of trading any price series even before a set of rules to generate buy and sell signals is applied.  We want several characteristics:
1.  Enough volatility so there is profit available by trading.
2.  Not so much volatility that whatever model is applied results in too much risk.
3.  High liquidity (and low bid-ask spread).
4.  Given that the issue passes the first three filters, then see if there is a trading model that can detect persistent patterns that precede profitable trades.

Use the accuracy / holding period charts to help focus on high CAR25.  The sweet spot is high accuracy and short holding period.  Again, description of the technique is in the chapter on risk in both books.

The list of issues that pass is fairly short.  The major sector exchange traded funds (XLB, XLE, XLF, ...) work, as do a few common stocks.  With the stocks, be careful to distinguish between a rising market and persistent patterns.  If you have the "Foundations" book, discussion of "The Prospector" begins on page 73 in Chapter 2 -- Risk.   In "QTA," it begins of page 124.

I recommend working with a single issue long / flat to begin.  If / when that works, extend to long / flat / short, and / or add additional systems, each a model and a single data stream.  The Bayesian sequential learning that is the basis for Dynamic Position Sizing ("QTA" beginning on page 385) works to switch between alternative systems.  Since CAR25 is a Dominant metric, the highest risk-normalized return comes from trading the best and switching as conditions change.

Best regards,  Howard


----------



## Quant

Triathlete said:


> Howard with your systems is there a specific group of companies that you concentrate on eg top 50 or 300 to generate your trades or do you also use  maybe speculative companies that are included as well...?????



I am actually in the middle of building a daily TF algo to trade the top 50 . The top 50 is ideal as they meet all of Howards requirement ^^above^^ and have well structured Price Action that i believe will be condusive to a positive expectant outcome . Im looking at this to trade a SMSF and i anticipate returns double the average fund with minimal time in the market by comparison  . Maybe someone should start a thread on this pursuit  ...   Great  topic by Howard , lucky to have him here   ...


----------



## Triathlete

Quant said:


> I am actually in the middle of building a daily TF algo to trade the top 50 . The top 50 is ideal as they meet all of Howards requirement ^^above^^ and have well structured Price Action that i believe will be condusive to a positive expectant outcome . Im looking at this to trade a SMSF and i anticipate returns double the average fund with minimal time in the market by comparison  . Maybe someone should start a thread on this pursuit  ...   Great  topic by Howard , lucky to have him here   ...




That sounds interesting Quant.

I have never really strayed outside the top 150 and as you have said the top 50 for a trader should provide more than enough action and with the added benefit of  plenty of liquidity.

If you can keep the forum updated on your outcome it would be great..

Also agree  that having Howard's insights on these topics is great.


----------



## Newt

Quant said:


> I am actually in the middle of building a daily TF algo to trade the top 50 . The top 50 is ideal as they meet all of Howards requirement ^^above^^ and have well structured Price Action that i believe will be condusive to a positive expectant outcome . Im looking at this to trade a SMSF and i anticipate returns double the average fund with minimal time in the market by comparison  . Maybe someone should start a thread on this pursuit  ...   Great  topic by Howard , lucky to have him here   ...



Yep, lots of us would be interested in updates, appropriate for what you're ok sharing of course.  By daily, do you mean interday, or intra?


----------



## Trembling Hand

No need to worry. We can all just buy an ETF and be a quant now..... 

http://www.quantxfunds.com/


----------



## Quant

Trembling Hand said:


> No need to worry. We can all just buy an ETF and be a quant now.....
> 
> http://www.quantxfunds.com/



Dont worry i have considered using ETFs in SMSF , the geared ones have appeal for sure


----------



## lftrader

Trembling Hand said:


> No need to worry. We can all just buy an ETF and be a quant now.....
> 
> http://www.quantxfunds.com/




  Everyone will be Dalio soon.


----------



## lftrader

This is intriguing how we have come to this stage. It seems complexity doubles every 5-10 years. What is next? 

http://www.priceactionlab.com/Blog/...short-equity-strategy-development-with-dlpal/


----------



## ThingyMajiggy

> Hedge funds using vast amounts of data, computing power, and machine-learning techniques to make money are drawing investors’ attention. But their brief track records show they suffer the same shortcomings as their more traditional peers. The Eurekahedge AI Hedge Fund Index, which tracks 12 of these money pools, has outperformed hedge fund peers since 2013 but failed to beat the S&P 500 Index.



Source - https://www.bloomberg.com/news/arti...organic&utm_source=facebook&utm_medium=social

So I guess Buffet's advice of buying an index and sitting on it is still the way to go then.


----------



## skc

ThingyMajiggy said:


> So I guess Buffet's advice of buying an index and sitting on it is still the way to go then.




FWIW... I think this advice will become problematic sooner or later.

It's not a bad idea when index investing represents only a small portion of total funds flow. 
But when index investing turns mainstream, it takes on a different life form.Index investing is an investor conceding that he/she cannot pick the right stocks to beat the market... and implying that the other market participants can better pick the winners. 

How does that work though when index investing goes mainstream? It means everyone is looking for someone else to be smart and sort out the "correct" relative weighting of the index. But who would that be? And those people won't have the size in FUM to counter the passive flows, so even if they are "correct" they can't win. f

Passive investing works up to a size, until it doesn't. I wonder when we'd get there (if not already).


----------



## Smurf1976

Largest market cap stocks go up, thus increasing their % share of investment via index funds, causing their price to go up further and the cycle repeats.

Looking at some of the biggest cap stocks in the US my personal opinion is that index investing is already a bubble. The higher those stocks go, the greater the share of the index they become and the greater the buying from the index funds. At some point, as with all bubbles, it won't end well.


----------



## minwa

skc said:


> But when index investing turns mainstream, it takes on a different life form.Index investing is an investor conceding that he/she cannot pick the right stocks to beat the market... and implying that the other market participants can better pick the winners.




I don't think that is possible with human nature. Most enter the market because they think they are better than the average. The group who enter the market with an already "defeated" attitude/believe and are willing to take index market average return minus fees will always be the minority.


----------



## howardbandy

Greetings --

I agree that Warren Buffett has been very successful with Berkshire-Hathaway.  But I believe he should be characterized as the CEO of a diversified coglomerate rather than as an investor.  He buys majority interests in companies in industries he understands well, then takes an active role in their management.  He has stated that he expects to hold a very long time.  In the financial crisis of 2008-2009, Berkshire stock dropped over 50%.  Indeed, Wikipedia describes Bershire Hathaway as a conglomerate:
https://en.wikipedia.org/wiki/Berkshire_Hathaway

If I was plaaning to emulate his success in the future, I would:
... Have a long time horizon.  Mr. Buffett began buying stock in Berkshire in 1962.
... Be well enough capitalized to be able to buy controlling interests in major companies.
... Have a staff of competent and experienced executives to manage those companies. 
... Have the tail winds provided by an expanding economy, favorable monetary environment, and stable political environment.

Unfortunately, I can meet none of the first three or those criteria, and I have very little confidence in continuation of the fourth. 

Trading an index, through a future or ETF, based on models developed using the scientific method is rational.  Buying and holding an index without a validated model or set of rules is an act of faith.

Best,  Howard


----------



## ThingyMajiggy

howardbandy said:


> Trading an index, through a future or ETF, based on models developed using the scientific method is rational.  Buying and holding an index without a validated model or set of rules is an act of faith.
> 
> Best,  Howard




That's all well and good, but the "act of faith" is still out performing the AI/ML going by that chart.


----------



## Gringotts Bank

ThingyMajiggy said:


> That's all well and good, but the "act of faith" is still out performing the AI/ML going by that chart.




It sure does.  Faith moves mountains.  Learn what that means.  There's not a single method that will perform without a solid belief in same.


----------



## howardbandy

ThingyMajiggy said:


> That's all well and good, but the "act of faith" is still out performing the AI/ML going by that chart.




Has done well in the past.  Estimating risk and profit potential in the future requires a testable system and returns metrics that can be used for management.  

As an alternative, consider James Simons' hedge fund:
http://fortune.com/2017/05/16/hedge-fund-james-simons-renaissance-technologies/

Best,  Howard


----------



## howardbandy

Gringotts Bank said:


> It sure does.  Faith moves mountains.  Learn what that means.  There's not a single method that will perform without a solid belief in same.




I have a solid belief in the scientific method.  I hope that counts.

Best,  Howard


----------



## Gringotts Bank

howardbandy said:


> I have a solid belief in the scientific method.  I hope that counts.
> 
> Best,  Howard



If I said 'yes it counts', you might misinterpret me.

What I said in the previous post points to a deeper possibility - whether the individual mind interacts with what we commonly refer to as 'consensus reality'.  There's piles of evidence to suggest it doesn't just _interact_, but in fact it _creates _reality.  I don't just mean a soft, dream-like personal reality, but all the way up to the hardest consensus reality.

I am a scientist myself, and I still use it in my work, but I'm reformed to some degree.  The scientific method has some very major weaknesses which rarely get mentioned, let alone discussed.


----------



## Trembling Hand

Astrology isn't science GB.


----------



## minwa

howardbandy said:


> As an alternative, consider James Simons' hedge fund:
> http://fortune.com/2017/05/16/hedge-fund-james-simons-renaissance-technologies/




Using one extreme outlier isn't really solid evidence. For example, I could pick a fundamentalist from the market wizards book that would blow out Simons's returns.

The stats show there is nothing market beating, on average, of employing AI/ML by those hedge funds.


----------



## Value Hunter

howardbandy said:


> It is my opinion that market efficiency limits the use of fundamental information and that it is of near-zero value to individuals other than those with a seat in the boardroom and their best friends.




Howard there are plenty of forum members like myself, Craft, ROE, Value Collector, and too many others to name that are doing very well out of fundamental information thank you very much and would strongly disagree with your assertion. People like you who believe that fundamental information is of little value allow people like us to keep make out-sized profits.


----------



## howardbandy

Gringotts Bank said:


> I am a scientist myself, and I still use it in my work, but I'm reformed to some degree.  The scientific method has some very major weaknesses which rarely get mentioned, let alone discussed.



Hi GB --
Can you point me to discussion of the weaknesses of the scientific method?
Best,  Howard


----------



## Gringotts Bank

howardbandy said:


> Hi GB --
> Can you point me to discussion of the weaknesses of the scientific method?
> Best,  Howard




On the smallest scale, what's 'out there' in the material world is a mass of waves/particles.  We construct reality (our very own reality, shared by no one) by taking that stuff and filtering it with our expectations, beliefs and memories.

Science has a very hard time controlling for the effect of expectation and belief.  We can't perceive things that are outside our beliefs, and we fully perceive things that others might say don't exist so long as we believe them.  So what is real becomes almost impossible to define.  All we can say is what's real for us.

"A fundamental conclusion of the new physics also acknowledges that the observer creates the reality. As observers, we are personally involved with the creation of our own reality. Physicists are being forced to admit that the universe is a “mental” construction. Pioneering physicist Sir James Jeans wrote: “The stream of knowledge is heading toward a non-mechanical reality; the universe begins to look more like a great thought than like a great machine. Mind no longer appears to be an accidental intruder into the realm of matter, we ought rather hail it as the creator and governor of the realm of matter. Get over it, and accept the inarguable conclusion. The universe is immaterial-mental and spiritual." (“The Mental Universe” ; Nature 436:29,2005)           ~~R.C. Henry, Professor of Physics and Astronomy

This article is also quite good.

https://www.scientificamerican.com/...f-expectations-can-allow-you-to-bend-reality/

If you were to take your best trading system and teach it to 10 people, each would create their own very personal results.  Some would do well, some would lose, some would lose heavily.  

If you had 10 companies with the same fully automated system and same infrastructure, software, latency and all that, some would do much better than others.


----------



## howardbandy

Gringotts Bank said:


> On the smallest scale, what's 'out there' in the material world is a mass of waves/particles.  We construct reality (our very own reality, shared by no one) by taking that stuff and filtering it with our expectations, beliefs and memories.



Are we discussing the same thing?  The scientific method I am referring to is the technique of postulating models, followed by out-of-sample validation.  This one:
https://en.wikipedia.org/wiki/Scientific_method
Best,  Howard


----------



## Gringotts Bank

howardbandy said:


> Are we discussing the same thing?  The scientific method I am referring to is the technique of postulating models, followed by out-of-sample validation.  This one:
> https://en.wikipedia.org/wiki/Scientific_method
> Best,  Howard




Yes, science is a method of determining what is true (knowledge of fact).  I'm saying facts are not possible to determine because for a fact to exist, there first has to be an individual mind in order to perceive it.  As soon as the mind arises in an attempt to perceive, it carries all sorts of confounding factors such as expectation and belief.

If I hypnotize you to believe that the number 4 does not exist, you literally will not see it or know it.  For the rest of us the number 4 exists and is real, but are we objective independent observers or were we hypnotized to believe it exists at an early age and just went along with it?   Strange example, but it shows how subjective things are.


----------



## howardbandy

Gringotts Bank said:


> Yes, science is a method of determining what is true (knowledge of fact).  I'm saying facts are not possible to determine because for a fact to exist, there first has to be an individual mind in order to perceive it.  As soon as the mind arises in an attempt to perceive, it carries all sorts of confounding factors such as expectation and belief.
> 
> If I hypnotize you to believe that the number 4 does not exist, you literally will not see it or know it.  For the rest of us the number 4 exists and is real, but are we objective independent observers or were we hypnotized to believe it exists at an early age and just went along with it?   Strange example, but it shows how subjective things are.




In which case there is nothing but illusion -- we are imaginary creatures who live in a pseudo-reality created by the dream of some other entity.  So much for free will.  

How could we tell?  What experiment could distinguish between being real and being imaginary?  How should we plan and live our lives given the uncertainty?  How does either help us buy and sell profitably?

Any statement asserted without evidence can be dismissed without evidence.

Best,  Howard


----------



## ThingyMajiggy

howardbandy said:


> Any statement asserted without evidence can be dismissed without evidence.




Mmmm...funny how it works the other way round


----------



## howardbandy

ThingyMajiggy said:


> Mmmm...funny how it works the other way round



Not really.  Wikipedia states it well: "Under modern interpretations, a scientific hypothesis must be falsifiable, implying that it is possible to identify a possible outcome of an experiment that conflicts with predictions deduced from the hypothesis; otherwise, the hypothesis cannot be meaningfully tested."


----------



## skc

howardbandy said:


> Any statement asserted without evidence can be dismissed without evidence.




There is evidence.


----------



## Gringotts Bank

howardbandy said:


> In which case there is nothing but illusion -- we are imaginary creatures who live in a pseudo-reality created by the dream of some other entity.  So much for free will.
> 
> How could we tell?  What experiment could distinguish between being real and being imaginary?  How should we plan and live our lives given the uncertainty?  How does either help us buy and sell profitably?
> 
> Any statement asserted without evidence can be dismissed without evidence.
> 
> Best,  Howard




Many of the most prominent figures throughout history believe we live in an illusory matrix.

http://www.independent.co.uk/news/s...that-shows-choice-could-just-be-a7008181.html

Some say it's possible to break the illusion.  Real people, not just movie actors.

Just carry on I guess...?  Or perhaps try to manipulate the software of the mind to your advantage.  There are primitive hunter-gatherers in the jungles of PNG who would completely deny the existence of advanced technology because it would be too threatening to their worldview.  If you gave them a box of smart phones and demonstrated how to operate them, they would not even entertain the idea; instead they'd burn and bury them.   We do the same thing in our own way.  There are quite a number of sane, reasonable and intelligent people who claim to have broken the illusion but we wouldn't give them the time of day.  It's too threatening.

https://www.theguardian.com/technol...usk-tesla-space-x-paypal-hyperloop-simulation

Howard you say "reality created by the dream of some other entity".  But that's not the only possibility.  It could be created by our true identity - an identity of which we are unaware.  Then the work would be to drop the false identity, as suggested by non-dualist philosophies and religions.


----------



## howardbandy

Gringotts Bank said:


> Many of the most prominent figures throughout history believe we live in an illusory matrix.




IMO these thoughts would be better addressed in a philosophical / theological thread.  Whatever reality actually is, I have no option other than treating what I experience as "the" reality. 

Best,  Howard


----------



## ThingyMajiggy

howardbandy said:


> I have no option other than treating what I experience as "the" reality.




That reality being that you don't actually trade if I recall correctly, is that correct?


----------



## howardbandy

ThingyMajiggy said:


> That reality being that you don't actually trade if I recall correctly, is that correct?



I do.


----------



## Gringotts Bank

howardbandy said:


> IMO these thoughts would be better addressed in a philosophical / theological thread.  Whatever reality actually is, I have no option other than treating what I experience as "the" reality.
> 
> Best,  Howard




That's traditionally where such ideas are kept.  But if such philosophies have any merit at all, then  they will be applicable to all aspects of life, including trading.

I'm not critisizing your method of trading.  I like it, use and promote it.  Just saying there might be more to life than what we imagine.  

You have a choice other than the one you mention.  It is to treat the current reality as if it's a simulation.  _*If*_ there's a way to break through, then that would be the path.  Unfortunately this is mere speculation on my part.  But I do understand the theory very well - as well as you understand statistical analysis.  And the theory seems very sound.


----------



## Wysiwyg

lftrader said:


> This is intriguing how we have come to this stage. It seems complexity doubles every 5-10 years. What is next?
> 
> http://www.priceactionlab.com/Blog/...short-equity-strategy-development-with-dlpal/




*"Conclusion*

The results in this article demonstrated at least two things:

(1) There is potential for significant edge in the indicators developed by DLPAL

(2) Long/short equity strategies have the potential of generating significant alpha during major stock market downtrends even if cost due to friction is high."

I spent many hours testing L/S strategies that switch on Index MA crosses and lo/behold they did exceptionally well during the GFC. Their conclusion is (1) a sales pitch (2) waiting for the next complete market destruction (potential).

Backtests mean little and this article is basic could have/should have stuff.


----------



## tech/a

People like Howard are experts in analysing data.
They have the ability to analyse data in a vast range of fields.

From medicine ( without being a Doctor or specialist )
For Engineering or Geology without building a single structure
A whole host of data anylists sent man to the moon
Probs to Mars, Jupiter, Astroids and beyond without leaving the
Planet.

Analysis of stock,futures or any other financial data is no different.

I'll bet the last thing employers who engage the services of the likes of Howard
Want to know is if they trade flown to the moon or cured a disease.

How's Pete's threads influenced your trading and search for something that works for you?
You don't need to follow the ideas of people like Howard.
Petes method doesn't.


----------



## Newt

Good post Tech.  The "Market Wizard" series are the classic reference for showing how many ways there are to skin the market cat, albeit after generally many years of hard work.

The title of this thread quite disturbed me, but we're lucky to have someone with Howard's experience taking the time to post to remind us how the jungle is changing.  I'd rather be concerned than ignorant.  The reality is hard enough but shame to see this uncertainty upsetting people to the level of personal attacks.  

All the effort we put into chasing down reliable profitability, but meanwhile that car is heading off down the road of increasingly complexity.


----------



## Skate

*Quoting Dr Howard Bandy*

_“The business of trading is changing with astonishing speed. It is now about applied mathematics, machine learning, Bayesian statistics. Traders without skills in math, programming, statistical analysis, and scientifically developed trading techniques are at a severe disadvantage. Stephen and his colleagues will "eat the lunch" of unprepared traders” _

*Quoting Dr Tomasz Janeczko (Founder of Amibroker)*

_“In my opinion throwing more languages (like Python) into the mix just makes things harder, not easier, and is not really necessary as everything is doable within AmiBroker itself, but @howardbandy and others have other opinion. People have tendency to think in terms "more = better". But the truth is that in programming less is more. And less is better. So don't add complexity when it is not needed”_

*Quoting ThingyMajiggy*

_“Would love to see some evidence of the far superior method in action”_

*Quoting tech/a*

_“People like Howard are experts in analysing data. They have the ability to analyse data in a vast range of fields. From medicine ( without being a Doctor or specialist ) For Engineering or Geology without building a single structure A whole host of data anylists sent man to the moon Probs to Mars, Jupiter, Astroids and beyond without leaving the Planet”_

Having a keen interest in the subject matter I would like to play the Devil's Advocate.

tech/a makes a disconnect between a data analysis and the end user of the data whereas Dr Howard Bandy has no disconnect, he is a "Data Analyst" and a "Trader" being in a unique position working both sides of the fence adding more weight to ThingyMajiggy reasonable request “to see some evidence of the far superior method in action”

I would like to hear from Dr Howard Bandy in regards to the comments made by Dr Tomasz Janeczko _“In my opinion throwing more languages (like Python) into the mix just makes things harder, not easier, and is not really necessary as everything is doable within AmiBroker itself”
_
Dr Howard Bandy uses languages like Python, ML and AI while at the same time having a deep understanding of Amibroker Formula Language and as fence sitter I’m not fully convinced that either software has an advantage over the other but if there is I would like to see evidence.

*My DISCLAIMER*

I respect Dr Howard Bandy and Dr Tomasz Janeczko holding them both in high regards.


----------



## tech/a

Skate

Do you really believe amilanguage is on a par with the power of Python?

What is enough?

A propeller is enough to fly a plane
A jet engine is better.

I'll leave the rest to the quants.


----------



## Skate

_*"Skate - Do you really believe amilanguage is on a par with the power of Python?"*_

tech/a I'm a fence sitter - _"I’m not fully convinced that either software has an advantage over the other but if there is I would like to see evidence"._

Dr Tomasz Janeczko and Dr Howard Bandy have differing opinions which I both respect.

_*What is enough?*_
"Sometimes good enough is good enough"

Playing the Devil's Advocate again.
"Sometimes good enough is not good enough when better is expected"

_*A propeller is enough to fly a plane
A jet engine is better.*_

Great point - _*A jet engine is better - *_for commercial applications, whereas for individuals - _*A propeller is enough to fly a plane *_and *What is enough? *sometimes enough is enough.

tech/a to use your analogy a dingy is more responsive and maneuverable compared to an Ocean liner, so quoting Dr Tomasz Janeczko again:

"_People have tendency to think in terms "more = better". But the truth is that in programming less is more. And less is better. So don't add complexity when it is not needed”
_
*Holding my position*
_"Dr Howard Bandy uses languages like Python, ML and AI while at the same time having a deep understanding of Amibroker Formula Language and as fence sitter I’m not fully convinced that either software has an advantage over the other but if there is I would like to see evidence"_


----------



## tech/a

When you say "see evidence"  do you understand it's about capability.

Python is the broadly accepted language for scientific research in a wide variety of fields.
It's not a restricted language as ami language is.

The complexity you keep referring to shouldn't be confused with thoroughness in investigation.


----------



## tech/a

Have you noticed the rich fly jets they know they are better and can afford them.

The poor own Cessna's they aren't as good as jets but that's all they can afford.


----------



## rb250660

Didn't Einstein say something like "Everything should be made as simple as possible, but not simpler."


----------



## Wysiwyg

_“Would love to see some evidence of the far superior method in action”

I think this question stems from the belief that with all Howard's knowledge and experience he would be in the top 1% of traders and exceptionally wealthy from trading markets. 


The markets are driven by human response mechanisms. I doubt that can ever be completely quantified._


----------



## Newt

That's what I love about Peter2's threads.  Real time, logical, hard slog to give positive expectancy. I suspect a lot of people would be happy to start (learning to trade) knowing profitability is possible, rather than the Lear Jet.  Nothing wrong with dreaming - it just doesn't seem pay well in the markets - sigh.


----------



## Newt

Skate said:


> *Quoting Dr Howard Bandy*
> 
> Having a keen interest in the subject matter I would like to play the Devil's Advocate.
> 
> tech/a makes a disconnect between a data analysis and the end user of the data whereas Dr Howard Bandy has no disconnect, he is a "Data Analyst" and a "Trader" being in a unique position working both sides of the fence adding more weight to ThingyMajiggy reasonable request “to see some evidence of the far superior method in action”
> 
> I would like to hear from Dr Howard Bandy in regards to the comments made by Dr Tomasz Janeczko _“In my opinion throwing more languages (like Python) into the mix just makes things harder, not easier, and is not really necessary as everything is doable within AmiBroker itself”
> _
> Dr Howard Bandy uses languages like Python, ML and AI while at the same time having a deep understanding of Amibroker Formula Language and as fence sitter I’m not fully convinced that either software has an advantage over the other but if there is I would like to see evidence.
> 
> *My DISCLAIMER*
> 
> I respect Dr Howard Bandy and Dr Tomasz Janeczko holding them both in high regards.





Devil's advocate is good.  Would it be pushing this too far to compare expensive tools versus knowledgeable tradesperson?  A great craftsman with cheap tools will probably beat the newbie starting with all the expensive toys everytime?  I think the original thread premise was that we're often up against master craftsman, who also have great tools?  (was  that too many questions in a row??)

Respect for  Tomasz J 2nd'ed.  Amibroker is an amazing software tool.


----------



## Wysiwyg

Newt said:


> Devil's advocate is good.  Would it be pushing this too far to compare expensive tools versus knowledgeable tradesperson?



That is right. Where do I start would be one question. Next stage is on the treadmill for countless hours, days, weeks before coming full circle. I do believe there is value in knowing information not broadly known and that is where there is an edge.


----------



## CanOz

tech/a said:


> Have you noticed the rich fly jets they know they are better and can afford them.
> 
> The poor own Cessna's they aren't as good as jets but that's all they can afford.




Not the most relevant analogy....python is free, amibroker is at least $250.00 US


----------



## skc

tech/a said:


> A propeller is enough to fly a plane
> A jet engine is better.




I also agree that a jet engine is more powerful than a propeller. 

But here's where the analogy fails, and reflecting what ThingyMajiggy has asked... what's the evidence?

Evidence has shown that a jet engine will move an aircraft faster than a propeller... and faster = better. Is there equivalent evidence that shows more powerful programming and analysis languages = better profits. 

How does anyone show or prove that? I have no idea.

Having said that, I also believe that my pocket calculator improves my trading. If I don't have a calculator I wouldn't be able to do some of the quick sums through the trading day (or do them slowly and with more mistakes), I may take less trades, or wrong trades, or have less conviction on the right trades because I couldn't do the calculations fast enough. So a calculator = better profits for me. But I am not sure if I have a calculator 5000x faster would make any further difference. I also know profitable traders who don't even own a calculator. 

A calculator vs manual calculation isn't the same as Python vs Amibroker which isn't the same as advanced big data quant vs others. So no analogy is perfect. I am not taking sides on the debate... I wish someone can make a winning argument.

P.S. ThingyMajiggy account is disabled?


----------



## tech/a

I think the original post was pointing out what retail traders are now up against.
10 yrs ago the competition in the market was less.
Now it's more
In 10 yrs time it will be even more.

Will I turn from Duck to Dinasour if I don't come up to a level that is competitive?

Without long trends it's far more difficult to profit.
10 yrs ago the taxi driver was making a profit in the market.

Choppy markets are very difficult to etch out $$s

Frankly I don't know how fundies make a buck today and I've not seen any evidence on this forum that any one of the many here ----are.
Trading short term choppy markets over a long period (years) with long term fundamental analysis doesn't equate to me.

Not saying technical analysis in un trained hands is any better.
But it is much more reactive in choppy markets.
There is no doubt that there are more tools and increasingly more 
Powerful tools available to the technical trader.


----------



## Trembling Hand

tech/a said:


> Not saying technical analysis in un trained hands is any better.
> But it is much more reactive in choppy markets.
> There is no doubt that there are more tools and increasingly more
> Powerful tools available to the technical trader.



Really? Tech those very same powerful tools you think you are talking about are doing the same for quantitative fundamental approaches. Exactly the same.


----------



## tech/a

Yeh I know 
But not by the Fundies on this forum
I doubt they are interested in implementing.

Those that I know of are looking for extremities in fundamentals
Setting up to take advantage of.

A combination of both would be interesting particularly in indexes,
Currencies and commodities.


----------



## skc

minwa said:


> I don't think that is possible with human nature. Most enter the market because they think they are better than the average. The group who enter the market with an already "defeated" attitude/believe and are willing to take index market average return minus fees will always be the minority.




sorry Minwa. I forgot to link further discussion on this topic is here https://www.aussiestockforums.com/threads/a-long-bet.33273/#post-951785


----------



## tech/a

Have you noticed Howard's absence?


----------



## Wysiwyg

Any tools being used by the trading competition besides a programming language?


----------



## Wysiwyg

> Hedge funds have long relied on computers to help make trades. According to market research firm Preqin, some 1,360 hedge funds make a majority of their trades with help from computer models—roughly 9 percent of all funds—and they manage about $197 billion in total. But this typically involves data scientists—or "quants," in Wall Street lingo—using machines to build large statistical models. These models are complex, but they're also somewhat static. As the market changes, they may not work as well as they worked in the past. And *according to Preqin's research, the typical systematic fund doesn't always perform as well as funds operated by human managers (see chart below*)


----------



## Wysiwyg

Information (and speed of) is the obvious key. Accessing and processing that informatiopn does require some computer skills. For example the list of shorted stocks from ASIC website would be handy automatic information when testing strategies. E.g. if > 10% short sold, then don't buy. E.g. seasonal affects on stocks, news articles, broker reports are information that delivered quickly or 'read' for significance would be an advantage.


----------



## minwa

tech/a said:


> Have you noticed Howard's absence?




Howards absence is voluntary. He was online today and his account still works.

Thingy's account is disabled. Click on it and it says disabled.

There be more to the story than just here. But if that resulted from just this exchange here then I am done here.


----------



## tech/a

minwa said:


> Howards absence is voluntary




I totally understand his non participation.
What I cant understand is that proof of trading prowess
evidently will enhance the credibility of a Maths Professor
who is talking on his topic of expertise.

Its like listening to a researcher who is looking for
a cure to cancer but rejecting their credibility because
they don't have evidence of curing it!


----------



## minwa

tech/a said:


> I totally understand his non participation.
> What I cant understand is that proof of trading prowess
> evidently will enhance the credibility of a Maths Professor
> who is talking on his topic of expertise.
> 
> Its like listening to a researcher who is looking for
> a cure to cancer but rejecting their credibility because
> they don't have evidence of curing it!




Never knew being a maths proffessor automatically makes you profitable in trading. We are talking about making money in trading here arnt we ?


----------



## tech/a

Oh I see!

Doesn't make you a Doctor/Engineer or Astronaut.

You are really missing the whole point of this thread.


----------



## Joe Blow

I'd like to have my two cents worth here.

Howard has contributed a lot of valuable content here at ASF and he has answered a lot of questions posed by a lot of people, especially on the topic of Amibroker. He is an acknowledged expert in his field who is trying to engage in a serious discussion on a topic of real importance to traders. So I think it is incumbent upon everyone to participate in the discussion in a constructive and respectful way. There is nothing wrong with questions or genuine disagreement but questions shouldn't seem like accusations. They should be framed in such a way as to advance and expand the discussion, not to tear it down or sabotage it.

If a post doesn't add something positive or constructive to the debate then perhaps it's something that is better left unsaid.

So, having said that, can we please get back to discussing the topic at hand? I will tidy up this thread a little later and remove as many off topic posts as is practicable.


----------



## McLovin

The metaphors in this thread are next level ridiculous. The poor buying Cessnas because they can't afford jets. 

Cessna make jets. I guess that would make you middle class if you're flying around in one of theirs.



			
				tech/a said:
			
		

> Its like listening to a researcher who is looking for
> a cure to cancer but rejecting their credibility because
> they don't have evidence of curing it!




This thread looks more like a researcher claiming to have cured cancer without any evidence of curing it. IMO, of course.


----------



## tech/a

Oh come on !!!! They are classic


----------



## howardbandy

Greetings --

I have not gone away.  But I am discouraged.  

When someone -- anyone -- asks how to learn trading system development, and specifically what books to read or courses to take, my first recommendation is to read Daniel Kahneman's "Thinking, Fast and Slow."  I have previously mentioned it on ASF and it is at the top of the bibliography I have posted.  Kahneman's thesis is that humans have two thinking systems. System 1 is fast and reacts in a way that reflects situations as WYSIATI -- What You See Is All There Is.  System 2 is conscious thought.  He explains in detail, supported by his own Nobel Prize winning research and experience as well as that of other psychologists and economists, the very many biases we all have.  When we allow our thinking to stop with System 1, we all are limited by our biases.  We are naturally expert at fooling ourselves.  

He points out the ease with which we assign causality when there is none, inflate probability of events based on recentness, modify our estimates toward arbitrary and often unnoticed anchors.  Without using mathematics, or even giving the names to the techniques, he describes the scientific method and Bayesian analysis and insists that all forecasting should follow them.

Some of his examples are of professional money managers.  

I am a member of a nonfiction book discussion club and Kahneman is the book for the July meeting.  One of our members is a clinical psychologist.  He said that Kahneman is among the required readings for the periodic re-certifications that his profession requires.  It is not an easy read, but it is an important one.  At very least, it will explain a lot of our thought processes; perhaps, it will change your life.

---------

About specific results associated with machine learning and artificial intelligence -- the topic I intended this thread to follow when I started it.

Every trader will make his or her own determination of personal risk tolerance, what to trade, how to trade, how to develop techniques, and how to build confidence in them.  I have posted some specific recommendations based on both academic-style research and actual trading results from my own experiences and those of the hedge fund where I was a senior analyst.  I have described the techniques in two books and a video posted to YouTube.  

The conclusions and results are crystal clear.  Begin with a statement of personal risk tolerance.  Let the Data Prospector evaluate the risk and profit potential inherent in the issue you are considering trading.  The data -- daily bars for most people, but the technique works with any bar length -- must satisfy three criteria:  enough volatility to offer profit from trading it; not so much volatility that risk exceeds the trader's tolerance for accuracy and holding period that is attainable; and persistent and discoverable signals that precede profitable trades.  The first two can be determined from the Data Prospector.  If he, the Data Prospector, passes the data series, then the trading system developer can try to find a set of rules that identify trades that offer profit potential adequate to compensate for the risk.

All of the above is independent of trading system development platform and the model that will signal the trades.  It applies in all trading circumstances.  

Much as traders wish it to be so, long holding periods, infrequent trading, portfolios of issues drawn from a large pool of issues, and extensive reliance on either judgment or backtesting alone cannot produce systems that pass validation that are sufficiently profitable for reasonable risk. 

Trading system development can be seen from two points of view -- 1.  Begin by identifying desirable trades, then see what patterns occurred earlier -- 2.  Begin with indicators, then see what trades occurred after.  These correspond to 1.  Machine learning / pattern recognition  -- and 2.  Traditional chart analysis / decision tree rules.

There is a steep learning curve to work with machine learning.  But there is evidence that some of the largest and most profitable trading houses and hedge funds are increasingly moving toward machine learning.  I have given some examples -- such as James Simons.

Is it important that I am personally and currently an active trader in order for my advice to be valuable?  Professionals in the field of data analysis, modeling, simulation, prediction, or whatever you wish to call it require three areas of expertise:  domain knowledge; programming skills; analysis skills.  I have all three.  More than that, I am expert in all three.  Sports coaches are no long playing in league matches, but they have valuable knowledge and experience and can pass that on.  They can provide the coaching for the frequent repetitions with prompt feedback that players require to fully develop.  A male data analyst does not need to personally use lipstick and skin conditioners in order to develop accurate forecasts for development and distribution of those products.  

So it is disappointing to me that the discussion left science-based development and went into full-throated criticism without justification.  One of my colleagues regularly reminds us that saying "just saying" does not automatically convert an intractable condition into one at the top of the list and worth consideration.  

Stay with traditional development if you wish.  It is familiar.  It may work for you.  There are two risks.  One, without understanding risk-normalized profit potential and applying it trade-by-trade, it will be difficult to manage trading if or when economic conditions change and performance declines.  Two, machine learning develops better systems.  Hands down better.  Always better.  No single decision tree model is ever at the top of a run where more capable models were included as alternatives.  No Kaggle competition is ever won by a decision tree.  In machine learning, simpler is not always better -- in fact, it is seldom better.  The winner of the Netflix million dollar prize was a very complex ensemble.  Read about the competition and some of the models that did well here:  http://blog.echen.me/2011/10/24/winning-the-netflix-prize-a-summary/ 

Back to my first piece of advice -- begin by reading Kahneman.
https://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555/

Thanks for listening.

Best,  Howard


----------



## Trembling Hand

Howard I think it would go 1000 steps in the right direction if you could post a simple decision tree model result and then an example of an improvement or better approach with machine learning. It's hard for someone to see new technology as better when you cannot see the new tech or how it is applied. 

I believe if you started this thread with an example we wouldn't have 384 posts of bitching and endless speculation about possibilities. We would have moved far further down the road but we are stuck at the denial stage....... still.


----------



## Skate

Trembling Hand said:


> *Howard I think it would go 1000 steps in the right direction if you could post a simple decision tree model result and then an example of an improvement or better approach with machine learning.* It's hard for someone to see new technology as better when you cannot see the new tech or how it is applied.
> 
> I believe if you started this thread with an example we wouldn't have 384 posts of bitching and endless speculation about possibilities. We would have moved far further down the road but we are stuck at the denial stage....... still.




Trembling Hand you nailed it.

*"if you could post a simple decision tree model result and then an example of an improvement or better approach with machine learning"*

*Exactly what I was asking in post #357 & #359*
_"Dr Howard Bandy uses languages like Python, ML and AI while at the same time having a deep understanding of Amibroker Formula Language and as a fence sitter I’m not fully convinced that either software has an advantage over the other but if there is I would like to see evidence"_


----------



## Lone Wolf

howardbandy said:


> Two, machine learning develops better systems.  Hands down better.  Always better.




It may be true that a machine learning system written by an expert will outperform a standard decision tree model. However, most of us are not experts in the field of machine learning. A more important question for a public trading forum might be whether the average Joe would be more profitable using a decision tree model or developing their own ML system. 

Most people work full-time jobs in unrelated fields, they first need to learn a programming language, then learn basic ML techniques. What are the chances of them using that basic level of competence to produce a trading system that significantly outperforms what they could have achieved through the decision tree model?

It could also be part of the reason for resistance to the idea that machine learning is better. ML is beyond most people, or so they believe. People want to believe that the path they can take is just as good as the path they can't take.


----------



## tech/a

Perfect
Can't agree more 
"like"


----------



## howardbandy

Greetings --

It is difficult to make a direct comparison between a system using a decision tree model developed using a traditional platform such as TradeStation or AmiBroker and a system using machine learning even if that model is decision tree.  Difficult because of the focus on indicator and rules versus focus on target.

-------------------

Using a traditional platform and a common mean reversion entry using the RSI indicator and impulse signals, the model looks like this in an AmiBroker or TradeStation-like pseudocode:
RSILookback = 2
RSIValue = RSI(C,RSILookback)
EntryLevel = 30
ExitLevel = 40
Buy = Cross(EntryLevel,RSIValue)
Sell = Cross(RSIValue,ExitLevel)

The result is a series of trades identified by the impulse signals Buy and Sell.  
There will be about 50 trades each year, each holding between 1 and 6 days.
When these trades are analyzed, there will be three points at which risk can be measured: Closed trade, Mark-to-market daily, Intra-day.
Since most people think in terms of daily data and daily accounting, Mark-to-market daily risk is the most appropriate metric.  But that is relatively difficult to use when trades are held more than one day.

We can use state signals -- beLong and beFlat -- without loss of accounting generality.  That gives a signal for every day so an n-day trade can be manipulated as if it is n 1-day trades.  This helps a lot and I recommend everyone use state signals even if they decide to stay with traditional platform.

The fitting process consists of varying the values of the three parameters:  RSILookback, EntryLevel, ExitLevel.
Find the best fit in-sample, test out-of-sample.  Then analyze the OOS trades as the best estimate of future performance.  Safe-f and CAR25 can be computed for these trades.

-----------------------

Using a machine learning program, the process looks like this:
RSILookback = 2
RSIValue = RSI(C,RSILookback)
Target = ref(C,1) > C
RSIValueY1 = ref(RSIValue,-1)
Model = DecisionTree

Pause for explanation before proceeding.

The target is True if tomorrow's close is higher than today's close.  That is the beLong state signal.
RSIValueY1 is the value of the RSIValue yesterday.  It is given its own variable because the data points will not be processed in sequence.  During fitting, there is no memory of yesterday's value.  If it is important, it must have its own variable.  Each data point must be self contained and independent of the others.
Note there is no 30 or 40.  If the fitting process identifies that entering when the RSIValue drops through 30, the data point for those bars will have RSIValue <30 and RSIValueY1 >30.  We could force the comparison to 30, but that would remove much of the advantage of using the machine learning.  We want the model to identify the pattern.  To pick the signal out of the noise.

The data that is important for the discussion is, at this point, two predictor columns -- RSIValue and RSIValueY1, and one column of Target.  Say there are 252 daily data points and the system behaves as if it is stationary for at least one year.

Split the data.  Store the predictors in a two dimensional array named X.  It has 252 rows, 2 columns.
Store the target in a one dimensional array named y.  It has 252 rows, 1 column.  

For a given index -- the index of any row is taken from the date -- row i of X and row i of y correspond.  y is the known target.

Split both X and y through a randomizing process on the indexes.  Separate about 60% of X and y to use to train the model.  Call the X portion XTrain and the y portion yTrain.  Let the machine learning function that implements decision tree find the best fit of XTrain predicting yTrain.

Save the model in an array named A.  A is essentially the coefficients of the solution to the linear algebra that fits XTrain to yTrain.

Call the portion of the X array that was left over when XTrain was removed XTest.  Call the y portion yTest.

Validate the model out-of-sample by applying the A matrix to XTest and have it predict what y would be according to the model.  Call those y values yPredict.  There is a state for every bar, day, row.  If the target is 1, the state is beLong, otherwise the state is beFlat.

The metric for goodness of fit is comparison between known yTest and predicted yPredict.  There are many metrics that can be used.  Confusion matrix is one.  Area under the ROC curve is another.  Or the distribution of the rows of yPredict that have the value 1 -- beLong -- can be used directly to calculate risk (safe-f) and profit potential (CAR25).

----------------------

The difficulty in comparison is that the trading system disappeared when it was translated into a machine learning application.  We can still test several values for RSILookback, but not for 30 and 40.

Let me say that again.  When a trading system is developed using machine learning, the rules have disappeared.  We are looking for patterns the precede desirable trades.  There is no direct comparison.  

-------------------

The "Quantitative Technical Analysis" book goes through all this in much more detail with charts, diagrams, and more detailed explanatins.  It also provides Python code that does what I have described using a logisic regression model.  The chart on page 382 shows some results -- XLV daily 2010 through 2014 has a TWR of about 2.4 traded at full fraction.

Additionally, read the chart on page 403 that compares OOS trades with buy and hold. And the charts on pages 416 and 417 that show the effect of using safe-f to manage trading day-by-day.

------------------

I hope this helps rather than clouds the discussion.  It should illustrate the process and also illustrate the difficulty of making an apples to apples comparison. The Python code in the QTA book can be downloaded and run.  

Best regards,  Howard


----------



## howardbandy

Lone Wolf said:


> It could also be part of the reason for resistance to the idea that machine learning is better. ML is beyond most people, or so they believe. People want to believe that the path they can take is just as good as the path they can't take.




These are some of the points Kahneman makes as well.  We look for confirmation of what we already believe.  When faced with a difficult question we substitute an easier question.

Best,  Howard


----------



## howardbandy

Lone Wolf said:


> It may be true that a machine learning system written by an expert will outperform a standard decision tree model. However, most of us are not experts in the field of machine learning. A more important question for a public trading forum might be whether the average Joe would be more profitable using a decision tree model or developing their own ML system.
> 
> Most people work full-time jobs in unrelated fields, they first need to learn a programming language, then learn basic ML techniques. What are the chances of them using that basic level of competence to produce a trading system that significantly outperforms what they could have achieved through the decision tree model?




We are all trading in the same arena.  James Simons and Goldman Sachs take their side of the trade without asking whether we are experts or novices.

It is hard to compare directly, but subjectively, if a machine learning trading system scores 100 arbitrary points, traditional platform decision trees seldom score above 70 or 80.

Stay with what you know and like.  But consider implementing daily mark-to-market accounting, use state signals in your trading model code, analyze the recent trade result about once a week, adjust safe-f based on recent trade results, compute CAR25 regularly, and take any system whose CAR25 drops (say to single digits) offline.

Best,  Howard


----------



## traderxxx

id just like to say that you dont need powerful fast programes to trade,
and like SKC i wouldnt be without my calculator,
some of my little algo/formulars are as simple as abc,
or i should say a+b+c
having said that i would still like to be working with a programmer to
help keep my too many systems sorted out.


----------



## Value Hunter

tech/a said:


> Frankly I don't know how fundies make a buck today and I've not seen any evidence on this forum that any one of the many here ----are.




tech/a are you simply not believing that people like Craft, Value Collector,  ROE, etc are making huge returns when they post return figures or talk about their positions? Do you expect them to send snapshots of bank statements/brokerage statements to a stranger on the internet? Why do successful fundamental investors on the forum have to prove anything to you or anybody else? The forum is not a dick swinging contest.

If somebody seems credible based on their post count and length of time on ASF, the quality of their writing/analysis and the stock picks/comments they publicly made in the past over a long period of time can't you just give them the benefit of the doubt that they are being truthful about the returns they are making?


----------



## Triathlete

Value Hunter said:


> * are making huge returns* when they post



Just so we can make our own comparison what are you calling *Huge returns* , and in that case what do you call a *small *and *medium return and over what timeframe.*


----------



## Value Hunter

Well its of course subjective but a good example is in the SMSF thread https://www.aussiestockforums.com/threads/smsf-returns.25070/page-2

Craft posted that his 9 year CAGR for his super fund was 34.4%. Now admittedly the post was in 2013 but I am sure if you asked Craft for an update on his SMSF it would still be powering along.

Or what about Value Collector who bought a large number of shares in Capilano Honey at under $3.00 per share and still owns them today (now $16+ per share)? Or the big win he had on Fortescue metals? Or ROE buying shares in Dominos Pizza under $4 and holding it until it was a 10+ bagger? And ROE buying Credit Corp at under $1.00 and still owning it today when its at $17+

If you follow the posts of the top fundamental investors over time you can get a sense of how well they are doing. You know stuff like that is likely to be true because the posting occurs in real time, for example ROE buying Credit Corp at under $1.00 and posting about how he bought some and why it is a good buy and then later coming in thread and saying he still owns it.


----------



## Value Hunter

Of course looking at a random assortment of an investors positions does not give you the full story but it certainly gives you a good glimpse.


----------



## minwa

Cool, now we have examples of both "dinosaur" methods (P2's a nice representation of a technical method) - still waiting on example from the supposed superior ML/AI method. And please don't use Renaissance/Goldman, 99.9% of us here can not relate to an institution with billions dollar worth of resources.


----------



## howardbandy

I give up.


----------



## tech/a

Howard

I would too
Clearly some don't take the time to read
Your posts-----from one who does a huge thanks.

Value Hunter.
We've all had multi bagged wins.ive had many over the years.
I'd be very surprised if any Fundamental trader here is clearing 25%
Or more----consistantly year in year out.
Particularly if they have a portfolio of over 10 stocks.


----------



## Trembling Hand

Can we cut and move the sh!te out of this thread?


----------



## Joe Blow

Trembling Hand said:


> Can we cut and move the sh!te out of this thread?



Absolutely. The question is how much needs to go?


----------



## satanoperca

howardbandy said:


> Greetings --
> 
> Daniel Kahneman's "Thinking, Fast and Slow."




Thanks Howard, looking forward to my bed time read.

There are some of us that sit in the darkness of this site, absorbing information and appreciate all the long time posters and their varied opinions.

ML does seem well beyond the minds of many, but I for one, like to have an understanding of all types of trading without having the need to be an expert in them all.

Please keep on posting.


----------



## Value Hunter

Tech/a clearly you glossed over my last post



Value Hunter said:


> Well its of course subjective but a good example is in the SMSF thread https://www.aussiestockforums.com/threads/smsf-returns.25070/page-2
> 
> Craft posted that his 9 year CAGR for his super fund was 34.4%. Now admittedly the post was in 2013 but I am sure if you asked Craft for an update on his SMSF it would still be powering along.




Is that not clear enough evidence for you? Do you need to see the accounts for Crafts SMSF before you believe it?

As for making 25%+ returns every single year no fundamental investor in the world will achieve that over the long term because returns from fundamental investing are lumpy and inconsistent by their nature. But there are definitely fundamental investors who can get a CAGR north of 25% over a long period of time as I proved above.


----------



## Value Hunter

For what its worth here was Buffetts take on algo/quant strategies. 

“In my opinion, investment success will not be produced by arcane formulas, computer programs or signals flashed by the price behaviour of stocks and markets,” that is what he wrote in his 1987 annual letter.


----------



## CanOz

Didn't buffet admit to missing the tech run altogether because he didn't know how to value those businesses?


----------



## Trembling Hand

Value Hunter said:


> For what its worth
> 
> here  in his 1987 annual letter.






Absolutely F'n nothin I would say.


----------



## Boggo

Value Hunter said:


> For what its worth here was Buffetts take on algo/quant strategies.
> 
> “In my opinion, investment success will not be produced by arcane formulas, computer programs or signals flashed by the price behaviour of stocks and markets,” that is what he wrote in his 1987 annual letter.




Nope, his idea instead was to buy $1 billion worth or 6.2% of Coca Cola and get himself on the board and a few years later get his son Howard on there too.
I can see all the fundies on here replicating that sort of influential impact on their holdings


----------



## Quant

Forums pffft who  needs em really , the progressive thinkers that might contribute here are drowned out by opinionated dinosaurs . This whole FA v TA v Quant thing isn't an issue , they ALL work to a degree and a hybrid of all is the grail  , so many crippled by closed minds . Its not the tools you use , its the skillset you bring to the game , never forget that . Although sharp tools obviously help ...  Words are wasted in these places much of the time , people are more interested in cutting down new and exciting theories/methods . its easier than learning something new . The political/general threads are a prime example of the lack of critical thinking in this world . Easier to be in a " team " than innovate  .  I'm glad what I do is difficult , sheep are easy to slaughter  .. 5% of the people make 99% of the money ( something like that )  BE the 5% !!!!!!  sharpen your tools / make new tools / be open to new , innovative tools


There I've said my piece !

If you are not progressive your dead


----------



## Quant

CanOz said:


> Not the most relevant analogy....python is free, amibroker is at least $250.00 US



Its not the cost of the software , its the time learning to drive .. as you know 250 bucks is NOTHING in the big picture  . The ability to code competently  is worth 100's of thousands of dollars  ..


----------



## Sir Burr

Howard, how far out do you think these tools fade?
ie: a 1hr, daily or weekly long term system.


----------



## Wysiwyg

Quant said:


> Its not the cost of the software , its the time learning to drive .. as you know 250 bucks is NOTHING in the big picture  . The ability to code competently  is worth 100's of thousands of dollars  ..



The ability to code competently for 100"s of thousand dollars would mean the 'coder' is best of 100's to 1000's of applicants. Then coding a winning trading strategy and eating the retailers lunch (thread context) would be the ultimate goal. Computer language = money tree? The market balances these things out.


----------



## Quant

Bit like Mr Bandy I give up . Coding is the biggest money on the planet , end of story . 4 largest cap publically traded companies on planet . 2 100% code driven  and the other 2 not far of it , what else needs to be said  . Give it a short amount of time and it will be 5 with FB closing in on top 5  .... Coding has no future apparently  , well if you listen to dinosaurs  ... pffffft







"  If your not progressive your dead "


----------



## Wysiwyg

Quant said:


> This whole FA v TA v Quant thing isn't an issue , they ALL work to a degree and a hybrid of all is the grail  , so many crippled by closed minds .



Great encouragement to jump on the tradmill and grind out hours of exploration and research with traditional tools/indicators. Nothing is new in this thread.  


> Its not the tools you use , its the skillset you bring to the game , never forget that . Although sharp tools obviously help ...  Words are wasted in these places much of the time , people are more interested in cutting down *new and exciting theories/methods*



 Myself and it appears others have simply asked for evidence/examples/results from the thread start yet to use the common reply, we are missing the point, what a joke.  

You got something eating retailers lunch or still on the treadmill?


----------



## DaveDaGr8

Wysiwyg said:


> Great encouragement to jump on the tradmill and grind out hours of exploration and research with traditional tools/indicators. Nothing is new in this thread.
> Myself and it appears others have simply asked for evidence/examples/results from the thread start yet to use the common reply, we are missing the point, what a joke.
> 
> You got something eating retailers lunch or still on the treadmill?




You ARE missing the point !!!

Spend the time, run the math, build the system, test it, refine it, learn it then come to your own conclusions. Howard has laid the foundation that you can follow in Python if you choose. He has written books with the code already there for you to plunder. 

If that's too hard take the easy route, google and Youtube, i just did both and got a bucketload of hits with code already there. I know for a fact that Github has the source for many ML trading applications based on Pythons scikit. You actually don't need to program anything, just put in some effort to investigate ( aka copy and paste ) and draw your own conclusions.

or don't, It's entirely up to you.

BUT ... Nobody is obliged to give you evidence/examples/results.


----------



## Wysiwyg

DaveDaGr8 said:


> You ARE missing the point !!!
> 
> Spend the time, run the math, build the system, test it, refine it, learn it then come to your own conclusions.



I tested systems from 2009 to 2015 so I am off the much encouraged treadmill now. Thanks for the advice. 
I was enquiring about anything I may not know about.


----------



## DaveDaGr8

Wysiwyg said:


> I tested systems from 2009 to 2015 so I am off the much encouraged treadmill now. Thanks for the advice.
> I was enquiring about anything I may not know about.




I take it you didn't find what you were looking for then ?


----------



## Quant

DaveDaGr8 said:


> BUT ... Nobody is obliged to give you evidence/examples/results.



Its ironic , people act like Rsoles and then demand proof . I'd be willing to share some evidence but never under duress/demand  .  I took 5 years to get here  , I aint handing it out like candy to help convince the unconvincable  ...  the curve fitting/datamining accusations id have to fight of , the 2 and froing to actually try and help someone while getting sand kicked in face with next to zero chance of any recognition even if I proved my point  . ASF isn't a very progressive place unfortunately . WHY would anyone do that  , the only person I have to convince anything works is me . Howard has tried and he does have a slight agenda being an educator but he isn't selling snake oil  , He is a very very intelligent man and the main reason I ever came here and id suggest he may have been driven away ..  go figure , the dinosaurs know better and yield big sticks    WHY


----------



## Wysiwyg

DaveDaGr8 said:


> I take it you didn't find what you were looking for then ?



No but I did come to a better understanding of the market place including company news effects, other companies buying/selling, sentiment, zones of interest, precise support/resistance, deliberate price spikes up or down, fading/exiting on such spikes, going with the trend, consolidation patterns (and the time in such), position sizing, patience and fast exits when necessary.


----------



## Gringotts Bank

Quant said:


> 1--- try and help someone while getting sand kicked in face with next to zero chance of any recognition even if I proved my point  .
> 2---the only person I have to convince anything works is me .




1--- A lot of people do that here.  It is hard though, when you think about what we're trying to do - compete for trading profits. I'm quite confident you wouldn't share anything that can be used profitably by someone posting on a public forum, and nor would I.  Howard's not going to be any different.  A common tactic is to scoff and mock anything one thinks might be useful, but where the OP hasn't given enough details for the attacker to profit from it.  The OP then feels obliged to back up his argument and share more, because if he doesn't he appears to have backed down from the argument and made to look silly.  So the tactic does have some power, but it won't work on those who are above the need for recognition.  Forums have limited usefulness if competition is the dominant meme.  Are we really competing against other traders, or just our own minds?

2--- you know that's not true.  Your first statement contradicts it.


----------



## tech/a

GB

Open your eyes

You've learned nothing from Petes stuff?
Over 100 charts in charts of interest that have fleshed out some rippers.
plenty for a discretionary trader.

if you can't learn from the above you've got no hope with Howard's stuff.
Your not his target market anyway.


----------



## Wysiwyg

Gringotts Bank said:


> I'm quite confident you wouldn't share anything that can be used profitably by someone posting on a public forum, and nor would I.




Duck made the good point there are short term and long term trading guides on ASF. Peter2 s the only poster I have seen with reguarly 'detailed and reasoned' trade activity posted. When he is buying and selling posted in advance. The time he has given to help others practically is remarkable and rare.


----------



## Wysiwyg

Gringotts Bank said:


> Are we really competing against other traders, or just our own minds?



You would have run thousands of strategies in Amibrioker so I assume you came to the conclusion that indicator based strategies are generally 50/50 over time and everything works sometimes? Then there are the biases and assumptions of getting price and quantity and other gotcha's. It's a treadmill with the "profitable trading system" carrot dangling up front. What people need to know is the market has countless nuances which leaves all participants at risk at all times. People don't "know", they play higher probabilities based on knowledge/expereince and set risk as desired. 
When I trade there is price movement and how I plan to react. Nothing else.


----------



## Newt

Yes, but after 5 years of so running Amibroker backtests, people would learn most trend following strategies deliver just under 50% win/loss but can still pull a profit over the medium term with proper position sizing, possibly a market filter.  e.g.  Simple 100 day BO strategy still worked over last 5 years last time I checked - about 17% compound on ASX.  

I wouldn't for one minute pretend that the edge is a anything other than a pretty fine line, but it is there.  Most people seem to get caught up in "being right" for a high % of the time, when its more like having a casino-like edge where you have to be playing very consistently over a long time to make money (the casino that is, not the players  )


----------



## Gringotts Bank

Wysiwyg said:


> You would have run thousands of strategies in Amibrioker so I assume you came to the conclusion that indicator based strategies are generally 50/50 over time and everything works sometimes?




I have some systems that definitely work.  Making enough to live off is another matter.

I like your Epictetus quote, but I'd change it slightly.


----------



## tech/a

Gringotts Bank said:


> I have some systems that definitely work.  Making enough to live off is another matter.
> 
> I like your Epictetus quote, but I'd change it slightly.




I would suggest it's more a case of having enough capital to
To make a living from.


----------



## Value Hunter

I am not saying that quant strategies do not work or that Howards ideas do not work. But as a fundamental investor it bothers me when quants attack fundamental investing when decades of evidence and proven results (and practitioners on this forum) show that it works. In particular Howard's baseless assertion earlier in the thread where he claimed that fundamental investing does not work. 

To Quote Howard "It is my opinion that market efficiency limits the use of fundamental information and that it is of near-zero value to individuals other than those with a seat in the boardroom and their best friends." Can Howard provide any proof or evidence whatsoever for his baseless and false assertion?

He provided no evidence of this claim and when I called him out on it I got silence in response.


----------



## Indoril

Value Hunter said:


> I am not saying that quant strategies do not work or that Howards ideas do not work. But as a fundamental investor it bothers me when quants attack fundamental investing when decades of evidence and proven results (and practitioners on this forum) show that it works. In particular Howard's baseless assertion earlier in the thread where he claimed that fundamental investing does not work.
> 
> To Quote Howard "It is my opinion that market efficiency limits the use of fundamental information and that it is of near-zero value to individuals other than those with a seat in the boardroom and their best friends." Can Howard provide any proof or evidence whatsoever for his baseless and false assertion?
> 
> He provided no evidence of this claim and when I called him out on it I got silence in response.




Hate to nitpick here but he did say it was his opinion. He didn't claim it as a fact.


----------



## Sir Burr

Newt said:


> Simple 100 day BO strategy still worked over last 5 years last time I checked - about 17% compound on ASX.




Quoting Howard, the days of long term holding, and simple trading algorithms are over.

I'm simple and can't understand why, maybe 17% is unacceptable with these new tools? What happens when everyone is using these new tools?


----------



## Trembling Hand

Sir Burr said:


> Quoting Howard, the days of long term holding, and simple trading algorithms are over.



Come on guys. You are all just being trolled. This thread was never about information or explanation of any method. If there was any intent to show the possibility within these pages of the usefulness of ML the OP would have shown it. But all that was ever given as input were inflammatory claims about other peoples approach and reference numbers to pages in his books that you have to purchase. You don't need a complex Machine Learning algo to draw the correct conclusion of expected outcome from those two inputs..


----------



## Sir Burr

Trembling Hand said:


> his books that you have to purchase. You don't need




Haha thanks, that's a relief!


----------



## Gringotts Bank

When people have their beliefs challenged, it's always aggressive.  

Give me all your intellectual property for free or I'll say it's bullsh1t.


----------



## Trembling Hand

Gringotts Bank said:


> When people have their beliefs challenged, it's always aggressive.
> 
> Give me all your intellectual property for free or I'll say it's bullsh1t.



Nah its not that at all. I'm actually using Python to find trades. Its not ML but I have spent a bit of time going down that rabbit hole too. But you cannot expect to make extraordinary claims without having extraordinary proof. Howard failed to offer anything close to that to further the discussion. He cannot rely on an appeal to authority as proof of an extraordinary claim. And along the way managed to kick sand in the face of everyone else who didn't follow his process. 

Hardly a helpful thread.


----------



## tech/a

Perhaps You could ask Stephen Simmon?


----------



## DaveDaGr8

Trembling Hand said:


> Nah its not that at all. I'm actually using Python to find trades. Its not ML but I have spent a bit of time going down that rabbit hole too. But you cannot expect to make extraordinary claims without having extraordinary proof. Howard failed to offer anything close to that to further the discussion. He cannot rely on an appeal to authority as proof of an extraordinary claim. And along the way managed to kick sand in the face of everyone else who didn't follow his process.
> 
> Hardly a helpful thread.




So make it helpful. You stated you went down that rabbit hole, what did you find ???


----------



## brajeevn

To Quote Howard "It is my opinion that market efficiency limits the use of fundamental information and that it is of near-zero value to individuals other than those with a seat in the boardroom and their best friends." Can Howard provide any proof or evidence whatsoever for his baseless and false assertion?
...In long term holding period (may be useful due to stock split/dividend etc..) is profitable. But Howard advocate for short term holding period due to RISK involved.  In that sense, I think his opinion is right especially if huge money is invested. Holding a stock for short term period do not need to consider fundamental value (Price action have no logic to define or predict). My trading success solely based on Howard's methods..... and always admire him for his contributions..


----------



## tech/a

What constitutes  " Help " 
What constitutes " Proof "

What exactly do you want to see?


----------



## Trembling Hand

DaveDaGr8 said:


> So make it helpful. You stated you went down that rabbit hole, what did you find ???



Actually, I was going to but was expecting Howard to follow through with something better than wild claims about us all being on the scrap heap and assumed his stuff would be far better than what I have been able to achieve. And had tried to move the thread on a few time asking for examples from Howard not because I wanted to steal his IP but it would have been a catalyst to move away from the Quant v TA V Fundie Bore feast. 

Maybe I'll have a go in another thread at another time.


----------



## Indoril

https://arxiv.org/abs/1208.4429
A paper on the common mistakes of machine learning in the markets. I haven't read it yet, I'm on mobile right now, but thought others might be interested. Its not a long read.


----------



## captain black

Indoril said:


> https://arxiv.org/abs/1208.4429
> A paper on the common mistakes of machine learning in the markets. I haven't read it yet, I'm on mobile right now, but thought others might be interested. Its not a long read.




Thanks for posting this @Indoril, always something new to learn


----------



## Gringotts Bank

https://www.wired.com/2016/01/the-rise-of-the-artificially-intelligent-hedge-fund/

"In the simplest terms, this means it creates a large and random collection of digital stock traders and tests their performance on historical stock data. After picking the best performers, it then uses their "genes" to create a new set of superior traders".


----------



## traderxxx

after reading these posts, here I am still sitting here with hand drawn charts and a calculator,
of course have a data supplier and charts on screen but wouldn't be without
my hand drawn charts.   never heard of excel until 6 months ago and have recently
learnt how to change cell colours, ( imm not kidding)
I also know I can protect a sheet using the protect/unprotect  button

but if someone could tell me how to unprotect certain cells on a protected sheet
that would be good.


----------



## captain black

traderxxx said:


> but if someone could tell me how to unprotect certain cells on a protected sheet
> that would be good.




A couple of links that may be of help:

https://www.technipages.com/excel-lock-unlock-cells

https://support.office.com/en-us/ar...orksheet-75481b72-db8a-4267-8c43-042a5f2cd93a


----------



## traderxxx

thanks for the links captain black,
all done and and worked out ok.


----------

