- Joined
- 4 February 2015
- Posts
- 10
- Reactions
- 0
Wow, that's a real treasure-trove of papers! Thanks for the link.
But, I don't really see any there that relate to crowd-sourcing predictions?
An Experiment: can crowds predict market movements?
- Thread starter BrianEastlake
- Start date
Wow, that's a real treasure-trove of papers! Thanks for the link.
But, I don't really see any there that relate to crowd-sourcing predictions?
- Joined
- 4 February 2015
- Posts
- 10
- Reactions
- 0
Thanks for the tip. It looks interesting, I'll probably have to dedicate a weekend to studying it properly. I'll also pass it on to Will - he'll have a better shot at understanding the maths in the paper.
It seems that these researchers have arrived at the conclusion that future price action can be predicted (in certain scenarios at least).
Cheers
-Brian
DeepState
Multi-Strategy, Quant and Fundamental
- Joined
- 30 March 2014
- Posts
- 1,615
- Reactions
- 81
Hi Brian
My last entry on this thread.
Key Points:
+ In something like financial markets, the participants as a whole cannot sustainably, meaningfully, outforecast their own subsequent actions if there exists motivation to profit from this forecast. In capital markets, that motivation is apparent.
+ It follows that a representative sample of this crowd will not do so either. Creating independence of the type that Surowiecki requires in an endogenous system sees it collapse on itself as a forecasting method. It requires a crowd of forecasters to stand completely apart from the crowd being forecast. That’s not practically feasible when they look at the same things and think/act in similar ways if the info is useful and there is logic to its use. They also live in the same political economy. If these points do not matter, there is no forecasting power anyway.
+ A crowd in which some are truly insightful and are better than others on average at forecasting is not any better at forecasting its overall actions for the presence of a few with insight. It does mean that those with insight will profit more than those without. Some degree of heterogeneity in ability is required for the healthy function of capital markets. It is true that, if you can identity those with insight, a survey of a crowd of those talented few will likely produce a more stable outcome than any particular individual within it chosen at random. The critical issue is to survey those only. Not the whole crowd or a representative sample of it.
+ There is motivation to game the preference provided by participants which you will not be able to reliably uncover. You will not be able to rely on the kindness of strangers, even if they are just curious.
+ The only type of crowd-based survey that matters for prediction of a type that aims to make money directly is one which surveys the current thoughts of those with real insight before it is known to the rest of the crowd. An historical example is given.
+ Will is wrong. He should pay-up now or leave the market…in which case he should pay-up now because that action demonstrates why he is wrong so long as a single person in the market cares about making money. I feel comfortable with that assumption. He may argue otherwise. If he protests, who cares, his thoughts about the market are random and can be disregarded as noise allowing others to find advantage for his absence of effort…..Hi Will.
+ An experimental statistical outline is suggested. The behavior of the EURUSD is such that, for short term forecasts like that suggested, a reasonable bogey is simply a forecast that next week’s exchange rate is today’s. If you get significant results, you’ll need to explain how a smart crowd managed to outsmart itself and intends to keep doing so when it is motivated to ensure that this does not occur. This will require the kind of arithmetic gymnastics that would be needed in explaining how a table of poker players managed to outguess itself as a whole.
------
In reality, you need at least some of this behavior to keep capital markets functioning, but having a lot of it doesn’t mean that the whole is more predictive. If you need a reference on this, try Grossman-Stiglitz.
As a result, the only survey that makes sense for making super-normal returns is one where the truly insightful ones are surveyed and the result compared with the rest of the crowd (whose outcome is sufficiently expressed in the prevailing price so as to make no practical differerence for a broad market like EURUSD). Surveying the crowd as a whole, which can be comprised of very smart people on average, but not all of whom can be above average, will yield nothing more than what the market is showing for the most part.
Except for pocket solutions for smaller markets than the type we are talking about, surveying the crowd will get nowhere sustainably. You need to survey the insightful. The rest, forget about it. It’s already visible in the price.
-----
http://www.bloomberg.com/news/artic...agrees-with-n-y-to-end-analyst-survey-program
You will note that people who work extremely hard to become influential in the financial market and know a lot about it tend to work for money, even if they are curious about it. They seek to monetize their views. That said, central bankers and policy officials are not particularly well paid and, yet, are influential. However, if they were to release information to your survey before it becomes public and are found to do so, they will be fired. If you knew you were receiving information from them and used it, you would also be imprisoned if discovered. If you were offering kickbacks to them for participation beyond curiosity, they would also be imprisoned. If I used the survey outcome knowing what you did, I would be imprisoned. I hope that’s not your underlying purpose here…
As for the rest of the possibilities: those who have no particular influence because their opinion is not known or cared for by others with money in the market ahead of the survey release; are not creating policy directly that influences outcomes; but yet are very insightful (and not malicious voters); are also keen to share those insights to an essentially private survey before otherwise releasing it elsewhere for curiosity; and for these outcomes not to be used in any way that influences the market and allows it to adapt….leads me to wonder about this element of your experimental design and how incentives for participation like platonic curiosity of the type you need can be expected to produce a sustainably useful outcome. It would certainly not be a representative sample of the crowd or market participants which is the underlying proposition about smart crowds and prediction in endogenous settings.
Some reading on game theory might be called for. I was only half-kidding (less, actually) when I talked about foxing in my prior sign-off.
If you try to force this survey design to meet the Surowiecki requirments, this design is the natural end point. All participants will be independent, somehow motivated, but none insightful (at least as referenced by usual powers of deduction) beyond individual idiosyncratic data not available to any other. A crowd standing fully independently of another crowd whose actions they are trying to estimate. A group of curious people in separate, (figurative) sub-terrainian chambers, estimating what the latest fashion is on the surface of the planet. Trying hard to emulate a situation like counting jellybeans from outside the jar or weighing oxen. Except this is not like that. Is this design issue not evident?
You will still get a probability distribution just for asking, as I would get when asking my dice via rolling them. It just won’t bear much resemblance to anything that might be useful beyond curiosity.
Nonetheless, one scenario is that Will demands evidence that it is not fully random. He can claim that a P-value of 0.00001 still left room for doubt or find some wrinkle as to why the survey was flawed and ask for another one with some new feature. You can counter with a request to provide evidence that it is random and show evidence which rejects complete uncertainty to high confidence but is not absolute. If you are both motivated from curiosity, please start a thread and attempt to claim this grey space between the two viewpoints! You will be inundated….with those saying it is not random (because ASF is a biased sample by headcount…something for you to consider in your experimental design from the outset). A crowd survey by straw poll, drawing on their collective wisdom, will slay Will’s hypothesis. That’s unless, knowing we are running an experiment on crowds, they try to spite us and vote in favour of Will out of mirth and curiosity (for there is no material monetary incentive on this outcome)…which will it be? …
In any case, the EURUSD weekly return as measured by log adjusted returns over the last 15 years (pretty much to the birth of the Euro although pre-dating the minting of coins etc.) suggests that there is no short term momentum in there at the basic level although someone out there will torture the heck out of it with some algo with an acronym and publish a statistically significant result, disavowing the other hundred they tried. Hence, for the most part (though not strictly speaking), the current level of the EURUSD now is pretty much the best estimate of where it will be in a week from now for the crowd as a whole given what it knows in aggregate (this is not true over longer periods). That can serve as your null hypothesis.
To be useful for money making and to obtain the upside kind of curiosity, the sample crowd outcome will need to be statistically better than the bogey above and also statistically positive in a net of t-cost P&L sense. That’s at least two hurdles. For implementation, experienced traders would consider a lot more before making something like this operational. Still, let’s start there.
If you manage to obtain some result which surpasses an arbitrary hurdle like a 5% confidence interval (note P&L will almost certainly be non-normal and require a different stat test than t-stat or anything based on student’s t or normal distribution assumptions. You are moving into non-parametric stuff here), you will still need to overcome an explanation of why the crowd managed to outsmart itself and kept on doing it.
Importantly, you will need to explain why a crowd which can outsmart itself intends to keep doing so and yet be serious enough to continue to regard itself as smart! Capital markets are driven by money and it does not seem like a good experimental assumption that those who reveal their preference out of curiosity, do so without malice, do not otherwise participate in the market, are independent in data, are so independent that they do not contact each other or share meaningful common links, yet are as smart or more predictive than the insightful ones with who have incentives which go beyond curiosity. That is a large set of compounding assumptions. It does not make it impossible. It falls just a little bit short of that.
A smart crowd in this setting learns until it cannot beat itself. Anything else is not smart or consistent with the presence of insight and the motivation to actually use it. A table of poker players does not beat itself even if all the players are just curious, play for toothpicks and possibly bragging rights, and don’t talk to each other. Neither can a group of curious people who cannot interact with the table or each other in any way. There is a whole world of HFT/Algo/Fundamental/Tech/Astrology/Séance working hard every day to make it hard for it to happen sustainably. Some of these efforts are outlined in Tech/A’s attachments. His do not specifically relate to crowd-sourced data. You might, however, be interested in this piece of fairly full-bore data-mining which is massively crowd sourced and goes beyond the use of simple, period ahead estimates, as single dot points. It is an effort at crowd-sourced artificial intelligence. 'Supervised learning':
![2015-02-10 18_24_58-[1402.1624] Using Twitter to Model the EUR_USD Exchange Rate - Internet Expl.jpg](https://aussiestockforums.b-cdn.net/data/attachments/54/54515-dcf215369a1522c51eb5facb4192345a.jpg "2015-02-10 18_24_58-[1402.1624] Using Twitter to Model the EUR_USD Exchange Rate - Internet Expl.jpg")
In contrast to surveying a crowd, a survey of those with real insight can and, routinely, does violate the Surowiecki requirements and yet reveal their superior wisdom in prediction. However, that is very different to the proposition that Surowiecki is talking about when he refers to wisdom of crowds as it relates to oxen. It is also a very different proposition to what is being presented here. What you want in this situation is the wisdom of the wisest in the crowd. Don’t bother with surveying the crowd as a whole, or a representative sub-sample consisting of the curious but otherwise uninterested/uninvolved. The hardest part is finding the most insightful/predictive part of the seething mass and motivating them in whatever way needed to give you their views ahead of general distribution. The next hardest is to use it without becoming incarcerated.
My last entry on this thread.
Key Points:
+ In something like financial markets, the participants as a whole cannot sustainably, meaningfully, outforecast their own subsequent actions if there exists motivation to profit from this forecast. In capital markets, that motivation is apparent.
+ It follows that a representative sample of this crowd will not do so either. Creating independence of the type that Surowiecki requires in an endogenous system sees it collapse on itself as a forecasting method. It requires a crowd of forecasters to stand completely apart from the crowd being forecast. That’s not practically feasible when they look at the same things and think/act in similar ways if the info is useful and there is logic to its use. They also live in the same political economy. If these points do not matter, there is no forecasting power anyway.
+ A crowd in which some are truly insightful and are better than others on average at forecasting is not any better at forecasting its overall actions for the presence of a few with insight. It does mean that those with insight will profit more than those without. Some degree of heterogeneity in ability is required for the healthy function of capital markets. It is true that, if you can identity those with insight, a survey of a crowd of those talented few will likely produce a more stable outcome than any particular individual within it chosen at random. The critical issue is to survey those only. Not the whole crowd or a representative sample of it.
+ There is motivation to game the preference provided by participants which you will not be able to reliably uncover. You will not be able to rely on the kindness of strangers, even if they are just curious.
+ The only type of crowd-based survey that matters for prediction of a type that aims to make money directly is one which surveys the current thoughts of those with real insight before it is known to the rest of the crowd. An historical example is given.
+ Will is wrong. He should pay-up now or leave the market…in which case he should pay-up now because that action demonstrates why he is wrong so long as a single person in the market cares about making money. I feel comfortable with that assumption. He may argue otherwise. If he protests, who cares, his thoughts about the market are random and can be disregarded as noise allowing others to find advantage for his absence of effort…..Hi Will.
+ An experimental statistical outline is suggested. The behavior of the EURUSD is such that, for short term forecasts like that suggested, a reasonable bogey is simply a forecast that next week’s exchange rate is today’s. If you get significant results, you’ll need to explain how a smart crowd managed to outsmart itself and intends to keep doing so when it is motivated to ensure that this does not occur. This will require the kind of arithmetic gymnastics that would be needed in explaining how a table of poker players managed to outguess itself as a whole.
------
The great bulk of this industry is about competition from within. If the industry as a whole were satisfied with just getting buy-hold returns, most of it would not exist. It exists because most people aren’t satisfied with that and want more. Overall, they will fail after frictions and fees are considered. On average, they will persist with that endeavor anyway. Welcome to human behavior. If you are interested in psychology of investing, this aspect bears a lot of worthy examination.
In reality, you need at least some of this behavior to keep capital markets functioning, but having a lot of it doesn’t mean that the whole is more predictive. If you need a reference on this, try Grossman-Stiglitz.
As a result, the only survey that makes sense for making super-normal returns is one where the truly insightful ones are surveyed and the result compared with the rest of the crowd (whose outcome is sufficiently expressed in the prevailing price so as to make no practical differerence for a broad market like EURUSD). Surveying the crowd as a whole, which can be comprised of very smart people on average, but not all of whom can be above average, will yield nothing more than what the market is showing for the most part.
The concept of regarding a crowd that learns off the smart ones as a smart predictor of its own behavior is erroneous. A crowd where elements are capable of some prediction (as must exist for a sustainably functioning capital market) consists of those who are insightful and those who are not. Insight is a relative concept in this world. Even the smart ones are surprised by what they find when they find it. The others are surprised when they find out a little later. On average, they are consistently being surprised. This is what makes aggregate survey outcomes non-predictive, but why, within a crowd, superior prediction leading to excess money-making can exist for a subset. Whilst individuals within this subset make predictions which are accurate, it is true that a wider survey of them will yield more stable outcomes….but these other participants also have to be insightful. The crowd as a whole cannot be more insightful than itself. That is why a survey of the whole population trying to predict itself is not a strong idea.
Except for pocket solutions for smaller markets than the type we are talking about, surveying the crowd will get nowhere sustainably. You need to survey the insightful. The rest, forget about it. It’s already visible in the price.
-----
If your crowd experiment doesn’t work, making it even more diverse to the point that you survey the entire populace of earth isn’t going to make it better. What will improve it is narrowing the survey to an extremely unrepresentative sample of those who truly know better than the crowd overall.
This is an extract from a report which talks about the kind of survey which actually works in real life. The idea was created by an Australian a long time ago although the news item refers to the US and a firm called BlackRock, currently the world’s largest money manager. The idea was so good that, in the end, the umpires effectively changed the rules or, alternatively, chose to enforce the rules differently. It is a survey of influential people in the crowd whose opinions are listened to, cared about, and thus influence the crowd. Know what they are thinking, and you have a good notion of what the wider crowd will be thinking.
http://www.bloomberg.com/news/artic...agrees-with-n-y-to-end-analyst-survey-program
You will note that people who work extremely hard to become influential in the financial market and know a lot about it tend to work for money, even if they are curious about it. They seek to monetize their views. That said, central bankers and policy officials are not particularly well paid and, yet, are influential. However, if they were to release information to your survey before it becomes public and are found to do so, they will be fired. If you knew you were receiving information from them and used it, you would also be imprisoned if discovered. If you were offering kickbacks to them for participation beyond curiosity, they would also be imprisoned. If I used the survey outcome knowing what you did, I would be imprisoned. I hope that’s not your underlying purpose here…
As for the rest of the possibilities: those who have no particular influence because their opinion is not known or cared for by others with money in the market ahead of the survey release; are not creating policy directly that influences outcomes; but yet are very insightful (and not malicious voters); are also keen to share those insights to an essentially private survey before otherwise releasing it elsewhere for curiosity; and for these outcomes not to be used in any way that influences the market and allows it to adapt….leads me to wonder about this element of your experimental design and how incentives for participation like platonic curiosity of the type you need can be expected to produce a sustainably useful outcome. It would certainly not be a representative sample of the crowd or market participants which is the underlying proposition about smart crowds and prediction in endogenous settings.
Some reading on game theory might be called for. I was only half-kidding (less, actually) when I talked about foxing in my prior sign-off.
They don’t have to discuss it. All they need to do is read the news and look at economic variables and also know the price series. It is through these issues that forecasters lose independence when the system is endogenous. This is not like being able to estimate the dimensions of an ox and having a sense of the mass per unit volume. It is the influence of these common variables that help cause the madness of crowds. I do not personally know the guy from Barclays Fixed Income Trading in the Square Mile, but I am reading some of the same stuff. If this stuff is useful for prediction, there will be correlation amongst those with predictive ability. This violates Surowiecki. If it doesn’t the experiment fails before it starts anyway and any positive statistical outcome which might arise is just an outlier.
In order to meet Surowiecki’s requirements within an endogenous system, you will essentially need to create an exogenous system within an otherwise endogenous one. This is pretty much the only way to reconcile them. Not only can they not talk to each other or even refer to past survey results, but they can’t even look at the price or have any information that might coordinate their estimates. Once again, the wisdom of crowds concept does not apply to financial markets unless it is an insightful subset that you are managing to survey. Integrity about not discussing their predictions with anyone doesn’t even matter except in fairly extreme situations like publishing their views as editor of the Financial Times. Survey participants can be mute, unable to type a word (but can type their estimate for next week’s EURUSD level or otherwise communicate it), not visible to one another, deaf and pious. If they use the same info, they will correlate unless that info is useless and/or they all don’t know anything about the info available and apply it randomly.But, probability estimates are exactly what we want, and I don't think we have to go to quite those extremes to get them.
If you try to force this survey design to meet the Surowiecki requirments, this design is the natural end point. All participants will be independent, somehow motivated, but none insightful (at least as referenced by usual powers of deduction) beyond individual idiosyncratic data not available to any other. A crowd standing fully independently of another crowd whose actions they are trying to estimate. A group of curious people in separate, (figurative) sub-terrainian chambers, estimating what the latest fashion is on the surface of the planet. Trying hard to emulate a situation like counting jellybeans from outside the jar or weighing oxen. Except this is not like that. Is this design issue not evident?
You will still get a probability distribution just for asking, as I would get when asking my dice via rolling them. It just won’t bear much resemblance to anything that might be useful beyond curiosity.
For a market to exist, price action cannot be fully random all the time. Grossman-Stiglitz explains this if not immediately evident. Price action exists in a strange and unstable space of near-randomness. Will can concede anytime he wants and pay up in any manner that you have pre-agreed. If he doesn’t, it doesn’t matter because he can sit out in that belief (for to participate is worse than gambling) and really ceases to have any relevance to the market. By doing so, he reduces its randomness as if fait accompli. If it were otherwise, there is no incentive for price discovery and there would be no capital market. A certain order exists in the markets which would not if it were fully uncertain. It’s not even vaguely close to that.
Nonetheless, one scenario is that Will demands evidence that it is not fully random. He can claim that a P-value of 0.00001 still left room for doubt or find some wrinkle as to why the survey was flawed and ask for another one with some new feature. You can counter with a request to provide evidence that it is random and show evidence which rejects complete uncertainty to high confidence but is not absolute. If you are both motivated from curiosity, please start a thread and attempt to claim this grey space between the two viewpoints! You will be inundated….with those saying it is not random (because ASF is a biased sample by headcount…something for you to consider in your experimental design from the outset). A crowd survey by straw poll, drawing on their collective wisdom, will slay Will’s hypothesis. That’s unless, knowing we are running an experiment on crowds, they try to spite us and vote in favour of Will out of mirth and curiosity (for there is no material monetary incentive on this outcome)…which will it be? …
In any case, the EURUSD weekly return as measured by log adjusted returns over the last 15 years (pretty much to the birth of the Euro although pre-dating the minting of coins etc.) suggests that there is no short term momentum in there at the basic level although someone out there will torture the heck out of it with some algo with an acronym and publish a statistically significant result, disavowing the other hundred they tried. Hence, for the most part (though not strictly speaking), the current level of the EURUSD now is pretty much the best estimate of where it will be in a week from now for the crowd as a whole given what it knows in aggregate (this is not true over longer periods). That can serve as your null hypothesis.
To be useful for money making and to obtain the upside kind of curiosity, the sample crowd outcome will need to be statistically better than the bogey above and also statistically positive in a net of t-cost P&L sense. That’s at least two hurdles. For implementation, experienced traders would consider a lot more before making something like this operational. Still, let’s start there.
If you manage to obtain some result which surpasses an arbitrary hurdle like a 5% confidence interval (note P&L will almost certainly be non-normal and require a different stat test than t-stat or anything based on student’s t or normal distribution assumptions. You are moving into non-parametric stuff here), you will still need to overcome an explanation of why the crowd managed to outsmart itself and kept on doing it.
Importantly, you will need to explain why a crowd which can outsmart itself intends to keep doing so and yet be serious enough to continue to regard itself as smart! Capital markets are driven by money and it does not seem like a good experimental assumption that those who reveal their preference out of curiosity, do so without malice, do not otherwise participate in the market, are independent in data, are so independent that they do not contact each other or share meaningful common links, yet are as smart or more predictive than the insightful ones with who have incentives which go beyond curiosity. That is a large set of compounding assumptions. It does not make it impossible. It falls just a little bit short of that.
A smart crowd in this setting learns until it cannot beat itself. Anything else is not smart or consistent with the presence of insight and the motivation to actually use it. A table of poker players does not beat itself even if all the players are just curious, play for toothpicks and possibly bragging rights, and don’t talk to each other. Neither can a group of curious people who cannot interact with the table or each other in any way. There is a whole world of HFT/Algo/Fundamental/Tech/Astrology/Séance working hard every day to make it hard for it to happen sustainably. Some of these efforts are outlined in Tech/A’s attachments. His do not specifically relate to crowd-sourced data. You might, however, be interested in this piece of fairly full-bore data-mining which is massively crowd sourced and goes beyond the use of simple, period ahead estimates, as single dot points. It is an effort at crowd-sourced artificial intelligence. 'Supervised learning':
In contrast to surveying a crowd, a survey of those with real insight can and, routinely, does violate the Surowiecki requirements and yet reveal their superior wisdom in prediction. However, that is very different to the proposition that Surowiecki is talking about when he refers to wisdom of crowds as it relates to oxen. It is also a very different proposition to what is being presented here. What you want in this situation is the wisdom of the wisest in the crowd. Don’t bother with surveying the crowd as a whole, or a representative sub-sample consisting of the curious but otherwise uninterested/uninvolved. The hardest part is finding the most insightful/predictive part of the seething mass and motivating them in whatever way needed to give you their views ahead of general distribution. The next hardest is to use it without becoming incarcerated.
I’ll sign off here and look forward to how this rolls out if you choose to pursue it and report it. Before discovery, curiosity usually featured somewhere. Enjoy your journey.
Similar threads
