be partially both. But it is important to try to differentiate. A risk premium can be expected to persist: the counterparty is paying for insurance against a risk. They may improve their pricing of the insurance, but they will probably continue to pay something.
By contrast, an inefficiency will last only until other people notice it. And failing to differentiate between a real opportunity and a chance event will only lead to losses.
Some traders will profit from inefficiencies. Not all traders will. A lot of traders will use meaningless or widely known information. Many forecasts are easy. I can predict the days the non-farm payroll will be released. I can predict what days fall on weekends. I can predict the stock market closes at 4 p.m. eastern time. In many cases, making a good prediction is the easy part. The hard part is that the forecast has to be better than the market's, which the consensus of everyone else's prediction is. For developed stock indices the correlation between the daily range on one day and the next is roughly between 65% and 70%. So a very good volatility estimator is that it will be what it was the day before (a few more insights like this will lead you to GARCH). It is both hard and profitable to make an even slightly better one-day forecast. And whether it is because the techniques that are used are published, employees leave and take information with them, or just that several people have a similar idea at the same time, these forecast edges don't last forever.
Aside: Alpha Decay
The extinction of floor traders is an example of a structural shift in markets destroying a job. Similar to most people, traders tend to think that their skills are special, and their jobs will always be around. This isn't true. The floors have gone. Fixed commissions have gone. Investment advisors are being replaced by robo-advisors. There are fewer option market-makers, each trading many more stocks than in the past. Offshoring will definitely come to trading, and it is quite possible that a market structure such as a once-a-day auction could replace continuous trading.
But as well as these structural changes, the alpha derived from market inefficiencies (as opposed to the beta of exposure to a mispriced risk factor) doesn't last forever. Depending on how easy it is to trade the effect, the half-life of an inefficiency-based strategy seems to be between 6 months and 5 years. Mclean and Pontiff (2016) showed that the publication of a new anomaly lessens its returns by up to 58%. And publication isn't the only thing that erodes alpha. Chordia et al. (2014) showed that increasing liquidity also reduces excess returns by about 50%. Sometimes the anomaly exists only because it isn't worth the time of large traders to get involved. A similar effect is that the easy access to data will kill strategies. Sometimes the alpha isn't due to a wrinkle in the financial market. It is due to the costs of processing information.
Just as some traders will profit by using a stupid idea like candlestick charting, some traders will succeed for a while with an overfit model. I'm in no way using this to condone data-mining, but we can learn a valid lesson from this. As Guns and Roses pointed out, “nothing lasts forever.” Lucky strategies will never last but even the best, completely valid strategy will have a lifetime. So, when you are making money don't think that being “prudent” is a good idea. The right thing to do is to be as aggressive as possible. Amateurs go broke for a lot of reasons, but professionals often suffer in bad times because they didn't fully capitalize on good times, instead thinking that making steady but small profits was the best thing to do.
They also spend too much in good times, forgetting that they won't last. I've had a floor trader tell me about his new Ferrari about an hour before laughing about the stupid spending habits of NFL and NBA players (the last I heard he was selling houses). Many times, traders have short careers because a valid strategy dies. Amateurs blow up, but professionals don't allow for alpha-decay. For example, many floor traders didn't survive the death of the open-outcry pits. Their edge disappeared, and their previous spending habits left them with little. (In this case “trickle-down” economics was correct, as profits from market-making trickled down to prostitutes, strippers, and cocaine dealers. At least it wasn't wasted.)
Behavioral Finance
Think about how stupid the average person is, then realize half of them are stupider than that.
—George Carlin
The history of markets is nowhere near as big as we often assume. For example, equity options have only been traded in liquid, transparent markets since the CBOE opened in 1973. S&P 500 futures and options have only been traded since 1982. The VIX didn't exist until 1990 and wasn't tradable until 2004. And the average lifetime of an S&P 500 company is only about 20 years. In the long term, values are related to macro variables such as inflation, monetary policy, commodity prices, interest rates, and earnings. And these change on the order of months and years. Even worse, they are all co-dependent.
So, what might seem like a decent length of history that we can study and look for patterns, quite possibly isn't (this does not apply to HFT or market-making where a huge number of data points can be collected in what is essentially a stationary environment). When it comes to volatility markets, I think that although there appear to be many thousands of data points, there might only be dozens. A better way to think of market data might be that we are seeing a small number of data points, and that they occur a lot of times.
I think this makes quantitative analysis of historical data much less useful than is commonly thought.
But there is something that has been constant: human nature.
Humans have been essentially psychologically unchanged for 300,000 years when Homo sapiens (us) first appeared. This means that any effect that can conclusively be attributed to psychology will effectively have 300,000 years of evidence behind it. This seems to be potentially a much better source for gaining clarity.
The problem with psychological explanations (for anything) is that they are incredibly easy to postulate. As the baseball writer Bill James was reported to say, “Twentieth-century man uses psychology exactly like his ancestors used witchcraft; anything you don't understand, it's psychology.” The finance media is always using this kind of pop psychology to justify what happened that day. “Traders are exuberant” when the market goes up a lot; “Traders are cautiously optimistic” when it goes up a little, and so on. I try not to do this, but I'm as guilty as anyone else. I think psychology could be incredibly helpful, but we have to be very careful in applying it. Ideally, we want several psychological biases pointing to one tradeable anomaly, and we want them to have been tested on a very similar situation to the one we intend to trade.
Further, traders aren't psychologists and reading behavioral finance at any level from pop psychology to real scientific journals is probably just going to lead to hunches and guesses. To be fair, traders currently make the same mistakes from reading articles about geopolitics or economics. One week, traders will be experts on the effects of tariffs on soybeans and the next week they will be talking about Turkish interest rates. It is far easier to sound knowledgeable than to actually be so. It isn't obvious that badly applied behavioral psychology is any more useful than badly applied macroeconomics. And it is obvious that traders can't do better than misapply, either.
After I explained this nihilistic view to an ex-employer he said, “Well, I have to do something.” And what we do is exactly what I've said isn't very good: we apply statistics and behavioral finance. These are far from perfect tools, but they are the best we have. The edges they give will be small, but some edges can be found. We will always know only a small part of what can be known. Making money is hard.
Proponents of behavioral finance contend that various psychological biases cause investors to systematically make mistakes that lead to market inefficiencies. Behavioral psychology was first applied to finance in the 1980s, but for decades before that psychologists were studying the ways people actually made decisions under uncertainty.
