reviewed with an emphasis on regularization methods, bias-variance and other tradeoffs, generalization error, the curse of dimensionality, and traps of overfitting. ML involves a wealth of parametric, nonparametric, deep, online, and latent structure algorithms, whose success is data-dependent according to the “No free lunch” theorem. Meta-learning methods include hyperparameter optimization, boosting, and other ensemble methods. An important context of financial ML is competition-based market efficiency imposing limits on the acceptable complexity and expected performance of predictive models. Some topics of active research such as “benign overfitting” in interpolating deep neural nets and other ML algorithms are also covered. Several approaches of combining multiple forecasts are discussed using secondary ML, dimensionality reduction, and other methods, while highlighting correlation-based limits on alpha diversification. Multi-factor risk models and trading costs are reviewed including both theoretical and empirical aspects relevant to portfolio construction. Effects of price impact on stock market macro elasticity are also discussed. A unified framework of multi-period portfolio optimization is presented with several special closed-form solutions with impact and slippage costs and approximations for efficient algorithmic approaches. Optimal portfolio capacity and leverage are discussed, including a critical review of the Kelly criterion. The book also presents a brief review of intraday algorithmic execution and high-frequency trading (HFT) and raises fundamental questions of more efficient market design to benefit the general investing public.
Acknowledgments
This book wouldn't be possible without the author's interaction with many colleagues in academia and coworkers, competitors, and friends in the financial industry. The role of the early mentors, Vladimir Yankov (in physics) and Aaron Sosnick (in finance), was especially valuable in forming the author's ways of thinking about challenging problems and asking better questions.
Special thanks to all my superiors in the industry for prudently hiring or dismissing me, as appropriate for each occasion, and to all my peers and direct reports for the opportunity to learn from them.
I would like to thank Marco Avellaneda and Jean-Philippe Bouchaud for encouraging me to write up this material, as well as Aaron for discouraging it. A few fellow quants including, but not limited to, Colin Rust and Alexander Barzykin provided valuable comments and critique on various parts of the book draft. Their feedback is gratefully acknowledged.
Warm regards to those interviewers and interviewees who made the endless Q&A sessions more fun than they are supposed to be.
And thank you, Angela, for food, books, love, and understanding.
The time needed to write this book was an unexpected byproduct of the spread of the SARS-CoV-2 virus, which may have caused a temporary loss of smell, taste, or job, but hopefully not of sense of humor.
Introduction
Science is what we understand well enough to explain to a computer. Art is everything else we do.
Donald Knuth
Financial investment is a way of increasing existing wealth by buying and selling assets of fluctuating value and bearing related risk. The value of a bona fide investment is expected to grow on average, or in expectation, albeit without a guarantee. The very fact that such activity, pure gambling aside, exists is rooted in the global accumulation of capital, or, loosely speaking, increase in commercial productivity through rational management and technological innovation. There are also demographic reasons for the stock market to grow—or occasionally crash.
Another important reason for investments is that people differ in their current need for money. Retirees have accumulated assets to spend while younger people need cash to pay for education or housing, entrepreneurs need capital to create new products and services, and so forth. The banking and financial industry serves as an intermediary between lenders and borrowers, facilitating loans, mortgages, and municipal and corporate bonds. In addition to debt, much of the investment is in equity. A major part of the US equity market is held by pension funds, including via mutual funds holdings.1 Aside from occasional crisis periods, the equity market has outperformed the inflation rate. Stock prices are correlated with the gross domestic product (GDP) in all major economies.2 Many index and mutual funds make simple diversified bets on national or global stock markets or industrial sectors, thus providing inexpensive investment vehicles to the public.
In addition to the traditional, long-only investments, many hedge funds utilize long-short and market-neutral strategies by betting on both asset appreciation and depreciation.3 Such strategies require alpha, or the process of continuous generation of specific views of future returns of individual assets, asset groups, and their relative movements. Quantitative alpha-based portfolio management is conceptually the same for long-only, long-short, or market-neutral strategies, which differ only in exposure constraints and resulting risk profiles. For reasons of risk and leverage, however, most quantitative equity portfolios are exactly or approximately market-neutral. Market-neutral quantitative trading strategies are often collectively referred to as statistical arbitrage or statarb. One can think of the long-only market-wide investments as sails relying on a breeze subject to a relatively stable weather forecast and hopefully blowing in the right direction, and market-neutral strategies as feeding on turbulent eddies and waves that are zero-mean disturbances not transferring anything material—other than wealth changing hands. The understanding and utilization of all kinds of pricing waves, however, involves certain complexity and requires a nontrivial data processing, quantitative, and operational effort. In this sense, market-neutral quant strategies are at best a zero-sum game with a natural selection of the fittest. This does not necessarily mean that half of the quants are doomed to fail in the near term: successful quant funds probably feed more on imperfect decisions and execution by retail investors, pension, and mutual funds than on less advanced quant traders. By doing so, quant traders generate needed liquidity for traditional, long-only investors. Trading profits of market-neutral hedge funds, which are ultimately losses (or reduced profits) of other market participants, can be seen as a cost of efficiency and liquidity of financial markets. Whether or not this cost is fair is hard to say.
_____________
1 1 Organization for Economic Co-operation and Development (OECD) presents a detailed analysis of world equity ownership: A. De La Cruz, A. Medina, Y. Tang, Owners of the World's Listed Companies, OECD Capital Market Series, Paris, 2019.
2 2 F. Jareño, A. Escribano, A. Cuenca, Macroeconomic variables and stock markets: an international study, Applied Econometrics and International Development, 19(1), 2019.
3 3 A.W. Lo, Hedge Funds: An Analytic Perspective - Updated Edition, Princeton University Press, 2010.
Historically, statistical arbitrage started as trading pairs of similar stocks using mean-reversion-type alpha signals
