Algorithms are a sequence of steps or rules designed to achieve a goal. They can take many forms and facilitate optimisation throughout the investment process, from idea generation to asset allocation, trade execution, and risk management.
Machine Learning (ML) involves algorithms that learn rules or patterns from data to achieve a goal such as minimising a prediction error.
How ML algorithms can extract information from data to support or automate key investment activities, which include:
- observing the market
- analysing data to form expectations about the future and decide on placing buy or sell orders
- managing the resulting portfolio to produce attractive returns relative to the risk
The goal of active investment management is to generate alpha, defined as portfolio returns in excess of the benchmark used for evaluation.
The fundamental law of active management postulates that the key to generating alpha is having accurate return forecasts combined with the ability to act on these forecasts.
This law defines the information ratio (IR) to express the value of active management as the ratio of the return difference between the portfolio and a benchmark to the volatility of those returns. It further approximates the IR as the product of the following:
- The information coefficient (IC), which measures the quality of forecasts as their rank correlation with outcomes
- The square root of the breadth of a strategy is expressed as the number of independent bets on these forecasts
Applications of ML for trading (ML4T) typically aim to make more efficient use of a rapidly diversifying range of data to produce both better and more actionable forecasts, thus improving the quality of investment decisions and results.
Covering the following topics:
- Key trends behind the rise of ML in the investment industry
- The design and execution of a trading strategy that leverages ML
- Popular use cases for ML in trading
The rise of ML in the investment industry
The trends that have propelled algorithmic trading and ML to their current prominence include:
- Changes in the market microstructure, such as the spread of electronic trading and the integration of markets across asset classes and geographies
- The development of investment strategies framed in terms of risk-factor exposure, as opposed to asset classes
- The revolutions in computing power, data generation and management, and statistical methods include breakthroughs in deep learning
- The outperformance of the pioneers in algorithmic trading relative to human, discretionary investors
From electronic to high-frequency trading
The 1997 order-handling rules by the SEC introduced competition to exchanges through electronic communication networks (ECNs). ECNs are automated alternative trading systems (ATS) that match buy-and-sell orders at specified prices, primarily for equities and currencies and are registered as broker-dealers. It allows significant brokerages and individual traders in different geographic locations to trade directly without intermediaries, both on exchanges and after hours.
Dark pools are another type of private ATS that allows institutional investors to trade large orders without publicly revealing their information, contrary to how exchanges managed their order books prior to competition from ECNs.
Dark pools do not publish pro-trade bids and offers, and trade prices only become public sometime after execution.
With the rise of electronic trading, algorithms for cost-effective execution developed rapidly and adoption spread quickly from the sell-side to the buy-side and across asset classes.
Direct market access (DMA) gives a trader greater control over execution by allowing them to send orders directly to the exchange using the infrastructure and market participant identification of a broker who is a member of an exchange. Sponsored access removes pre-trade risk controls by the brokers and forms the basis for high-frequency trading (HFT).
HFT refers to automated trades in financial instruments that are executed with extremely low latency in the microsecond range and where participants hold positions for very short periods. The goal is to detect and exploit inefficiencies in the market microstructure, and the institutional infrastructure of trading venues.
HFT strategies aim to earn small profits per trade using passive or aggressive strategies.
Passive strategies include arbitrage trading to profit from very small price differentials for the same asset, or its derivatives, traded on different venues.
Aggressive strategies include order anticipation or momentum ignition. Order anticipation, also known as liquidity detection, involves algorithms that submit small exploratory orders to detect hidden liquidity from large institutional investors and trade ahead of a large order to benefit from subsequent price movements.
Momentum ignition implies an algorithm executing and cancelling a series of orders to spoof other HFT algorithms into buying (or selling) more aggressively and benefitting from the resulting price changes.
Factor investing and smart beta funds
To the extent that specific risk characteristics predict returns, identifying and forecasting the behaviour of these risk factors becomes a primary focus when designing an investment strategy. It yields valuable trading signals and is the key to superior active-management results.
The industry’s understanding of risk factors has evolved very substantially over time and has impacted how ML is used for trading.
Modern portfolio theory (MPT) introduced the distinction between idiosyncratic and systematic sources of risk for a given asset.
Idiosyncratic risk can be eliminated through diversification, but the systematic risk cannot.
The recognition that the risk of an asset does not depend on the asset in isolation, but rather on how it moves relative to other assets and the market as a whole, was a major conceptual breakthrough. In other words, assets earn a risk premium based on their exposure to underlying, common risks experienced by all assets, not due to their specific, idiosyncratic characteristics.
Multifactor models define risks in broader and more diverse terms than just the market portfolio.
A particularly attractive aspect of risk factors is their low or negative correlation. Value and momentum risk factors, for instance, are negatively correlated, reducing the risk and increasing risk-adjusted returns above and beyond the benefit implied by the risk factors.
Furthermore, using leverage and long-short strategies, factor strategies can be combined into market-neutral approaches. The combination of long positions in securities exposed to positive risks with underweight or short positions in securities exposed to negative risks allows for the collection of dynamic risk premiums.
Smart beta funds take a passive strategy but modify it according to one or more factors, such as cheaper stocks or screening them according to dividend payouts, to generate better returns. This growth has coincided with increasing criticism of the high fees charged by traditional active managers as well as heightened scrutiny of their performance.
The ongoing discovery and successful forecasting of risk factors that, either individually or in combination with other risk factors, significantly impact future asset returns across asset classes is a key driver of the surge in ML in the investment industry.
Algorithmic pioneers outperform humans
The track record and growth of assets under management (AUM) of firms that spearheaded algorithmic trading has played a key role in generating investor interest and subsequent industry efforts to replicate their success.
Systematic funds differ from HFT in that trades may be held significantly longer while seeking to exploit arbitrage opportunities as opposed to advantages from sheer speed.
Investments in strategic capabilities
Three trends have boosted the use of data in algorithmic trading strategies and may further shift the investment industry from discretionary to quantitative styles:
- The exponential increase in the availability of digital data
- The increase in computing power and data storage capacity at the lower cost
- The advances in statistical methods for analysing complex datasets
ML and alternative data
Hedge funds have long looked for alpha through information advantage and the ability to uncover new uncorrelated signals. Historically, this included things such as proprietary surveys of shoppers, or of voters ahead of elections or referendums.
Designing and executing and ML-driven strategy
Sourcing and managing data
The dramatic evolution of data availability in terms of volume, variety, and velocity is a key complement to the application of ML to trading, which in turn has boosted industry spending on the acquisition of new data sources. However, the proliferating supply of data requires careful selection and management to uncover the potential value, including the following steps:
- Identify and evaluate market, fundamental, and alternative data sources containing alpha signals that do not decay too quickly.
- Deploy or access a cloud-based scalable data infrastructure and analytical tools like Hadoop or Spark to facilitate fast, flexible data access.
- Carefully manage and curate data to avoid look-ahead bias by adjusting it to the desired frequency on a point-in-time basis. This means that data should reflect only information available and known at the given time. ML algorithms trained on distorted historical data will almost certainly fail during live trading.
From alpha factor research to portfolio management
Alpha factors are designed to extract signals from data to predict returns for a given investment universe over the trading horizon. A typical factor takes on a single value for each asset when evaluated at a given point in time, but it may combine one or several input variables or time periods.
The research phase
The research phase includes the design and evaluation of alpha factors.
A predictive factor captures some aspect of a systematic relationship between a data source and an important strategy input like asset returns. Optimising the predictive power requires creative feature engineering in the form of effective data transformations.
The execution phase
During the execution phase, alpha factors emit signals that lead to buy or sell orders. The resulting portfolio holdings, in turn, have specific risk profiles that interact and contribute to the aggregate portfolio risk. Portfolio management involves optimising position sizes to achieve a balance of return and risk of the portfolio that aligns with the investment objectives.
Strategy backtesting
To obtain unbiased performance estimates for a candidate strategy, we need a backtesting engine that simulates its execution in a realistic manner. In addition to the potential biases introduced by the data or a flawed use of statistics, the backtesting engine needs to accurately represent the practical aspects of trade-signal evaluation, order placement, and execution in line with market conditions.
ML for trading – strategies and use cases
Today, traders pursue a range of different objectives when using algorithms to execute rules:
- Trading execution algorithms that aim to achieve favourable pricing
- Short-term trades that aim to profit from small price movements, for example, due to arbitrage
- Behavioural strategies that aim to anticipate the behaviour of other market participants
- Trading strategies based on absolute and relative price and return predictions
Trading-execution programs aim to limit the market impact of trades and range from the simple slicing of trades to matching time-weighted or volume-weighted average pricing. Simple algorithms leverage historical patterns, whereas more sophisticated versions take into account transaction costs, implementation shortfall, or predicted price movements.
HFT funds most prominently rely on very short holding periods to benefit from minor price movements based on bid-ask or statistical arbitrage.
Behavioural algorithms usually operate in lower-liquidity environments and aim to anticipate moves by a larger player with significant price impact, based, for example, on sniffing algorithms that generate insights into other market participants’ strategies.
