How to Develop a

Reinforcement Learning
Trading System
Learning Objectives
● Identify the components of an
RL trading system

● Understand the steps required

to develop a trading strategy
using deep reinforcement
learning strategies

● Identify the final strategy checks

required to go live with an RL
trading system
Agenda
Components of a Reinforcement
Learning Trading System

Steps Required to Develop a Deep

Reinforcement Learning Strategy

Final Checks Before Going Live

with Your Strategy
RL Trading System

Historical Data Policy

Data Analysis Optimization

Live
Trading
RL Trader Backtester

Orders
Operator?
RL Trading System

Historical Data Policy

Data Analysis Optimization

Live
RL Trader Trading
Backtester
(RL Algo)

Orders
Operator?
RL Trading System

Historical Data Policy

Data Analysis Optimization

Live
Trading
RL Trader Backtester

Orders
Operator?
RL Trading System

Historical Data Policy

Data Analysis Optimization

Live
Trading
RL Trader Backtester

Orders
Operator?
Agenda
Components of a Reinforcement
Learning Trading System

Steps Required to Develop a Deep

Reinforcement Learning Strategy

Final Checks Before Going Live

with Your Strategy
DRL Algorithm Development
1. Choose your instrument(s) 5. Define your action space and
decide on a training method
2. Model trading costs and other
potential drags on performance 6. Train and backtest your RL
Trader
3. Obtain and cleanse a sufficient
amount of historical data 7. Go live or retrain your RL Trader

4. Create an ensemble of algos

and experiment with different
inputs
Choose the Instrument
You Will Trade
● Make sure sufficient data is
available
● Low-volume / low-liquidity
markets can be more profitable
● High volatility instruments such
as Crypto can also have higher
performance potential
● Need to be sure you have the
risk appetite and budget
needed to trade effectively
Model Trading Costs
● Slippage is the difference
between the expected price of
a trade and the price at which it
is actually executed
● Slippage is highest during
periods of high volatility when
market orders are used
● You need bid-ask price data at a
minimum to model slippage in
backtesting (visibility into the
order book is better though)
Model Trading Costs
● Slippage is the difference
between the expected price of
a trade and the price at which it
is actually executed
● Slippage is highest during
periods of high volatility when
market orders are used
● You need bid-ask price data at a
minimum to model slippage in
backtesting (visibility into the
order book is better though)
Model Trading Costs
● Slippage is the difference
between the expected price of
a trade and the price at which it
is actually executed
● Slippage is highest during
periods of high volatility when
market orders are used
● You need bid-ask price data at a
minimum to model slippage in
backtesting (visibility into the
order book is better though)
Collect/Cleanse Data
● Training an RL algo is very
sensitive to gaps or outliers in
the data
● Need to interpolate missing
values and “correct” outliers
● If data gaps are too big, remove
the data from the training set
Ensemble of Algorithms
● You need a variety of algos as
each will perform differently in
different trading environments
● Give algos that perform better a
higher weight for trading
decisions
● Keep backtesting on new data
and re-weight algos
immediately
Use Different Inputs
● Can you use both price and
sentiment data to train your RL
algorithm?
● Better to use separate neural
networks for each type of data
and feed their representations
to your RL Trader
● Stepwise structure with
different NNs at each step
works best
RL Algorithm
● Responsible for taking the
inputs from the deep recurrent
neural network (DRNN) and
making a decision
● Buy, Sell, or Hold
● Needs to learn actions based on
analysis of a continuous data
feed (like a human trader)
● Many RL methods but literature
and experience support
actor-based methods
Training Your RL Trader
● Present the RL Trader with a
window of data that you have
normalized
● Allow it to experiment with
different actionable scenarios
● Calculate PnL for initial training
then add Sharpe ratios for each
scenario
● Your RL Trader will eventually
develop a trading style that will
maximize PnL and/or Sharpe
ratio
Backtest Your RL Trader
● Use more recent out-of-sample
and unseen data set
● Data set should include a range
of market situations
(uptrending, downtrending,
volatile, steady)
● If RL Trader maintains or
improves performance then you
are ready to go live or paper
trade
Agenda
Components of a Reinforcement
Learning Trading System

Steps Required to Develop a Deep

Reinforcement Learning Strategy

Final Checks Before Going Live

with Your Strategy
Transition to Live Trading
● Don’t be hasty
● It took you a lot of time and
effort to get to this stage
● You have already spent heavily
in time and costs
● The rewards can be substantial
if your trader is successful and
keeps its edge
Go-Live Checklist
● Your RL Trader seems to
perform well for a long period in
the past and in different market
conditions
● Your RL Trader seems to
perform well on another
security that you haven’t trained
it on
● Your algorithm does get decent
performance in the test set and
has done well in training and
development sets
Go-Live Checklist
● If your algo gets destroyed in
the testing data set it means
that it has been overfitted and
you need to either:
○ Bring in more data
○ Look for less deep neural
network architectures
○ Increase your dropout
percentage
Dropout Rate
● As a neural network learns,
neuron weights settle into their
context within the network
● If neurons are dropped this
results in the network learning
multiple independent
representations
● The network becomes less
sensitive to the specific weights
of neurons and is better able to
generalize and not overfit
Go-Live Checklist
● Your algorithm performs very
badly in the training set and
cannot be trained
○ Use more inputs for your NN
○ Increase depth of NN
○ Adjust hyperparameters
○ Make sure you initialize the
weights before training
Go-Live Checklist
● Your algorithm performs well in
all train and test sets and you
want to try live data?
● You need to keep on training
with smaller learning rates for
your deep neural network. There
is more potential performance
to squeeze out of it before live
trading
Go-Live Checklist
● Your live environment is
extremely volatile
● Use a demo account to paper
trade for a while and see what
happens
● Try live trading but make micro
trades until you gain confidence
in the strategy
Investment and Trading Risk
Management
Learning Objectives
● Distinguish between risk
management in trading vs
investment portfolios

● Understand how diversification

is achieved in a trading portfolio

● Identify optimization criteria for

managing strategy risk and
portfolio risk
Agenda
Investment Risk Management

Trading Strategy Risk Management

Trading Portfolio Risk Management

Maximizing
Risk-Adjusted Return
● Identify undervalued assets

● Reduce risk by combining

assets in a portfolio
Asset Price Volatility:
Systematic Risk
● Systematic risk influences the
value of financial assets in
general
● A significant change in the
performance of the economy
would affect most of the assets
in an investor’s portfolio
Asset Price Volatility:
Systematic Risk
This risk is generally measured by:

● Beta
● Other indicators of an
investment’s correlation with
the overall economy
● Measures of the volatility of
market indexes such as the VIX
Asset Price Volatility:
Systematic Risk
● This kind of risk affects a single
asset
● An example is news that affects a
specific stock such as a corporate
credit rating downgrade
● Diversification is the most effective
strategy for investors to limit their
exposure to unsystematic risk
● This risk is generally measured by
an estimate of the volatility of a
specific asset
Diversification and Risk
● The diversification effect on
portfolios can be decomposed
into two components:
○ Risk reduction as a result of
holding imperfectly
correlated securities
○ Risk reduction as number of
securities in a portfolio
increases
Diversification and Risk
● The risk of a portfolio is
measured by the ratio of the
covariance of a portfolio’s
return to the variance of the
market return (portfolio beta)

● A domestic portfolio that is

fully diversified would have a
beta of 1
 Portfolio Risk Reduction Through Diversification

Portfolio
Risk

Number of Stocks in
Portfolio
International
Diversification and Risk
● The total risk of any portfolio is
therefore composed of
systematic risk and
unsystematic risk

● Increasing the number of

securities in the portfolio
reduces the unsystematic risk
component leaving the
systematic risk component
unchanged
 Risk Reduction with International Diversification

Portfolio Risk

Number of Stocks in
Portfolio
Market Correlations and Volatility
Market Correlations and Volatility
International
Diversification and Risk
● The FX risks of a portfolio are
reduced through international
diversification

● FX as an asset category also

provides diversification benefits
Investment Diversification
and Risk
● Most asset managers are
constrained by their mandate to
invest in a single asset category.

● Diversification with other asset

categories is usually
accomplished by the investor
Agenda
Investment Risk Management

Trading Strategy Risk Management

Trading Portfolio Risk Management
Trading Capital and Risk
Management
● Zero or minimal operator
intervention means risk strategy
design is key
● Risk management usually
handled at strategy level and
portfolio level
Strategy Level Risk
Management
● Stop Losses

○ Static
○ Dynamic
○ Variable

● Set a maximum loss or risk on

each trade or strategy
Strategy Level Risk
Management
● Return metrics

○ Win/Loss %
○ Average Win
○ Average Loss

● Risk-to-Reward Ratio set by

your RL Algorithm to maximize
strategy return
Strategy Level Risk
Management
● RL Algorithm will optimize:

○ Profit target
○ Stop loss level(s)
○ Time outs

● Too high a profit target or loose

stop loss levels can dramatically
lengthen the average trade
completion time
Strategy Level Risk
Management
● Long trade completion time:

○ Increases capital needs

○ Reduces number of trades
○ Reduces period returns

● RL reward design must

incorporate all of these factors
Portfolio Level Risk
Management
● Global Stop Losses:

○ Strategy or firm level risk

limits
○ Usually a fixed percentage of
trading capital
○ Closes out all open trades

● Key factor in survival of a

trading firm
Portfolio Level Risk
Management
● Strategy Diversification

○ Multiple strategies
○ Multiple asset classes
○ Multiple assets within each
asset class
Portfolio Level Risk
Management
● Pairs Trading Diversification

○ Pairs chosen using PCA

analysis, fundamental
analysis, and momentum
○ Multiple pairs chosen for each
strategy
○ Strategies applied to different
asset classes (Stocks, FX,
Crypto…)
Agenda
Investment Risk Management

Trading Strategy Risk Management

Trading Portfolio Risk Management

Portfolio Level Risk
Management
● Measuring risk reduction in a
portfolio of trading assets

○ Level of correlation between

trading assets
○ Number of trading assets
held in portfolio
Notebook: Measuring Risk Reduction in a Portfolio*
From Principles: Life and Work by Ray Dalio

“From my earlier failures, I knew that no matter how confident I was in making any
one bet I could still be wrong—and that proper diversification was the key to
reducing risks without reducing returns. If I could build a portfolio filled with
high-quality return streams that were properly diversified (they zigged and
zagged in ways that balanced each other out), I could offer clients an overall
portfolio return much more consistent and reliable than what they could get
elsewhere.”

*Based on https://lambdaclass.com/finance_playground/diversification-dalio-holy-grail.html
Portfolio Risk Reduction
In [1]:
%config InlineBackend.figure_format = "retina"

import numpy as np
import pandas as pd
import altair as alt

np.random.seed(42) # Set seed for reproducibility

Portfolio Risk Reduction
In [2]:
def correlated_streams(n, mean, risk, corr):
"""Generates `n` return streams with given average `mean` and
`risk`,
and with an average correlation level `corr`.
"""
num_samples = 10_000
means = np.full(n, mean)

corr_mat = np.full((n, n), corr, dtype=np.dtype("d"))

np.fill_diagonal(corr_mat, 1,)
cov_mat = corr_mat * risk**2

streams = np.random.multivariate_normal(means, cov_mat,

size=num_samples)

return streams.T
Portfolio Risk Reduction
In [3]:
n=5
mean, std, corr = 10, 15, 0.6
streams = correlated_streams(n, mean, std, corr)

In [4]:
streams.mean(axis=1)

Out[4]:
array([10.12229747, 9.92797016, 9.98877207, 10.05103342,
9.90978558])

In [5]:
streams.std(axis=1)
Portfolio Risk Reduction
Out[5]:
array([15.07254044, 15.05168254, 15.17926238, 15.2192544 ,
15.14908131])

In [6]:
np.corrcoef(streams)

Out[6]:
array([[1. , 0.60676484, 0.61222918, 0.61179636, 0.60301561],
[0.60676484, 1. , 0.61036834, 0.61049393, 0.61073826],
[0.61222918, 0.61036834, 1. , 0.61526424, 0.61265281],
[0.61179636, 0.61049393, 0.61526424, 1. , 0.605607 ],
[0.60301561, 0.61073826, 0.61265281, 0.605607 , 1. ]])
Portfolio Risk Reduction
In [7]:
def aggregate_risk(return_streams, n):
"""Returns the pooled risk (std) of the `n` first streams
in `return_streams`
"""
assert len(return_streams) >= n

aggregate_returns = np.sum(return_streams[:n], axis=0) / n

return aggregate_returns.std()
Portfolio Risk Reduction
In [8]:
max_assets = 20
assets = range(1, max_assets+1)

mean = 10 # Avg mean return of 10%

risk_levels = range(1, 15)

index = pd.MultiIndex.from_product([risk_levels, assets],

names=["risk_level", "num_assets"])
simulated_data = pd.DataFrame(index=index)

for risk in risk_levels:

for corr in np.arange(0.0, .8, 0.1):
return_streams = correlated_streams(max_assets, mean, risk,
corr)
risk_level = np.zeros(max_assets)
for num_assets in assets:
risk_level[num_assets-1] = aggregate_risk(return_streams,
num_assets)
simulated_data.loc[(risk, ), round(corr, 1)] = risk_level
simulated_data.columns.names = ["correlation"]
Portfolio Risk Reduction
In [9]:
simulated_data.query("risk_level == 14")
Portfolio Risk Reduction
In [10]: In [10]: (continued)
def plot_risk_level(data, risk_level): points = base.mark_circle().encode(
subset = data.query("risk_level == @risk_level") opacity=alt.value(0)
stacked = subset.stack().reset_index(name="risk") ).add_selection(
stacked.head() highlight
).properties(
chart = alt.Chart(data=stacked) height=400,
width=600,
highlight = alt.selection(type="single", on="mouseover", title="Risk % by number of assets in portfolio"
fields=["correlation"], nearest=True) )
lines = base.mark_line().encode(
base = chart.encode( size=alt.condition(~highlight, alt.value(1), alt.value(3)),
alt.X("num_assets", axis=alt.Axis(title="Number of tooltip=["correlation"]
Assets")), )
alt.Y("risk", axis=alt.Axis(title="Risk %")), return points + lines
alt.Color("correlation:N", scale=alt.Scale(scheme="set2")))
In [11]:
plot_risk_level(simulated_data, 10)
Out[11]:
Portfolio Risk Reduction
Portfolio Level Risk
Management
● Reduced risk through exposure
to different sources of income

○ Combine uncorrelated
revenue streams
○ From a number of trading
assets
○ Capture “true” alpha and use
leverage to increase returns in
a “Risk Parity” strategy*
* https://www.bridgewater.com/resources/risk-parity-is-about-balance.pdf