Approaching Real-Time Business Intelligence

Trading at the Speed of Light

Sean McClure, Ph.D.

Business Analytics, Excellerate Inc.

smcclure@excellerate4success.com

www.excellerate4success.com
 Overview
Themes
Introducing Excellerate
Metrics
Real-Time BI and HFT

Information at High Frequency Data Mining

Strategies at High Frequency

Big Data

Developing and Deploying Models

Meta Data
Executing and Monitoring Real-Time
Systems
Prediction
Summary

www.excellerate4success.com
 About Us!

Business Intelligence Service Providers

• Dedicated to bringing top quality business

intelligence expertise to successful
growing organizations (SGOs);

• Aggressively researching industry best

practices and best-in-breed software tools
to deliver high-end analytics and data
mining expertise;

• Business Intelligence model supported by

Subject Matter Experts (SMEs) in key
business areas.
www.excellerate4success.com
 Real -Time Business Intelligence
Defining “Real-Time”
Three types of latency1:

• Data latency: time taken to collect and store the data;

• Analysis latency: time taken to analyze the data and turn it

into actionable information; and

• Action latency: the time taken to react to the information and

take action.

Approaching “zero” latency

• Real-time business intelligence technologies are designed to reduce all three
latencies as close to zero as possible;

• Traditional BI only seeks to reduce data latency.

www.excellerate4success.com
 Real -Time Business Intelligence
Real Time BI in various industries

Debit and Credit Fraud Detection

Marketing
1
Inventory Control
Supply-chain Optimization
Customer relationship management (CRM)
Dynamic pricing and yield management
Data validation
Operational intelligence and risk management 2

Call center optimization

Transportation industry
Finance (biggest candidates)

www.excellerate4success.com
3
 High Frequency Trading (HFT)
Best Case Study for “Real-Time” Intelligence

High
• Trading platform transacts

Execution Latency
large number of orders at Traditional long-
very fast speeds; term Investing

• Complex algorithms
analyze multiple markets and Algorithmic/
High-
execute orders based on electronic trading
frequency
market conditions; Trading

• Faster execution speeds

Low
more profitable than slower
Short Position Holding Period Long
execution speeds.
Figure 1 – Types of Trading6

In the U.S., high-frequency trading accounts for ~73%

of all equity trading volume5
www.excellerate4success.com
 High Frequency Information
Properties of Tick Data – Quote, Trade, Price and Volume Information

Date/time quote
originated
Provided by other
Highest price available for market participants
sale of the security through limit orders
• Timestamp
• Security ID Lowest price entered for
• Bid Price buying the security
• Ask Price
• Available bid volume Total demand
• Available ask volume
Total supply
• Last trade price
• Last trade size Price at which the last trade in the
• Option-specific data security cleared

Actual size of the last

www.excellerate4success.com
executed trade
 High Frequency Information
Recent microstructure research and advances in econometric
modeling tell us there are unique characteristics to tick data;

irregularly spaced Tick Data

quotes arriving randomly
very short time intervals
time
(compare to low-frequency)

Irregularities wealth of information not available in low-frequency

data;

Inter-trade durations signal changes in market volatility,

liquidity, and other variables.

Volume of data allows for statistically precise inferences.

www.excellerate4success.com
Number of observations in single day of tick data = 30 years of daily observations
 High Frequency Information
Modeling the Arrivals of Tick Data creates a host of opportunities not
available at low-frequency
• Time distance between quote arrivals carries information
time

quote processes
Duration models
trade processes
price processes
Estimate the factors affecting the
volume processes duration between ticks
High Trade
Duration
Higher likelihood of
unobserved bad
news
Low Trade
Low Price
Duration Low Volume
Duration
Higher likelihood Duration
Increased
of unobserved Increased Liquidity
Volatility
good news

Absence of Trade
Lack of news, low levels
of liquidity, trading halts,
www.excellerate4success.com
trader motivations
 High Frequency Information
Data sampling methods overcome irregularities in high-frequency data
for ease of processing
Traditional Approach Linear Time-Weighted Interpolation7

Minute 1 Minute 2 Minute 3 Minute 1 Minute 2 Minute 3

Figure A Figure B
t tlast
quote qˆ t qt ,last qˆt qt ,last (qt ,next qt ,last )
t next tlast
Most modern computational techniques have been developed
to work with regularly spaced data (easy to process)

High frequency data-sampling methods developed to

overcome irregularities in tick data by sampling at
www.excellerate4success.com
predetermined periods of time
 High Frequency Information
Security Price Adjustments to Information
The price of the security in the inefficient market begins adjusting before/after
the news becomes public ( “information leakage” and “overshooting”)

Many solid trading strategies exploit both the information leakage and
overshooting to generate consistent profits

Efficient and Inefficient Markets

Inefficient market response
Good News Bad News

Efficient Efficient market response

market
response

Information Arrival Time Information Arrival Time

Inefficient market response

Figure 2 Incorporation
www.excellerate4success.com of information in efficient and inefficient markets6
 High-Frequency Strategies
Trading on High-Frequency Information
Traders leverage state-of-the-art IT technology to implement trading
strategies that have high-frequency opportunities;

High-frequency trading strategies typically fall into four main categories8.

HFT-based Strategies

Electronic
Statistical Liquidity Others
Liquidity
Arbitrage Detection
Provision

Spread Market Neutral Sniffing/Pinging/ Latency

Capturing Arbitrage Sniping Arbitrage

Cross
Asset, Cross Quote Matching Short Term
Rebates
Market & ETF Momentum
www.excellerate4success.com
 High-Frequency Strategies
Liquidity Provision Strategies - Spread Capturing

Liquidity providers profit from the spread between bid and ask prices by
continuously buying and selling securities;

Executed predominantly using limit orders

Ask Asking Price

Market Buy Orders Market Sell Orders

Bid-Ask Spread Market Price

Limit Buy Orders Limit Sell Orders

Bid Offer Price

High-speed transmission of orders and

low-latency execution required for
successful implementation of liquidity
provision strategies. Market Transactions
www.excellerate4success.com
 High-Frequency Strategies
Predictions based on Real-Time access to the Limit Order Book (LOB)

Direction of market price • Shape of limit order book is

movement
predictive of impending
changes in market price10

• Exploited by market-maker
traders;
buy sell • Depends on probability
distribution for arriving market
orders;
Direction of market price
movement • Shape can be estimated when
book not observable.

www.excellerate4success.com
buy sell
 High-Frequency Strategies
Statistical Arbitrage

“Stat-Arb” rests squarely on data mining. It finds

statistical relationships in large amounts of data
and builds a model of those relationships;

Leverages states of the art technology to profit

from small and short-lived discrepancies
between securities;

Arbitrageurs generate profits by selling the

asset on the market where it is valued higher
and simultaneously buying it on another
market where it is valued lower.

www.excellerate4success.com
 High-Frequency Strategies
Detecting Statistical Anomalies in Price Levels Identify securities
that trade in
frequency unit
Once gap in prices
reverse, close out
position/stop loss

Measure difference
between prices of Sij ,t Si ,t S j ,t ,t 1, T
identified securities
Monitor and act
upon differences
in security prices

St Si , S j, E S 2 S
St Si , S j, E S 2 S Select most stable
T 2

relationships mini, j t 1
Sij ,t

T
Estimate T
1 2 distributional 1
St St E St E St St
T 1t properties of the T
1 t 1
www.excellerate4success.com difference
 High-Frequency Strategies
Fundamental Arbitrage Strategies by Asset Class6

Asset Class Fundamental Arbitrage Strategy

Foreign Exchange Triangular Arbitrage
Foreign Exchange Uncovered Interest Parity (UIP)
Arbitrage
Equities Different Equity Classes of the
Same Issuer
Equities Market Neutral Arbitrage
Equities Liquidity Arbitrage
Equities Large-to-Small Information
Spillovers
Futures and the Underlying Asset Basis Trading
Indexes and ETFs Index Composition Arbitrage
Options Volatility Curve Arbitrage
www.excellerate4success.com
 Model Development/Deployment
Model Development
Models used in HFT Ideas
• Linear Econometric Models
• Academic research and
• Autoregressive (AR) Estimation proprietary extensions
• Moving Average (MA) Estimation
• Autoregressive Moving Average (ARMA)
• Cointegration Tools
Volatility Modeling
• To model observed volatility • Modeling predominantly
clustering = ARMA or GARCH in Matlab /R,
• c++ for back-tests and
transition into production
NonLinear Econometric Models
Allows for modeling of complex nontrivial
relationships in data Back Testing
• Taylor series expansion
• Threshold autoregressive model
• Modeled relationships tested on lengthy
• Markov switching model spans of tick data
• Nonparametric estimation • Forecasting validity
www.excellerate4success.com
• Neural Networks • Various market situations
 Model Development/Deployment
Back-Testing Econometric Models Model Accuracy Analysis6

Point Forecasts Accuracy Curve

• predict price will reach certain
level /point Random Forecast
• regression of realized values 100
from historical data against out of Model C
sample forecasts Model A

Hit Rate (%)

Directional Forecasts
• makes decisions to enter into
positions based on expectations
of system going up or down
(without target) Model B

Accuracy Curves 0.0

Miss Rate (%) 100 %
• compares the accuracy of
probabilistic forecasts
• HFT models done with TSA
curves
www.excellerate4success.com
 Executing Real-Time Systems
Execution Optimization Algorithms

• Algorithms spanning order-execution processes

• Designed to optimize trading execution once the buy-

and-sell decisions have been made elsewhere

best way to route the order to the exchange

best point in time to execute a submitted order (non-market order)
best sequence of sizes in which the order should be optimally processed

Common Types

1)
1) Market Aggressiveness Selection algorithms designed to choose between market
and limit orders for optimal execution;

2)
1) Price-Scaling algorithms designed to select the best execution price according to
the pre-specified trading benchmarks; and

3)
1) Size-optimization algorithms that determine the optimal ways to break down large
trading lots into smaller parcels to minimize adverse costs (cost of market impact)
www.excellerate4success.com
 Executing Real-Time Systems
Execution Optimization Algorithms
Market Aggressiveness Price-Scaling Size Optimization
Selection
• Tries to obtain the best • Tries to trade with
• Balances passive and
price for the strategy position undetected
aggressive trading using
optimization Strike Algorithm
• Large order packets are
min Cost ( ) Risk( ) • Minimizes the cost of broken up for least
execution relative to a
benchmark
amount of market impact
Cost ( ) Eo P( ) Pb (“Stealth Trading”)
• Designed to capture gains in
Risk ( ) ( ( )) periods of favorable prices
P( ) P f ( X , ) g( X ) ( ) 2
min Et Pt 1 ( t ) Pb ,t
Pb Benchmark execution price t

P(a) Realized execution price Pt 1 ( t ) Realized price

(a) Deviation of trading outcome
t Trading aggressiveness
P Market price at order entry
Pb ,t Benchmark price
f ( X , a) Market Impact due to trade
Plus Algorithm
g( X ) Price impact due to info leak
Wealth Algorithm
www.excellerate4success.com
 Executing Real-Time Systems
HFT Business Cycle 1 1 – 4: run-time
Receive/archive real-
time tick data on 5 – 6: post-trade
securities of interest

6 2
Ensure trading costs incurred Apply back-tested
during execution are within econometric models to the
acceptable ranges tick data obtained in 1
Each functions built with
independent alert systems
that notify monitoring
personnel of problems,
5 unusual patterns etc.
Evaluate trading
performance relative to
predetermined benchmarks 3
Send orders and keep track of
open positions/P&L values

4
Monitor run-time trading
behavior, compare with predefined
parameters, manage the run-time
www.excellerate4success.com trading risk
 Summary
Themes

Introducing Excellerate Metrics

Real-Time BI and HFT

Data Mining
Information at High Frequency
Big Data
Strategies at High Frequency

Developing and Deploying Models Meta Data

Executing and Monitoring Real-Time

Systems Prediction

www.excellerate4success.com
 Thank You
Sean McClure, Ph.D.
Business Analytics, Excellerate Inc.

smcclure@excellerate4success.com

www.excellerate4success.com
 References
1) Richard Hackathorn, "Active Data Warehousing: From Nice to Necessary," Teradata Magazine (June 2006), AR-4835

2) cdn.avangate.com/web/images/articles/fraud-lock.jpg

3) genesissolutions.com/wp-content/uploads/2009/10/3.3.3-MROSupply-307x195.jpg

4) partnerc.com/images/iStock_000007068822Small.jpg

5) http://www.economist.com/node/5475381?story_id=E1_VQSVPRT

6) High-Frrequency Trading, A Practical Guide to Algorithmic Strategies and Trading Systems, Aldridge, Wiley
Trading, 2010

7) Dacorogna, M.M., R. Gencay, U.A. Muller, R. Olsen and O.V. Pictet, 2001. An introduction to High-Frequency
Finance. Academic Press: San Diego, CA.

8) High-Frequency Trading; Gomber, Arndt, Lutat, Uhle, Deutsche Borse Group

9) http://www.mathestate.com/tools/Financial/wp2/HowMarketsWork.html

10) Cao, C., O. Hansch and X. Wang, 2004. “The Informational Content of an Open Limit Order Book” Penn State
University.

www.excellerate4success.com