Big Data Analytics and Use Cases

Steve M Stennett
US Federal Information Management CTO
smstenne@us.ibm.com © 2012 IBM Corporation
703-444-8845
Most Requested Uses of Big Data

Log Analytics & Storage

Fusion of multi-INT (multi-formats)

RFID Tracking & Analytics

Fraud Detection & Modeling

Risk Modeling & Management

360°View of a Person, Place, or Thing

Warehouse Extension (case patterns)

Email / Call Center Transcript Analysis

Call Detail Record Analysis

IBM Watson

2 © 2012 IBM Corporation

A Challenging Environment ata 1024
YottaByte
r D
1021
n so e ZettaByte
Se lum 1018
Vo ExaByte

1015
PetaByte
1012
TeraByte

3 © 2012 IBM Corporation

Integrated Layers Address These Challenges

Visual Analysis & Collaboration

Visual Analysis, Link Analysis, Shared Workspaces, Case
Management, Cross-Functional Publishing & Collaboration

Visual & Link Workflow & Case Operational

Analysis Management Dashboards
Analytics
Descriptive & Predictive Analytics Against Structured,
Semi-Structured & Unstructured Information; High-Scale
"Sense Making"; High-Velocity Data Stream Analytics
Descriptive & Content & Streams
Predictive Analytics Sentiment Analytics
Analytics

Trusted Information
Establish, Manage, Share & Deliver Information MDM
that is Accurate, Complete, & In-Context
Persistent
Relationship Analytic Information sharing via
Awareness Repositories NIEM and other standards

Structured & Unstructured Content from

Databases, Open Sources, Human
Intelligence, Signals Intelligence, etc…
Source Systems
4 © 2012 IBM Corporation
Real Time Marine Mammal Position and Behavior Modeling

Filter wind &

wave noise

Model Marine
Mammal
+ + = environment

Correlate to
Galway Bay
ecosystem
Analytics & InfoSphere Streams Advanced
Sensors Acoustical Analytics

5 © 2012 IBM Corporation

What We Have Learned
Big Data Requires A Disruptive Approach – It Breaks The Traditional Model

Traditional Approach Big Data Approach

Business Users IT
Determine what Delivers a platform to
question to ask enable creative discovery

IT Business
Structures the Explores what
data to answer questions could be
that question asked

Structured & Repeatable Analytics Iterative & Exploratory Analytics

•Query Based -- Questions Drive Data •Autonomic -- Insight Drives Answers
•Citizen Surveys VS. •Citizen Sentiment
•Monthly, Weekly, Daily •Persistent & Ad Hoc
•Data At Rest •Data In Motion
6 © 2012 IBM Corporation
 Who’s talking to Whom?

Stream Denoising &

A Social Network
Conversation Pairing Speaker Detection Analysis
A B Olivier Mihalis
talks to talks to

C D Ching-Yung Upendra
Stream talks to talks to
B
Stream C

E Deepak

After denoising

- Just-in-time - Just-in-time - Social network

Stream D - Features: - Features: GSM - Fusion technique
Volumetrics domain
- Iterative method
- Very high accuracy - High accuracy
- Very low complexity - Moderate complexity

7 - Robust to noise - Robust to noise © 2012 IBM Corporation

Real Time Detection and Management of Wildfires

Wildfire Management
application

Real-time US map of wildfire
risk

Detect wildfire smoke

Detailed smoke dispersion
prediction model

Task NOAA satellite and NASA
UAV to monitor wildfire

Generate health alerts

8 © 2012 IBM Corporation

New analytic applications require a big data platform

Advanced Analytic Applications

• Integrate and manage the full variety,

velocity and volume of data

• Apply advanced analytics to

information in its native form

• Visualize all available data for ad-hoc

analysis
Big Data Platform
• Development environment for building
Process and analyze any type of data new analytic applications
Accelerators
• Workload optimization and scheduling

• Security and Governance

9 © 2012 IBM Corporation

 IBM Big Data Platform
Over 100 sample applications and toolkits with
Without a Big Data Platform industry focused toolkits with 300+ functions and
operators
You Code…
Analytic Applications
BI / Exploration / Functional Industry Predictive Content
Reporting Visualization App App Analytics Analytics

Event Custom SQL

Handling and IBM Big Data Platform
Scripts
Visualization Application Systems
Multithreading & Discovery Development Management

Check Application
Pointing Management Accelerators
HA Accelerators
and
Toolkits

Hadoop Stream Data

System Computing Warehouse
Performance Debug
Connectors
Optimization

Information Integration & Governance

Security

“TerraEchos developers can deliver applications 45% faster due to

10 the agility of Streams Processing Language…” © 2012 IBM Corporation
– Alex Philip, CEO and President, TerraEchos
 Analytic Accelerators Designed for Variety

Text Simple &

(listen, verb), Acoustic
(radio, noun) Advanced Text

Mining in Advanced
Microseconds Mathematical Models

Predictive ∑ R( s , a )
population
t t Statistics

GeoSpatial Image & Video

11 © 2012 IBM Corporation

 IBM Big Data Platform – Hadoop Distribution-Agnostic

InfoSphere BigInsights

Visualization & Exploration

Analytic Applications
Development Tools
BI / Exploration / Functional Industry Predictive Content
Reporting Visualization App App Analytics Analytics
Enterprise
capabilities Advanced Engines
IBM Big Data Platform

Connectors Visualization Application Systems

& Discovery Development Management

Workload Optimization
Accelerators

Administration & Security Hadoop Stream Data

System Computing Warehouse

Open source
components IBM-certified or or … Information Integration & Governance
Apache Hadoop

Apache

12 © 2012 IBM Corporation

Imagine the Possibilities of Analyzing All Available Data
Faster, More Comprehensive, Less Expensive

Cyber Threat Predictive Sentiment

Analysis Modelling & risk Analysis
analysis

Accurate and timely Tendency Analysis Low-latency

threat detection information sharing

13 © 2012 IBM Corporation

ABA - Activity Based Analysis Patterns
Converge Multiple inter-related Domains • Tendency analysis
Observation Space • Sentiment analysis
(Threat Analysis)
• Integrated prospective
Environmental employee behavior
Human Domain Human modeling
Domain Domain
• Predictive modeling

Observation Space
Observation Space

(past history)
• System log analytics
Cyber Political (reduce operational risk)
Domain ABA Domain

The Human Domain

intersects with and is both
influenced and constrained
by the other domains
Human Human
Economic Domain – People are a constant in
Domain
Domain each domain.
– Other domains have a
• Geospatially impact
• Causal Impact
• Temporal Impact.

Observation Space
ABA is the convergence in

time and Space of the other
domains
14 © 2012 IBM Corporation
 Big Data Platform Video/Imagery Analytics

Real-time Events Cognos/i2/BigSheets/Browser Visualization

Tracking and Linking
(Actionable Intelligence) 1 3 Historical View
Broadcast Video

Visual Semantic Classification

User-Generated 2 Machine Learning
Content Sites
Transport
System S Data Fabric
Operating System

X86 X86 FPGA X86 Cell

Box Blade Blade Blade Blade

InfoSphere BigInsights

4 Bootstrap and Enrich

Video Blogs
Real-Time
Real-TimeVideo
VideoAnalytics
Analytics Batch
BatchVideo
VideoAnalytics
Analytics
15 © 2012 IBM Corporation
Covert Intrusion Detection

State-of-the-art covert surveillance

system based on Streams platform

Acoustic signals from buried fiber optic

cables are monitored, analyzed and
reported in real time for necessary
action

Currently designed to scale up to 1600

streams of raw binary data

16 © 2012 IBM Corporation

Brocade - Extensible Solution Architecture – Sentiment Analysis Plus

17 © 2012 IBM Corporation

Identifying Hidden Asset (People, Places or Things) Patterns
(IBM Business Partner)
PMML Model Generation
Auto-Generated PMML Models - for Historica
lHistorica
Data
Cyber Interrogation lHistorica
Data
lHistorica
Data
l Data

Unstructured
Semi-structured
Structured
Video

Current Hadoop
Data System
Stream
Computing

Data
Warehouse Mission
Analyst

Real Time
Data Future – Near Real Time Model Changes
18 © 2012 IBM Corporation
Adelos – S4 Workflow: Real-Time GeoOntology through Streams-Based\Attribute Correlation and Stream Fusion

Adelos
MASINT

S-1 S-2 S-3

Connected to Adelos
HPC Classified S Patterns
Cloud 98.9%
114.554W –
45.3566N
17:56:7856
November 14, 2009
S-4
PMML – Prediction
Metadata Chips
87.9%
77.2% 114.55W –
114.55W – 45.356N
45.3N 17:56:7856
17:56:78
SML – Universal Wrapper
19 © 2012 IBM Corporation
Example – Teleco Solution Architecture

Real-time Summary Offer-section

Statistics

Joint Offer Decisions

Summary Statistic Extraction
KPI Monitoring
Offer Models Model (call pattern based)

Influencers and relationships

(social network analysis based)

Graph Edges
and Nodes

Graph Construction
Telecom Data

Social Network Analysis -

Data Preprocessing
Call Detail Records
Preprocessed BigInsights
CDRs

Create Offer Models

- Complex Decision Tree
- Calling patterns/ user contracts
20 © 2012 IBM Corporation
 Traffic Management for Sustainability and Efficiency
Multimodal Data Streams
• GPS
• Cell-phones (location tracking)
• Public Transport (bus, docking)
• Pollution measurements
• Weather Conditions (including road conditions)
• Optical traffic flow detectors
• Travel time data based on plate recognition
• Induction loop detector data
• Accidents in network as they are being recorded
• Road closures (road work, etc)
• Still pictures from road cameras Real Time
GPS Real Time Real Time Real Time
Speed &
Data Transformation Geo Aggregates
Real Time Traffic Monitoring & Streams Logic Mapping
Heading
Estimation
& Statistics
Information
(Multimodal) Travel Planner
Storage
Interactive adapters
visualization

Data
Only 4 x86 Blade servers to process Web
Server
Warehouse

250,000 GPS probes per second, Google Offline

maps of 630,000 line segments Earth statistical
analysis

21 © 2012 IBM Corporation

 Real Time Geo Mapping & Speed Estimation

GPS probe
Matching map artifact
Estimated path
Estimated speed & heading

22 © 2012 IBM Corporation

The Grand Challenge: Analyze a Large Volume and Variety of
Streaming and Static Data to Produce Actionable Intelligence
Patterns of Life and Behavior
Modeling

Unmanned Correlation

System of Reference: Social, political

Weather, etc influences and
constraints on Observation Space

Find the Relevant Dots

Connect Them Activity Detection and Tracking
Tell Me What I don’t know
Keep it Up to Date
Entity Relationships and
Contextual Relevance

Historical data Anomaly Detection

Predictive Modeling and

Cognitive Awareness
23 © 2012 IBM Corporation
24 © 2012 IBM Corporation