3rd grade

Big Data Analytics

Dr. Nesma Mahmoud
Lecture 3: More on
Big Data Analytics
What will we learn in this lecture?

01. Intro to Big Data Analytics Lifecycle

02. Detailed Big Data Analytics Lifecycle with Case Study

03. Real Project Example

 Intro to Big Data
01. Analytics Lifecycle
Intro
● To address the distinct requirements for performing
analysis on Big Data, a step-by-step methodology is
needed to organize the activities and tasks involved with
acquiring, processing, analyzing and repurposing data.
Big Data Analytics Lifecycle can be …
Big Data Analytics Lifecycle can be …
Big Data Analytics Lifecycle can be …
Big Data Analytics Lifecycle can be …
● This is a specific data analytics
lifecycle that organizes and manages
the tasks and activities associated
with the analysis of Big Data.
 Detailed Big Data
01. Analytics Lifecycle with
Case Study
ETI Insurance Company (Case Study)
● ETI’s Big Data journey has reached the stage where its IT team
possesses the necessary skills and the management is convinced of the
potential benefits that a Big Data solution can bring in support of the
business goals.

● The CEO and the directors are eager to )‫ (متشوق لي‬see Big Data in action.
○ In response to this, the IT team, in partnership with the business
personnel, take on ETI’s first Big Data project.

○ After a thorough evaluation process, the “detection of fraudulent

claims” objective is chosen as the first Big Data solution.

● The team then follows a step-by-step approach as set forth by the Big
Data Analytics Lifecycle in pursuit of achieving this objective.
Stage 1: Business Case Evaluation
● Also called “define the problem”

● Each Big Data analytics lifecycle must begin

with a well-defined business case that
presents a clear understanding of the
justification, motivation and goals of
carrying out the analysis.

● The Business Case Evaluation stage

requires that a business case be created,
assessed and approved prior to proceeding
with the actual hands-on analysis tasks
Stage 1: Business Case Evaluation
● Based on business requirements that are
documented in the business case, it can be
determined whether the business problems being
addressed are really Big Data problems.
○ In order to qualify as a Big Data problem, a
business problem needs to be directly related
to one or more of the Big Data characteristics
of volume, velocity, or variety
● Another outcome of this stage is the determination
of the underlying budget required to carry out the
analysis project.
○ Any required purchase, such as tools,
hardware and training, must be understood
in advance(‫ مقدما‬- ‫ )في األول‬so that the
anticipated investment can be weighed
against the expected benefits of achieving
the goals.
Stage 1: Business Case Evaluation (Case Study)
● Carrying out Big Data analysis for the “detection of fraudulent claims”
○ corresponds to a decrease in monetary loss) ‫ (خسارة مالية‬and hence
carries complete business backing.

● For keeping the analysis somewhat straightforward)‫(مباشره‬, the scope of

Big Data analysis is limited to identification of fraud in the building
sector.
○ ETI provides building and contents insurance)‫ (تأمين‬to both domestic
and commercial ( ‫)محلي و تجاري‬customers.

● To measure the success of the Big Data solution for fraud detection, one
of the KPIs set is the reduction in fraudulent claims by 15%.
Stage 1: Business Case Evaluation (Case Study)
● Taking their budget into account)‫(في الحسبان – في عين االعتبار‬,
○ the team decides that their largest expense will be in the procuring)‫ الحصول على‬- ‫(التوريد‬
○ of new infrastructure that is appropriate for building a Big Data solution environment.
○ They realize that they will be leveraging open source technologies to support batch
processing and therefore do not believe that a large, initial up-front investment is
required for tooling.
○ However, when they consider the broader Big Data analytics lifecycle, the team
members realize that they should budget for the acquisition of additional data
quality and cleansing tools and newer data visualization technologies.
○ After accounting for these expenses, a cost-benefit analysis reveals that the
investment in the Big Data solution can return itself several times over if the targeted
fraud-detecting KPIs can be attained.
○ As a result of this analysis, the team believes that a strong business case exists for
using Big Data for enhanced data analysis.
Stage 2: Data Identification
● This stage is dedicated to identifying the
datasets required for the analysis project
and their sources.

● Identifying a wider variety of data sources

may increase the probability of finding
hidden patterns and correlations.

● Depending on the business scope of the

analysis project and nature of the business
problems being addressed, the required
datasets and their sources can be internal
and/or external to the enterprise.
Stage 2: Data Identification
● In the case of internal datasets,
○ a list of available datasets from internal
sources, such as data marts and operational
systems, are typically compiled and matched
against a predefined dataset specification.

● In the case of external datasets,

○ a list of possible third-party data providers,
such as data markets and publicly available
datasets, are compiled.
○ Some forms of external data may be
embedded within blogs or other types of
content-based web sites, in which case they
may need to be harvested)‫ (الحصول عليها‬via
automated tools.
Stage 2: Data Identification (Case Study)
A number of internal and external datasets are identified.

Internal data includes External data includes

• policy data, • social media data (Twitter feeds),
• insurance application documents, • weather reports,
• claim data, • geographical (GIS) data
• claim adjuster notes, • and census data)‫(بيانات التعداد السكاني‬.
• incident photographs,
• call center agent notes and emails.

• Nearly all datasets go back(Cover past) five years in time.

• The claim data consists of historical claim data(historical records) consisting of
multiple fields where one of the fields specifies if the claim was fraudulent or
legitimate.). ‫(احتيالية أو شرعية‬
Stage 3: Data Acquisition and Filtering
● In this stage, the data is gathered from all of the
data sources that were identified during the
previous stage.

● The acquired data is then subjected to automated

filtering for the removal of corrupt data or data that
has been deemed to have)‫ (يعتبر أن لديه‬no value to the
analysis objectives.

● Depending on the type of data source, data may

come as a collection of files, such as data
purchased from a third-party data provider, or
may require API integration, such as with Twitter.
Stage 3: Data Acquisition and Filtering
● In many cases, especially where external,
unstructured data is concerned, some or most of
the acquired data may be irrelevant (noise) and
can be discarded as part of the filtering process.

● Data classified as “corrupt” can include records

with missing or nonsensical values)‫ (قيم ال معنى لها‬or
invalid data types.

● Data that is filtered out for one analysis may

possibly be valuable for a different type of
analysis.
○ Therefore, it is advisable to store a verbatim copy of
the original dataset before proceeding with the
filtering. To minimize the required storage space, the
verbatim ) ‫(حرفي‬copy can be compressed.
Stage 3: Data Acquisition and Filtering
● Both internal and external data needs to be
persisted(stored) once it gets generated or enters
the enterprise )‫(المؤسسة‬boundary.
○ For batch analytics, this data is persisted to
disk prior to analysis.
○ For real-time analytics, the data is analyzed
first and then persisted to disk.
● As evidenced in this Figure, metadata can be
added via automation to data from both internal
and external data sources to improve the
classification and querying.
Examples of appended metadata include dataset size and structure, source information,
date and time of creation or collection and language-specific information. It is vital that
metadata be machine-readable and passed forward along subsequent analysis stages. This
helps maintain data provenance)‫ (المنشأ‬throughout the Big Data analytics lifecycle, which helps
to establish and preserve )‫ (يحفظ‬data accuracy and quality.
Stage 3: Data Acquisition and Filtering (Case Study)

Data Its Sources

policy data policy administration system
claim data, incident photographs
claims management system
and claim adjuster notes
insurance application documents document management system
currently embedded within the
The claim adjuster notes claim data → a separate process is
used to extract them
Call center agent notes and emails CRM system
rest of the datasets third-party data providers
 Stage 3: Data Acquisition and Filtering (Case Study)
● A compressed copy of the original version of all of the datasets is stored
on-disk.

● From a provenance perspective )‫ (من حيث المنشأ‬, the following metadata is

tracked to capture the pedigree of each dataset: dataset’s name, source,
size, format, checksum, acquired date and number of records.

● A quick check of the data qualities of Twitter feeds and weather reports
suggests that around four to five percent of their records are corrupt.
○ Consequently, two batch data filtering jobs are established to remove
the corrupt records.
Stage 4: Data Extraction
● Some of the data identified as input for the
analysis may arrive in a format incompatible
with the Big Data solution.

● The need to address disparate )‫ (مختلفة‬types

of data is more likely with data from external
sources.

● The Data Extraction lifecycle stage is

dedicated to extracting disparate data and
transforming it into a format that the
underlying Big Data solution can use for the
purpose of the data analysis.
Stage 4: Data Extraction
● The extent ) ‫(مدى‬of extraction and transformation
required depends on the types of analytics and
capabilities of the Big Data solution.
○ For example, extracting the required fields
from delimited textual data, such as with
webserver log files, may not be necessary if
the underlying Big Data solution can already
directly process those files.
○ Similarly, extracting text for text analytics,
which requires scans of whole documents,
is simplified if the underlying Big Data
solution can directly read the document in its
native format
 Stage 4: Data Extraction

It demonstrates the extraction of the latitude and

longitude coordinates of a user from a single JSON
It illustrates the extraction of comments and a field.
user ID embedded within an XML document
without the need for further transformation.

Further transformation is needed in order to separate the data into two separate fields as required
by the Big Data solution.
Stage 4: Data Extraction (Case Study)
● The IT team observes that some of the datasets will need to be pre-
processed in order to extract the required fields.
● For example,
○ the tweets dataset is in JSON format.
■ In order to be able to analyze the tweets, the user id,
timestamp and the tweet text need to be extracted and
converted to tabular form.
○ the weather dataset arrives in a hierarchical format (XML),
■ fields such as timestamp, temperature forecast, wind speed
forecast, wind direction forecast, snow forecast and flood
forecast are also extracted and saved in a tabular form.
Stage 5: Data Validation and Cleansing
● It is dedicated to establishing often complex
validation rules and removing any known
invalid data.
● Invalid data can skew and falsify analysis
results.
● Unlike traditional enterprise data, where the
data structure is pre-defined and data is
pre-validated, data input into Big Data
analyses can be unstructured without any
indication of validity.
○ Its complexity can further make it
difficult to arrive at a set of suitable
validation constraints.
Stage 5: Data Validation and Cleansing
● Big Data solutions often receive redundant data
across different datasets.
○ This redundancy can be exploited to explore
interconnected datasets in order to
assemble validation parameters and fill in
missing valid data.
○ For example, as illustrated in this Figure:
■ • The first value in Dataset B is
validated against its corresponding
value in Dataset A.
■ • The second value in Dataset B is not
validated against its corresponding
value in Dataset A.
■ • If a value is missing, it is inserted
from Dataset A.
Stage 5: Data Validation and Cleansing
● For batch analytics,
○ data validation and cleansing can be
achieved via an offline ETL operation.
● For real-time analytics,
○ a more complex in-memory system is
required to validate and cleanse the
data as it arrives from the source.

● Provenance can play an important role in

determining the accuracy and quality of
questionable data.
Stage 5: Data Validation and Cleansing (Case Study)
● To keep costs down,
○ ETI is currently using free versions of the weather and the census
datasets that are not guaranteed to be 100% accurate.
■ As a result, these datasets need to be validated and cleansed.
○ Based on the published field information,
■ the team is able to check the extracted fields for typographical
errors and any incorrect data as well as data type and range
validation.
■ A rule is established that a record will not be removed if it
contains some meaningful level of information even though
some of its fields may contain invalid data.
Stage 6: Data Aggregation and Representation
● This stage is dedicated to integrating
multiple datasets together to arrive at a
unified view.
● Data may be spread across multiple
datasets, requiring that datasets be joined
together via common fields, for example
date or ID.
○ In other cases, the same data fields
may appear in multiple datasets, such
as date of birth.
● Either way, a method of data reconciliation
is required or the dataset representing the
correct value needs to be determined.
Stage 6: Data Aggregation and Representation
● Performing this stage can become complicated because of differences
in:
○ • Data Structure – Although the data format may be the same, the
data model may be different.
○ • Semantics – A value that is labeled differently in two different
datasets may mean the same thing, for example “surname” and
“last name.”

● The large volumes processed by Big Data solutions can make data
aggregation a time and effort-intensive operation.

● Reconciling these differences can require complex logic that is

executed automatically without the need for human intervention.
Stage 6: Data Aggregation and Representation
● A data structure standardized by the Big Data solution can act as a common
denominator that can be used for a range of analysis techniques and projects.
○ This can require establishing a central, standard analysis repository, such as
a NoSQL database, as shown in the following Figure.

A simple example of data aggregation where two datasets are aggregated together using
the Id field.
Stage 6: Data Aggregation and Representation
● This Figure shows the same piece of data stored in two different formats.
○ Dataset A contains the desired piece of data, but it is part of a BLOB that is
not readily accessible for querying.
○ Dataset B contains the same piece of data organized in column-based
storage, enabling each field to be queried individually.

Dataset A and B can be combined to create a standardized data structure with a Big Data
solution.
Stage 6: Data Aggregation and Representation (Case
Study)
● For meaningful analysis of data,
○ it is decided to join together policy data, claim data and call center
agent notes in a single dataset that is tabular in nature where each
field can be referenced via a data query.
○ It is thought that this will not only help with the current data analysis
task of detecting fraudulent claims but will also help with other data
analysis tasks, such as risk evaluation and speedy settlement of
claims.
○ The resulting dataset is stored in a NoSQL database.
Stage 7: Data Analysis
● The Data Analysis stage is dedicated to carrying out the
actual analysis task, which typically involves one or more
types of analytics.
● This stage can be iterative in nature,
○ especially if the data analysis is predictive analytics,
in which case analysis is repeated until the
appropriate pattern or correlation is uncovered.
● Depending on the type of analytic result required,
○ this stage can be as simple as querying a dataset to
compute an aggregation for comparison.
○ On the other hand, it can be as challenging as
combining data mining and complex statistical
analysis techniques to discover patterns and
anomalies or to generate a statistical or
mathematical model to depict relationships between
variables.
Stage 7: Data Analysis (Case Study)
● The IT team involves the data analysts at this stage as it does not have the right
skillset for analyzing data in support of detecting fraudulent claims.
● In order to be able to detect fraudulent transactions,
○ first the nature of fraudulent claims needs to be analyzed in order to find
which characteristics differentiate a fraudulent claim from a legitimate claim.
○ For this, the predictive data analysis approach is taken. As part of this
analysis, a range of analysis techniques are applied.
● This stage is repeated a number of times as the results generated after the first
pass are not conclusive enough to comprehend what makes a fraudulent claim
different from a legitimate claim.
● As part of this exercise, attributes that are less indicative of a fraudulent claim are
dropped while attributes that carry a direct relationship are kept or added.
Stage 8: Data Visualization
● The ability to analyze massive amounts of data and
find useful insights carries little value if the only
ones that can interpret the results are the analysts.
● The Data Visualization stage is dedicated to using
data visualization techniques and tools to
graphically communicate the analysis results for
effective interpretation by business users.
● The results of completing the Data Visualization
stage provide users with the ability to perform
visual analysis, allowing for the discovery of
answers to questions that users have not yet even
formulated.
Stage 8: Data Visualization
Stage 8: Data Visualization (Case Study)
● The team has discovered some interesting findings and now needs to
convey the results to the actuaries, underwriters and claim adjusters.

● Different visualization methods are used including bar and line graphs
and scatter plots.
○ Scatter plots are used to analyze groups of fraudulent and
legitimate claims in the light of different factors, such as customer
age, age of policy, number of claims made and value of claim.
Stage 8: Data Visualization (Case Study)
Stage 9: Utilization of Analysis Results
● Subsequent to analysis results being made available to
business users to support business decision-making, such
as via dashboards, there may be further opportunities to
utilize the analysis results.
● This stage is dedicated to determining how and where
processed analysis data can be further leveraged.
● Depending on the nature of the analysis problems being
addressed, it is possible for the analysis results to produce
“models” that encapsulate new insights and understandings
about the nature of the patterns and relationships that exist
within the data that was analyzed.
○ A model may look like a mathematical equation or a set of rules.
○ Models can be used to improve business process logic and
application system logic, and they can form the basis of a new
system or software program.
Stage 9: Utilization of Analysis Results (Case Study)
● Based on the data analysis results, the underwriting and the claims
settlement users have now developed an understanding of the nature of
fraudulent claims.

● However, in order to realize tangible benefits from this data analysis

exercise, a model based on a machine-learning technique is generated,
which is then incorporated into the existing claim processing system to
flag fraudulent claims.
03. Real Project Example
Real Project From Kaggle
● Insurance Claims - Fraud Detection

○ Steps: https://medium.com/@sauravkarki10.12/insurance-claim-
fraud-detection-project-c700e31c7602
○ https://www.kaggle.com/datasets/mykeysid10/insurance-claims-
fraud-detection/data
 Thanks!
Do you have any questions?

CREDITS: This presentation template was created by Slidesgo, and includes

icons by Flaticon, and infographics & images by Freepik