Data Warehousing & Mining

MEDIA
mathematics
HEALTH ENGINEERING
DESIGN
management
EDUCATION
MUSIC
GEOGRAPHY E
ART
C PHYSICS
O
law
L
O
agriculture
BIOTECHNOLOGY G
Y
LANGUAGE
CHEMISTRY history
MECHANICS
psychology
DATA WAREHOUSING AND DATA

MINING
Subject: DATA WAREHOUSING AND DATA MINING Credits: 4
SYLLABUS
Basic Concepts of Data Warehousing

Introduction, Meaning and characteristics of Data Warehousing, Online Transaction Processing (OLTP), Data
Warehousing Models, Data warehouse architecture & Principles of Data Warehousing Data Mining.
Building a Data Warehouse Project

Structure of the Data warehouse, Data warehousing and Operational Systems, Organizing for building data
warehousing, Important considerations – Tighter integration, Empowerment, Willingness Business
Considerations: Return on Investment Design Considerations, Technical Consideration, Implementation
Consideration, Benefits of Data warehousing.
Managing and Implementing a Data Warehouse Project

Project Management Process, Scope Statement, Work Breakdown Structure and Integration, Initiating a data
warehousing project Project Estimation, Analyzing Probability and Risk, Managing Risk: Internal and External,
Critical Path Analysis.
Data Mining
What is Data mining (DM)? Definition and description, Relationship and Patterns, KDD vs Data mining,
DBMS vs Data mining, Elements and uses of Data Mining, Measuring Data Mining Effectiveness :
Accuracy,Speed & Cost Data Information and Knowledge, Data Mining vs. Machine Learning, Data Mining
Models. Issues and challenges in DM, DM Applications Areas.
Techniques of Data Mining

Various Techniques of Data Mining Nearest Neighbour and Clustering Techniques, Decision Trees, Discovery
of Association Rules, Neural Networks, Genetic Algorithm.
OLAP
Need for OLAP, OLAP vs. OLTP Multidimensional Data Model Multidimensional verses Multirelational
OLAP Characteristics of OLAP: FASMI Test (Fast, Analysis Share, Multidimensional and Information),
Features of OLAP, OLAP Operations Categorization of OLAP Tools: MOLAP, ROLAP
Suggested Readings:
1. Pieter Adriaans, Dolf Zantinge Data Mining, Pearson Education

2. George M. Marakas Modern Data Warehousing, Mining, and Visualization: Core Concepts, Prentice
Hall, 1st edition
3. Alex Berson, Stephen J. Smith Data Warehousing, Data Mining, and OLAP (Data Warehousing/Data
Management), McGraw-Hill
4. Margaret H. Dunham Data Mining, Prentice Hall, 1st edition,
5. David J. Hand Principles of Data Mining (Adaptive Computation and Machine Learning), Prentice Hall,
1st edition
6. Jiawei Han, Micheline Kamber Data Mining, Prentice Hall, 1st edition
7. Michael J. Corey, Michael Abbey, Ben Taub, Ian Abramson Oracle 8i Data Warehousing McGraw-Hill
Osborne Media, 2nd edition
DATA WAREHOUSING AND DATA MINING
MCA
COURSE OVERVIEW
The last few years have seen a growing recognition of informa- sional verses Multirelational OLAP, OLAP Operations and
tion as a key business tool. In general, the current business Categorization of OLAP Tools: MOLAP and ROLAP.
market dynamics make it abundantly clear that, for any com- Armed with the knowledge of data warehousing technology,
pany, information is the very key to survival. the student continues into a discussion on the principles of
If we look at the evolution of the information processing business analysis, models and patterns and an in-depth analysis
technologies, we can see that while the first generation of client/ of data mining.
server systems brought data to the desktop, not all of this data
Prerequisite
was easy to understand, unfortunately, and as such, it was not
Knowledge of Database Management Systems
very useful to end users. As a result, a number of new tech-
nologies have emerged that are focused on improving the Objective
information content of the data to empower the knowledge Ever since the dawn of business data processing, managers
workers of today and tomorrow. Among these technologies are have been seeking ways to increase the utility of their informa-
data warehousing, online analytical processing (OLAP), and data tion systems. In the past, much of the emphasis has been on
mining. automating the transactions that move an organization through
Therefore, this book is about the need, the value and the the interlocking cycles of sales, production and administration.
technological means of acquiring and using information in the Whether accepting an order, purchasing raw materials, or paying
information age. employees, most organizations process an enormous number

of transactions and in so doing gather an even larger amount
From that perspective, this book is intended to become the
of data about their business.
handbook and guide for anybody who’s interested in planning,
or working on data warehousing and related issues. Meaning Despite all the data they have accumulated, what users really
and characteristics of Data Warehousing, Data Warehousing want is information. In conjunction with the increased amount
Models, Data warehouse architecture & Principles of Data of data, there has been a shift in the primary users of comput-
Warehousing, topics related to building a data warehouse ers, from a limited group of information systems professionals
project are discussed along with Managing and implementing a to a much larger group of knowledge workers with expertise in
data warehouse project. Using these topics as a foundation, this particular business domains, such as finance, marketing, or
book proceeds to analyze various important concepts related to manufacturing. Data warehousing is a collection of technologies
Data mining, Techniques of data mining, Need for OLAP, designed to convert heaps of data to usable information. It
OLAP vs. OLTP, Multidimensional data model, Multidimen-
i
does this by consolidating data from diverse transactional The newest, hottest technology to address these concerns is data
systems into a coherent collection of consistent, quality-checked mining. Data mining uses sophisticated statistical analysis and
databases used only for informational purposes. Data ware- modeling techniques to uncover pattern and relationships
houses are used to support online analytical processing (OLAP). hidden in organizational databases – patterns that ordinary
However, the very size and complexity of data warehouses methods might miss.
make it difficult for any user, no matter how knowledgeable in The objective of this book is to have detailed information
the application of data, to formulate all possible hypotheses about Data warehousing, OLAP and data mining. I have
that might explain something such as the behavior of a group brought together these different pieces of data warehousing,
of customers. How can anyone successfully explore databases OLAP and data mining and have provided an understandable
containing 100 millions rows of data, each with thousands of and coherent explanation of how data warehousing as well as
attributes? data mining works, plus how it can be used from the business
perspective. This book will be a useful guide.
ii

MCA
CONTENT
. Lesson No. Topic Page No.
Lesson Plan vi
Data Warehousing
Lesson 1 Introduction to Data Warehousing 1
Lesson 2 Meaning and Characteristics of Data Warehousing 5
Lesson 3 OnLine Transaction Processing 9

Lesson 4 Data warehousing Models 13
Lesson 5 Architecture and Principles of Data warehousing 19
Building a Data Warehouse Project
Lesson 6 Data warehousing and Operational Systems 25
Lesson 7 Building Data Warehousing, Important Considerations 32
Lesson 8 Building Data Warehousing - 2 36

Lesson 9 Business Considerations: Return on Investment
Design Considerations 39
Lesson 10 Technical Consideration, Implementation Consideration 42

Lesson 11 Benefits of Data Warehousing 45
Managing and Implementing a Data Warehouse Project
Lesson 12 Project Management Process, Scope Statement 48
Lesson 13 Work Breakdown Structure 52
Lesson 14 Project Estimation, analyzing Probability and Risk 55
Lesson 15 Managing Risk: Internal and External, Critical Path Analysis 59
Data Mining
Lesson 16 Data Mining Concepts 63
Lesson 17 Data Mining Concepts-2 67
Lesson 18 Elements and uses of Data Mining 73
Lesson 19 Data Information and Knowledge 78
Lesson 20 Data Mining Models 82
Lesson 21 Issues and challenges in DM, DM Applications Areas 87
Data Mining Techniques
Lesson 22 Various Techniques of Data Mining Nearest Neighbor
and Clustering Techniques 93
Lesson 23 Decision Trees 98
iv
MCA
CONTENT
Lesson No. Topic Page No.
Lesson 24 Decision Trees - 2 103
Lesson 25 Neural Networks 107
Lesson 26 Neural Networks 112
Lesson 27 Association Rules and Genetic Algorithm 118
OLAP
Lesson 28 Online Analytical Processing, Need for OLAP
Multidimensional Data Model 124
Lesson 29 OLAP vs. OLTP, Characteristics of OLAP 129
Lesson 30 Multidimensional verses Multirelational OLAP,
Features of OLAP 132
Lesson 31 OLAP Operations 136
Lesson 32 Categorization of OLAP Tools Concepts used
in MOLAP/ ROLAP 141
v
CHAPTER 1
LESSON 1 DATA WAREHOUSING
INTRODUCTION TO
DATA WAREHOUSING
Structure communication requirements it is possible to incorporate

• Objective additional or expert information it is.
• Introduction The logical link between what the managers see in their decision
Support EIS application and the company’s operational
• Meaning of Data warehousing
activities Johan McIntyre of SAS institute Inc.
• History of Data warehousing
In other words the data warehouse provides warehouse
• Traditional Approaches To Historical Data provides data that is already transformed and summarized,
• Data from legacy systems therefore making it an appropriate environment for the more
• Extracted information on the Desktop efficient DSS and EIS applications.
• Factors, which Lead To Data Warehousing A data warehouse is a collection of corporate information,
derived directly from operational system and some
Objective external data sources.
The main objective of this lesson is to introduce you with the
basic concept and terminology relating to Data Warehousing. Its specific purpose is to support business decisions, not
By the end of this lesson you will be able to understand: business ask “What if?” questions. The answer to these
questions will ensure your business is proactive, instead of
• Meaning of a Data warehouse reactive, a necessity in today’s information ago.
• Evolution of Data warehouse The industry trend today is moving towards more powerful
Introduction hardware and software configuration, we now have the ability to
Traditionally, business organizations create billions of bytes of process vast volumes of information analytically, which would
data about all aspects of business everyday, which contain have been unheard of tenor even five years ago. A business
millions of individual facts about their customers, products, today must we able to use this emerging technology or run the
operations, and people. However, this data is locked up and is risk if being information under loaded. As you read that
extremely difficult to get at. Only a small fraction of the data correctly - under loaded - the opposite of over loaded. Over-
that is captured, processed, and stored in the enterprise is loaded means you are so determine what is important. If you
actually available to executives and decision makers. are under loaded, you are information deficient. You cannot
Recently, new concepts and tools have evolved into a new cope with decision – making expectation because you do not
technology that make it possible to provide all the key people know where you stand. You are missing critical pieces of
within the enterprise with access to whatever level of informa- information required to make informed decisions.
tion needed for the enterprise to survive and prosper in an To illustrate the danger of being information under loaded,
increasingly competitive world. The term that is used for this consider the children’s story of the country mouse is unable to
new technology is “data warehousing”. In this unit I will be cope with and environment its does not understand.
discussing about the basic concept and terminology relating to What is a cat? Is it friend or foe?
Data Warehousing.
Why is the chess in the middle of the floor on the top of a
The Lotus was your first test of “What if “processing on the platform with a spring mechanism?
Desktop. This is what a data warehouse is all about using
Sensory deprivation and information overload set in. The
information your business has gathered to help it react better,
picture set country mouse cowering in the corner. If is stays
smarter, quicker and more efficiently.
there, it will shrivel up and die. The same fate awaits the
Meaning of Data Warehousing business that does not respond to or understand the environ-
Data warehouse potential can be magnify if the appropriate data ment around it. The competition will moves in like cultures and
has been collected and stored in a data warehouse. A data exploit all like weaknesses.
warehouse is a relational database management system In today’s world, you do not want to be the country mouse. In
(RDBMS) designed specifically to meet the needs of transaction today’s world, full of vast amounts of unfiltered information, a
processing system. It can be loosely defined as any centralized business that does not effectively use technology to shift
data repository, which can be queried for business benefit, but through that information will not survive the information age.
this will be more clearly defined letter. Data warehouse is a new Access to, and the understating of, information is power. This
powerful technique making. It possible to extract archived power equate to a competitive advantage and survival. This unit
operational data and over come inconsistencies between will discuss building own data warehouse-a repository for
different legacy data formats, as well as integrating data through- storing information your business needs to use if it hopes to
out an enterprise, regardless of location, format, or survive and thrive in the information age. We will help you
1
understand what a data warehouse is and what it is not. You the disadvantages faced it led to the development of the new
will learn what human resources are required, as well as the roles application called Data Warehousing
and responsibilities of each player. You will be given an
Factors, which Lead To Data Warehousing
overview of good project management techniques to help
Many factors have influenced the quick evolution of the data
ensure the data warehouse initiative dose not fail due the poor
warehousing discipline. The most important factor has been the
project management. You will learn how to physically imple-
advancement in the hardware and software technologies.
ments a data warehouse with some new tools currently available
to help you mine those vast amounts of information stored Hardware and Software prices: Software and hardware prices
with in the warehouse. Without fine running this ability to have fallen to a great extent. Higher capacity memory chips are
mine the warehouse, even the most complete warehouse, available at very low prices.
would be useless. • Powerful Preprocessors: Today’s preprocessor are many
times powerful than yesterday’s mainframes: e.g. Pentium III
History of Data Warehousing
and Alpha processors
Let us first review the historical management schemes of the
analysis data and the factors that have led to the evolution of • Inexpensive disks: The hard disks of today can store
the data warehousing application class. hundreds of gigabytes with their prices falling. The amount
of information that can be stored on just a single one-inch
Traditional Approaches to Historical Data high disk drive would have required a roomful of disk drives
Throughout the history of systems development, the primary in 1970’s and early eighties.
emphasis had been given to the operational systems and the
data they process. It was not practical to keep data in the
• Desktop powerful for analysis tools: Easy to use GUI
interfaces, client/server architecture or multi-tier computing
operational systems indefinitely; and only as an afterthought
can be done on the desktops as opposed to the mainframe
was a structure designed for archiving the data that the opera-
computers of yesterday.
tional system has processed. The fundamental requirements of
the operational and analysis systems are different: the opera- • Server software: Server software is inexpensive, powerful,
tional systems need performance, whereas the analysis systems and easy to maintain as compared to that of the past.
need flexibility and broad scope. Example of this is Windows NT that have made setup of
powerful systems very easy as well as reduced the cost.
Data from Legacy Systems
Different platforms have been developed with the development The skyrocketing power of hardware and software, along with
of the computer systems over past three decades. In the 1970’s, the availability of affordable and easy-to-use reporting and
business system development was done on the IBM mainframe analysis tools have played the most important role in evolution
computers using tools such as Cobol, CICS, IMS, DB2, etc. of data warehouses.
With the advent of 1980’s computer platforms such as AS/400 Emergence of Standard Business
and VAX/VMS were developed. In late eighties and early Applications
nineties UNIX had become a popular server platform introduc- New vendors provide to end-users with popular business
ing the client/server architecture which remains popular till date. application suites. German software vendor SAP AG, Baan,
Despite all the changes in the platforms, architectures, tools, and PeopleSoft, and Oracle have come out with suites of software
technologies, a large number of business applications continue that provide different strengths but have comparable function-
to run in the mainframe environment of the 1970’s. The most ality. These application suites provide standard applications that
important reason is that over the years these systems have can replace the existing custom developed legacy applications.
captured the business knowledge and rules that are incredibly This has led to the increase in popularity of such applications.
difficult to carry to a new platform or application. These systems Also, the data acquisition from these applications is much
are, generically called legacy systems. The data stored in such simpler than the mainframes.
systems ultimately becomes remote and becomes difficult to get End-user more Technology Oriented
at. One of the most important results of the massive investment
Extracted Information on the Desktop in technology and movement towards the powerful personal
During the past decade the personal computer has become very computer has been the evolution of a technology-oriented
popular for business analysis. Business Analysts now have business analyst. Even though the technology-oriented end
many of the tools required to use spreadsheets for analysis and users are not always beneficial to all projects, this trend certainly
graphic representation. Advanced users will frequently use has produced a crop of technology-leading business analysts
desktop database programs to store and work with the that are becoming essential to today’s business. These technol-
information extracted from the legacy sources. ogy-oriented end users have frequently played an important role
in the development and deployment of data warehouses. They
The disadvantage of the above is that it leaves the data frag-
have become the core users that are first to demonstrate the
mented and oriented towards very specific needs. Each
initial benefits of data warehouses. These end users are also
individual user has obtained only the information that she/he
critical to the development of the data warehouse model: as
requires. The extracts are unable to address the requirements of
they become experts with the data warehousing system, they
multiple users and uses. The time and cost involved in
train other users.
addressing the requirements of only one user are large. Due to
2
Discussions

• Write short notes on:
• Legacy systems
• Data warehouse
• Standard Business Applications
• What is a Data warehouse? How does it differ from a
database?
• Discuss various factors, which lead to Data Warehousing.
• Briefly discuss the history behind Data warehouse.
References
1. Adriaans, Pieter, Data mining, Delhi: Pearson Education
Asia, 1996.
2. Anahory, Sam, Data warehousing in the real world: a practical
guide for building decision support systems, Delhi: Pearson
Education Asia, 1997.
3. Berry, Michael J.A. ; Linoff, Gordon, Mastering data mining
: the art and science of customer relationship management,
New York : John Wiley & Sons, 2000
4. Corey, Michael, Oracle8 data warehousing, New Delhi: Tata
McGraw- Hill Publishing, 1998.
5. Elmasri, Ramez, Fundamentals of database systems, 3rd ed.
Delhi: Pearson Education Asia, 2000.
3
Notes
4
CHAPTER 1: DATA WAREHOUSING

LESSON 2
MEANING AND CHARACTERISTICS OF DATA WAREHOUSING
Structure Data warehousing evolved with the integration of a number of

• Objective different technologies and experiences over the last two decades,
• Introduction which have led to the identification of key problems.
• Data warehousing Data Warehousing

Because data warehouses have been developed in numerous
• Operational vs. Informational Systems
organizations to meet partic-ular needs, there is no single,
• Characteristics of Data warehousing canonical definition of the term data warehouse.1 Profes-sional
• Subject oriented magazine articles and books in the popular press have elabo-
• Integrated rated on the meaning in a variety of ways. Vendors have
capitalized on the popularity of the term to help mar-ket a
• Time variant
variety of related products, and consultants have provided a
• Non-volatiles large variety of services, all under the data-warehousing banner.
Objective However, data warehouses are quite distinct from traditional
The objective of this lesson is to explain you the significance databases in their structure, functioning, performance, and
and difference between Operational systems and Informational purpose.
systems. This lesson also includes various characteristics of a Operational vs. Informational Systems
Data warehouse. Perhaps the most important concepts that has come out of the
Introduction Data Warehouse movement is the recognition that there are two
In the previous section, we have discussed about the need of fundamentally different types of information systems in all
data warehousing and the factors that lead to it. In this section I organizations: operational systems and informational systems.
will explore the technical concepts relating to data warehousing “Operational systems” are just what their name implies, they are
to you. the systems that help us run the enterprise operate day-to-day.
These are the backbone systems of any enterprise, our “order
A company can have data items that are unrelated to each other.
entry’, “inventory”, “manufacturing”, “payroll” and “account-
Data warehousing is the process of collecting together such data
ing” systems. Because of their importance to the organization,
items within a company and placing it in an integrated data
operational systems were almost always the first parts of the
store. This integration is over time, geographies, and application
enterprise to be computerized. Over the years, these operational
platforms. By adding access methods (on-line querying,
systems have been extended and rewritten, enhanced and
reporting), this converts a ‘dead’ data store into a ‘dynamic’
maintained to the point that they are completely integrated into
source of information. In other words, turning a liability into
the organization. Indeed, most large organizations around the
an asset. Some of the definitions of data warehousing are:
world today couldn’t operate without their operational systems
“A data warehouse is a single, complete, and consistent store of and that data that these systems maintain.
data obtained from a variety of sources and made available to
end users in a way they can understand and use in a business On the other hand, there are other functions that go on within
context.” (Devlin 1997) the enterprise that have to do with planning, forecasting and
managing the organization. These functions are also critical to
“Data warehousing is a process, not a product, for assembling the survival of the organization, especially in our current fast
and managing data from various sources for the purpose of paced world. Functions like “marketing planning”, “engineering
gaining a single, detailed view of part or all of the business.” planning” and “financial analysis” also require information
(Gardner 1998) systems to support them. But these functions are different
A Data Warehouse is a capability that provides comprehensive from operational ones, and the types of systems and informa-
and high integrity data in forms suitable for decision support to tion required are also different. The knowledge-based functions
end users and decision makers throughout the organization. A are informational systems.
data warehouse is managed data situated after and outside the
operational systems. A complete definition requires discussion “Informational systems” have to do with analyzing data and
of many key attributes of a data warehouse system Data making decisions, often major decisions about how the
Warehousing has been the result of the repeated attempts of enterprise will operate, now and in the future. And not only do
various researchers and organizations to provide their organiza- informational systems have a different focus from operational
tions flexible, effective and efficient means of getting at the ones, they often have a different scope. Where operational data
valuable sets of data. needs are normally focused upon a single area, informational
data needs often span a number of different areas and need
large amounts of related operational data.
5
In the last few years, Data Warehousing has grown rapidly from subject contain only the information necessary for decision
a set of related ideas into architecture for data delivery for support processing.
enterprise end user computing.
Integrated
They support high-performance demands on an organization’s When data resides in money separate applications in the
data and information. Several types of applications-OLAP, DSS, operational environment, encoding of data is often inconsis-
and data mining applications-are supported. OLAP (on-line tent. For instance in one application, gender might be coded as
analytical processing) is a term used to describe the analysis of “m” and “f ” in another by o and l. When data are moved from
complex data from the data warehouse. In the hands of skilled the operational environment in to the data warehouse, when
knowledge workers. OLAP tools use distributed computing data are moved from the operational environment in to the
capabilities for analyses that require more storage and processing data warehouse, they assume a consistent coding convention
power than can be economically and efficiently located on an e.g. gender data is transformed to “m” and “f ”.
individual desktop. DSS (Decision-Support Systems) also
known as EIS (Executive Information Systems, not to be
Time variant
The data warehouse contains a place for storing data that are five
confused with enterprise integration systems) support an
to ten years old, or older, to be used for comparisons, trends,
organization’s leading deci-sion makers with higher-level data
and forecasting. These data are not up dated.
for complex and important decisions. Data mining is used for
knowledge discovery, the pro-cess of searching data for Non-volatile
unanticipated new knowledge. Data are not update or changed in any way once they enter the
Traditional databases support On-Line Transaction Processing data warehouse, but are only loaded and accessed.
(OLTP), which includes insertions, updates, and deletions, Data warehouses have the following distinctive characteristics.
while also supporting information query requirements. • Multidimensional conceptual view.
Traditional relational databases are optimized to process queries
• Generic dimensionality.
that may touch a small part of the database and transactions
that deal with insertions or updates of a few tuples per relation • Unlimited dimensions and aggregation levels.
to process. Thus, they cannot be optimized for OLAP, DSS, or • Unrestricted cross-dimensional operations.
data mining. By contrast, data warehouses are designed precisely • Dynamic sparse matrix handling.
to support efficient extraction, process-ing, and presentation for
• Client-server architecture.
analytic and decision-making purposes. In comparison to tradi-
tional databases, data warehouses generally contain very large • Multi-user support.
amounts of data from multiple sources that may include • Accessibility.
databases from different data models and sometimes lies • Transparency.
acquired from independent systems and platforms. • Intuitive data manipulation.
A database is a collection of related data and a database system • Consistent reporting performance.
is a database and database software together. A data warehouse
• Flexible reporting
is also a collection of information as well as supporting system.
However, a clear distinction exists, Traditional databases are Because they encompass large volumes of data, data ware-
transactional: relational, object-oriented, network, or hierarchical. houses are generally an order of magnitude (sometimes two
Data warehouses have the distinguishing characteristic that they orders of magnitude) larger than the source databases. The
are mainly intended for decision-support applications. They are sheer volume of data (likely to be in terabytes) is an issue that
optimized for data retrieval, not routine transaction processing. has been dealt with through enterprise-wide data warehouses,
virtual data warehouses, and data marts:
Characteristics of Data Warehousing
As per W. H. Inmon, author of building the data warehouse
• Enterprise-wide data warehouses are huge projects requiring
massive investment of time and resources.
and the guru who is ready widely considered to be the origina-
tor of the data warehousing concept, there are generally four • Virtual data warehouses provide views of operational
character that describe a data warehouse: databases that are materialized for efficient access.
W. H. Inmon characterized a data warehouse as “a subject- • Data marts generally are targeted to a subset of the
oriented, integrated, nonvola-tile, time-variant collection of data organization, such as a dependent, and are more tightly
in support of management’s decisions.” Data ware-houses focused.
provide access to data for complex analysis, knowledge discov- To summarize the above, here are some important points to
ery, and decision-making. remember about various characteristics of a Data warehouse:
Subject Oriented • Subject-oriented
Data are organized according to subject instead of application • Organized around major subjects, such as customer,
e.g. an insurance company using a data warehouse would product, sales.
organize their data by costumer, premium, and claim, instead
of by different products (auto. Life etc.). The data organized by
6
• Focusing on the modeling and analysis of data for

decision making, not on daily operations or transaction
processing.
• Provide a simple and concise view around particular
subject by excluding data that are not useful in the
decision support process.
• Integrated
• Constructed by integrating multiple, heterogeneous
data sources as relational databases, flat files, on-line
transaction records.
• Providing data cleaning and data integration
techniques.
• Time variant
• The time horizon for the data warehouse is
significantly longer than that of operational systems.
• Every key structure in the data warehouse contains an
element of time (explicitly or implicitly).
• Non-volatile
• A physically separate store of data transformed from
the operational environment.
• Does not require transaction processing, recovery, and
concurrency control mechanisms.
• Requires only two operations in data accessing: initial
loading of data and access of data (no data updates).
Discussions
• Metadata
• Operational systems
• OLAP
• DSS
• Informational Systems
• What is the need of a Data warehouse in any
organization?
• Discuss various characteristics of a Data warehouse.
• Explain the difference between non-volatile and
Subject-oriented data warehouse.
References
Asia, 1996.
: the art and science of customer relationship management, New
York : John Wiley & Sons, 2000
7
8
LESSON 3
ONLINE TRANSACTION PROCESSING
Structure Similarities and Differences in OLTP and Data

• Objective Warehousing
• Introduction
• Data warehousing and OLTP systems
OLTP Data Warehouse
• Similarities and Differences in OLTP and Data Warehousing
Processes in Data Warehousing OLTP Run day-to-day Information retrieval
Purpose
operation and analysis
• What is OLAP?
• Who uses OLAP and WHY? Structure RDBMS RDBMS
• Multi-Dimensional Views Data Model Normalized Multi-dimensional
• Benefits of OLAP SQL plus data analysis
Access SQL
Objective extensions
Type of Data that run the Data that analyses the
The main objective of this lesson is to introduce you with
Data business business
Online Transaction Processing. You will learn about the
Condition Changing
importance and advantages of an OLTP system. Historical descriptive
of Data incomplete
Introduction
Relational databases are used in the areas of operations and
control with emphasis on transaction processing. Recently The data warehouse server a different purpose from that of
relational databases are used for building data warehouses, OLTP systems by allowing business analysis queries to be
which stores tactical information (<1year into the future) that answered as opposed to “simple aggregation” such as ‘what is
answers who and what questions. In contrast OLAP uses MD the current account balance for this customer?’ Typical data
views of aggregate data to provide access strategic information. warehouse queries include such things as ‘which product line
OLAP enables users to gain insight to a wide variety of possible sells best in middle America and how dose this correlate to
views of information and transforms raw data to reflect the demographic data?
enterprise as understood by the user e.g., Analysts, managers Processes in Data Warehousing OLTP
and executives. The first step in data warehousing is to “insulate” your current
Data Warehousing and OLTP Systems operational information, i.e. to preserve the security and
A data base which in built for on line transaction processing, integrity of mission- critical OLTP applications, while giving
OLTP, is generally regarded as inappropriate for warehousing as you access to the broadest possible base of data. The resulting
they have been designed with a different set of need in mind database or data warehouse may consume hundred of
i.e., maximizing transaction capacity and typically having gigabytes-or even terabytes of disk space, what is required than
hundreds of table in order not to look out user etc. Data are capable efficient techniques for storing and retrieving massive
warehouse are interested in query processing as opposed to amounts of information. Increasingly, large organizations have
transaction processing. found that only parallel processing systems offer sufficient
bandwidth.
OLTP systems cannot be receptacle stored of repositories of
facts and historical data for business analysis. They cannot be The data warehouse thus retrieves data from a varsity of
quickly answer adhoc queries is rapid retrieval is almost impos- heterogeneous operational database. The data is than trans-
sible. The data is inconsistent and changing, duplicate entries formed and delivered to the data warehouse/ store based in a
exist, entries can be missing and there is an absence of historical selected modal (or mapping definition). The data transforma-
data, which is necessary to analyses trends. Basically OLTP offers tion and movement processes are completed whenever an
large amounts of raw data, which is not easily understood. The update to the warehouse data is required so there should some
data warehouse offers the potential to retrieve and analysis from of automation to manage and carry out these functions.
information quickly and easily. Data warehouse do have The information that describes the modal metadata is the
similarities with OLTP as shown in the table below. means by which the end user finds and understands the data in
the warehouse and is an important part of the warehouse. The
metadata should at the very least contain:
• Structure of the data;
• Algorithm used for summarization;
9
• Mapping from the operational environment to the data • In contrast OLAP uses Multi-Dimensional (MD) views of
warehouse. aggregate data to provide access strategic information.

Data cleansing is an important viewpoint of creating an efficient • OLAP enables users to gain insight to a wide variety of
data warehouse of creating an efficient data warehouse in that is possible views of information and transforms raw data to
the removal of creation aspects Operational data such as low reflect the enterprise as understood by the user e.g. Analysts,
level transaction information which sloe down the query times. managers and executives.
The cleansing stage has to be as dynamic as possible to accom- • In addition to answering who and what questions OLAPs
modate all types of queries even those, which may require can answer “what if “ and “why”.
low-level information. Data should be extracted from produc-
• Thus OLAP enables strategic decision-making.
tion sources at regular interval and pooled centrally but the
cleansing process has to remove duplication and reconcile • OLAP calculations are more complex than simply summing
differences between various styles of data collection. data.
Once the data has been cleaned it is than transfer to the data • However, OLAP and Data Warehouses are complementary
warehouse, which typically is a large database on a high perfor- • The data warehouse stores and manages data while the
mance box, either SMP Symmetric Multi- Processing or MPP, OLAP transforms this data into strategic information.
Massively parallel Processing Number crunching power is Who uses OLAP and WHY?
another importance aspect of data warehousing because of the
complexity involved in processing adhoc queries and because of
• OLAP applications are used by a variety of the functions of
an organisation.
the vast quantities of data that the organization want to use in
the warehouse. A data warehouse can be used in different ways, • Finance and accounting:
for example it can be a central store against which the queries are Budgeting
run of it can be used like a data mart, data mart which are small Activity-based costing
warehouses can be established to provide subsets of the main
Financial performance analysis
store and summarized information depending on the require-
ments of a specific group/ department. The central stores And financial modelling
approach generally uses every simple data structures with very • Sales and Marketing
little assumptions about the relationships between data where Sales analysis and forecasting
as marts often uses multidimensional data base which can
Market research analysis
speed up query processing as they can have data structures which
reflect the most likely questions. Promotion analysis
Many vendors have products that provide on the more of the Customer analysis
above data warehouse functions. However, it can take a Market and customer segmentation
significant amount of work and specialized programming to • Production
provide the interoperability needed between products form.
Production planning
Multiple vendors to enable them to perform the required data
warehouse processes a typical implementation usually involves a Defect analysis
mixture of procedure forma verity of suppliers. Thus, OLAP must provide managers with the information they
Another approach to data warehousing is the Parsaye Sandwich need for effective decision-making. The KPI (key performance
paradigm put forward by Dr. Kamran Parsaye , CED of indicator) of an OLAP application is to provide just-in-time
information discovery, Hermosa beach. This paradigm or (JIT) information for effective decision-making. JIT informa-
philosophy encourages acceptance of the probability that the tion reflects complex data relationships and is calculated on the
first iteration of data warehousing effort will require consider- fly. Such an approach is only practical if the response times are
able revision. The Sandwich paradigm advocates the following always short The data model must be flexible and respond to
approach. changing business requirements as needed for effective decision
making.
• Pre-mine the data to determine what formats and data are
needed to support a data- mining application; In order to achieve this in widely divergent functional areas
OLAP applications all require:
• Build a prototype mini- data warehouse i.e. the, the meat of
sandwich most of features envisaged for the and product; MD views of data
• Revise the strategies as necessary; Complex calculation capabilities
• Build the final warehouse. Time intelligence
What is OLAP? Multi-Dimensional Views

• Relational databases are used in the areas of operations and • MD views inherently represent actual business models,
control with emphasis on transaction processing. which normally have more than three dimensions e.g., Sales
data is looked at by product, geography, channel and time.
• Recently relational databases are used for building data
warehouses, which stores tactical information (< 1 year into
the future) that answers who and what questions.
10
• MD views provide the foundation for analytical processing • Identify various benefits of OLTP.

through flexible access to information. • “OLAP enables organisations as a whole to respond more
• MD views must be able to analyse data across any dimension quickly to market demands, which often results in increased
at any level of aggregation with equal functionality and ease revenue and profitability”. Comment.
and insulate users from the complex query syntax • Who are the primary users of Online Transaction Processing
• What ever the query is they must have consistent response System?
times. • “The KPI (key performance indicator) of an OLAP
• Users queries should not be inhibited by the complex to application is to provide just-in-time (JIT) information for
form a query or receive an answer to a query. effective decision-making”. Explain.
• The benchmark for OLAP performance investigates a server’s References
ability to provide views based on queries of varying
complexity and scope.
Basic aggregation on some dimensions Education Asia, 1997.
More complex calculations are performed on other 2. Adriaans, Pieter, Data mining, Delhi: Pearson Education
dimensions Asia, 1996.
• Ratios and averages 3. Corey, Michael, Oracle8 data warehousing, New Delhi: Tata
• Variances on sceneries McGraw- Hill Publishing, 1998.
• A complex model to compute forecasts 4. Elmasri, Ramez, Fundamentals of database systems, 3rd ed.
• Consistently quick response times to these queries are Delhi: Pearson Education Asia, 2000.
imperative to establish a server’s ability to provide MD views
of information.
Benefits of OLAP
• Increase the productivity of manager’s developers and whole
organisations.
• Users of OLAP systems become more self-sufficient eg
managers no longer depend on IT to make schema changes.
• It allows managers to model problems that would be
impossible with less flexible systems
• Users have more control and timely access to relevant
strategic information which results in better decision
making.(timeliness, accuracy and relevance)
• IT developers also benefit form using OLAP specific
software as they can deliver applications to users faster.
• Thus, reducing the application backlog and ensure a better
service.
• OLAP further reduces the backlog by making its users self-
sufficient to build their own models but yet not
relinquishing control over the integrity of the data
• OLAP software reduces the query load and network traffic on
OLTP systems and data warehouses.
• Thus, OLAP enables organisations as a whole to respond
more quickly to market demands, which often results in
increased revenue and profitability. The goal of every
organisation.
Discussions
• Multi-Dimensional Views
• Operational Systems
• What is the significance of an OLTP System?
• Discuss OLTP related processes used in a Data warehouse.
• Explain MD views with an example.
11
12
LESSON 4
DATA WAREHOUSING MODELS
Structure
• Introduction
• Objective
• The Date warehouse Model
• Data Modeling for Data Warehouses
• Multidimensional models
• Roll-up display
• A drill-down display
• Multidimensional Schemas
• Star Schema
• Snowflake Schema
Objective
The main objective of this lesson is to make you understand a
data warehouse model. It also explains various types of
multidimensional models and Schemas.
Figure 2: The structure of data inside the data warehouse
Introduction
The current detail data is central in importance as it:
Data warehousing is the process of extracting and transforming
operational data into informational data and loading it into a • Reflects the most recent happenings, which are usually the
central data store or warehouse. Once the data is loaded it is most interesting;
accessible via desktop query and analysis tools by the decision • It is voluminous as it is stored at the lowest level of
makers. granularity;
The Data Warehouse Model • it is always (almost) stored on disk storage which is fast to
The data warehouse model is illustrated in the following access but expensive and complex to manage.
diagram. Older detail data is stored on some form of mass storage, it is
infrequently accessed and stored at a level detail consistent with
current detailed data.
Lightly summarized data is data distilled from the low level of
detail found at the current detailed level and generally is stored
on disk storage. When building the data warehouse have to
consider what unit of time is summarization done over and
also the contents or what attributes the summarized data will
contain.
Highly summarized data is compact and easily accessible and can
even be found outside the warehouse.
Metadata is the final component of the data warehouse and is
Figure 1: A data warehouse model
really of a different dimension in that it is not the same as data
The data within the actual warehouse itself has a distinct drawn from the operational environment but is used as:
structure with the emphasis on different levels of summariza- • a directory to help the DSS analyst locate the contents of the
tion as shown in the figure below. data warehouse,
• a guide to the mapping of data as the data is transformed
from the operational environment to the data warehouse
environment,
• a guide to the algorithms used for summarization between
the current detailed data and the lightly summarized data and
the lightly summarized data and the highly summarized
data, etc.
13
The basic structure has been described but Bill Inmon fills in
In the figure, there is a three-dimensional data cube that
the details to make the example come alive as shown in the organizes product sales data by fiscal quarters and sales regions.
following diagram. Each cell could contain data for a specific prod-uct, specific fiscal
quarter, and specific region. By including additional dimensions,
a data hypercube could be produced, although more than three
dimensions cannot be easily visualized at all or presented
graphically. The data can be queried directly in any combination
of dimensions, by passing complex database queries. Tools
exist for viewing data Data Modeling for Data Warehouses
Figure 3: An example of levels of summarization of data

inside the data warehouse
The diagram assumes the year is 1993 hence the current detail
data is 1992-93. Generally sales data doesn’t reach the current
level of detail for 24 hours as it waits until it is no longer
available to the operational system i.e. it takes 24 hours for it to
get to the data warehouse. Sales details are summarized weekly
by subproduct and region to produce the lightly summarized
detail. Weekly sales are then summarized again to produce the
highly summarized data.
According to the user’s choice of dimensions. Changing from
Data Modeling for Data Warehouses one dimensional hierarchy -(orientation) to another is easily
Multidimensional models take advantage of inherent accomplished in a data cube by a technique called pivoting (also
relationships in data to populate data in multidimensional called rotation). In this technique, the data cube can be thought
matrices called data cubes. (These may be called hypercube if of as rotating to show a different orientation of the axes. For
they have more than three dimensions.) For data that lend example, you might pivot the data cube to show regional sales
themselves to dimensional Formatting, query performance in revenues as rows, the fiscal quarter revenue totals as columns,
multidimensional matrices can be much better than in the and company’s products in the third dimension. Hence, this
relational data model. Three examples of dimensions in a technique is equivalent to having a regional sales table for each
corporate data warehouse would be the corporation’s fiscal product separately, where each table shows quarterly sales for
periods, products, and regions. that product region by region.
A standard spreadsheet is a two-dimensional matrix. One Multidimensional models lend themselves readily to hierarchical
example would be a spreadsheet of regional sales by product views in what is known as roll-up display and drill-down
for a particular time period. Products could be shown as rows, display.
with sales revenues for each region comprising the columns. • Roll-up display moves up the hierar-chy, grouping into
Adding a time dimension, such as an organiza-tion’s fiscal larger units along a dimension (e.g., summing weekly data by
quarters, would produce a three-dimensional matrix, which quar-ter, or by year). One of the above figures shows a roll-
could be repre-sented using a data cube. up display that moves from individual products to a coarser
grain of product categories.
• A drill-down display pro-vides the opposite capability,
furnishing a finer-grained view, perhaps disaggregating
country sales by region and then regional sales by sub region
and also breaking up prod-ucts by styles.
The multidimensional storage model involves two types of
tables: dimension tables and fact tables. A dimension table
consists of tuples of attributes of the dimension. A fact table
can be thought of as having tuples, one per a recorded fact. This
fact contains some measured or observed variable(s) and
identifies it (them) with pointers to dimension tables. The fact
14
table contains the data and the dimensions identify each tuple in

that data. An example of a fact table that can be viewed from
the perspec-tive of multiple dimensional tables.
Two common multidimensional schemas are the star schema

and the snowflake schema.
The star schema consists of a fact table with a single table for
each dimension.
The snowflake schema is a variation on the star schema in
which the dimensional tables from a star schema are organized
into a hierarchy by normalizing them. Some installations are
normalizing data warehouses up to the third normal form so
that they can access the data warehouse to the finest level of
detail. A fact con-stellation is a set of fact tables that share some
dimension tables. Following figure shows a fact constellation
with two fact tables, business results arid business forecast.
These share the dimension table called product. Fact constella-
tions limit the possible queries for the ware-house.
Data warehouse storage also utilizes indexing techniques to
support high perfor-mance access. A technique called bitmap
indexing constructs a bit vector for each value in a domain
(column) being indexed.
15
a. Give three dimension data elements and two fact data
It works very well for domains of low-cardinality. There is a 1
bit placed in the jth position in the vector if the jth row elements that could be in the database for this data
contains the value being indexed. For example, imagine an warehouse. Draw a data cube, for this database.
inventory of 100,000 cars with a bitmap index on car size. If b. State two ways in which each of the two fact data elements
there are four-car sizes--economy, compact, midsize, and full could be of low quality in some respect.
size-there will be four bit vectors, each containing 100,000 bits
2. You have decided to prepare a budget for the next 12
(12.5 K) for a total index size of 50K. Bitmap indexing can
months based on your actual expenses for the past 12. You
provide consider-able input/output and storage space advan-
need to get your expense information into what is in effect a
tages in low-cardinality domains. With bit vec-tors a bitmap
data warehouse, which you plan to put into a spreadsheet for
index can provide dramatic improvements in comparison,
easy sorting and analysis.
aggregation, and join performance. In a star schema, dimen-
sional data can be indexed to tuples in the fact table by join a. What are your information sources for this data warehouse?
indexing. Join indexes are traditional indexes to maintain b. Describe how you would carry out each of the five steps of
relationships between primary key and foreign key values. They data preparation for a data warehouse database, from
relate the values of a dimension of a star schema to rows in the extraction through summarization. If a particular step does
fact table. For example, consider a sales fact table that has city not apply, say so and justify your statement.
and fiscal quarter as dimensions. If there is a join index on city,
References
for each city the join index maintains the tuple IDs of tuples
containing that city. Join indexes may involve multiple dimen- 1. Adriaans, Pieter, Data mining, Delhi: Pearson Education
sions. Asia, 1996.
Data warehouse storage can facilitate access to summary data by 2. Anahory, Sam, Data warehousing in the real world: a practical
taking further advantage of the nonvolatility of data ware- guide for building decision support systems, Delhi: Pearson
houses and a degree of predictability of the analyses that will be Education Asia, 1997.
performed using them. Two approaches have been used.(1) 3. Berry, Michael J.A. ; Linoff, Gordon, Mastering data mining
smaller tables including summary data such as quarterly sales or : the art and science of customer relationship management, New
revenue by product line, and (2) encoding of level (e.g., weekly, York : John Wiley & Sons, 2000
quarterly, annual) into existing tables. By comparison, the 4. Corey, Michael, Oracle8 data warehousing, New Delhi: Tata
overhead of creating and maintaining such aggregations would McGraw- Hill Publishing, 1998.
likely be excessive in a volatile, transaction-oriented database.
Discussions Delhi: Pearson Education Asia, 2000.
• What are the various kinds of models used in Data
warehousing?
• Discuss the following:
• Roll-up display
• Drill down operation
• Star schema
• Snowflake schema
• Why is the star schema called by that name?
• State an advantage of the multidimensional database
structure over the relational database structure for data
warehousing applications.
• What is one reason you might choose a relational structure
over a multidimensional structure for a data warehouse
database? .
• Clearly contrast the difference between a fact table and a
dimension table.
Exercises
1. Your college or university is designing a data warehouse to
enable deans, department chairs, and the registrar’s office to
optimize course offerings, in terms of which courses are
offered, in how many sections, and at what times. The data
warehouse planners hope they will be able to do this better
after examining historical demand for courses and
extrapolating any trends that emerge.
16
17
Notes
18
LESSON 5
ARCHITECTURE AND PRINCIPLES OF DATA WAREHOUSING
Structure them a better understanding of their data and a better under-

• Objective standing of their business in relation to their competitors, and
it lets them provide better customer service.
• Introduction
So, what exactly is a data warehouse? Should your company
• Structure of a Data warehouse
have one, and if so, what should it look like?
• Data Warehouse Physical Architectures
Structure of a Data Warehouse
• Generic Two-Level Essentially, a data warehouse provides historical data for
• Expanded Three-Level decision-support applications. Such applications include
• Enterprise data warehouse (EDW) reporting, online analytical processing (OLAP), executive
• Data marts information systems (EIS), and data mining.
• Principles of a Data warehousing According to W. H. Inmon, the man who originally came up
with the term, a data warehouse is a centralized, integrated
Objective repository of information. Here, integrated means cleaned up,
The objective of this lesson is to let you know the basic merged, and redesigned. This may be more or less complicated
structure of a Data warehouse. You will also learn about Data depending on how many systems feed into a warehouse and
warehouse physical architecture and various principles of a Data how widely they differ in handling similar information.
warehousing.
But most companies already have repositories of information
Introduction in their production systems and many of them are centralized.
Let me start the lesson with an example, which illustrates the Aren’t these data warehouses? Not really.
importance and need of a data warehouse. Until several years Data warehouses differ from production databases, or online
ago Coca Cola had no idea how many bottles of Coke it transaction-processing (OLTP) systems, in their purpose and
produced each day because production data were stored on 24 design. An OLTP system is designed and optimized for data
different computer systems. Then, it began a technique called entry and updates, whereas a data warehouse is optimized for
Data warehousing. One airline spent and wasted over $100 data retrieval and reporting, and it is usually a read-only system.
million each year on inefficient mass media advertising cam- An OLTP system contains data needed for running the day-to-
paigns to reach frequent flyers…then it began data warehousing. day operations of a business but a data warehouse contains
Several years ago, the rail industry needed 8 working days to data used for analyzing the business. The data in an OLPT
deliver a freight quote to a customer. The trucking industry, by system is current and highly volatile, which data elements that
contrast, could deliver a freight quote to a customer on the may be incomplete or unknown at the time of entry. A
phone instantly, because unlike the rail industry, truckers were warehouse contains historical, nonvolatile data that has been
using…data warehousing. adjusted for transactions errors. Finally, since their purposes are
A data warehouse is a data base that collects current informa- so different, OLPT systems and data warehouses use different
tion, transforms it to ways it can be used by the warehouse data-modeling strategies. Redundancy is almost nonexistent in
owner, transforms that information for clients, and offers OLTP systems, since redundant data complicates updates. So
portals of access to members of your firm to help them make OLPT systems are highly normalized and are usually based on a
decisions and future plans. relational model. But redundancy is desirable in a data ware-
Data warehousing is the technology trend most often associated house, since it simplifies user access and enhances performance
with enterprise computing today. The term conjures up images by minimizing the number of tables that have to be joined.
of vast data banks fed from systems all over the globe, with Some data warehouses don’t use a relational model at all,
legions of corporate analysts mining them for golden nuggets preferring a multidimensional design instead.
of information that will make their companies more profitable. To discuss data warehouses and distinguish them from
All of the developments in database technology over the past transactional databases calls for an appropriate data model. The
20 years have culminated in the data warehouse. Entity- multidimensional data model is a good fit for OLAP and
relationship modeling, heuristic searches, mass data storage, decision-support technologies. In contrast to multi-databases,
neural networks, multiprocessing, and natural-language which provide access to disjoint and usually heterogeneous
interfaces have all found their niches in the data warehouse. But databases, a data warehouse is frequently a store of integrated
aside from being a database engineer’s dream, what practical data from multiple sources, processed for storage in a multidi-
benefits does a data warehouse offer the enterprise? mensional model. Unlike most transactional databases, data
warehouses typically support time-series and trend analysis,
When asked, corporate executives often say that having a data
both of which requires more historical data than are generally
warehouse gives them a competitive advantage, because it gives
19
maintained in transactional databases. Compared with transac- Data Warehousing, OnLine Analytical Processing (OLAP) and
tional databases, data warehouses are nonvolatile. That means Decision Support Systems - apart from being buzz words of
that information in the data warehouse changes far less often today IT arena - are the expected result of IT systems and
and may be regarded as non-real-time with periodic updating. current needs. For decades, Information Management Systems
In transactional systems, transactions are the unit and are the have focused exclusively in the gathering and recording into
agent of change a database; by contrast, data warehouse Database Management Systems data that corresponded to
information is much more coarse grained and is refreshed everyday simple transactions, from which the name OnLine
according to a careful choice of refresh policy, usually incremen- Transaction Processing (OLTP) comes from.
tal. Warehouse updates are handled by the warehouse’s Managers and analysts now need to go steps further from the
acquisition component that provides all required preprocessing. simple data storing phase and exploit IT systems by posing
We can also describe data warehousing more generally as “a complex queries and requesting analysis results and decisions
collection of decision support technologies, aimed at enabling that are based on the stored data. Here is where OLAP and Data
the knowledge worker (executive, manager, ana-lyst) to make, Warehousing is introduced, bringing into business the necessary
better and faster decisions.” The following Figure gives an system architecture, principles, methodological approach and -
overview of the conceptual structure of a data warehouse. It finally - tools to assist in the presentation of functional
shows the entire data warehousing process. This process Decision Support Systems.
includes possible cleaning and reformatting of data before it’s I.M.F. has been working closely with the academic community -
warehousing. At the end of the process, OLAP, data mining, which only recently followed up the progress of the commercial
and DSS may generate new relevant information such as rules; arena that was boosting and pioneering in the area for the past
this information is shown in the figure going back into the decade - and adopted the architecture and methodology
warehouse figure also shows that data sources may include files. presented in the following picture. This is the result of the
ESPRIT funded Basic Research project, “Foundations of Data
Warehouse Quality - DWQ”.
20
Being basically dependent on architecture in concept, a Data
Warehouse - or an OLAP system - is designed by applying data
warehousing concepts on traditional database systems and
using appropriate design tools. Data Warehouses and OLAP
applications designed and implemented comply with the
adopted methodology by IMF.
The final deployment takes place through the use of specialized
data warehouse and OLAP systems, namely MicroStrategy’s
DSS Series. MicroStrategy Inc. is one of the most prominent
and accepted international players on data warehousing systems
and tools, offering solutions for every single layer of the DW
architecture hierarchy.
Data Warehouse Physical Architectures

• Generic Two-Level
• Expanded Three-Level Fig : Generic Two-Level Physical Architecture
• Enterprise data warehouse (EDW) - single source of data for
decision making
• Data marts - limited scope; data selected from EDW
21
Principles of a Data Warehousing
• Load Performance
Data warehouses require increase loading of new data on a
periodic basic within narrow time windows; performance on
the load process should be measured in hundreds of
millions of rows and gigabytes per hour and must not
artificially constrain the volume of data business.
• Load Processing
Many steps must be taken to load new or update data into
the data warehouse including data conversion, filtering,
reformatting, indexing and metadata update.
• Data Quality Management
Fact-based management demands the highest data quality.
The warehouse must ensure local consistency, global
consistency, and referential integrity despite “dirty” sources
and massive database size.
• Query Performance
Fact-based management must not be slowed by the
performance of the data warehouse RDBMS; large, complex
queries must be complete in seconds not days.
• Terabyte Scalability
Data warehouse sizes are growing at astonishing rates. Today
these range from a few to hundreds of gigabytes and
terabyte-sized data warehouses.
Fig : Expanded Three-Level Physical Architecture Discussions
Associated with the three-level physical architecture • Write short notes on:
• Operational Data • Data Quality Management
Stored in the various operational systems throughout • OLAP
the organization • DSS
• Reconciled Data • Data marts
The data stored in the enterprise data warehouse • Operational data
Generally not intended for direct access by end users • Discuss Three-Layer Data Architecture with the help of a
• Derived Data diagram.
The data stored in the data marts • What are the various Principles of Data warehouse?
Selected, formatted, and aggregated for end user • What is the importance of a data warehouse in any
decision-support applications organization? Where it is required?
Self Test
As set of multiple choices is given with every question, Choose
the correct answer for the Following questions.
1. Data warehouse cannot deal with
a. Data analysis
b. Operational activities
c. Information extraction
d. None of these
2. A data warehouse system requires
a. Only current data
b. Data for a large period
c. Only data projections
d. None of these
Fig : Three-Layer Data Architecture
22
References

Asia, 1996.
23
Notes
24
CHAPTER 2
LESSON 6 BUILDING A DATA
DATA WAREHOUSING AND WAREHOUSE PROJECT
OPERATIONAL SYSTEMS
Structure A typical operational system deals with one order, one account,

one inventory item. An operational system typically dea1s with
• Objective
predefined events and, due to the nature of these events,
• Introduction requires fast access. Each transaction usually deals with small
• Operational Systems amounts of data.
• Warehousing” data outside the operational systems Most of the time, the business needs of an operational system
• Integrating data from more than one operational system do not change much. The application that records the transac-
tion, as well as the application that controls access to the
• Differences between transaction and analysis processes
information, that is, there porting side of the- banking
• Data is mostly non-volatile business does not change much over time. In this type of
• Data saved for longer periods than in transaction systems system, the information required, when a customer initiates a
• Logical transformation of operational data transaction must be current. Before a bank will allow a with-
drawal, it must first be certain of your current balance.
• Structured extensible data model
• Data warehouse model aligns with the business structure “Warehousing” Data outside the
Operational Systems
• Transformation of the operational state information
The primary concept of data warehousing is that the data stored
Objective for business analysis can most effectively be accessed by
The aim of this lesson is to explain you the need and impor- separating it from the data in the operational systems. Many of
tance of an Operational Systems in a Data warehouse. the reasons for this separation have evolved over the years. In
Introduction the past, legacy systems archived data onto tapes as it became
Data warehouse, a collection of data designed to support inactive and many analysis reports ran from these tapes or
management decision-making. Data warehouses contain a wide mirror data sources to minimize the performance impact on the
variety of data that present a coherent picture of business operational systems.
conditions at a single point in time. These reasons to separate the operational data from analysis
Development of a data warehouse includes development of data have not significantly changed with the evolution of the
systems to extract data from operating systems plus installation data warehousing systems, except that now they are considered
of a warehouse database system that provides managers flexible more formally during the data warehouse building process.
access to the data. Advances in technology and changes in the nature of business
have made many of the business analysis processes much more
The term data warehousing generally refers to the combination
complex and sophisticated. In addition to producing standard
of many different databases across an entire enterprise.
reports, today’s data warehousing systems support very
Operational Systems sophisticated online analysis including multi-dimensional
Up to now, the early database system of the primary purpose analysis.
was to, meet the needs of operational systems, which are
Integrating Data from more than one
typically transactional in nature. Classic examples of operational
Operational System
systems include
Data warehousing systems are most successful when data can be
• General Ledgers combined from more than one operational system. When the
• Accounts Payable data needs to be brought together from more than one source
• Financial Management application, it is natural that this integration be done at a place
independent of the source applications. Before the evolution of
• Order Processing
structured data warehouses, analysts in many instances would
• Order Entry combine data extracted from more than one operational system
• Inventory into a single spreadsheet or a database. The data warehouse
Operational systems by nature are primarily concerned with the may very effectively combine data from multiple source
handling of a single transaction. Look at a banking system, applications such as sales, marketing, finance, and production.
when you, the customer, make a deposit to your checking Many large data warehouse architectures allow for the source
account, the banking operational system is responsible for applications to be integrated into the data warehouse incremen-
recording the transaction to ensure the corresponding debit tally.
appears in your account record. The primary reason for combining data from multiple source
applications is the ability to cross-reference data from these
25
applications. Nearly all data in a typical data warehouse is built Even though many of the queries and reports that are run
around the time dimension. Time is the primary filtering against a data warehouse are predefined, it is nearly impossible
criterion for a very large percentage of all activity against the data to accurately predict the activity against a data warehouse. The
warehouse. An analyst may generate queries for a given week, process of data exploration in a data warehouse takes a business
month, quarter, or a year. Another popular query in many data analyst through previously undefined paths. It is also common
warehousing applications is the review of year-on-year activity. to have runaway queries in a data warehouse that are triggered
For example, one may compare sales for the first quarter of this by unexpected results or by users’ lack of understanding of the
year with the sales for first quarter of the prior years. The time data model. Further, many of the analysis processes tend to be
dimension in the data warehouse also serves as a fundamental all encompassing whereas the operational processes are well
cross-referencing attribute. For example, an analyst may attempt segmented. A user may decide to explore detail data while
to access the impact of a new marketing campaign run during reviewing the results of a report from the summary tables.
selected months by reviewing the sales during the same periods. After finding some interesting sales activity in a particular
The ability to establish and understand the correlation between month, the user may join the activity for this month with the
activities of different organizational groups within a company is marketing programs that were run during that particular month
often cited as the single biggest advanced feature of the data to further understand the sales. Of course, there would be
warehousing systems. instances where a user attempts to run a query that will try to
The data warehouse system can serve not only as an effective build a temporary table that is a Cartesian product of two tables
platform to merge data from multiple current applications; it containing a million rows each! While an activity like this would
can also integrate multiple versions of the same application. unacceptably degrade an operational system’s performance, it is
For example, an organization may have migrated to a new expected and planned for in a data warehousing system.
standard business application that replaces an old mainframe- Data is mostly Non-volatile
based, custom-developed legacy application. The data Another key attribute of the data in a data warehouse system is
warehouse system can serve as a very powerful and much that the data is brought to the warehouse after it has become
needed platform to combine the data from the old and the new mostly non-volatile. This means that after the data is in the data
applications. Designed properly, the data warehouse can allow warehouse, there are no modifications to be made to this
for year-on-year analysis even though the base operational information. For example, the order status does not change,
application has changed. the inventory snapshot does not change, and the marketing
Differences between Transaction and Analysis promotion details do not change. This attribute of the data
Processes warehouse has many very important implications for the kind
The most important reason for separating data for business of data that is brought to the data warehouse and the timing
analysis from the operational data has always been the potential of the data transfer.
performance degradation on the operational system that can Let us further review what it means for the data to be non-
result from the analysis processes. High performance and quick volatile. In an operational system the data entities go through
response time is almost universally critical for operational many attribute changes. For example, an order may go through
systems. The loss of efficiency and the costs incurred with many statuses before it is completed. Or, a product moving
slower responses on the predefined transactions are usually easy through the assembly line has many processes applied to it.
to calculate and measure. For example, a loss of five seconds of Generally speaking, the data from an operational system is
processing time is perhaps negligible in and of itself; but it triggered to go to the data warehouse when most of the activity
compounds out to considerably more time and high costs once on these business entity data has been completed. This may
all the other operations it impacts are brought into the picture. mean completion of an order or final assembly of an accepted
On the other hand, business analysis processes in a data product. Once an order is completed and shipped, it is unlikely
warehouse are difficult to predefine and they rarely need to have to go back to backorder status. Or, once a product is built and
rigid response time requirements. accepted, it is unlikely to go back to the first assembly station.
Operational systems are designed for acceptable performance for Another important example can be the constantly changing data
pre-defined transactions. For an operational system, it is that is transferred to the data warehouse one snapshot at a time.
typically possible to identify the mix of business transaction The inventory module in an operational system may change
types in a given time frame including the peak loads. It also with nearly every transaction; it is impossible to carry all of these
relatively easy to specify the maximum acceptable response time changes to the data warehouse. You may determine that a
given a specific load on the system. The cost of a long response snapshot of inventory carried once every week to the data
time can then be computed by considering factors such as the warehouse is adequate for all analysis. Such snapshot data
cost of operators, telecommunication costs, and the cost of any naturally is non-volatile.
lost business. For example, an order processing system might It is important to realize that once data is brought to the data
specify the number of active order takers and the average warehouse, it should be modified only on rare occasions. It is
number of orders for each operational hour. Even the query very difficult, if not impossible, to maintain dynamic data in the
and reporting transactions against the operational system are data warehouse. Many data warehousing projects have failed
most likely to be predefined with predictable volume. miserably when they attempted to synchronize volatile data
between the operational and data warehousing systems.
26
Data saved for longer periods than in transaction Logical Transformation of Operational

systems Data
Data from most operational systems is archived after the data This sub-section explores the concepts associated with the data
becomes inactive. For example, an order may become inactive warehouse logical model. The data is logically transformed
after a set period from the fulfillment of the order; or a bank when it is brought to the data warehouse from the operational
account may become inactive after it has been closed for a period systems. The issues associated with the logical transformation
of time. The primary reason for archiving the inactive data has of data brought from the operational systems to the data
been the performance of the operational system. Large warehouse may require considerable analysis and design effort.
amounts of inactive data mixed with operational live data can The architecture of the data warehouse and the data warehouse
significantly degrade the performance of a transaction that is model greatly impact the success of the project. This section
only processing the active data. Since the data warehouses are reviews some of the most fundamental concepts of relational
designed to be the archives for the operational data, the data database theory that do not fully apply to data warehousing
here is saved for a very long period. systems. Even though most data warehouses are deployed on
In fact, a data warehouse project may start without any specific relational database platforms, some basic relational principles are
plan to archive the data off the warehouse. The cost of knowingly modified when developing the logical and physical
maintaining the data once it is loaded in the data warehouse is model of the data warehouses.
minimal. Most of the significant costs are incurred in data Structured Extensible Data Model
transfer and data scrubbing. Storing data for more than five The data warehouse model outlines the logical and physical
years is very common for data warehousing systems. There are structure of the data warehouse. Unlike the archived data of the
industry examples were the success of a data warehousing legacy systems, considerable effort needs to be devoted to the
project has encouraged the managers to expand the time data warehouse modeling. This data modeling effort in the
horizon of the data stored in the data warehouse. They may early phases of the data warehousing project can yield significant
start with storing the data for two or three years and then benefits in the form of an efficient data warehouse that is
expand to five or more years once the wealth of business expandable to accommodate all of the business data from
knowledge in the data warehouse is discovered. The falling multiple operational applications.
prices of hardware have also encouraged the expansion of
The data modeling process needs to structure the data in the
successful data warehousing projects.
data warehouse independent of the relational data model that
may exist in any of the operational systems. As discussed later
in this paper, the data warehouse model is likely to be less
Order processing
•2 second response time
Data normalized than an operational system model. Further, the
Daily cl
•Last 6 months orders
osed orders Warehouse operational systems are likely to have large amounts of
•Last 5 years data overlapping business reference data. Information about current
•Response time 2 seconds products is likely to be used in varying forms in many of the
ry
Product Price/inventory vento to 60 minutes
ce/In
ct pri
•10 second response time
eekly p
rodu
•Data is not modified operational systems. The data warehouse system needs to
W
•Last 10 price changes
ms
consolidate all of the reference data. For example, the opera-
•Last 20 inventory transactions gra
g pro tional order processing system may maintain the pricing and
etin
ark
Marketing eekl
ym physical attributes of products whereas the manufacturing floor
W
•30 second response time application may maintain design and formula attributes for the
•Last 2 years programs
same product. The data warehouse reference table for products
•Different performance requirements
would consolidate and maintain all attributes associated with
•Combine data from multiple applications products that are relevant for the analysis processes. Some
•Data is mostly non-volatile attributes that are essential to the operational system are likely to
•Data saved for a long time period be deemed unnecessary for the data warehouse and may not be
Figure 3. Reasons for moving data outside the operations systems loaded and maintained in the data warehouse.
In short, the separation of operational data from the analysis

data is the most fundamental data-warehousing concept. Not
only is the data stored in a structured manner outside the
operational system, businesses today are allocating considerable
resources to build data warehouses at the same time that the
operational applications are deployed. Rather than archiving
data to a tape as an afterthought of implementing an opera-
tional system, data warehousing systems have become the
primary interface for operational systems.
Figure 3 highlights the reasons for separation discussed in this
section.
27
with a bank customer. The retail operational system may
provide some attributes such as social security number, address,

and phone number. A mortgage system or some purchased
database may provide with employment, income, and net
Orders worth information.
Enterprise Data Warehouse

The structure of the data in any single source application is likely
to be inadequate for the data warehouse. The structure in a
Product single application may be influenced by many factors, including:
• Purchased Applications: The application data structure
may be dictated by an application that was purchased from a
Future software vendor and integrated into the business. The user
of the application may have very little or no control over the
data model. Some vendor applications have a very generic
Future data model that is designed to accommodate a large number
and types of businesses.
• Legacy Application: The source application may be a very
old mostly homegrown application where the data model
has evolved over the years. The database engine in this
•Setup framework for Enterprise data warehouse application may have been changed more than once without
•Start with few a most valuable source applications anyone taking the time to fully exploit the features of the
•Add additional applications as business case can be made new engine. There are many legacy applications in existence
today where the data model is neither well documented nor
Figure 4. Extensible data warehouse understood by anyone currently supporting the application.
• Platform Limitations: The source application data model
may be restricted by the limitations of the hardware/
The data warehouse model needs to be extensible and struc- software platform or development tools and technologies. A
tured such that the data from different applications can be database platform may not support certain logical
added as a business case can be made for the data. A data relationship or there may be physical limitations on the data
warehouse project in most cases cannot include data from all attributes.
possible applications right from the start. Many of the
successful data warehousing projects have taken an incremental
approach to adding data from the operational systems and Order processing
Customer Product
Data
aligning it with the existing data. They start with the objective orders price
Available Inventory Warehouse

of eventually adding most if not all business data to the data Customers
warehouse. Keeping this long-term objective in mind, they Product Price/inventory Products
may begin with one or two operational applications that Product

price
Product
Inventory
Orders
provide the most fertile data for business analysis. Figure 4 Product Price changes
Product Inventory
illustrates the extensible architecture of the data warehouse. Product Price
Marketing
Data Warehouse model aligns with the business Customer
Profile
Product
price
structure Marketing programs
A data warehouse logical model aligns with the business

structure rather than the data model of any particular applica- •No data model restrictions of the source application
tion. The entities defined and maintained in the data •Data warehouse model has business entities
warehouse parallel the actual business entities such as custom- Figure 5. Data warehouse entities align with the business structure
ers, products, orders, and distributors. Different parts of an
organization may have a very narrow view of a business entity
such as a customer. For example, a loan service group in a bank Figure 5 illustrates the alignment of data warehouse entities
may only know about a customer in the context of one or more with the business structure. The data warehouse model breaks
loans outstanding. Another group in the same bank may know away from the limitations of the source application data models
about the same customer in context of a deposit account. The and builds a flexible model that parallels the business structure.
data warehouse view of the customer would transcend the view This extensible data model is easy to understand by the
from a particular part of the business. A customer in the data business analysts as well as the managers.
warehouse would represent a bank customer that has any kind
of business with the bank.
The data warehouse would most likely build attributes of a
business entity by collecting data from multiple source applica-
tions. Consider, for example, the demographic data associated
28
inventory may be reduced by an order fulfillment transaction or

OPERATIONAL SYSTEMS DATA WAREHOUSE
this quantity may be increased with receipt of a new shipment
• PROCESSES • PROCESSES ONE of the product. If this order processing system executes ten
THOUSANDS OF TRANSACTION thousand transactions in a given day, it is likely that the actual
MILLIONS OF DAILY THAT
TRANSACTIONS CONTAINS inventory in the database will go through just as many states or
DAILY. MILLIONS OF snapshots during this day. It is impossible to capture this
RECORDS. constant change in the database and carry it forward to the data
warehouse. This is still one of the most perplexing problems
• Deals with one account • Deals with a with the data warehousing systems. There are many approaches
at a time. summary of multiple
accounts to solving this problem. You will most likely choose to carry
periodical snapshots of the inventory data to the data ware-
• Runs day-to-day • Creates reports for
house. This scenario can apply to a very large portion of the
operations. strategic decision- data in the operational systems. The issues associated with this
makers. get much more complicated as extended time periods are
considered.
• Exists on different • Exists on a single
machines, dividing machine, providing a
university resources. centralized resource.
Order Processing
System Data
Daily cl
• Provides instantaneous • Adds layers of osed orders Warehouse
snapshot of an snapshots on a
Editor:
organization's affairs. Data regularly scheduled Orders (Closed)
changes from moment to basis. Data changes Order Please add Open,
Backorder, Shipped, Inventory snapshot 1
Closed to the arrow
moment. only when DW around the order
Inventory snapshot 2
Team updates the shot
warehouse. snap
ntory
kly inve
Down
Wee
UpInventory
Transformation of the Operational State
Information
It is essential to understand the implications of not being able
•Operational state information is not carried to the data warehouse
to maintain the state information of the operational system
•Data is transferred to the data warehouse after all state changes
when the data is moved to the data warehouse. Many of the
•Or, data is transferred with period snapshots
attributes of entities in the operational system are very dynamic
and constantly modified. Many of these dynamic operational
Figure 6. Transformation of the operational state information
system attributes are not carried over to the data warehouse;
others are static by the time they are moved to the data ware-
house. A data warehouse generally does not contain Figure 6 illustrates how most of the operational state informa-
information about entities that are dynamic and constantly tion cannot be carried over the data warehouse system.
going through state changes.
De-normalization of Data
To understand what it means to lose the operational state
Before we consider data model de-normalization in the context
information, let us consider the example of an order fulfillment
of data warehousing, let us quickly review relational database
system that tracks the inventory to fill orders. First let us look
concepts and the normalization process. E. F. Codd developed
at the order entity in this operational system. An order may go
relational database theory in the late 1960s while he was a
through many different statuses or states before it is fulfilled or
researcher at IBM. Many prominent researchers have made
goes to the “closed” status. Other order statuses may indicate
significant contributions to this model since its introduction.
that the order is ready to be filled, it is being filled, back ordered,
Today, most of the popular database platforms follow this
ready to be shipped, etc. This order entity may go through
model closely. A relational database model is a collection of
many states that capture the status of the order and the
two-dimensional tables consisting of rows and columns. In
business processes that have been applied to it. It is nearly
the relational modeling terminology, the tables, rows, and
impossible to carry forward all of attributes associated with
columns are respectively called relations, attributes, and tuples.
these order states to the data warehousing system. The data
The name for relational database model is derived from the
warehousing system is most likely to have just one final
term relation for a table. The model further identifies unique
snapshot of this order. Or, as the order is ready to be moved
keys for all tables and describes the relationship between tables.
into the data warehouse, the information may be gathered from
multiple operational entities such as order and shipping to Normalization is a relational database modeling process where
build the final data warehouse order entity. the relations or tables are progressively decomposed into
smaller relations to a point where all attributes in a relation are
Now let us consider the more complicated example of inven-
very tightly coupled with the primary key of the relation. Most
tory data within this system. The inventory may change with
data modelers try to achieve the “Third Normal Form” with all
every single transaction. The quantity of a product in the
29
of the relations before they de-normalize for performance or these price changes may be carried to the data warehouse with a
other reasons. The three levels of normalization are briefly periodic snapshot of the product price table. In a data ware-
described below: housing system you would carry the list price of the product
• First Normal Form: A relation is said to be in First Normal when the order is placed with each order regardless of the
Form if it describes a single entity and it contains no arrays selling price for this order. The list price of the product may
or repeating attributes. For example, an order table or change many times in one year and your product price database
relation with multiple line items would not be in First snapshot may even manage to capture all these prices. But, it is
Normal Form because it would have repeating sets of nearly impossible to determine the historical list price of the
attributes for each line item. The relational theory would call product at the time each order is generated if it is not carried to
for separate tables for order and line items. the data warehouse with the order. The relational database
theory makes it easy to maintain dynamic relationships between
• Second Normal Form: A relation is said to be in Second
business entities, whereas a data warehouse system captures
Normal Form if in addition to the First Normal Form
relationships between business entities at a given time.
properties, all attributes are fully dependent on the primary
key for the relation.
• Third Normal Form: A relation is in Third Normal Form if Order processing
in addition to Second Normal Form, all non-key attributes Customer Product
Data
Extensible data warehouse

orders price
are completely independent of each other. Available Inventory Warehouse

The process of normalization generally breaks a table into many De-normalized Customers
data Products
independent tables. While a fully normalized database can yield Product Price/inventory
Product Product Orders
fantastically flexible model, it generally makes the data model price Inventory
Transform Product Inventory
more complex and difficult to follow. Further, a fully normal- Product Price changes State
Product Price
ized data model can perform very inefficiently. A data modeler Marketing
in an operational system would take normalized logical data Customer
Profile
Product
price
model and convert it into a physical data model that is signifi- Marketing programs
cantly de-normalized. De-normalization reduces the need for

database table joins in the queries. •Structured extensible data model
•Data warehouse model aligns with the business structure
Some of the reasons for de-normalizing the data warehouse •Transformation of the state information
model are the same as they would be for an operational system, •Data is de-normalized because the relationships are static
namely, performance and simplicity. The data normalization in Figure 7. Logical transformation of application data
relational databases provides considerable flexibility at the cost
of the performance. This performance cost is sharply increased
in a data warehousing system because the amount of data Logical transformation concepts of source application data
involved may be much larger. A three-way join with relatively described here require considerable effort and they are a very
small tables of an operational system may be acceptable in important early investment towards development of a success-
terms of performance cost, but the join may take unacceptably ful data warehouse. Figure 7 highlights the logical
long time with large tables in the data warehouse system. transformation concepts discussed in this section.
Static Relationships in Historical Data Physical Transformation of Operational

Another reason that de-normalization is an important process Data
in data warehousing modeling is that the relationship between Physical transformation of data homogenizes and purifies the
many attributes does not change in this historical data. For data. These data warehousing processes are typically known as
example, in an operational system, a product may be part of the “data scrubbing” or “data staging” processes. The “data
product group “A” this month and product group “B” starting scrubbing” processes are some of the most labor-intensive and
next month. In a properly normalized data model, it would be tedious processes in a data warehousing project. Yet, without
inappropriate to include the product group attribute with an proper scrubbing, the analytical value of even the clean data can
order entity that records an order for this product; only the be greatly diminished. Physical transformation includes the use
product ID would be included. The relational theory would call of easy-to-understand standard business terms, and standard
for a join on the order table and product table to determine the values for the data. A complete dictionary associated with the
product group and any other attributes of this product. This data warehouse can be a very useful tool. During these physical
relational theory concept does not apply to a data warehousing transformation processes the data is sometimes “staged” before
system because in a data warehousing system you may be it is entered into the data warehouse. The data may be com-
capturing the group that this product belonged to when the bined from multiple applications during this “staging” step or
order was filled. Even though the product moves to different the integrity of the data may be checked during this process.
groups over time, the relationship between the product and the The concepts associated with the physical transformation of the
group in context of this particular order is static. data are introduced in this sub-section. Historical data and the
Another important example can be the price of a product. The current operational application data is likely to have some
prices in an operational system may change constantly. Some of missing or invalid values. It is important to note that it is
essential to manage missing values or incomplete transforma-
30
tions while moving the data to the data warehousing system.

The end user of the data warehouse must have a way to learn
about any missing data and the default values used by the
transformation processes.
Operational terms transformed into uniform
business terms
The terms and names used in the operational systems are
transformed into uniform standard business terms by the data
warehouse transformation processes. The operational applica-
tion may use cryptic or difficult to understand terms for a variety
of different reasons. The platform software may impose length
and format restriction on a term, or purchased application may
be using a term that is too generic for your business. The data
warehouse needs to consistently use standard business terms
that are self-explanatory.
A customer identifier in the operational systems may be called
cust, cust_id, or cust_no. Further, different operational
applications may use different terms to refer to the same
attribute. For example, a customer in the loan organization in a
bank may be referred to as a Borrower. You may choose a
simple standard business term such as Customer Id in the data
warehouse. This term would require little or no explanation
even to the novice user of the data warehouse.
Discussions
• Give some examples of Operational systems.
• Explain how the logical transformation of operational data
takes place.
• Describe the Differences between transaction and analysis
processes
• Explain the Physical transformation of operational data
• Discuss the need of de-normalizing the data.
References
Asia, 1996.
Notes
31
LESSON 7
BUILDING DATA WAREHOUSING, IMPORTANT CONSIDERATIONS
Structure companies of a large corporation may have different fiscal

• Objective calendars with quarters ending on different dates, making it
difficult to aggregate financial data by quarter. Various credit
• Introduction
cards may report their transactions differently, making it
• Building a Data Warehouse difficult to -compute all credit sales. These format
• Nine decisions in the design of a data warehouse inconsistencies must be resolved.
Objective • The data must be cleaned to ensure validity. Data cleaning is
The objective of this lesson is to introduce you with the basic an involved and involved and complex process that has been
concepts behind building a data warehouse. identified as the largest labor-demanding component -data
Introduction warehouse construction. For input data, cleaning must occur
There are several reasons why organizations consider data before the data are loaded into the warehouse. There is
warehousing a critical need. These drivers for data warehousing nothing about cleaning data that is specific to data
can be found in the business climate of a global marketplace, in warehousing and that could not be applied to a host
the changing organizational structures of successful corpora- database. However, since input data must be examined and
tions, and in the technology. formatted consistently, data warehouse builders should take
this opportunity to check for validity and quality.
Traditional databases support On-Line Transaction Processing
Recognizing erroneous and incomplete data is difficult to
(OLTP), which includes insertions, updates, and deletions,
automate, and cleaning that requires automatic error
while also supporting information query requirements.
correction can be even tougher. Some aspects, such as
Traditional relational databases are optimized to process queries
domain checking, are easily coded into data cleaning routines,
that may touch a small part of the database and transactions
but automatic recognition of other data problems can be
that deal with insertions or updates of a few tuples per relation
more challenging. (For example, one might require that City
to process. Thus, they cannot be optimized for OLAP, DSS, or
= ‘San Francisco’ together with State = ‘CT’ be recognized as
data mining. By contrast, data warehouses are designed precisely
an incorrect combination.)
to support efficient extraction, process-ing, and presentation for
After such problems have been taken care of, similar data from
analytic and decision-making purposes. In comparison to tradi-
different sources must be coordinated for loading into the
tional databases, data warehouses generally contain very large
warehouse. As data managers in the organization. Discover that
amounts of data from multiple sources that may include
their data are being cleaned for input into the warehouse; they
databases from different data models and sometimes lies
will likely want to upgrade their data with the cleaned data. The
acquired from independent systems and platforms.
process of returning cleaned data to the source is called back
Building a Data Warehouse flushing.
In constructing a data warehouse, builders should take a broad • The data must be fitted into the data model of the
view of the anticipated use of the warehouse. There is no way warehouse. Data from the various sources must be installed
to anticipate all possible queries or analyses during the design in the data model of the warehouse. Data may have to be
phase. However, the design should specifically support ad-hoc converted from relational, object-oriented, or legacy databases
querying, that is, accessing data with any meaningful combina- (network and/or hierarchical) to a multidimensional model.
tion of values for the attributes in the dimension or fact tables.
• The data must be loaded into the warehouse. The sheer
For example; a marketing-intensive consumer-products
volume of data in the warehouse makes loading the data a
company -would require different ways of organizing the data
significant task. Monitoring tools for loads as well as
warehouse than would a nonprofit charity focused on fund
methods to recover from incomplete or incorrect loads are
raising. An appropriate schema should be chosen that reflects
required. With the huge volume of data in the warehouse,
anticipated usage. incremental updating is usually the only feasible approach.
Acquisition of data for the warehouse involves the following The refresh policy will probably emerge as a compromise that
steps: takes into account the answers to the following questions:
• The data must be extracted from multiple, heterogeneous • How up-to-date must the data be?
sources; for example, databases or other data feeds such as • Can the warehouse go off-line, and for how long?
those containing financial market data or environmental data. • What are the data interdependencies?
• Data must be formatted for consistency within the • What is the storage availability?
warehouse. Names, meanings, and domains of data from
unrelated sources must be reconciled. For instance, subsidiary
• What are the distribution requirements (such as for
replication and partitioning)?
32
• What is the loading time (including cleaning, formatting, descriptions, warehouse operations and maintenance, and access

copying, transmitting, and overhead such as index support functionality. The second, business metadata, includes
rebuilding)? the relevant business rules and organizational details support-
As we have said, databases must strike a balance between ing the warehouse.
efficiency in transaction processing and supporting query The architecture of the organization’s distributed computing
requirements (ad hoc user requests), but a data warehouse is environment is a major -determining characteristic for the
typically optimized for access from a decision maker’s, needs. design of the warehouse. There are two basic distributed
Data storage in a data warehouse reflects this specialization and architectures: the distributed warehouse and the federated
involves the following processes: warehouse. For a distributed warehouse, all the issues of
distributed databases are relevant, for example, replication,
• Storing the data according to the data model of the
partitioning, communications, and consistency concerns. A
warehouse.
distributed architecture can provide benefits particularly
• Creating and maintaining, required data structures. important to warehouse performance, such as improved load
• Creating and maintaining appropriate access paths. balancing, scalability of performance, and higher availability. A
• Providing for time-variant data as new data are added. single replicated metadata repository would reside at each
distribution site. The idea of the federated warehouse is like
• Supporting the updating of warehouse data.
that of the federated database: a decentralized confederation of
• Refreshing the data. autonomous data warehouses, each with its own metadata
• Purging data repository. Given the magnitude of the challenge inherent to
Although adequate time can be devoted initially to constructing data warehouses, it is likely that such federations will consist of
the warehouse, the sheer volume of data in the warehouse smaller-scale components, such as data marts. Large organiza-
generally makes it impossible to simply reload the warehouse in tions may choose -to federate data marts rather than build huge
its entirety later on. Alternatives include selective (partial) data warehouses.
refreshing of data and separate warehouse versions (requiring, Nine Decisions in the design of a Data Warehouse
double, storage capacity for the ware-house). When the The job of a data warehouse designer is a daunting one. Often
warehouse uses an incremental data refreshing mechanism, data the newly appointed data warehouse designer is drawn to the
may need to be periodically purged; for example, a warehouse job because of the high vis-ibility and importance of the data
that maintains data on the pre-vious twelve business quarters warehouse function. In effect, management says to the designer:
may periodically purge its data each year. “Take all the enterprise data and make it available to manage-
Data warehouses must also be designed with full consideration ment so that they can answer all their questions and sleep at
of the environment in which they will reside. Important design night. And please do it very quickly, and we’re sorry, but we can’t
considerations include-the following: add any more staff until the proof of concept is successful.”
• Usage projections. This responsibility is exciting and very visible, but most
• The fit of the data model. designers feel over-whelmed by the sheer enormity of the task.
Something real needs to be accom-plished, and fast. Where do
• Characteristics of available sources.
you start? Which data should be brought up first? Which
• Design of the metadata component. management needs are most important? Does the design
• Modular component design. depend on the details of the most recent interview, or are there
• Design for manageability and change. some underlying and more constant design guidelines that you
can depend on? How do you scope the proj-ect down to
• Considerations of distributed and parallel architecture
something manageable, yet at the same time build an extensible
We discuss each of these in turn. Warehouse design is initially architecture that will gradually let you build a comprehensive
driven by usage pro-jections; that is, by expectations about who data warehouse environment?
will use the warehouse and in what way. Choice of a data model These questions are close to causing a crisis in the data ware-
to support this usage is a key initial decision. Usage projections house industry. Much of the recent surge in the industry toward
and the characteristics of the warehouse’s data sources are both “data marts” is a reaction to these very issues. Designers want to
taken into account. Modular design is a practical necessity to do something simple and achievable. No one is willing to sign
allow the warehouse to evolve with. the organization and its up for a galactic design that must somehow get every-thing
information environment. In addition, a well-built data right on the first try. Everyone hopes that in the rush to
warehouse must be designed for maintainability, enabling the simplification, the long-term coherence and extendibility of the
warehouse managers to effectively plan for and manage change design will not be compro-mised. Fortunately, a pathway
while providing optimal support to users. through this design challenge achieves an implementable
Metadata is defined as - description of a database including its immediate result, and at the same time it continuously mends
schema definition. The metadata repository is a key data the design so that eventually a true enterprise-scale data
warehouse component. The metadata repository includes both warehouse is built. The secret is to keep in mind a design
technical and business metadata. The first, technical metadata, methodology, which Ralph Kimball calls the “nine-step
covers details of acquisition processing, storage structures, data method”(see Table 7.1).
33
As a result of interviewing marketing users, finance users, sales designs that has a multipart key. Each component of the
force users, operational users, first- and second-level manage- multipart key is a foreign key to an individual dimension table.
ment, and senior manage-ment, a picture emerges of what is In the example of customer invoices, the “grain” of the fact
keeping these people awake at night. table is the individual line item on the customer invoice. In
Table 7.1 Nine-Step Method in the Design of a Data other words, a line item on an invoice is a single fact table
Warehouse record, and vice versa. Once the fact table representation is
decided, a coherent discussion of what the dimen-sions of the
1. Choosing the subject matter
data mart’s fact table are can take place.
2. Deciding what a fact table represents
Step 3: Identifying and conforming the dimensions. The
3. Identifying and conforming the dimensions. dimensions are the drivers of the data mart. The dimensions
4. Choosing the facts are the platforms for brows-ing the allowable constraint values
5. Storing precalculations in the fact table. and launching these constraints. The dimensions are the source’
of row headers in the user’s final reports; they carry the
6. Rounding out the dimension tables
enterprise’s vocabulary to the users. A well-architect set of
7. Choosing the duration of the database dimensions makes the data mart understandable and easy to
8. The need to track slowly changing dimensions use. A poorly presented or incomplete set of dimensions robs
9. Deciding the query priorities and the query modes the data mart of its usefulness. Dimensions should be chosen
with the long-range data warehouse in mind. This choice pre-
Can list and prioritize the primary business issues facing the
sents the primary moment at which the data mart architect must
enterprise. At the same time you should conduct a set of
disregard the data mart details and consider the longer-range
interviews with the legacy systems’ DBAs who will reveal which
plans. If any dimension occurs in two data marts, they must be
data sources are clean, which contain valid and consistent data,
exactly the same dimension, or one must be a mathematical
and which will remain supported over the next few years.
subset of the other. Only in this way can two data marts share
Preparing for the design with a proper set of interviews is one or more dimensions in the same application. When a
crucial. Inter-viewing is also one of the hardest things to teach dimen-sion is used in two data marts, this dimension is said to
people. I find it helpful to reduce the interviewing process to a be conformed. Good examples of dimensions that absolutely
tactic and an objective. Crudely put, the tactic is to make the end- must be conformed between data marts are the customer and
users talk about what they do, and the objective is to gain product dimensions in an enterprise. If these dimensions .are
insights that feed the nine design decisions. The tricky part is allowed drifting out of synchronization between data marts,
that the interviewer can’t pose the design questions directly to the overall data warehouse will fail, because the two data marts
the end users. End users talk about what are important in their will not be able to be used together. The requirement to
business lives. End users are intim-idated by system design conform dimensions across data marts i£ very strong. Careful
questions, and they are quite right when they insist that system thought must be given to this requirement before the firs: data
design is IT responsibility, not theirs. Thus, the challenge of the mart is implemented. The data mart team must figure out what
data mart designer is to meet the users tar more than half way. an enter-prise customer ID is and what an enterprise product
In any event, armed with both the top-down view (what keeps ID is. If this task is done correctly, successive data marts can be
management awake) and the bottom-up view (which data built at different times, on different machines, and by different
sources are available), the data warehouse designer may follow development teams, and these data marts
these steps: Will merge coherently into an overall data warehouse. In
Step 1: Choosing the subject matter of a particular data mart. particular, if the dimen-sions of two data marts are conformed,
The first data mart you build should be the one with the most it is easy to implement drill across by sending separate queries to
bangs for the buck. It should simultaneously answer the most the two data marts, and then sort-merging the answer sets on a
important business questions and be the most accessible in set of common row headers. The row headers can be made be
terms of data extraction. According to Kimball, a great place to common only if they are drawn from a conformed dimension
start in most enterprises is to build a data mart that consists of common to the two data marts.
cus-tomer invoices or monthly statements. This data source is With these first three steps correctly implemented, designers can
probably fairly accessible and of fairly high quality. One of attack last six steps (see Table 7.1). Each step gets easier if the
Kimball’s laws is that the best data source in any enterprise is preceding steps have been performed correctly.
the record of “how much money they owe us.” Unless costs
and profitability are easily available before the data mart is even Discussions
designed, it’s best to avoid adding these items to this first data • Write short notes on:
mart. Nothing drags down a data mart implementation faster • Data cleaning
than a heroic or impossible mission to provide activity-based
• Back flushing
costing as part of the first deliverable.
• Heterogeneous Sources
Step 2: Deciding exactly what a fact table record represents. This
step, according to R. Kimball, seems like a technical detail at this • Metadata repository
early point, but it is actually the secret to making progress on the
design. A table is the large central table in the dimensional
34
• Discuss various steps involved in the acquisition of data for

the data warehouse.
• List out various processes involved in data storage in a data
warehouse.
• What are the important design considerations, which need to
be thought of, while designing a data warehouse?
• Explain the difference between distributed warehouse and
the federated warehouse.
• What are the nine decisions in the design of a data
warehouse?
References
Notes
35
LESSON 8
BUILDING DATA WAREHOUSING - 2
Structure spend the extra time and build a core data warehouse first, and
then use this as the basis to quickly spin off many data marts.
• Objective
Disadvantage is this approach takes longer to build initially since
• Introduction
time has to be spent analyzing data requirements in the full –
• Data warehouse Application blown warehouse, identifying the data elements that will be
• Approaches used to build a Data Warehousing used in numerous marts down the road.
• Important considerations The advantage is that once you go to build the data mart, you
• Tighter Integration already have the warehouse to draw from.
• Empowerment Bottom-Up Approach, implying that the business priorities
resulted in developing individual data marts, which are then
• Willingness
integrated into the enterprise data warehouse. In this approach
• Reason for Building a Data Warehousing we need to build a workgroup specific data mart first. This
Objective approach gets data into your users hands quicker but the work it
The aim of this lesson is to study about Data warehouse takes to get the information into the data mart may not be
applications and various approaches that are used to build a reusable when moving the same data into a warehouse or trying
Data warehouse. to use similar data in different data mart.
Advantage is you gain speed but not portability
Introduction
It is the professional warehouse team deals with issues and We don’t care. In fact, we’d like to coin the term “ware mart”.
develops solutions, which wi1l best suit the needs of the The answer to which approach is correct depends on a number
analytical user community. A process of negotiation and, of vectors. You can take the approach that gets information in
sometimes of give and take is used to address issues that have to your user’s hands quickest. In our experience, that arguing
common ground between the players in the data warehouse over the marts and evolving into a warhorse. Results are what
delivery team. count not arguing over the data mart v/s the warehouse
approach.
Data Warehouse Application
In application, data warehouse application is different transac- Are they Different?
tion deals with large amounts of data, which is aggregate in Well, you can give many reasons your operational system and
nature, a data warehouse application answers questions like your data warehouses are not the same. The data need to
support operational needs is differently then the data needed to
• What is the average deposit by branch?
support analytical processing. In fact the data are physically
• Which day of the week is busiest? stored quite differently. An operationa1 system is optimized for
• Which customers with high average balances currently are not transactional updates, while a data warehouse system is
participating in a checking- plus account) optimized-for large queries dealing with large data sets. These
Because we are dealing with questions, each request is unique. differences become apparent. When you begin to monitor
The interface that supports this end user must be flexible by central processing unit (CPU) usage on a computer that contains
design. You have many different applications accessing the same a data warehouse v / s CPU usage on a system that contains an
information. operational system.
Each with a particular strength. A data mart is typically a subset Important Considerations
of your warehouse with a specific purpose in mind. For • Tighter Integration
example, you might have a financial mart and a marketing mart;
• Empowerment
each designed to feed information to a specific part of your
corporate business community. A key issue in the industry • Willingness
today is which approach should you take when building a Tighter Integration
Decision Support System? The term back end describes the data repository used to
Approaches used to build a Data Warehouse support the data warehouse coupled with the software that
We have experiences with two approaches to the build process: supports the repository. For example, Oracle 7 Cooperative
Server Front end describes the tools used by the warehouse end-
Top-Down Approach, meaning that an organization has
users to support their decision making activities, With classic
developed an enterprise data model, collected enterprise wide
operational systems sometimes ad-hoc dialog exist between the
business requirements, and decided to build an enterprise data
front end and back end to support specialties; with the data
warehouse with subset data marts. In this approach, we need to
36
warehouse team, this dialog must be ongoing as the warehouse c. Working management

springs to life. Unplanned fixes may be come necessary for the d. None of these
back end to support what the user community daces to do it
3. Which is not a key to developing data warehouse?
data warehouse system. This tighter integration between the
front end and the back end requires continual communication a. Structure of organization
between the data warehouse project team players. Some tools b. Change management
exist for end user analytic processing that require specific data c. User requirement
structures in the warehouse and in some cases specific naming
d. Methodology
standards, as result sometimes the nature of the front end tool
drives the design of the data structure in the warehouse. 4. Which of these factors does not track project development?
a. Performance measures based requirements
Empowerment
Users to control their own destiny- no running to the program- b. Enterprise requirements gathering
mers asking for reports programmed to satisfy burning needs. c. Cultural requirements
How many times in operational environments are new reports d. None of these
identified, scoped and programmed, only to discover by the tie
5. Which one is not a consideration for building a data
they are ready, the user has found other sources for the desired
warehouse?
output? An inevitable time delay exists between the identifica-
tion of new requirements and their delivery. This is no one’s a. Operating efficiency
fault it simple reality. b. Improved customer service
Willingness c. Competitive advantage
With the data warehouse Resistance often exists to making d. None of these
change, but then a warehouse initiative is rolled out with a set Discussion
of executives, manager clerks and analysts it will succeed. With
1. Write short notes on:
continual input and attention to detail the warehouse team
spends most of its time working wit the hands -on involve- • Tighter Integration
ment of the uses. All but the most stubborn users embrace the • Empowerment
new technology because it can provide them with answers to • Willingness
their question almost immediately. In shops here the move to a
2. Discuss different approaches used to build a Data
windows-like environment has no yet been made the simulta-
Warehousing. Which approach is generally used for building
neous rollout of a point-and- click interface and the placement
a data warehouse?
of power in the hands of the consumer is bound to succeed.
3. Discuss various applications of a data warehouse.
Reason for Building a Data Warehousing
Opportunity References
Considering a data warehouse development project within every 1. Anahory, Sam, Data warehousing in the real world: a practical
organization, many of the same frustration exist. These guide for building decision support systems, Delhi: Pearson
common themes are true business opportunity that a data Education Asia, 1997.
\warehouse can assist in achieving. Whenever one or more of 2. Corey, Michael, Oracle8 data warehousing, New Delhi: Tata
the following is heard, it is a signal that a true business reason McGraw- Hill Publishing, 1998.
exists for building a data warehouse
• We have all the data in our system. We just need access to it Delhi: Pearson Education Asia, 2000.
to make a better decision for running the business.
4. Alex Berson, Stephen J.Smith, Data warehousing, data mining
• We need an easier way to analyze information than training and OLTP: Tata McGraw- Hill Publishing, 2004
our business people in SQL.
Self Test
1. Identify the factors which does not facilitate change
management
a. Training
b. Communication
c. Rigid structure
d. None of these
2. Which group is not responsible for successful warehouse ?
a. Users
b. Top management
37
38
LESSON 9
BUSINESS CONSIDERATIONS: RETURN ON INVESTMENT DESIGN CONSIDERATIONS
Structure • The amount of data is doubling every 18 months, which

affects response time and the sheer ability to comprehend its
• Objective
content.
• Introduction
• Competition is. Heating up in the areas of business
• Need of a Data warehouse intelligence. And added Information value.
• Business Considerations: Return on Investment In addition, the necessity for data warehouses has increased as
• Approach organizations dis-tribute control away from the middle-
• Organizational issues management layer, which has traditionally provided and
• Design Considerations screened business information. As users depend more on
informa-tion obtained from Information Technology (IT)
• Data content systems - from critical-success measures to vital business-event-
• Metadata related information - the need to provide an information
• Data distribution warehouse for the remaining staff to use becomes more critical.
• Tools There are several technology reasons for the existence of data
• Performance considerations warehousing. First, the data warehouse is designed to address
the incompatibility of infor-mational and operational transac-
Objective tional systems. These two classes of informa-tion systems are
The objective of this lesson is to learn about the need of a data designed to satisfy different, often incompatible, requirements.
warehouse. It also includes topics related to various business At the same time, the IT infrastructure is changing rapidly, and
considerations and performance considerations. its capabilities are increasing, _s evidenced by the following:
Introduction • The price of MIPS (computer processing speed) continues to
The data warehouse is an environment, not a product. It is an decline, while
architectural construct of information systems that provides • The power of microprocessors doubles every 2 years.
users with current and historical decision support information
that is hard to access or present in traditional operational data
• The price of digital storage is rapidly dropping.
stores. In fact, the data warehouse is a cornerstone of the • Network bandwidth is increasing, while the price of high
organization’s ability to do effective information processing, bandwidth is decreasing.
which, among other things, can enable and shear the discovery • The workplace is increasingly heterogeneous with respect to
and exploration of important business trends and dependen- both the hardware and software.
cies that otherwise would have gone unnoticed. • Legacy systems need to, and can, be integrated with new
In principle, the data warehouse can meet informational needs applications.
of knowledge workers and can provide strategic business These business and technology drivers often make building a
opportunities by allowing customers and vendors access to data warehouse a strategic imperative. This chapter takes a close
corporate data while maintaining necessary security measures. look at what it takes to build a successful data warehouse.
There are several reasons why organizations consider data
warehousing a critical need. Business Considerations: Return on
Investment
Need of a Data Warehouse
There are several reasons why organizations consider data Approach
warehousing a crit-ical need. These drivers for data warehousing The information scope of the data warehouse varies with the
can be found in the business cli-mate of a global marketplace, in business require-ments, business priorities, and even magni-
the changing organizational structures of successful corpora- tude of the problem. The subject-oriented nature of the data
tions, and in the technology. warehouse means that the nature of the subject determines the
scope (or the coverage) of the warehoused information. Specif-
From a business perspective, to survive and succeed in today’s
ically, if the data warehouse is implemented to satisfy a specific
highly com-petitive global environment, business users
subject area (e.g., human resources), such a warehouse is
demand business answers mainly because
expressly designed to solve busi-ness problems related to
• Decisions need to be made quickly and correctly, using all personnel. An organization may choose to build another
available data. . Users are business domain experts, not warehouse for its marketing department. These two ware-
computer professionals. houses could be implemented independently and be completely
stand-alone applica-tions, or they could be viewed as compo-
39
nents of the enterprise, interacting with each other, and using a warehouse design: a business driven, continuous, iterative
common enterprise data model. As defined earlier the indi- warehouse engineering approach. In addition to these
vidual warehouses are known as data marts. Organizations general considerations, there are several specific points
embarking on data warehousing development can chose one of relevant to the data warehouse design.
the two approaches:
Data content
• The top-down approach, meaning that an organization has One common misconception about data warehouses is that
developed an enterprise data model, collected enterprise wide they should not con-tain as much detail-level data as operational
business requirements, an..: decided to build an enterprise systems used to source this data in. In reality, however, while
data warehouse with subset data marts the data in the warehouse is formatted differently from the
• The bottom-up approach, implying that the business operational data, it may be just as detailed. Typically, a data
priorities resulted i.e. developing individual data marts, warehouse may contain detailed data, but the data is cleaned up
which are then integrated into the enter-prise data warehouse and-transformed to fit the ware-house model, and certain
• The bottom-up approach is probably more realistic, but the transactional attributes of the data are filtered out. These
complexity of the integration may become a serious obstacle, attributes are mostly the ones used for the internal transaction
and the warehouse designer’s should carefully analyze each system logic, and they are not meaningful in the context of
data mart for integration affinity. analysis and decision-making.
The content and structure of the data warehouse are reflected in
Organizational Issues
its data model. The data model is the template that describes
Most IS an organization has considerable expertise in develop-
how information will be organized within the integrated
ing operational systems. However; the requirements and
warehouse framework. It identifies major sub-jects and
environments associated with the informational applications of
relationships of the model, including keys, attribute, and
a data warehouse are different. Therefore, an organization will
attribute groupings. In addition, a designer should always
need to employ different development practices than the ones it
remember that decision sup-port queries, because of their
uses for operational applications.
broad scope and analytical intensity, require data models to be
The IS department will need to bring together data that cuts optimized to improve query performance. In addition to its
across a com-pany’s operational systems as well as data from effect on query performance, the data model affects data storage
outside the company. But users will also need to be involved requirements and data loading performance.
with a data warehouse implementation since they are closest to
Additionally, the data model for the data warehouse may be
the data. In many ways, a data warehouse implemen-tation is
(and quite often is) different from the data models for data
not truly a technological issue; rather, it should be more
marts. The data marts, discussed in the previous chapter, are
concerned with identifying and establishing information
sourced from the data warehouse, and may contain highly
requirements, the data sources to fulfill these requirements, and
aggregated and summarized data in the form of a specialized
timeliness.
demoralized relational schema (star schema) or as a multidi-
Design Considerations mensional data cube. The key point is, however, that in a
To be successful, a data warehouse designer must adopt a dependent data mart environment, the data mart data is cleaned
holistic approach consider all data warehouse components as up, is transformed, and is consistent with the data warehouse
parts of a single complex system and take into the account all and other data marts sourced from the same warehouse.
possible data sources and all known usage requirements. Failing
Metadata
to do so may easily result in a data warehouse design that is
As already discussed, metadata defines the contents and location
skewed toward a particular business requirement, a particular
of data (data model) in the warehouse, relationships between
data source, or a selected access tool.
the operational databases and the data warehouse, and the
In general, a data warehouse’s design point is to consolidate business views of the warehouse data that are accessible by end-
data from mul-tiple, often heterogeneous, sources into a query user tools. Metadata is searched by users to find data
database. This is also one of the reasons why a data warehouse defini-tions or subject areas. In other words, metadata provides
is rather difficult to build. The main factors include decision-support oriented pointers to warehouse data, and thus
• Heterogeneity of data sources, which affects data conversion, provides a logical link between warehouse data and the decision
quality, and time-lines. Use of historical data, which implies support application. A data warehouse design should ensure
that data may be “old”. Tendency of databases to grow very that there is mechanisms that populates and maintains the
large. Another important point concerns the experience and metadata repository, and that all access paths to the data
accepted practices. Basi-cally, the reality is that the data warehouse have metadata as an entry point. To put it another
warehouse design is different from traditional OLTP. Indeed, way, the warehouse design should prevent any direct access to
the data warehouse is business-driven (not IS-driven, as in the warehouse data (especially updates) if it does not use
OLTP), requires continuous interactions with end users, and metadata definitions to gain the access.
is never finished, since both requirements and data sources
change. Understanding these points allows developers to
avoid a number of pitfalls relevant to data warehouse
development, and justifies a new approach to data
40
Data Distribution Thus, traditional database design and tuning techniques don’t

One of the biggest challenges when designing a data warehouse always work in the data warehouse arena. When designing a
is the data placement and distribution strategy. This follows data warehouse, therefore, the need to clearly understand users
from the fact that as the data Volumes continue to grow; the informational requirements becomes mandatory. Specifically,
database size may rapidly outgrow a single server. Therefore, it knowing how end users need to access various data can help
becomes necessary to know how the data should be divided design warehouse data-bases to avoid the majority of the most
across multiple servers, and which users should get access to expensive operations such as multitable scans and joins. For
which types of data. The data placement and distribution example, one design technique is to populate the ware-house
design should consider several options, including data distribu- with a number of demoralized views containing summarized,
tion by subject area (e.g., human resources, marketing), location derived, and aggregated data. If done correctly, many end-user
(e.g., geographic regions), or time (e.g., current, monthly, queries may execute directly against these views, thus maintain-
Quarterly). The designers should be aware that, while the ing appropriate overall perfor-mance levels.
distribution solves a number of problems, it may also create a
Discussions
few of its own; for example, if the warehouse servers are
distributed across multiple locations, a query that spans several • Write short notes on:
servers across the LAN or WAN may flood the network with a • Meta data
large amount of data. Therefore, any distribution strategy • Data distribution
should take into account all possible access needs for the • Data content
warehouse data.
• CASE tools
Tools
• Data marts
A number of tools available today are specifically designed to
help in the i1hple-mentation of a data warehouse. These tools • Discuss various design considerations, which are taken into
provide facilities for defining the transformation and cleanup account while building a data warehouse.
rules, data movement (from operational sources into the • Explain the need and importance of a data warehouse.
warehouse), end-user query, reporting, and data analysis. Each • Describe the organization issues, which are to be considered
tool takes a slightly different approach to data warehousing and while building a data warehouse.
often maintains its own version of the metadata, which is
• Explain the need of Performance Considerations
placed in a tool-specific, proprietary meta-data repository. Data
warehouse designers have to be careful not to sacrifice the • “Organizations embarking on data warehousing
overall design to fit a specific tool. At the same time, the development can choose one of the two approaches”.
designers have to make sure that all selected tools are compat- Discuss these two approaches in detail.
ible with the given data warehouse envi-ronment and with each • Explain the business considerations, which are taken into
other. That means that all selected tools can use a com-mon account while building a data warehouse
metadata repository. Alternatively, the tools should be able to References
source the metadata from the warehouse data dictionary (if it
exists) or from a CASE tool used to design the warehouse
database. Another option is to use metadata gate-ways that
translate one tool’s metadata into another tool’s format. If
these requirements are not satisfied, the resulting warehouse 2. Adriaans, Pieter, Data mining, Delhi: Pearson Education
environment may rapidly become unmanageable, since every Asia, 1996.
modification to the warehouse data model may involve some 3. Corey, Michael, Oracle8 data warehousing, New Delhi: Tata
significant and labor-intensive changes to the meta-data’ McGraw- Hill Publishing, 1998.
definitions for every tool in the environment. And then, these 4. Elmasri, Ramez, Fundamentals of database systems, 3rd ed.
changes would have to be verified for consistency and integrity. Delhi: Pearson Education Asia, 2000.
Performance Considerations
Although the data warehouse design point does not include
sub second response times typical of OLTP systems, it is
nevertheless a clear business requirement that an ideal data
warehouse environment should support inter-active query
processing. In fact, the majority of end-user tools are designed
as interactive applications. Therefore, “rapid” query processing is
a highly desired feature that should be designed into the data
warehouse. Of course, the actual performance levels are
business-dependent and vary widely from one environ-ment to
another. Unfortunately, it is relatively difficult to predict the
perfor-mance of a typical data warehouse. One of the reasons
for this is the unpredictable usage pattern against the data.
41
LESSON 10
TECHNICAL CONSIDERATION, IMPLEMENTATION CONSIDERATION
Structure • The communications infrastructure that connects the

• Objective warehouse, data marts,
• Introduction • Operational systems, and end users
• Technical Considerations • The hardware platform and software to support the
• Hardware platforms metadata repository
• Balanced approach • The systems management framework that enables centralized

management and administration of the entire environment.
• Optimal hardware architecture for parallel query scalability
Let’s look at some of these issues in more detail.
• Data warehouse and DBMS specialization
• Communications infrastructure
Hardware Platforms
Since many data warehouse implementations are developed into
• Implementation Considerations already exist-ing environments, many organizations tend to
• Access tools leverage the existing platforms and skill base to build a data
Objective warehouse. This section looks at the hardware platform
The purpose of this lesson is to take a close look at what it selection from an architectural viewpoint: what platform is best
takes to build a successful data warehouse. Here, I am focusing to build a successful data warehouse from the ground up.
on the Technical Issues, which are to be considered when An important consideration when choosing a data warehouse
designing and implementing a data warehouse. server is its capacity for handling the volumes of data required
by decision support appli-cations, some of which may require a
Introduction
significant amount of historical (e.g., up to 10 years) data. This
There are several technology reasons for the existence of data
capacity requirement can be quite large. For example, in general,
warehousing. First, the data warehouse is designed to address
disk storage allocated for the warehouse should be 2 to 3 times
the incompatibility of informational and operational transac-
the size of the data component of the warehouse to accommo-
tional system. These two classes of informa-tion systems are
date DSS processing, such as sorting, storing of intermediate
designed to satisfy different, often incompatible, requirements.
results, summarization, join, and format-ting. Often, the
At the same time, the IT infrastructure is changing rapidly, and
platform choice is the choice between a mainframe and non-
its capabilities are increasing, _s evidenced by the following:
MVS (UNIX or Windows NT) server.
• The price of MIPS (computer processing speed) continues to
Of course, a number of arguments can be made for and against
decline, while
each of these choices. For example, a mainframe is based on a
• The power of microprocessors doubles every 2 years. proven technology; has large data and throughput capacity; is
• The price of digital storage is rapidly dropping. reliable, available, and serviceable; and may support the legacy
• Network bandwidth is increasing, while the price of high databases that are used as sources for the data warehouse. The
bandwidth is decreasing. data warehouse residing on the mainframe is best suited for
situations in which large amounts of legacy data need to be
• The workplace is increasingly heterogeneous with respect to
stored in the data warehouse. A mainframe system, however, is
both the hardware and software.
not as open and flexible as a contemporary client/server system,
• Legacy systems need to, and can, be integrated with new and is not optimized for ad hoc query processing. A mod-ern
applications. server (no mainframe) can also support large data volumes and
These business and technology drivers often make building a a large number of flexible GUI-based end-user tools, and can
data warehouse a strategic imperative. This chapter takes a close relieve the mainframe from ad hoc query processing. However,
look at what it takes to build a successful data warehouse. in general, non-MVS servers are not as reliable as mainframes,
Technical Considerations are more difficult to manage and integrate into the existing
A number of technical issues are to be considered when environment, and may require new skills and even new
designing and imple-menting a data warehouse environment. organizational structures.
These issues include: From the architectural viewpoint, however, the data warehouse
• The hardware platform that would house the data server has to be specialized for the tasks associated with the data
warehouse warehouse, an main frame can be well suited to be a data
warehouse server. Let’s look at the hardware features that make
• The database management system that supports the a server-whether it is mainframe, UNIX-, or NT-based-an
warehouse database appropriate technical solution for the data warehouse.
42
To begin with, the data warehouse server has to be able to induced data skew is more severe in the low-density asymmetric

support large data volumes and complex query processing. In connection architectures (e.g., daisy-chained, 2-D and 3-D
addition, it has to be scalable, since the data warehouse is never mesh), and is virtu-ally nonexistent in symmetric connection
finished, as new user requirements, new data sour and more architectures (e.g., cross-bar switch). Thus, when selecting a
historical data are continuously incorporated into the ware- hardware platform for a data warehouse, take into account the
house, a clear as the user population of the data warehouse fact that the system-architecture-induced data skew can over-
continues to grow. Therefore, a clear -requirement for the data power even the best data layout for parallel query execution, and
warehouse server is the scalable high performance data loading can force an expen-sive parallel computing system to process
and ad hoc query processing as well as the ability to support queries serially.
databases in a reliable, efficient fashion. Chapter 4 briefly
Data warehouse and DBMS
touched on various design points to enable server specialization
Specialization
for scalability in performance throughput, user support, and
To reiterate, the two important challenges facing the developers
very large database (VLDB) processing.
of data ware-houses are the very large size of the databases and
Balanced Approach the need to process complex ad hoc queries in a relatively short
An important design point when selecting a scalable computing time. Therefore, among the most important requirements for
platform is the right balance between all computing component the data warehouse DBMS are performance, throughput, and
example, between the number of processors in a multiproces- scalability.
sor system an the I/O bandwidth. Remember that the lack of The majority of established RDBMS vendors have imple-
balance in a system inevitably results in a bottleneck! mented various degrees of parallelism in their respective
Typically, when a hardware platform is sized to accommodate products. Although any relational database management
the data house, this sizing is frequently focused on the number system-such as DB2, Oracle, Informix, or Sybase--supports
and size of disks. A typical disk configuration. Includes 2.5 to 3 parallel database processing, some of these products have been
times the amount of raw important consideration-disk architect to better suit the specialized requirements of the data
throughput comes from the actual number of disks, and not warehouse.
the total disk space. Thus, the number of disks has direct on In addition to the “traditional” relational DBMSs, there are
data parallelism. To balance the system, it is very important to databases that have been optimized specifically for data
correct number of processors to efficiently handle all disk I/O warehousing, such as Red Brick Warehouse from Red Brick
operations. If this allocation is not balanced, an expensive data Systems.
warehouse platform can rapidly become CPU-bound. Indeed,
since various processors have widely performance ratings and Communications Infrastructure
thus can support a different number of CPU, data warehouse When planning for a data warehouse, one often-neglected
designers should carefully analyze the disk I/O processor aspect of the archi-tecture is the cost and efforts associated with
capabilities to derive an efficient system configuration. For if it bringing access to corporate data directly to the desktop. These
takes a CPU rated at 10 SPECint to efficiently handle one 3- costs and efforts could be significant, since many large organiza-
Glry- _ drive, then a single 30 SPECint processor in a tions do not have a large user population with direct electronic
multiprocessor system can handle three disk drives. Knowing access to information, and since a typical data warehouse user
how much data needs to be processed, should give you an idea requires a rela-tively large bandwidth to interact with the data
of how big the multiprocessor ‘system should be. A consider- warehouse and retrieve a sig-nificant amount of data for the
ation is related to disk controllers. A disk controller can support analysis. This may mean that communications networks have to
a -amount of data throughput (e.g., 20 Mbytes/s). Knowing be expanded, and new hardware and software may have to be
the per-disk through-put ratio and the total number of disks purchased.
can tell you how many controller given type should be config- Implementation Considerations
ured in the system. A data warehouse cannot be simply bought and installed-its
The idea of a balanced approach can (and should) be carefully implementation requires the integration of many products
extended to all system components. The resulting system within a data warehouse. The caveat here is that the necessary
configuration will easily handle known workloads and provide a customization drives up the cost of implementing a data
balanced and scalable computing platform for future growth. warehouse. To illustrate the complexity of the data warehouse
implemen-tation, let’s discuss the logical steps needed to build a
Optimal hardware architecture for parallel query data warehouse:
scalability
An important consideration when selecting a hardware platform • Collect and analyze business requirements.
for a data warehouse is the -ability. Therefore, a frequent • Create a data model and a physical design for the data
approach to system selection is to take of hardware parallelism warehouse.
that comes in the form of shared-memory symmetric multipro- • Define data sources.
cessors (SMPs), clusters, and shared-nothing • Choose the database technology and platform for the
distributed-memory system terns (MPPs). As was shown in warehouse.
Chap. 3, the scalability of these systems can be seriously affected
by the system-architecture-induced data skew. This architecture-
43
• Extract the data from the operational databases, transform it, Discussions
and clean it up. • Write short notes on:

• And load it into the database. • Access tools
• Choose database access and reporting tools. • Hardware platforms
• Choose database connectivity software. • Balanced approach
• Choose data analysis and presentation software. • Discuss Data warehouse and DBMS specialization
• Update the data warehouse. • Explain Optimal hardware architecture for parallel query
When building the warehouse, these steps must be performed scalability
within the constraints of the current state of data warehouse • Describe the Technical issues, which are to be considered
technologies. . while building a data warehouse.
Access Tools • Explain the Implementation Considerations, which are
Currently, no single tool on the market can handle all possible taken into account while building a data warehouse
data warehouse Access needs. Therefore, most implementations
References
rely on a suite of tools. The best way to choose this suite
includes the definition of different types of access. The data and 1. Anahory, Sam, Data warehousing in the real world: a practical
selecting the best tool for that kind of access. Examples of guide for building decision support systems, Delhi: Pearson
access types include Education Asia, 1997.
• Simple tabular form reporting 2. Adriaans, Pieter, Data mining, Delhi: Pearson Education
Asia, 1996.
• Ranking.
• Multivariable analysis
• Time series analysis
4. Berson, Smith, Data warehousing, Data Mining, and OLAP,
• Data visualization, graphing, charting, and pivoting. New Delhi: Tata McGraw- Hill Publishing, 2004
Complex textual search
• Statistical analysis Delhi: Pearson Education Asia, 2000.
• Artificial intelligence techniques for testing of hypothesis,
trends discovery definition, and validation of data clusters
and segments
• Information mapping (i.e., mapping of spatial data in
geographic’ information systems)
• Ad hoc user-specified queries
• Predefined repeatable queries
• Interactive drill-down reporting and analysis
• Complex queries with multitable joins, multilevel
subqueries, and sophisticated search criteria
In addition, certain business requirements often exceed existing
tool capabili-ties and may require building sophisticated
applications to retrieve and analyze warehouse data. These
applications often take the form of custom-developed screens
and reports that retrieve frequently used data and format it in a
pre-defined standardized way. This approach may be very useful
for those data warehouse users who are not yet comfortable
with ad hoc queries.
There are a number of query tools on the market today. Many
of these tools are designed to easily compose and execute ad
hoc queries and build cus-tomized reports with little knowledge
of the underlying database technology, SQL, or even the data
model (i.e., Impromptu from Cognos, Business Objects, etc.),
while others (e.g., Andyne’s GQL) provide relatively low-level
capabilities for an expert user to develop complex ad hoc queries
in a fashion similar to developing SQL queries for relational
databases. Business requirements that exceed the capabilities of
ad hoc query and reporting tools are fulfilled by dif-ferent classes
of tools: OLAP and data mining tools.
44
LESSON 11
BENEFITS OF DATA WAREHOUSING
Structure at your organization is not striking it rich, you might want to

ask your data warehousing experts if they’ve ever heard of
• Objective
fool’s gold.
• Introduction
Benefits of Data Warehousing
• Benefits of Data warehousing
Data warehouse usage includes
• Tangible Benefits
• Locating the right information
• Intangible Benefits
• Presentation of information (reports, graphs). Testing of
• Problems with data warehousing hypothesis
• Criteria for a data warehouse • Discovery of information
Objective • Sharing the analysis
The aim of this lesson is to study various benefits provided by Using better tools to access data can reduce outdated, historical
a data warehouse. You will also learn about the problems with data. Arise; users can obtain the data when they need it most,
data warehousing and the criteria for Relational Databases for often during business. Decision processes, not on a schedule
building a data warehouse predetermined months earlier by the department and computer
Introduction operations staff.
Today’s data warehouse is a user-accessible database of historical Data warehouse architecture can enhance overall availability of
and functional company information fed by a mainframe. business intelligence data, as well as increase the effectiveness
Unlike most systems, it’s set up according to business rather and timeliness of business decisions.
than computer logic. It allows users to dig and churn through
large caverns of important consumer data, looking for relation-
Tangible Benefits
Successfully implemented data warehousing can realize some
ships and making queries. That process-where users sift
significant tangible benefits. For example, conservatively
through piles of facts and figures to discover trends and
assuming an improvement in out-of-stock conditions in the
patterns that suggest new business opportunities-is called data
retailing business that leads to 1 percent increase in sales can
mining.
mean a sizable cost benefit (e.g., even for a small retail business
All that shines is not gold, however. Data warehouses are not with $200 million in annual sales, a conservative 1 percent
the quick-hit fix that some assume them to be. A company improvement in salary yield additional annual revenue of $2
must commit to maintaining a data warehouse, making sure all million or more). In fact, several retail enterprises claim that data
of the data is accurate and timely. warehouse implementations have improved out-of stock
Benefits and rewards abound for a company that builds and conditions to the extent that sales increases range from 5 to 20
maintains a data warehouse correctly. Cost savings and increases percent. This benefit is in addition to retaining customers who
in revenue top the list for hard returns. Add to that an increase might not have returned if, because of out-of-stock problems,
in analysis of marketing databases to cross-sell products, less they had to do business with other retailers.
computer storage on the mainframe and the ability to identify Other examples of tangible benefits of a data warehouse
and keep the most profitable customers while getting a better initiative include the following:
picture of who they are, and it’s easy to see why data warehous-
ing is spreading faster than a rumor of gold at the old mill.
• Product inventory turnover is improved.
For example, the telecom industry uses data warehouses to

• Costs of product introduction are decreased with improved
selection of target markets.
target customers who may want certain phone services rather
than doing “blanket” phone and mail campaigns and aggravat- • More cost-effective decision making is enabled by separating
ing customers with unsolicited calls during dinner. (ad hoc) query
Some of the soft benefits of data warehousing come in the • Processing from running against operational databases.
technology’s effect on users. When built and used correctly, a • Better business intelligence is enabled by increased quality
warehouse changes users’ jobs, granting them faster access to and flexibility of market analysis available through multilevel
more accurate data and allowing them to give better customer data structures, which may range from detailed to highly
service. summarize. For example, determining the effectiveness of
A company must not forget, however, that the goal for any data marketing programs allows the elimination of weaker pro-
warehousing project is to lower operating costs and generate grams and enhancement of stronger ones.
revenue—this is an investment, after all, and quantifiable ROI • Enhanced asset and liability management means that a data
should be expected over time. So if the data warehousing effort warehouse can provide a “big” picture of enterprise wide
45
purchasing and inventory patterns, and can indicate • Data Quality Management - The shift to fact-based
otherwise unseen credit exposure and opportunities for cost management demands the highest data quality. The
savings. warehouse must ensure local consistency, global consistency,
and referential integrity despite “dirty” sources and massive
Intangible Benefits
database size. While loading and preparation are necessary
In addition to the tangible benefits outlined above, a data
steps, they are not sufficient. Query throughput is the
warehouse provides a number of intangible benefits. Although
measure of success for a data warehouse application. As
they are more difficult to quantify, intangible benefits should
more questions are answered, analysts are catalyzed to ask
also be considered when planning for the data ware-house.
more creative and insightful questions.
Examples of intangible benefits are:
1. Improved productivity, by keeping all required data in a
• Query Performance - Fact-based management and ad-hoc
analysis must not be slowed or inhibited by the performance
single location and eliminating the rekeying of data
of the data warehouse RDBMS; large, complex queries for
2. Reduced redundant processing, support, and software to key business operations must complete in seconds not days.
support overlapping decision support applications.
• Terabyte Scalability - Data warehouse sizes are growing at
3. Enhanced customer relations through improved knowledge astonishing rates. Today these range from a few to hundreds
of individual requirements and trends, through of gigabytes, and terabyte-sized data warehouses are a near-
customization, improved communica-tions, and tailored term reality. The RDBMS must not have any architectural
product offerings limitations. It must support modular and parallel
4. Enabling business process reengineering-data warehousing management. It must support continued availability in the
can provide useful insights into the work processes event of a point failure, and must provide a fundamentally
themselves, resulting in developing breakthrough ideas for different mechanism for recovery. It must support near-line
the reengineering of those processes mass storage devices such as optical disk and Hierarchical
Problems with Data Warehousing Storage Management devices. Lastly, query performance must
One of the problems with data mining software has been the not be dependent on the size of the database, but rather on
rush of companies to jump on the band wagon as the complexity of the query.
these companies have slapped ‘data warehouse’ labels on • Mass User Scalability - Access to warehouse data must no
traditional transaction-processing products, and co-opted longer be limited to the elite few. The RDBMS server must
the lexicon of the industry in order to be considered support hundreds, even thousands, of concurrent users
players in this fast-growing category. while maintaining acceptable query performance.
Chris Erickson, president and CEO of Red Brick (HPCwire, • Networked Data Warehouse - Data warehouses rarely exist
Oct. 13, 1995) in isolation. Multiple data warehouse systems cooperate in a
larger network of data warehouses. The server must include
Red Brick Systems have established a criteria for a relational
tools that coordinate the movement of subsets of data
database management system (RDBMS) suitable for data
between warehouses. Users must be able to look at and work
warehousing, and documented 10 specialized requirements for
with multiple warehouses from a single client workstation.
an RDBMS to qualify as a relational data warehouse server, this
Warehouse managers have to manage and administer a
criteria is listed in the next section.
network of warehouses from a single physical location.
According to Red Brick, the requirements for data warehouse
• Warehouse Administration - The very large scale and time-
RDBMSs begin with the loading and preparation of data for
cyclic nature of the data warehouse demands administrative
query and analysis. If a product fails to meet the criteria at this
ease and flexibility. The RDBMS must provide controls for
stage, the rest of the system will be inaccurate, unreliable and
implementing resource limits, chargeback accounting to
unavailable.
allocate costs back to users, and query prioritization to
Criteria for a Data Warehouse address the needs of different user classes and activities. The
The criteria for data warehouse RDBMSs are as follows: RDBMS must also provide for workload tracking and tuning
• Load Performance - Data warehouses require incremental so system resources may be optimized for maximum
loading of new data on a periodic basis within narrow time performance and throughput. “The most visible and
windows; performance of the load process should be measurable value of implementing a data warehouse is
measured in hundreds of millions of rows and gigabytes per evidenced in the uninhibited, creative access to data it
hour and must not artificially constrain the volume of data provides the end user.
required by the business. • Integrated Dimensional Analysis - The power of
• Load Processing - Many steps must be taken to load new multidimensional views is widely accepted, and dimensional
or updated data into the data warehouse including data support must be inherent in the warehouse RDBMS to
conversions, filtering, reformatting, integrity checks, physical provide the highest performance for relational OLAP tools.
storage, indexing, and metadata update. These steps must be The RDBMS must support fast, easy creation of
executed as a single, seamless unit of work. precomputed summaries common in large data warehouses.
It also should provide the maintenance tools to automate
the creation of these precomputed aggregates. Dynamic
46
calculation of aggregates should be consistent with the

interactive performance needs.
• Advanced Query Functionality - End users require
advanced analytic calculations, sequential and comparative
analysis, and consistent access to detailed and summarized
data. Using SQL in a client/server point-and-click tool
environment may sometimes be impractical or even
impossible. The RDBMS must provide a complete set of
analytic operations including core sequential and statistical
operations.
Discussions
• Query Performance
• Integrated Dimensional Analysis
• Load Performance
• Mass User Scalability
• Terabyte Scalability
• Discuss in brief Criteria for a data warehouse
• Explain Tangible benefits. Provide suitable examples for
explanation.
• Discuss various problems with data warehousing
• Explain Intangible benefits. Provide suitable examples for
explanation.
• Discuss various benefits of a Data warehouse.
References
Asia, 1996.
New Delhi: Tata McGraw- Hill Publishing, 2004
Notes
47
CHAPTER 3
LESSON 12 MANAGING AND IMPLEMENTING A
PROJECT MANAGEMENT PROCESS, DATA WAREHOUSE PROJECT
SCOPE STATEMENT
Structure Every project must have a defined start and end. If you are
• Objective unclear about what must be done to complete the project, don’t
start it.
• Introduction
If at any point in the project if becomes clear that the end
• Project Management Process
product of the project cannot be delivered then the project
• Scope Statement should be cancelled. This also applies to a data warehouse
• Project planning project.
• Project scheduling One must have a clear idea of what you are building and why it
• Software Project Planning is unique.
• Decision Making It is not necessary to understand why your product or service is
unique from day one. But through the process of developing
Objective the project plan, the uniqueness of the project. Only with a
define end do you have a project that can succeed
various topics related to Process Management Process. It also
includes the need of Scope statement; project planning, Project In addressing the unique aspect or providing business users
Scheduling Software Project Planning and decision-making. with timely access to data amidst constantly changing business
conditions, Whether you are embarking on a customer relation-
Introduction ship management initiative, a balanced scorecard
Now here we will talk about project management when we implementation, a risk management system or other, decision
haven’t yet defined the term “project”, For this definition, we support applications, there is a large body of knowledge about
for to the source - the project management institute, The Project what factors contributed the most to the failures of these type
management institute is a nonprofit professional organization of initiative. We, as an industry: need to leverage this knowledge
dedicated to advancing the state of the art in the management and learn these lessons so we don’t repeat them in our own
of projects. projects.
Membership is open to anyone actively engaged or interested in A data warehouse is often the foundation for many of decision
the application practice teaching, and researching of project support initiatives. Many studies have show that a significant
management principles and techniques. reason for the failure of data warehousing
According to the book. A Guide to the project management And decision support projects is not failure of the technology,
body knowledge, published by the project management but rather, inappropriate project management including lack of
institute standards committee. A project is a temporary integration, lack of communication and lack of clear linkages to
endeavor undertaken to create u unique product or service. business objectives and to benefits achievement.
One key point here is temporary endeavor. Any project must
The Scope Statement
have a defined start and end, If you are unclear about what
must be done to complete the project, don’t start it. This is a One of the major proven techniques we can use to help us with
sure path to disaster. In addition, if at any point in the project this discovery process is called a scope statement. A scope
if becomes clear that the end product of the project cannot be statement is a written document by which you begin to define
delivered then the project should be called. This also applies to a the job at hand and all the key deliverables. In fact, we feel it is
data warehouse project. Another key point is to create a unique good business practice not to begin work on any project until
product or service. You must have a clear idea of what you are you have developed a scope statement. No work should begin
building and why it is unique. It is not necessary for you to on, a project until a scope statement has been developed. These
understand why your product or service is unique from day are the major elements in the breakdown of a scope statement.
one. But through the process of developing the project plan the 1.Project Title and Description: Every project should have a
uniqueness of the project. Only with a define end do you have a clear name and description of what you are trying to accom-
project that can succeed. plish.
Project Management Process 2.Project Justification: clear description of why this project is
The project management institute is a nonprofit professional being done. What is the goal of the project?
organization dedicated to advancing the state of the art in the 3. Project Key Deliverables: a 1ist of keys items that must be
management of projects. accomplished so this project can be completed. What must be
A project is a temporary endeavor undertaken to create a unique done for us to consider the project done?
product or service.
48
4. Project Objective: an additional list of success criteria. During early stages of planning, critical decisions must be made

These items must be measurable: a good place to put any time, due to estimation results e.g.
money, or resource constraints. • Whether or not to bid on a contract, and if so, for how
Think of scope statement as your first stake in the ground. much
What’s important is that a scope statement provides a docu- • Whether or not build software, or to purchase existing
mented basis for building a common understanding among all software that has much of the desired functionality
the shareholders of the project at hand. It is crucial that you
• Whether or not to subcontract (outsource) some of the
begin to write down and document the project and all its
software development
assumptions. If you do not do this, you will get burned. All a
stakeholder remembers about a hallway conversation is the The project plan is the actual description of:
deliverable, not the constraints. Many project managers have • The tasks to be done
been burned by making a hallway statement, such as: “If the • Who is responsible for the tasks
new network is put into place by July I, feel comfortable saying
• What order tasks are to be accomplished
we can provide you with the legacy data by July 10, not All the
stakeholder will remember will remember is the date of July 10, • When the tasks will be accomplished, and how long they will
not the constraint associated with it. take
A scope stmt provides a documented basis for building a The Project Plan is updated throughout the project.
common understanding among all the shareholders of the Project planning is only part of the overall management of the
project at hand. project. Other project management activities include:
It is crucial that you begin to write down and document the • Risk analysis and management
project and all its assumptions. If you do not do this, you • Assigning and organizing project personnel
might get into some problem.
• Project scheduling and tracking
Project Planning • Configuration management
Project planning is probably the most time-consuming project
Project Management (across disciplines) is actually considered a
management activity. It is a continuous activity from initial
whole separate field from software engineering. The Project
concept through to system delivery. Plans must be regularly
Management Institute (http://www.pmi.org) has actually
revised as new information becomes available Various different
developed an entire Body of Knowledge for project manage-
types of plan may be developed to support the main software
ment, called PMBOK. However, there are some aspects of
project plan that is concerned with schedule and budget.
software project management that are unique to the manage-
Types of project plan are: ment of software.
• Quality plan There are four dimensions of development:
• Validation plan • People
• Configuration management plan • Process
• Maintenance plan • Product
• Staff development plan • Technology
A project plan is usually of the following structure: The resources a manager should consider during project
• Introduction planning are:
• Project organization • Human Resources: This includes “overhead”, and by far is
• Risk analysis the most dominant aspect of software costs and effort.
• Hardware and software resource requirements • Software Resources: COTS, in-house developed software and
tools, reusable artifacts (e.g. design patterns), historical data
• Work breakdown
(helps with cost/effort estimation)
• Project schedule
Note that there is usually not much involving physical re-
• Monitoring and reporting mechanisms sources, except as it relates to overhead.
Three important concepts in project planning are:
Project Scheduling
• Activities in a project should be organised to produce
During project scheduling:
tangible outputs for management to judge progress
• Split the project into tasks and estimate time and resources
• Milestones are the end-point of a process activity
required to complete each task.
• Deliverables are project results delivered to customers
• Organize tasks concurrently to make optimal use of work
The waterfall process allows for the straightforward definition force
of progress milestones.
• Minimize task dependencies to avoid delays caused by one
task waiting for another to complete
49
This activity is highly dependent on project managers intuition • Set policies and procedures
and experience. Why do we need to make a project plan?

Some scheduling problems are: There are two quotations that I’d like to include here:
• Estimating the difficulty of problems and hence the cost of • No one plans to fail, they just fail to plan.
developing a solution is hard
• Plan the flight, fly the plan.
• Productivity is not proportional to the number of people
Planning is about making a schedule that involves using tools
working on a task
like
• Adding people to a late project makes it later because of
• Activity Networks: PERT/CPM
communication overheads
• Gantt Chart
• For further reading on this, look at Frederick Brooks’ book
“The Mythical Man Month” • Tabular Notation
• The unexpected always happens; always allow contingency in Activity networks show:
planning. • Activities: Task + Duration + Resources
Scheduling starts with estimation of task duration: • Milestones: No Duration, No Resources
• Determination of task duration Critical Path Method
• Estimation of anticipated problems Duration of an activity is calculated with the formula
• Consideration of unanticipated problems Duration = (a + 4m + b)/6
• Use of previous project experience Where a is the pessimistic, m is the most likely and b is the
Activity charts are used to show task dependencies. They can be optimistic estimation of the duration of that activity.
used to identify the critical path i.e. the project duration. Terminologies relating to timing between activities are:
Bar charts and activity networks are: • ES: early start, soonest an activity can begin
• Graphical notations used to illustrate the project schedule • LS: late start, latest time an activity can begin
• Show project breakdown into tasks • EF: early finish, earliest an activity can finish
• Tasks should not be too small. They should take about a • LF: late finish, latest an activity can finish
week or two
• Slack: difference between ES and LS, if zero critical path
• Activity charts show task dependencies and the the critical
The scope of the project describes the desired functionality of
path
the software, and what constraints are involved. Therefore,
• Bar charts show schedule against calendar time project scope is determined as part of the analysis process by the
• Used to show activity timelines requirements team. The project manager then uses the scope
• Used to show staff allocations that has been defined in order to determine project feasibility,
and to estimate what costs, total effort and project duration will
Software Project Planning be involved in engineering the software.
Software project planning is the process of So does that mean that planning should come after require-
• Determining what the scope and available resources for a ments? Definitely no. Project planning is actually an iterative
project, process, which starts at the beginning of the project, and
• Estimating the costs, total effort and duration of the project continues throughout.
required to develop and deliver the software, Decision Making
• Determining the feasibility of the project, Decision-making during software project planning is simply
• Deciding on whether or not to go forward with project option analysis. In a project, the manager must achieve a set of
implementation, essential project goals. He also may have desirable (but not
• Creating a plan for the successful on-time and within budget essential) organisational goals. He must plan within
delivery of reliable software that meets the client’s needs, and organisational constraints. There are usually several different
ways of tackling the problem with different results as far as goal
• Updating the plan as necessary throughout the project
achievement is concerned.
lifetime.
Decision-making is usually based on estimated value: by
Planning is primarily the job of the “Project Manager” and its
comparing the values of possible alternatives.
steps are
A few examples of goals and constraints are:
• Develop Goals
• Project goals: high reliability, maintainability, development
• Identify the end state
within budget, development time <2 years
• Anticipate problems
• Desirable goals: staff development, increase organisational
• Create alternative courses of action profile in application area, reusable software components
• Define strategy
50
• Organisational constraints: use of a specific programming Discussions

language or tool set, limited staff experience in application • Explain the following with related examples:
area.
• Scope Statement
In decision-making, it is advised to make goals as explicit and
• Goals
(wherever possible) quantified as possible. Then score possible
options against each goal, and use a systematic approach to • Deliverables
option analysis to select the best option • Organizational Constraints
It always helps to use graphs and visual aids during decision- • Decision making
making. When a single value is used, a list is usually sufficient. • Milestones
When more than one value is considered during decision-
• What is Project Management Process?
making, it is advised to use Polar Graphs for presenting this
information. • Describe the need of Project planning. Can a project be
completed in time without Project Planning? Explain with
As a practice, consider the following problem:
an example.
You are a project manager and the Rational salesman tells you
• Explain the significance of Project scheduling
that if you buy the Rational Suite of tools, it will pay for itself
in just a few months. • What is Software Project Planning?
The software will cost you $9600. You talk to other companies • Describe the contents that are included in a Project Plan. Why
and find that there is a 50% chance that the software will cut do we need to make a project plan?
defect rework costs by $32000. Reference
There is a 20% chance that there will be no improvement and a 1. Hughes, Bob, Software project management, 2nd ed. New
30% chance that negative productivity will increase project costs Delhi: Tata McGraw- Hill Publishing, 1999.
by $20000. 2. Kelkar, S.A., Software project management: a concise study, New
Should you buy this software package? Delhi: Prentice Hall of India, 2002
Summary 3. Meredith, Jack R.; Mantel, Samuel J., Project management: a
• Good project management is essential for project success managerial approach, New York: John Wiley and Sons, 2002.
• The intangible nature of software causes problems for 4. Royce, Walker, Software project management: a unified
management framework, Delhi: Pearson Education Asia, 1998.
• Managers have diverse roles but their most significant 5. Young, Trevor L., Successful project management, London:
activities are planning, estimating and scheduling Kogan Page, 2002.
• Planning and estimating are iterative processes which

continue throughout the course of a project
• A project milestone is a predictable state where some formal
report of progress is presented to management.
• The primary objective of software project planning is to plan
for the successful on-time and within budget delivery of
reliable software that meets the client’s needs
• Project scope should be determined through requirements
analysis and used for planning by the project manager
• The primary resources needed for a software project are the
actual developers
• Early on in a project decisions must be made on whether or
not to proceed with a project, and whether or not to buy or
build the software
• The project tasks, schedule and resource allocation is all part
of the Project Plan
• Project planning is only part of the project management
process
• Effective management depends on good planning
• Planning and estimating are iterative process
• Milestones should occur regularly
• Option analysis should be carried out to make better
decision
51
LESSON 13
WORK BREAKDOWN STRUCTURE
Structure Revise the work breakdown structure whenever design concepts

• Objective or other key requirements change during the project.
• Introduction How to build a WBS (a serving

• Work Breakdown Structure
suggestion)
1. Identify the required outcome of the project, the overall
• How to build a WBS
objective you want to achieve. Write it on a Post-It Note and
• To create work breakdown structure stick it high on a handy wall.
• From WBS to Activity Plan 2. Brainstorm with your project team and sponsor:
• Estimating Time • Identify all the processes, systems and other things you will
Objective need to do in the project.
When you will complete this lesson you should be able to: • Write each component on a separate Post-It Note.
• Understand the purpose of a Work Breakdown Structure. • Stick the components on the wall, arranged randomly under
• Construct a WBS the first Post-It Note.
• Every process should have an outcome (otherwise, why
Introduction
follow the process?). Identify which Post-Its represent
Once Project scope stmt is completed, another useful technique
processes and break them down into their outcomes:
is called Work Breakdown Structure.
products, services, documents or other things you can
Work Breakdown Structure is exactly as it sounds a breakdown deliver. Then discard the process Post-Its.
of all the work that must be done. This includes all deliverables.
3. Have a break.
For e.g.., if you are expected to provide the customer with
weekly status reports, this should also be in the structure. 4. Over the next few hours (or days, for a large or complex
project), return to the wall and move the Post-It Notes
Work Breakdown Structure (WBS) around so that they are grouped in a way that makes sense.
In order to identify the individual tasks in a project it is useful
to create a Work Breakdown Structure. Get the team together
• Every team member should have a chance to do this, either
in a group or individually.
and brainstorm all of the tasks in the project, in no particular
order. Write them down on sticky notes and put them on a • Everyone has an equal say about what groupings make sense.
whiteboard. Once everyone has thought of as many tasks as • If an item seems to belong in two or more groups, make
they can arrange the sticky notes into groups under the major enough copies of the Post-It Note to cover all the relevant
areas of activity. groups.
A work breakdown structure (WBS) is a document that shows 5. When the team has finished its sorting process, reconvene the
the work involved in completing your project. group and agree on titles (labels) for each group of Post-Its.
Keep it simple: 6. Draw your WBS diagram.
• A one-page diagram can convey as much information as a 10- Whether the WBS should be activity-oriented or deliverable-
page document. oriented is a subject of much discussion. There are also various
• As a general rule, each box on your diagram is worth about approaches to building the WBS for a project. Project manage-
80 hours (three weeks) of work. ment software, when used properly, can be very helpful in
developing a WBS, although in early stages of WBS develop-
• About 3-4 levels is plenty of detail for a single WBS diagram.
ment, plain sticky notes are the best tool (especially in teams).
If you need to provide more detail, make another diagram.
An example of a work breakdown for painting a room (activity-
Draw a work breakdown structure early in the project. It will
oriented) is:
help you:
• Clearly explain the project to your sponsor and stakeholders;
• Prepare materials
• Develop more detailed project plans (schedules, Gantt charts

• Buy paint
etc) in an organized, structured way. • Buy a ladder
The work breakdown structure is part of the project plan. The • Buy brushes/rollers
sponsor and critical stakeholders should sign it off before the • Buy wallpaper remover
project begins.
52
• Prepare room Time dependencies:

• Remove old wallpaper • End-start (task 1 must end before task 2 can begin)
• Remove detachable decorations • Start-start (two tasks must begin at the same time, e. g.
• Cover floor with old newspapers simultaneous PR and advertising; forward military
movement and logistics train activities.)
• Cover electrical outlets/switches with tape
• Cover furniture with sheets
• End-end: two tasks must end at the same time. E. G.
cooking: several ingredients must be ready at the right
• Paint the room moment to combine together.
• Clean up the room • Staggered start. Tasks that could possibly be done start-start,
• Dispose or store left over paint but might not actually require it, may be done staggered-start
• Clean brushes/rollers for external convenience-related reasons.
• Dispose of old newspapers Estimating Time
• Remove covers Lapsed time: end date minus start date. Takes into account
The size of the WBS should generally not exceed 100-200 weekends, other tasks.
terminal elements (if more terminal elements seem to be Task time: just the actual days required to do the job.
required, use subprojects). The WBS should be up to 3-4 levels Hofstadter’s law: things take longer than you planned for,
deep. even if you took Hofstadter’s law into account.
To Create Work Breakdown Structure Discussion:
Work breakdown structure should be at a level where any
• Write short notes on
stakeholder can understand all the steps it will take to accom-
plish each task and produce each deliverable. • Activity Plan
1. Project Management • Dependencies
1.1 Administrative • Estimated Time
1.1.1 Daily Mgmt • Activity-oriented
1.1.2 Daily Communication • Explain the need and importance of a Work Breakdown
structure.
1.1.3 Issue Resolution
• Construct a WBS for an Information Technology upgrade
1.2 Meeting
project. Breakdown the work to at least third level for one of
1.2.1 Client Meetings the WBS items. Make notes of questions you had while
1.2.2 Staff Meetings completing this exercise.
2.Technical Editor • Create WBS for the following:
2.1 Choosing Technical Editor • Hotel Management System
2.1.1 Determine skill set • Railway Reservation System
2.1.2 Screen Candidates • Hospital Management System
2.1.3 Determine appropriate Candidates Reference
2.1.4 Choose Candidates 1. Carl L. Pritchard. Nuts and Bolts Series 1: How to Build a
2.2 Working with Technical Editor Work Breakdown Structure.
2.2.1 Establish Procedure 2. Project Management Institute. Project Management Institute
2.2.2 Agree on Rule of Engagement. Practice Standard for Work Breakdown Structures.
3. Gregory T. Haugan. Effective Work Breakdown Structures
From WBS to Activity Plan
(The Project Management Essential Library Series).
1. Identify irrelevant tasks
More information
2. Identify resources needed
1. Noel N Harroff (1995, 2000) “The Work Breakdown
3. Decide time needed (task time, not lapsed time) for the
Structure”. Available at http://www.nnh.com/ev/
smallest units of WBS.
wbs2.html
4. Identify dependencies (many are already in the WBS, but
2. 4pm.com (undated) “Work Breakdown Structure”. Available
there are probably also cross-dependencies.) There are two
at http://www.4pm.com/articles/
types:
work_breakdown_structure.htm
• Logical dependency: carpet cannot be laid until painting
3. University of Nottingham (November 2002) “Work
is complete.
Breakdown Structure” for building the Compass student
• Resource dependency: if same guy is to paint, he portal. Available at http://www.eis.nottingham.ac.uk/
cannot paint two rooms at the same time compass/workstructure.html
53
4. NASA Academy of Program and Project Leadership
(undated) “Tools: work breakdown structure”. Collection of

documents available at http://appl.nasa.gov/perf_support/
tools/tools_wbs.htm
5. Kim Colenso (2000) “Creating The Work Breakdown
Structure”. Available at http://www.aisc.com/us/lang_en/
library/white_papers/
Creating_a_Work_Breakdown_Structure.pdf
Notes
54
LESSON 14
PROJECT ESTIMATION, ANALYZING PROBABILITY AND RISK
Structure This paper proposes that project managers be trained to be

more closely aware of the limitations of their own estimation
• Objective
powers, which will lead to project plans that explicitly match the
• Introduction project capabilities and risks, as well as contingency plans that
• Project Estimation should be more meaningful.
• Analyzing probability & Risk Most organizations have difficulty estimating what it takes to
deliver a data warehouse (DW). The time and effort to imple-
Objective
ment an enterprise DW, let alone a pilot, is highly variable and
When you will complete this lesson you should be able to:
dependent on a number of factors. The following qualitative
• Understand the importance of Project Estimation. discussion is aimed toward estimating a pilot project. It can also
• Study about probability and Risk analysis. be useful in choosing an acceptable pilot that falls within a time
Introduction schedule that is acceptable to management.
To estimate budget and control costs, project managers and Each of the following points can make a substantial difference
their teams must determine what physical resources (people, in the cost, risk, time and effort of each DW project.
equipment and materials) and what quantities of those 1. What are the number of internal source systems and
resources are required to complete the project. The nature of database/files?
project and the organization will affect resource planning.
Each source system along with its database and files will take
Expert judgment and the availability of alternatives are the only
additional research, including meetings with those who have
real tools available to assist in resource planning. It is important
knowledge of data. The time to document the results of the
to have people who have experience and expertise in similar
research and meeting should also be included in the estimates.
projects and with the organization performing this particular
project help determine what resources are necessary. 2. How many business processes are expected for the pilot?
Accurately planning and estimating software projects is an (Examples: analyze sales, analyze markets, and analyze financial
extremely difficult software management function. Few accounts.) a pilot should be limited to just one business
organizations have established formal estimation processes, process. If management insists on more than one, the time and
despite evidence that suggests organizations without formal effort will be proportionally greater.
estimation are four times more likely to experience cancelled or 3. How many subject areas are expected for the pilot?
delayed projects. (Examples: customer, supplier/vendor, store/location,
Project Estimation product, organizational unit, demographic area of market
In the 1970s, geologists at Shell were excessively confident when segment, general ledger account, and promotion/campaign.)
they predicted the presence of oil or gas. They would for If possible, a pilot should be limited to just one subject area.
example estimate a 40% chance of finding oil, but when ten If management insists on more than one, the time and effort
such wells were actually drilled, only one or two would produce. will be proportionally greater.
This overconfidence cost Shell considerable time and money. 4. Will a high-level enterprise model be developed during
Shell embarked on a training programme, which enabled the the pilot?
geologists to be more realistic about the accuracy of their
Ideally, an enterprise model should have been developed prior
predictions. Now, when Shell geologists predict a 40% chance
to the start of the DW pilot, if the model has not been
of finding oil, four out of ten are successful.
finished and the pilot requires its completion. The schedule for
Software project managers are required to estimate the size of a the pilot must be adjusted.
project. They will usually add a percentage for ‘contingency’, to
5. How many attributes (fields, columns) will be selected
allow for their uncertainty. However, if their estimates are
for the pilot?
overconfident, these ‘contingency’ amounts may be insufficient,
and significant risks may be ignored. Sometimes several such The more attributes to research understand, clean, integrate and
‘contingency’ amounts may be multiplied together, but this is a document, the longer the pilot and the greater the effort.
clumsy device, which can lead to absurdly high estimates, while 6. Are the source files well modeled and well documented?
still ignoring significant risks. In some cases, higher manage- Documentation is critical to the success of the pilot. Extra time
ment will add additional ‘contingency’, to allow for the fallibility and effort must be included if the source files and databases
of the project manager. Game playing around ‘contingency’ is have not been well documented.
rife.
55
7. Will there be any external data (Lundberg, A. C. 14. Will it be necessary to synchronize the oper5ational
Nielsen. Dun and Bradstreet) in the pilot system? Is the system with the data warehouse?
external system well documented? This is always difficult and will require initial planning, genera-
External data is often not well documented and usually does tion procedures and ongoing effort form operations.
not follow the organization standards. Integrating external data 15. Will a new hardware platform be required? If so, will to
is often difficult and time consuming. different than the existing platform?
8. Is the external data modeled? (Modeled, up-to-date, The installation of new hardware always requires planning and
accurate, actively being used and comprehensive; a high execution effort.
level accurate and timely mode! Exists: an old, out of
If it is to be a new type of platform, operations training and
date model exists; no model exists)
familiarization takes time and efforts, a new operating system
Without a model, the effort to understand the source external requires work by the technical support staff. There are new
data is significantly greater. It’s unlikely that the external data has procedures to follow, new utilities to learn and the shakedown
been modeled, but external data vendors should find the sale and testing efforts for anything new is always time consuming
of their data easier when they have models that effectively and riddled with unexpected problems.
document their products.
16. Will new desktop swill require?
9. How much cleaning will the source data require? (Data
New desktops require installation, testing and possible training
need no cleaning, minor complexity, transformations,
of the users of the desktops.
medium/moderate complexity; and very complicated
transformation required) 17. Will new network work be required?
Data cleansing both with and without software tools to aid the If a robust network (one that one handle the additional load
process is tedious and time consuming organizations usually form the data warehouse with acceptable performance) is already
overestimate the quality of their data and always underestimate in place, shaken-out and tested, a significant amount of work
the effort to clean the data. and risk will be eliminated.
10. How much integration will be required? (None 18. Will network people be available?
required, moderate integration required, serious and If network people are available, it will eliminate the need to
comprehensive integration required) recruit or train
An example of integration is pulling customer data together 19. How many query tools will be chosen?
form multiple internal files as well as from external data. The Each new query tool takes time to train those responsible for
absence of consistent customer identifiers can cause significant support and time to train the end users.
work to accurately integrate customer data.
20. Is user management sold on and committed to this
11. What the estimated size of the pilot database? project and what is the organization level at which the
Data warehouse Axiom #A – Large DW databases (100GB TO commitment was made?
500GB) will always have performance problems. Resolving If management is not sold on the project, the risk is signifi-
those problems (living within an update/refresh/backup cantly greater. For the project manager, lack of management
window, providing acceptable query performance) will always commitment means far more difficulty in getting resources
take significant time and effort. (money, involvement) and getting timely answers.
Data Warehouse Axiom #B – A pilot greater than 500 GB is 21. Where does the DW project manager’s report in the
not a pilot; it’s a disaster waiting to happen. organization?
12. What is the service level requirement? (Five days/week, The higher up the project mangers report, the greater the
eight hours/day; six days/weeks, eighteen hours/day; management commitment, the more visibility and the more the
seven days/week, 24 hours/day) indication that the project is important to the organization.
It is always easier to establish an operational infrastructure as 22. Will the appropriate user be committed and available
well as a develop the update/refresh/backup scenarios for an for the project?
8*% than for a 24*& schedule. It’s also easier for operational
If people important to the project are not committed and
people and DBAs to maintain a more limited scheduled up
available, it will take far longer for the project to complete. User
time.
involvement is essential to the success of any DW project.
13. How frequently will the data be loaded/updated/
23. Will knowledgeable application developers
refreshed? (Monthly, weekly, daily, hourly)
(programmers) be available for the migration process?
The more frequent the load/ update/refresh, the greater the
These programmers need to be available when they are needed
performance impact, If real time is every being considered, the
unavailability means the project will be delayed.
requirement is for an operational, not a decision support
system. 24. How many trained and experienced programmer/
analysts will be available for system testing?
If these programmer/analysts are not available, the will have to
be recruited and / or trained.
56
25. How many, trained an experienced systems analysts will the level of sophistication of the users? (Power users,

be assigned to the project full time? occasional users, little desktop experience)
If an adequate number of these trained and experienced system The users; level of familiarity and sophistication will dictate the
analysts are not available full time, the will have to be recruited amount of training and user support required.
and/or trained. A data warehouse project is not a part-time 33. Will migration software will be chosen and used for the
option. It requires the full dedication of team members. pilot?
26. Will the DBAs be familiar with the chosen relational Migration software will require training and time in the learning
database management systems (RDBMS), will they curve, but should decrease the overall data migration effort.
experienced in database design and will they be
34. Will a repository be used for the pilot?
available for this project full time?
A repository of some type is highly recommended of the DW
A new RDBMS requires recruitment or training, and there is
but it will require training, establishing standards for metadata
always a ramp up effort with software as critical as RDBMS.
and repository use as well as the effort to populate the reposi-
DBAs have different areas of expertise and to all DBAs will
tory.
have database design skill. Database design is a skill that is
mandatory for designing a good DW. The DBA for the DW 35. Are there any serious security issues? What audit
will be very important and will require a full-time effort. requirements need to be followed of the pilot?
27. Will technical support people be available for capacity Security and audit require additional procedures (or at the least
planning, performance monitoring and trouble- the implementation of existing procedures) and the time to test
shooting? and satisfy those procedures.
A Data warehousing of any size will require the involvement of Analyzing Probability & Risk
technical support people. Capacity planning is very difficult in In life, most things result in a bell curve. The figure below
this environment as the ultimate size of the DW, user volumes shows a sample bell curve that measures the likelihood of on-
and user access patterns can rarely be anticipated. However, time project delivery. The graph measures the number of
What can be anticipated are performance problems that will projects delivered late, on time, and early.
require the constant attention of the technical support staff. As you can see in the figure the most likely outcome fails to the
28. Will be necessary to get an RFP (Request for Proposal) center of the curve. This type of graph is skewed in the center;
for any of the data warehouse software tools? hence, the terminology bell curve is taken from the shape. The
Some organizations require an RFP on software over a specific following table1 summarizes the data presented in the figure.
dollar amount. RFPs take time to write and to evaluate. It’s What this teaches us is we currently do time estimates incor-
easier to get the information other ways without the need for rectly. That is, trying to predict a single point will never work.
and RFP. The law of averages works against/us.
29. Will a CASE tool be chosen and will the data
administrators who use the CASE tool be available and Table 1: Bell Curve Data Summary
experienced?
A CASE tool should facilitate documentation, communication Percentage Delivered Days fro, Expected
Of projects Delivery
with the users and provide one of the sources of the DW
metadata. Data administrators are usually the folks who use the 25 33 days early or 35 days late
CASE tools. Their experience with one of the tools will 50 25 days early or 25 days late
expedite the documentation, communication and meta-data
75 8 days early or 13 late
population processes.
30. How many users are expected for the pilot?
The more users, the more training, user support and the more Table 2 : Three-point Time Estimate Worksheet
performance issues that can be anticipated. More users means
more changes and requests for additional function, and more
Task Subtasks Best Most Worst
problems that are uncovered that require resolution. Case Likely
31. How many queries/day/’user are expected and what is Choosing The Determine 1.0 3.0 5.0
the level of complexity? (simple, moderately complex, Technical Skill set
very complex) editor
The higher the volume of queries and the greater their complex- Screen candidates 1.0 2.0 3.0
ity would indicate performance problems more user training
Choose candidate 0.5 1.0 2.0
required and misunderstanding of the ways to write accurate
queries. Total 2.5 6.0 10.0
32. How comfortable and familiar are the users with
desktop computers and with the operating system (Win-
dows, NT, OS/2)? (Very familiar and familiarity) What is
57
We should predict project time estimates like we predict rolling Reference

dice. Experience has taught us when a pair of dice is rolled; the 1. Hughes, Bob, Software project management, 2nd ed. New
most likely number to come up is seven. When you look at Delhi: Tata McGraw- Hill Publishing, 1999.
alternatives, the odds of a number other than seven coming up 2. Kelkar, S.A., Software project management: a concise study, New
are less. You should test a three-point estimate the optimistic Delhi: Prentice Hall of India, 2002
view, pessimistic view, and the most likely answer?
3. Meredith, Jack R.; Mantel, Samuel J., Project management: a
Based on those answers. You can determine a time estimate managerial approach, New York: John Wiley and Sons, 2002.
Table 2 shows an example of a three point estimated
4. Royce, Walker, Software project management: a unified
worksheet.
framework, Delhi: Pearson Education Asia, 1998.
As you can see from, Table 2 just the task of choosing the
5. Young, Trevor L., Successful project management, London:
technical editor has considerable latitude in possible outcomes,
Kogan Page, 2002.
yet each one of these outcomes, yet each one of these outcomes
has a chance of becoming reality. Within a given projection
Notes
many of the tasks would come in on the best case guess and
many of the tasks will also come in on the worst case guess. In
addition, each one of these outcomes has associated measurable
risks.
We recommend you get away fro, single point estimates and
move toward three point estimates. By doing this, you will start
to get a handle on your true risk. By doing this exercise with
your team members you will set everyone thinking about the
task and all the associated risks, what if a team member gets
sick? What if the computer breaks down what if someone gets
pulled away on another tasks? These things do happen and they
do affect the project.
You are now also defining the acceptable level of performance.
For example, if project team members came in with 25 days to
choose a technical editor, we would consider this irresponsible.
WE would require a great deal of justification.
Another positive aspect to the three-point estimate is it
improves the stakeholder’s morale. The customers will begin to
feel more comfortable because he or she will have an excellent
command on the project. At the same time, when some tasks
do fall behind, everyone realizes this should be expected.
Because the project takes all outcomes into consideration. You
could still come in within the acceptable timelines. Point
estimates and improve the level of accuracy.
Discussions
• Write notes on:
• Integration
• RFP
• External Data
• Discuss various points that can make a substantial difference
in the cost, risk, time and effort of each DW project.
• Discuss Probability & Risk Analysis
58
LESSON 15
MANAGING RISK: INTERNAL AND EXTERNAL, CRITICAL PATH ANALYSIS
Structure
Objective
Introduction Minimum The lowest possible cost.
Risk Analysis
The most likely cost.
Risk management Mode
Objective This is the highest point on the curve.
When you have completed this lesson you should be able to:
• Understand the importance of Project Risk Management. The midway cost of n projects.
• Understand what is Risk. Median (In other words, n/2 will cost less than the
• Identify various types of Risks. median, and n/2 will cost more.)
• Describe Risk Management Process
• Discuss Critical Path Analysis Average
The expected cost of n similar projects,
divided by n.
Introduction
Project Risk Management is the art and science of identifying, Reasonable The highest possible cost, to a 95%
assigning and responding to risk throughout the life of a maximum certainty.
project and in the best interests of meeting project objectives.
Risk management can often result in significant improvements Absolute The highest possible cost, to a 100%
in the ultimate success of projects. Risk management can have a maximum certainty.
positive impact on selecting projects, determining the scope of
projects, and developing realistic schedules and cost estimates. It
helps stakeholders understand the nature of the project, On this curve, the following sequence holds:
involves team members in defining strengths and weaknesses, Minimum < mode < median < average < reasonable maxi-
and help to integrate the other project management areas. mum < absolute maximum
Risk Analysis Note the following points:
At a given stage in a project, there is a given level of knowledge
and uncertainty about the outcome and cost of the project. The
probable cost can typically be expressed as a skewed bell curve, • The absolute maximum cost may be infinite, although there
since although there is a minimum cost; there is no maximum is an infinitesimal tail. For practical purposes, we can take the
cost. reasonable maximum cost. However, the reasonable
maximum may be two or three times as great as the average
cost.
• Most estimation algorithms aim to calculate the mode. This
means that the chance that the estimates will be achieved on a
single project is much less than 50%, and the chances that the
total estimates will be achieved on a series of projects is even
lower. (In other words, you do not gain as much on the
roundabouts as you lose on the swings.)
• A tall thin curve represents greater certainty, and a low broad
curve represents greater uncertainty.
There are several points of particular interest on this curve: • Risk itself has a negative value. This is why investors
demand a higher return on risky ventures than on ‘gilt-
edged’ securities. Financial number-crunchers use a measure
of risk called ‘beta’.
• Therefore, any information that reduces risk has a positive
value.
59
This analysis yields the following management points: To mitigate the risks in a software development project, a
• A project has a series of decision points, at each of which the management strategy for every identified risk must be devel-
sponsors could choose to cancel. oped.
• Thus at decision point k, the sponsors choose between Risk monitoring steps are:
continuation, which is then expected to cost Rk, and • Assess each identified risks regularly to decide whether or not
cancellation, which will cost Ck. it is becoming less or more probable
• The game is to reduce risk as quickly as possible, and to place • Also assess whether the effects of the risk have changed
decisions at the optimal points. • Each key risk should be discussed at management progress
• This means we have to understand what specific information meetings
is relevant to the reduction of risk, plan the project so that
Managing Risks: Internal & External
this information emerges as early as possible, and to place
When a closer look at risk. When you do get caught this is
the decision points immediately after this information is
typically due to one of three situations:
available.
1. Assumptions – you get caught by unvoiced assumptions
Risk Management which were never spelled out.
Risk management is concerned with identifying risks and
2. Constraints – you get caught by restricting factors, which
drawing up plans to minimize their effect on a project. A risk is
were not fully understood.
a probability that some adverse circumstance will occur:
3. Unknowns – items you could never predict, by they are acts
• Project risks affect schedule or resources
of God or human errors.
• Product risks affect the quality or performance of the
The key to risk management Is to do our best to identify the
software being developed
source of all risk and the likelihood of its happening.,
• Business risks affect the organization developing or
For example when we project plan, we typically do not take
procuring the software
work stoppages into account. But if we were working for and
The risk management process has four stages: airline that was under threat of major strike, we might re
• Risk identification: Identify project, product and business evaluate the likelihood of losing valuable project time.
risks Calculate the cost to the project if the particular risk happens
• Risk analysis: Assess the likelihood and consequences of and make decision. You can decide either to accept it, find a way
these risks to avoid it or to prevent it. Always look for ways around the
• Risk planning: Draw up plans to avoid or minimize the obstacles.
effects of the risk Internal and External Risks
• Risk monitoring: Monitor the risks throughout the project Duncan Nevison lists the following types of internal risks:
Some risk types that are applicable to software projects are: 1. Project Characteristics
• Technology risks Schedule Bumps
• People risks Cost Hiccups
• Organizational risks Technical Surprises
• Requirements risks 2. Company politics
• Estimation risks Corporate Strategy Change
Risk Analysis Departmental Politics
• Assess probability and seriousness of each risk 3. Project Stakeholders
• Probability may be very low, low, moderate, high or very high Sponsor
• Risk effects might be catastrophic, serious, tolerable or Customer
insignificant Subcontractors
Risk planning steps are: Project Team
1. Consider each risk and develop a strategy to manage that risk As well as the following external risks;
2. Avoidance strategies: The probability that the risk will arise is 1. Economy
reduced • Currency Rate Change
3. Minimization strategies: The impact of the risk on the • Market Shift
project or product will be reduced
• Competitors Entry OR Exit
4. Contingency plans: If the risk arises, contingency plans are
• Immediate Competitive Actions
plans to deal with that risk
• Supplier Change
60
2. Environment Self Test

• Fire, Famine, Flood As set of multiple choices given with every question, choose the
correct answer for the following question.
• Pollution
1. Which on is not an environmental risk
• Raw Materials
a. Technological changes
3. Government
b. Legal requirements
• Change in Law
c. Government decisions
• Change in Regulation
d. Office politics
By going through at this risk, you get a sense fo all the influ-
ences that may impact your particular project. You should take 2. What is a economic risk faced by project
the time to assess and reassess these. For example; if your a. Competitor’s exit or entry
project is running severely behind schedule, is there another b. Supplier’s change
vendor waiting to try to take the business? If your project is
c. Market shift
running way over budget, is there a chance, the funding may get
cut? We must always be aware of the technology with which we d. All of the above
are working. Familiarity with technology is important; we need 3. Which of these is not internal risk
to know if what we are working with is a new release of the a. Policy change
software or a release that has been out for a long time.
b. Department structures
Critical Path Analysis c. Sponsor
After your determination what must be done and how long it
d. None of these
will take, you are ready to start looking for your critical paths and
dependencies. These critical paths are yet another form of risk 4. Which of these is an internal risk
within a project. For example, there a re inherent dependencies a. Customer expectations
among many project activities b. Regulatory changes
A technical editor must be selected before the editing cycle of a c. Market factors
chapter can be completed. These are examples of dependency
d. Monetary
analysis.
5. The project estimation can get delayed because fo the reason
Let’s say we are working on a project with three unique lists of
activities associated with it. Each unique path (A, B, C) of a. Wrong assumptions
activities is represented based on its dependencies. Each cell b. Constraints
represents the number of days that activity would take. c. Unknown factors
By adding all the rows together, you tell the duration of each d. All of the above
path. This is shown in table 3.
6. Which one is not an environmental risk
Start A represents a part of the project with three steps, which a. Technological changes
will take a total of eight days. Start B represents a part of the
project with three steps, which will take a total of six days. Start b. Legal requirements
C represents a path with two steps, which will take a total of 20 c. Government decisions
days d. Office politics
7. What is a economic risk faced by project
Unique Part # Part # Part # Total
a. Competitor’s exit or entry
Task 1 2 3
Start A 1 3 4 8 days b. Suppliers change
c. Market shift
Start B 1 2 3 6 days
d. All of the above
Start C 15 5 20 days 8. Which of these is not an internal risk
a. Policy change
b. Department Structures
Table 3 : Critical Path Analysis c. Sponsor
The critical path is Start C. You must begin this as soon as d. None of these
possible. In fact, this tells us the sooner this project can be done 9. Which of these is an internal risk
is 20 days. If you do not start the activity that takes 15 days first,
a. Customer expectations
it will delay the entire project ending one day for each day you
wait. b. Regulatory changes
c. Market factors
d. Monetary
61
10. The project estimation can get delayed because of the reason
a. Wrong assumptions
b. Constraints
c. Unknown factors
d. All of the above
Reference
1. Hughes, Bob, Software project management, 2nd ed. New
Delhi: Tata McGraw- Hill Publishing, 1999.
2. Kelkar, S.A., Software project management: a concise study, New
Delhi: Prentice Hall of India, 2002
3. Meredith, Jack R.; Mantel, Samuel J., Project management: a
managerial approach, New York: John Wiley and Sons, 2002.
4. Royce, Walker, Software project management: a unified
framework, Delhi: Pearson Education Asia, 1998.
5. Young, Trevor L., Successful project management, London:
Kogan Page, 2002.
Notes
62
CHAPTER 4
LESSON 16 DATA MINING
DATA MINING CONCEPTS
Structure Data storage became easier as the availability of large amounts

of computing power at low cost i.e., the cost of processing
• Objective
power and storage is falling, made data cheap. There was also
• Introduction the introduction of new machine learning methods for
• Data mining knowledge representation based on logic programming etc. in
• Data mining background addition to traditional statistical analysis of data. The new
• Inductive learning methods tend to be computationally intensive hence a demand
for more processing power.
• Statistics
Having concentrated so much attention on the accumulation of
• Machine Learning data the problem was what to do with this valuable resource? It
Objective was recognized that information is at the heart of business
When you have completed this lesson you should be able to: operations and that decision-makers could make use of the data
• Understand the basic concepts of data mining. stored to gain valuable insight into the business. Database
Management systems gave access to the data stored but this was
• Discuss data mining background
only a small part of what could be gained from the data.
• Learn various concepts like Inductive learning, Statistics and Traditional on-line transaction processing systems, OLTPs, are
machine learning good at putting data into databases quickly, safely and efficiently
Introduction but are not good at delivering meaningful analysis in return.
Data mining is a discovery process that allows users to under- Analyzing data can provide further knowledge about a business
stand the substance of and the relationships between, their by going beyond the data explicitly stored to derive knowledge
data. Data mining uncovers patterns and rends in the contents about the business. This is where Data Mining or Knowledge
of this information. Discovery in Databases (KDD) has obvious benefits for any
enterprise.
I will briefly review the state of the art of this rather extensive
field of data mining, which uses techniques from such areas as The term data mining has been stretched beyond its limits to
machine learning, statistics, neural networks, and genetic apply to any form of data analysis. Some of the numerous
algorithms. I will highlight the nature of the information that definitions of Data Mining, or Knowledge Discovery in
is discov-ered, the types of problems faced in databases, and Databases are:
potential applications. Data Mining, or Knowledge Discovery in Databases
Data Mining (KDD) as it is also known, is the nontrivial extraction of
The past two decades has seen a dramatic increase in the implicit, previously unknown, and potentially useful
amount of information or data being stored in electronic information from data. This encompasses a number of
format. This accumulation of data has taken place at an different technical approaches, such as clustering, data
explosive rate. It has been estimated that the amount of summarization, learning classification rules, finding
information in the world doubles every 20 months and the size dependency net works, analyzing changes, and detecting
and number of databases are increasing even faster. The increase anomalies.
in use of electronic data gathering devices such as point-of-sale William J Frawley, Gregory Piatetsky-Shapiro and Christopher J
or remote sensing devices has contributed to this explosion of Matheus
available data. Figure 1 from the Red Brick Company illustrates Data mining is the search for relationships and global
the data explosion. patterns that exist in large databases but are ‘hidden’
among the vast amount of data, such as a relationship
between patient data and their medical diagnosis. These
relationships represent valuable knowledge about the
database and the objects in the database and, if the data-
base is a faithful mirror, of the real world registered by the
database.
Marcel Holshemier & Arno Siebes (1994)
The analogy with the mining process is described as:
Data mining refers to “using a variety of techniques to
identify nuggets of information or decision-making
Figure 1: The Growing Base of Data
63
knowledge in bodies of data, and extracting these in such a reconfigured to ensure a consistent format as there is a
way that they can be put to use in the areas such as decision possibility of inconsistent formats because the data is drawn
support, prediction, forecasting and estimation. The data is from several sources e.g. sex may recorded as f or m and also
often voluminous, but as it stands of low value as no direct as 1 or 0.
use can be made of it; it is the hidden information in the • Transformation - the data is not merely transferred across
data that is useful” but transformed in that overlays may added such as the
Clementine User Guide, a data mining toolkit demographic overlays commonly used in market research.
Basically data mining is concerned with the analysis of data and The data is made useable and navigable.
the use of software techniques for finding patterns and • Data mining - this stage is concerned with the extraction of
regularities in sets of data. It is the computer, which is respon- patterns from the data. A pattern can be defined as given a
sible for finding the patterns by identifying the underlying rules set of facts(data) F, a language L, and some measure of
and features in the data. The idea is that it is possible to strike certainty C a pattern is a statement S in L that describes
gold in unexpected places as the data mining software extracts relationships among a subset Fs of F with a certainty c such
patterns not previously discernable or so obvious that no one that S is simpler in some sense than the enumeration of all
has noticed them before. the facts in Fs.
Data mining analysis tends to work from the data up and the • Interpretation and evaluation - the patterns identified by
best techniques are those developed with an orientation the system are interpreted into knowledge which can then be
towards large volumes of data, making use of as much of the used to support human decision-making e.g. prediction and
collected data as possible to arrive at reliable conclusions and classification tasks, summarizing the contents of a database
decisions. The analysis process starts with a set of data, uses a or explaining observed phenomena.
methodology to develop an optimal representation of the
Data Mining Background
structure of the data during which time knowledge is acquired.
Data mining research has drawn on a number of other fields
Once knowledge has been acquired this can be extended to
such as inductive learning, machine learning and statistics etc.
larger sets of data working on the assumption that the larger
data set has a structure similar to the sample data. Again this is Inductive Learning
analogous to a mining operation where large amounts of low- Induction is the inference of information from data and
grade materials are sifted through in order to find something of inductive learning is the model building process where the
value. environment i.e. database is analyzed with a view to finding
patterns. Similar objects are grouped in classes and rules
The following diagram summarizes the some of the stages/
formulated whereby it is possible to predict the class of unseen
processes identified in data mining and knowledge discovery by
objects. This process of classification identifies classes such that
Usama Fayyad & Evangelos Simoudis, two of leading expo-
each class has a unique pattern of values, which forms the class
nents of this area.
description. The nature of the environment is dynamic hence
the model must be adaptive i.e. should be able learn.
Generally, it is only possible to use a small number of proper-
ties to characterize objects so we make abstractions in that
objects, which satisfy the same subset of properties, are
mapped to the same internal representation.
Inductive learning where the system infers knowledge itself
from observing its environment has two main strategies:
• Supervised learning - this is learning from examples where
a teacher helps the system construct a model by defining
classes and supplying examples of each class. The system has
to find a description of each class i.e. the common properties
in the examples. Once the description has been formulated
the description and the class form a classification rule, which
The phases depicted start with the raw data and finish with the can be used to predict the class of previously unseen objects.
extracted knowledge, which was acquired as a result of the This is similar to discriminate analysis as in statistics.
following stages: • Unsupervised learning - this is learning from observation
• Selection - selecting or segmenting the data according to and discovery. The data mine system is supplied with objects
some criteria e.g. all those people who own a car, in this way but no classes are defined so it has to observe the examples
subsets of the data can be determined. and recognize patterns (i.e. class description) by itself. This
• Preprocessing - this is the data cleansing stage where certain system results in a set of class descriptions, one for each class
information is removed which is deemed unnecessary and discovered in the environment. Again this similar to cluster
may slow down queries for example unnecessary to note the analysis as in statistics.
sex of a patient when studying pregnancy. Also the data is
64
Induction is therefore the extraction of patterns. The quality of • Illustrate various stages of Data mining with the help of a

the model produced by inductive learning methods is such that diagram.
the model could be used to predict the outcome of future • “Inductive learning is a system that infers knowledge itself
situations in other words not only for states encountered but from observing its environment has two main strategies”.
rather for unseen states that could occur. The problem is that Discuss these two main strategies.
most environments have different states, i.e. changes within,
• Explain the need of Data mining.
and it is not always possible to verify a model by checking it for
all possible situations. • Explain how Data mining helps in decision-making?
Given a set of examples the system can construct multiple References
models some of which will be simpler than others. The simpler 1. Adriaans, Pieter, Data mining, Delhi: Pearson Education
models are more likely to be correct if we adhere to Ockhams Asia, 1996.
razor, which states that if there are multiple explanations for a 2. Berson, Smith, Data warehousing, Data Mining, and OLAP,
particular phenomena it makes sense to choose the simplest New Delhi: Tata McGraw- Hill Publishing, 2004
because it is more likely to capture the nature of the phenom-
enon. 3. Elmasri, Ramez, Fundamentals of database systems, 3rd ed.
Statistics
Statistics has a solid theoretical foundation but the results from Related Websites
statistics can be overwhelming and difficult to interpret, as they • www-db.stanford.edu/~ullman/mining/mining.html
require user guidance as to where and how to analyze the data. • www.cise.ufl.edu/class/cis6930fa03dm/notes.html
Data mining however allows the expert’s knowledge of the data • www.ecestudents.ul.ie/Course_Pages/Btech_ITT/
and the advanced analysis techniques of the computer to work Modules/ET4727/lecturenotes.htm
together.
Statistical analysis systems such as SAS and SPSS have been used
by analysts to detect unusual patterns and explain patterns
using statistical models such as linear models. Statistics have a
role to play and data mining will not replace such analyses but
rather they can act upon more directed analyses based on the
results of data mining. For example statistical induction is
something like the average rate of failure of machines.
Machine Learning
Machine learning is the automation of a learning process and
learning is tantamount to the construction of rules based on
observations of environmental states and transitions. This is a
broad field, which includes not only learning from examples,
but also reinforcement learning, learning with teacher, etc. A
learning algorithm takes the data set and its accompanying
information as input and returns a statement e.g. a concept
representing the results of learning as output. Machine learning
examines previous examples and their outcomes and learns
how to reproduce these and make generalizations about new
cases.
Generally a machine learning system does not use single
observations of its environment but an entire finite set called
the training set at once. This set contains examples i.e. observa-
tions coded in some machine-readable form. The training set is
finite hence not all concepts can be learned exactly.
Discussions
• KDD
• Machine Learning
• Inductive Learning
• Statistics
• What is Data mining?
65
Notes
66
LESSON 17
DATA MINING CONCEPTS-2
Structure William J Frawley, Gregory Piatetsky-Shapiro and Christopher J

• Objective Matheus
• Introduction Some important points about Data mining:
• What is Data Mining? • Data mining finds valuable information hidden in large
• Data Mining: Definitions volumes of data.
• KDD vs Data mining, • Data mining is the analysis of data and the use of software
• Stages of KDD techniques for finding patterns and regularities in sets of
data.
• Selection
• The computer is responsible for finding the patterns by
• Preprocessing
identifying the underlying rules and features in the data.
• Transformation
• It is possible to “strike gold” in unexpected places as the data
• Data mining mining software extracts patterns not previously discernible
• Data visualization or so obvious that no one has noticed them before.
• Interpretation and Evaluation • Mining analogy:
• Data Mining and Data Warehousing • Large volumes of data are sifted in an attempt to find
• Machine Learning vs Data mining something worthwhile.
• DBMS vs Data mining • In a mining operation large amounts of low-grade
materials are sifted through in order to find something
• Data Warehousing
of value.
• Statistical Analysis
What is Data Mining?
• Difference between Database management systems (DBMS),
Data mining in the non-trivial process of identifying valid,
Online Analytical Processing (OLAP) and Data Mining
novel, potentially useful, and ultimately understandable
Objective patterns in data. With the widespread use of databases and the
When you have completed this lesson you should be able to: explosive growth in their sizes, organizations are faced with the
• Understand the basic concepts of data mining. problem of information overload. The problem of effectively
utilizing these massive volumes of data is becoming a major
• Study the concept of Knowledge Discovery of Data (KDD)
problem for all enterprises. Traditionally, we have been using
• Study the relation between KDD and Data mining data for querying a reliable database repository via some well-
• Identify various stages of KDD circumscribed application or canned report-generating utility.
• Understand the difference between Data mining and Data While this mode of interaction is satisfactory for a large class of
warehousing applications, there exist many other applications, which demand
exploratory data analyses. We have seen that in a data ware-
• Learn various concepts like Inductive learning, Statistics and
house, the OLAP engine provides an adequate interface for
machine learning
querying summarized and aggregate information across
• Understand the difference between data mining and DBMS different dimension-hierarchies. Though such methods are
• Understand the difference between Database management relevant for decision support systems, these lack the exploratory
systems (DBMS), Online Analytical Processing (OLAP) and characteristics of querying. The OLAP engine for a data ware-
Data Mining house (and query languages for DBMS) supports query-
Introduction triggered usage of data, in the sense that the analysis is based
Lets revise the topic done in the previous lesson. on a query posed by a human analyst. On the other hand, data
mining techniques support automatic exploration of data. Data
Can you define the term “data mining”? mining attempts to source out [patterns and trends in the data
Here is the answer: “Simply put, data mining is used to discover and infers rules from these patterns. With these rules the user
patterns and relationships in your data in order to help you will be able to support, review and examine decisions in some
make better business decisions.” related business or scientific area. This opens up the possibility
Herb Edelstein, Two Crows of a new way of interacting with databases and data ware-
“The non trivial extraction of implicit, previously unknown, houses. Consider, for example, a banking application where the
and potentially useful information from data” manager wants to know whether there is a specific pattern
followed by defaulters. It is hard to formulate a SQL query for
67
such information. It is generally accepted that if we know the information. Then, can we say that extracting the average age o f
query precisely we can turn to query language to formulate the the employees of a department from the employees database
query. But if we have some vague idea and we do not know the (which stores the date-of-birth of every employee) is a data-
precisely query, then we can resort to data mining techniques. mining task? The task is surely ‘non-trivial’ extraction of
The evolution of data mining began when business data was implicit information. It is needed a type of data mining task,
first stored in computers, and technologies were generated to but at a very low level. A higher-level task would, for example,
allow users to navigate through the data in real time. Data be to find correlations between the average age and average
mining takes this evolutionary process beyond retrospective income of individuals in an enterprise.
data access and navigation, to prospective and proactive 2. Data mining is the search for the relationships and global
information delivery. This massive data collection, high patterns that exist in large databases but are hidden among
performance computing and data mining algorithms. vast amounts of data, such as the relationship between
We shall study some definitions of term data mining in the patient data and their medical diagnosis. This relationship
following section. represents valuable knowledge about the databases, and the
objects in the database, if the database is a faithful mirror of
Data Mining: Definitions the real world registered by the database.
Data mining, the extraction of the hidden predictive informa-
Consider the employee database and let us assume that we have
tion from large databases is a powerful new technology with
some tools available with us to determine some relationships
great potential to analyze important information in the data
between fields, say relationship between age and lunch-patterns.
warehouse. Data mining scours databases for hidden patterns,
Assume, for example, that we find that most of employees in
finding predictive information that experts may miss, as it goes
there thirties like to eat pizzas, burgers or Chinese food during
beyond their expectations. When implemented on a high
their lunch break. Employees in their forties prefer to carry a
performance client/server or parallel processing computers, data
home-cooked lunch from their homes. And employees in their
mining tolls can analyze massive databases to deliver answers to
fifties take fruits and salads during lunch. If our tool finds this
questions such as which clients are most likely to respond to the
pattern from the database which records the lunch activities of
next promotions mailing. There is an increasing desire to use
all employees for last few months, then we can term out tool as
this new technology in the new application domain, and a
a data-mining tool. The daily lunch activity of all employees
growing perception that these large passive databases can be
collected over a reasonable period fo time makes the database
made into useful actionable information.
very vast. Just by examining the database, it is impossible to
The term ‘data mining’ refers to the finding of relevant and notice any relationship between age and lunch patterns.
useful information from databases. Data mining and knowl-
3. Data mining refers to using a variety of techniques to
edge discovery in the databases is a new interdisciplinary field,
identify nuggets of information or decision-making
merging ideas from statistics, machine learning, databases and
knowledge in the database and extracting these in such a way
parallel computing. Researchers have defined the term ‘ data
that they can be put to use in areas such as decision support,
mining’ in many ways.
prediction, forecasting and estimation. The data is often
We discuss a few of these definitions below. voluminous, but it has low value and no direct use can be
1. Data mining or knowledge discovery in databases, as it is made of it. It is the hidden information in the data that is
also known is the non-trivial extraction of implicit, useful.
previously unknown and potentially useful information Data mining is a process of finding value from volume. In any
from the data. This encompasses a number of technical enterprise, the amount of transactional data generated during
approaches, such as clustering, data summarization, its day-to-day operations is massive in volume. Although these
classification, finding dependency networks, analyzing transactions record every instance of any activity, it is of little use
changes, and detecting anomalies. in decision-making. Data mining attempts to extract smaller
Though the terms data mining and KDD are used above pieces of valuable information from this massive database.
synonymously, there are debates on the difference and similarity 4. Discovering relations that connect variables in a database is
between data mining and knowledge discovery. In the present the subject of data mining. The data mining system self-
book, we shall be suing these two terms synonymously. learns from the previous history of the investigated system,
However, we shall also study the aspects in which these two formulating and testing hypothesis about rules which
terms are said to be different. systems obey. When concise and valuable knowledge about
Data retrieval, in its usual sense in database literature, attempts the system of interest is discovered, it can and should be
to retrieve data that is stored explicitly in the database and interpreted into some decision support system, which helps
presents it to the user in a way that the user can understand. It the manager to make wise and informed business decision.
does not attempt to extract implicit information. One may Data mining is essentially a system that learns from the existing
argue that if we store ‘date-of-birth’ as a field in the database data. One can think of two disciplines, which address such
and extract ‘age’ from it, the information received from the problems- Statistics and Machine Learning. Statistics provide
database is not explicitly available. But all of us would agree that sufficient tools for data analysis and machine learning deals with
the information is not ‘non-trivial’. On the other hand, if one different learning methodologies. While statistical methods are
attempts to as a sort of non-trivial extraction of implicit theory-rich-data-poor, data mining is data-rich-theory-poor
68
approach. On the other hand machine-learning deals with Stages OF KDD

whole gamut of learning theory, which most often data mining The stages of KDD, starting with the raw data and finishing
is restricted to areas of learning with partially specified data. with the extracted knowledge, are given below.
5. Data mining is the process of discovering meaningful, new Selection
correlation patterns and trends by sifting through large This stage is concerned with selecting or segmenting the data
amount of data stored in repositories, using pattern that are relevant to some criteria. For example, for credit card
recognition techniques as well as statistical and mathematical customer profiling, we extract the type of transactions for each
techniques. type of customers and we may not be interested in details of
One important aspect of data mining is that it scans through a the shop where the transaction takes place.
large volume of to discover patterns and correlations between Preprocessing
attributes. Thus, though there are techniques like clustering, Preprocessing is the data cleaning stage where unnecessary
decision trees, etc., existing in different disciplines, are not information is removed. For example, it is unnecessary to note
readily applicable to data mining, as they are not designed to the sex of a patient when studying pregnancy! When the data is
handle Ii amounts of data. Thus, in order to apply statistical drawn from several sources, it is possible that the same
and mathematical tools, we have to modify these techniques to information represented in different sources in different
be able efficiently sift through large amounts of data stored in formats. This stage reconfigures the data to ensure a consistent
the secondary memory. format, as there is a possibility of inconsistent formats.
KDD vs. Data Minning Transformation
Knowledge Discovery in Database (KDD) was formalized in The data is not merely transferred across, but transformed in
1989, with reference to the general concept of being broad and order to be suitable the task of data mining. In this stage, the
high level in the pursuit of seeking knowledge from data. The data is made usable and navigable.
term data mining was then coined; this high-level application
technique is used to present and analyze data for decision- Data Mining
makers. This stage is concerned with the extraction of patterns from the
data.
Data mining is only one of the many steps involved in
knowledge discovery in databases. The various Steps in the Interpretation and Evaluation
knowledge discovery process include data selection, data cleaning The patterns obtained in the data mining stage are converted
and preprocessing, data transformation and reduction, data into knowledge, which turn, is used to support decision-
mining algorithm selection and finally the post-processing and making.
the interpretation of the discovered knowledge. The KDD Data Visualization
process tends to be highly iterative and interactive. Data mining Data visualization makes it possible for the analyst to gain a
analysis tends to work up from the data and the best techniques deeper, more intuitive understanding of the data and as such
are developed with an orientation towards large volumes of can work well alongside data mining. Data mining allows the
data, making use of as much data as Possible to arrive at reliable analyst to focus on certain patterns and trends and explore them
conclusions and decisions. The analysis process starts with a set in depth using visualization. On its own, data visualization can
of data, and uses a methodology to develop an optimal be overwhelmed by the volume of data in a database but in
representation of the structure of data, during which knowl- conjunction with data mining can help with exploration.
edge is acquired. Once knowledge is acquired, this can be
Data visualization helps users to examine large volumes of data
extended to large sets of data on the assumption that the large
and detect patterns visually. Visual displays of data such as
data set has a structure similar to the simple data set.
maps, charts and other graphics representations allow data to be
Fayyad et al. distinguish between KDD and data mining by presented compactly to the users. A single graphical screen can
giving the following definitions. encode as much information as can a far larger, number of text
Knowledge Discovery in Databases is, “Process of identifying a screens. For example, if a user wants to find out whether the
valid, potentially useful and ultimately understandable structure production problems at a plant are correlated to the location of
in data. This process involves selecting or sampling data from a the plants, the problem locations can be encoded in a special
data warehouse, cleaning or preprocessing it, transforming or color, say red, on a map. The user can then discover locations in
reducing it (if needed), applying a data-mining component to which the problems are occurring. He may then form a hypoth-
produce a structure, and then evaluating the derived structure. esis about why problems are occurring in those locations, and
Data Mining is a step in the KDD process concerned with the may verify the hypothesis against the database.
algorithmic means by which patterns or structures are enumer- Data Mining and Data Warehousing
ated from the data under acceptable computational efficiency The goal of a data warehouse is to support decision making
limitations”. with data. Data mining can be used in conjunction with a data
Thus, the structures that arc the outcome of the data mining warehouse to help with certain types of decisions. Data mining
process must meet certain conditions so that these can be can be applied to operational databases with individual
considered as knowledge. These conditions are: validity, transactions. To make data mining more efficient, the data
understandability, utility, novelty and interestingness. warehouse should have an aggregated’ or summarized collec-
69
tion of data. Data mining helps in extracting meaningful new • Requires expert user guidance
patterns that cannot be found necessarily by merely querying or Difference between Database management systems
processing data or metadata in the data warehouse. Data mining (DBMS), Online Analytical Processing (OLAP) and Data
applications should therefore be strongly considered early, Mining
during the design of a data warehouse. Also, data mining tools
should be designed to facilitate their use in conjunction with
data ware-houses. In fact, for very large databases ‘running into
Area DBMS OLAP Data Mining
terabytes of data, successful use of database mining applica-
Knowledge
tions will depend, first on the construction of a data Extraction of
Summaries, trends discovery of
warehouse. Task detailed and
and forecasts hidden patterns
summary data
and insights
Machine Learning vs. Data Mining
Type of Insight and
• Large Data sets in Data Mining result
Information Analysis
Prediction
• Efficiency of Algorithms is important Deduction
Multidimensional Induction (Build
(Ask the
• Scalability of Algorithms is important Method question,
data modeling, the model, apply
Aggregation, it to new data, get
• Real World Data verify with
Statistics the result)
data)
• Lots of Missing Values Who
What is the average Who will buy a
• Pre-existing data - not user generated Example
purchased
income of mutual mutual fund in
mutual funds
• Data not static - prone to updates question
in the last 3
fund buyers by the next 6
region by year? months and why?
• Efficient methods for data retrieval available for use years?
• Domain Knowledge in the form of integrity constraints

available.
Data Mining vs. DBMS Discussion
• Example DBMS Reports • What is data mining?
• Last months sales for each service type • Discuss the role of data mining in data warehousing.
• Sales per service grouped by customer sex or age • How a data mining different from KDD?
bracket • Explain the stages of KDD.
• List of customers who lapsed their policy • Write an essay on “Data mining: Concepts, Issues and
• Questions answered using Data Mining Trends”.
• What characteristics do customers that lapse their policy • How can you link data mining with a DBMS?
have in common and how do they differ from • Explain the difference between a KDD and Data mining.
customers who renew their policy?
• Correctly contrast the difference between Database
• Which motor insurance policy holders would be Management System, OLAP and Data mining?
potential customers for my House Content Insurance
policy? References
Data Warehouse
Asia, 1996.
• Data Warehouse: centralized data repository, which can be
queried for business benefit.
• Data Warehousing makes it possible to
• Extract archived operational data Delhi: Pearson Education Asia, 2000.
• Overcome inconsistencies between different legacy data
Related Websites
formats
• www-db.stanford.edu/~ullman/mining/mining.html
• Integrate data throughout an enterprise, regardless of
location, format, or communication requirements • www.cise.ufl.edu/class/cis6930fa03dm/notes.html
• Incorporate additional or expert information • www.ecestudents.ul.ie/Course_Pages/Btech_ITT/
Modules/ET4727/lecturenotes.htm
Statistical Analysis
• Ill-suited for Nominal and Structured Data Types
• Completely data driven - incorporation of domain
knowledge not possible
• Interpretation of results is difficult and daunting
70
71
Notes
72
LESSON 18
ELEMENTS AND USES OF DATA MINING
Structure 4. Analyze the data by application software.

5. Present the data in a useful format such as a graph or a table.
• Objective Relationships & Patterns
• Introduction • Discovering relationships is key to successful marketing.
• Elements and uses of Data Mining • In operational or data warehouse system, the data architect
• Relationships & Patterns and design personnel have meticulously defined entities and
• Data mining problems/issues relationships.
• Limited Information • An entity is a set of information containing fact about a
related set of data. The discovery process in a data mining
• Noise and missing values
exercise sheds light on relationships hidden deep down in
• Uncertainty many layers of corporate data.
• Size, updates, and irrelevant fields • The benefits of pattern discovery to a business add real value
• Potential Applications to a data mining exercise. No one can accurately predict that
• Retail/Marketing person X is going to perform activity in close proximity with
activity Z.
• Banking
• Insurance and Health Care
• Using data mining techniques & systematic analysis on
warehouse data, however this prediction can be backed up by
• Transportation the detection of patterns behavior.
• Medicine • Patterns are closely related to habit; in other words, the
• Data Mining and Data Warehousing likelihood of an activity being performed in closely
Data Mining as a Part of the Knowledge Discovery Process proximity to another activity is discovered in the midst of
Objective identifying a pattern.
At the end of this lesson you will be able to • Some operational systems in the midst of satisfying daily
• Understand various elements and uses of Data mining business requirements create vast amount of data.
• Study the importance and role of relationships and patterns • The data is complex and the relationships between elements
in data mining are not easily found by the naked eye.
• Study various issues related to Data mining • We need Data mining Softwares to do these tasks.
• Identify potential applications of data mining • Insightful Miner
• Understand how data mining is a part of KDD • XAffinity
Introduction Data Mining Problems/Issues

After studying the previous lessons, you must have understood Data mining systems rely on databases to supply the raw data
the meaning and significance of Data mining. In this lesson, I for input and this raises problems in that databases tend be
will explain you about various elements and uses of Data dynamic, incomplete, noisy, and large. Other problems arise as a
mining. You will study the importance and role of relationships result of the adequacy and relevance of the information stored.
and patterns in data mining. There are various issues related to Limited Information
Data mining, I will discuss all these issues in this lesson. You A database is often designed for purposes different from data
will also study various potential applications of data mining. mining and sometimes the properties or attributes that would
Apart from the above topics, I will also tell you how data simplify the learning task are not present nor can they be
mining is a part of KDD. requested from the real world. Inconclusive data causes
Elements and uses of Data Mining problems because if some attributes essential to knowledge
Data mining consists of five major elements: about the application domain are not present in the data it may
be impossible to discover significant knowledge about a given
1. Extract, transform and load transaction data onto the data
domain. For example cannot diagnose malaria from a patient
warehouse system.
database if that database does not contain the patients’ red
2. Store and manage the data in a multidimensional database blood cell count.
system.
3. Provided data access to business analysts and information
technology professionals.
73
Noise and missing values • Predict customers likely to change their credit card affiliation
Databases are usually contaminated by errors so it cannot be • Determine credit card spending by customer groups
assumed that the data they contain is entirely correct. Attributes,
• Find hidden correlations between different financial
which rely on subjective or measurement judgments, can give
indicators
rise to errors such that some examples may even be mis-
classified. Error in either the values of attributes or class • Identify stock trading rules from historical market data
information are known as noise. Obviously where possible it is Insurance and Health Care
desirable to eliminate noise from the classification information • Claims analysis - i.e which medical procedures are claimed
as this affects the overall accuracy of the generated rules. together
Missing data can be treated by discovery systems in a number of • Predict which customers will buy new policies
ways such as;
• Identify behaviour patterns of risky customers
• simply disregard missing values
• Identify fraudulent behaviour
• omit the corresponding records
Transportation
• infer missing values from known values
• treat missing data as a special value to be included
• Determine the distribution schedules among outlets
additionally in the attribute domain • Analyse loading patterns
• or average over the missing values using Bayesian techniques. Medicine
Noisy data in the sense of being imprecise is characteristic of all • Characterise patient behaviour to predict office visits
data collection and typically fit a regular statistical distribution • Identify successful medical therapies for different illnesses
such as Gaussian while wrong values are data entry errors.
Statistical methods can treat problems of noisy data, and Data Mining and Data Warehousing
separate different types of noise. The goal of a data warehouse is to support decision making
with data. Data mining can be used in conjunction with a data
Uncertainty warehouse to help with certain types of decisions. Data mining
Uncertainty refers to the severity of the error and the degree of can be applied to operational databases with individual
noise in the data. Data precision is an important consideration transactions. To make data mining more efficient, the data
in a discovery system. warehouse should have an aggregated’ or summarized collec-
Size, Updates, and Irrelevant Fields tion of data. Data mining helps in extracting meaningful new
Databases tend to be large and dynamic in that their contents patterns that cannot be found necessarily by merely querying or
are ever-changing as information is added, modified or processing data or metadata in the data warehouse. Data mining
removed. The problem with this from the data mining applications should therefore be strongly considered early,
perspective is how to ensure that the rules are up-to-date and during the design of a data warehouse. Also, data mining tools
consistent with the most current information. Also the learning should be designed to facilitate their use in conjunction with
system has to be time-sensitive as some data values vary over data ware-houses. In fact, for very large databases ‘running into
time and the discovery system is affected by the ‘timeliness’ of terabytes of data, successful use of database mining applica-
the data. tions will depend, first on the construction of a data
warehouse.
Another issue is the relevance or irrelevance of the fields in the
database to the current focus of discovery for example post Data Mining as a Part of the Knowledge Discovery
codes are fundamental to any studies trying to establish a Process
geographical connection to an item of interest such as the sales Knowledge Discovery in Databases, frequently abbreviated as
of a product. KDD, typically encompasses more than data mining. The
knowledge discovery process comprises six phases: 6 data
Potential Applications
selection, data cleansing, enrichment, data transformation or
Data mining has many and varied fields of application some of
encoding, data mining, and the reporting and display of the
which are listed below.
discovered information.
Retail/Marketing As an example, consider a transaction database maintained by a
• Identify buying patterns from customers specialty consumer goods retailer. Suppose the client data
• Find associations among customer demographic includes a customer name, zip code, phone num-ber, date of
characteristics purchase, item code, price, quantity, and total amount. A variety
• Predict response to mailing campaigns of new knowledge can be discovered by KDD processing on
this client database. During data selec-tion, data about specific
• Market basket analysis items or categories of items, or from stores in a specific region
Banking or area of the country, may be selected. The data cleansing
• Detect patterns of fraudulent credit card use process then may correct invalid zip codes or eliminate records
with incorrect phone prefixes. Enrichment typically enhances the
• Identify ‘loyal’ customers
data with additional sources of information. For example,
74
given the client names and phone numbers, the store may The term data mining is currently used in a very broad sense. In

purchase other data about age, income, and credit rating and some situation includes statistical analysis and constrained
append them to each record. Data transformation and encoding optimization as well as machine. There is no sharp line
may be done to reduce the amount of data. For instance, item separating data mining from these disciplines. It is beyond our
codes may be grouped in terms of product categories into scope, therefore, to discuss in detail the entire range of applica-
audio, video, supplies, electronic gadgets, camera, accessories, tions that make up this body of work.
and ;0 on. Zip codes may be aggregated into geographic Types of Knowledge Discovered during Data Mining
regions; incomes may be divided into Len ranges, and so on. The term “knowledge” is broadly interpreted as involving some
Goals of Data Mining and Knowledge degree of intelligence. Knowledge is often class inductive and
Discovery deductive. Data mining addresses inductive knowledge.
Broadly speaking, the goals of data mining fall into the Knowledge can be represented in many forms in an unstruc-
following classes: Prediction, identification, classification, and tured sense; it can be represented by rules, or prepositional logic.
opti-mization. In a structured form, it may be represented in decision trees,
semantic networks, neural works, or hierarchies of classes or
• Prediction - Data mining can show how certain attributes
frames. The knowledge discovered during data mining can be
within the data will behave in the future. Examples of
described in five ways, as follows.
predictive data mining include the analysis of buy-ing
transactions to predict what consumers will buy under certain 1. Association. Rules-These rules correlate the presence of a
discounts, how much sales volume a store would generate in set of items another range of values for another set of
a given period, and whether deleting a product line would variables. Examples: (1) When a _ retail shopper buys a
yield more profits. In such applications, business logic is handbag, she is likely to buy shoes. (2) An X-ray image
used cou-pled with data mining. In a scientific context, maintaining characteristics a and b is likely to also exhibit
certain seismic wave patterns may pre-dict an earthquake with characteristic c.
high probability. 2. Classification hierarchies-The goal is to work from an
• Identification - Data patterns can be used to identify the existing set of even a transaction to create a hierarchy of
existence of an item, an event, or an activity. For example, classes. Examples: (1) A population may be -divided into
intruders trying to break a system may be identified by the five ranges of credit worth -iness based on a history of
programs executed, files accessed, and CPU time per session. previous co transactions. (2) A model may be developed for
In biological applica-tions, existence of a gene may be the factors that determine desirability of location of a. store-
identified by certain sequences of nucleotide symbols in the on a 1-10 scale. (3) Mutual funds may be classified based on
DNA sequence. The area known as authentication is a form performance data using characteristics such as growth,
of identification .It ascertains whether a user is indeed a’ income, and stability.
specific’ user or one from an authorized class; it involves a 3. Sequential patterns-A sequence of actions or events is
comparison of parameters or images or signals against a sought. Example: If a patient underwent cardiac bypass
database. surgery for blocked arteries and an aneurysm and later
• Classification - Data mining can partition the data so that developed high blood urea within a year of surgery, he or she
different classes or categories can be identified based on is likely to suffer from kidney failure within the next 18
combinations of parameters. For example, customer in a months. Detection of sequential pat-terns is equivalent to
supermarket can be categorized into discount-seeking detecting association among events with certain temporal
shoppers, shoppers in a rush, loyal regular shoppers, and relationships.
infrequent shoppers. This classification may be in different 4. Patterns within time series-Similarities can be detected
analyses of customer buying transactions as a post-mining within positions of the time series. Three examples follow
activity times classification based on common domain with the stock market price data as a time series: (1) Stocks of
knowledge is used as an input to decompose the mining a utility company ABC Power and a financial company XYZ
problem and make it simpler. For instance, health foods, Securities show the same pattern during 1998 in terms of
party foods, or school lunch foods are distinct categories in closing stock price. (2) Two products show the same selling
the supermarket business. It -makes sense to analyze pattern in summer but a different one in win-ter. (3) A
relationships within and across categories as separate prob- pattern in solar magnetic wind may be used to predict
lems. Such, categorization may be used to encode the data changes in earth atmospheric conditions
appropriately before subjecting it to further data mining. 5. Categorization and segmentation-: A given population of
• Optimization - One eventual goal of data mining may be to events or items can be partitioned (segmented) into sets of
optimize the use off limited resources such as time, space, “similar” elements. Examples: (1) An entire population of
money, or materials and to maximize output variables such treatment data on a disease may be divided into groups
as sales or profits under a given set of constraints. As such, based on the similarity of side effects produced. (2) The
this goal of mining resembles the objective function used in adult popu1ation in the United States may be categorized
operations research problems deals with optimization under into five groups from “most likely to buy” to “least likely to
constraints. buy” a new product. (3) The web accesses made by a
collection of users against a set of documents (say, in a
75
digital library) may be analyzed in. terms of the key-words of
documents to reveal clusters or categories of Users.

For most applications, the desired knowledge is a combination
of the above types. We expand on each of the above knowledge
types in the following subsections.
Discussions
• Association Rules
• Discuss various elements and uses of data mining.
• Explain the significance of relationship and pattern in data
mining.
• Give some examples of Operational systems.
References
Asia, 1996.
Related Websites
• www.cise.ufl.edu/class/cis6930fa03dm/notes.html
• www.ecestudents.ul.ie/Course_Pages/Btech_ITT/
76
77
Notes
LESSON 19
DATA INFORMATION AND KNOWLEDGE
Structure Knowledge
• Objective Information can be converted into knowledge about historical
• Introduction patterns and future trends. For example summary information
on retail supermarket sales can be analyzed in light of promo-
• Data, Information and Knowledge
tional efforts to provide knowledge of consumer buying
• Information behavior. Thus, a manufacturer or retailer could determine
• Knowledge which items are most susceptible to promotional efforts.
• Data warehouse Data Warehouse
• What can Data Mining Do? Dramatic advances in data capture, processing power, data
• How Does Data Mining Work? transmission and storage capabilities are enabling organization
to integration their various databases into data warehouses.
• Data Mining in a Nutshell Data warehousing is defined as a process of centralized data
• Differences between Data Mining and Machine Learning management and retrieval. Data warehousing, like data mining,
Objective is a relatively new term although the concept itself has been
At the end of this lesson you will be able to around for years. Data warehousing represents an ideal vision
of maintaining a central repository of all organizational data.
• Understand the meaning and difference of Data,
Centralization of data is needed to maximize user access and
Information and knowledge
analysis. Dramatic technological advances are making this vision
• Study the need of data mining a reality for many companies. And, equally dramatic advances in
• Understand the working of data mining data analysis software is what support data mining.
• Study the difference between Data Mining and Machine What can Data Mining do?
Learning Data mining is primarily used today by companies with a strong
Introduction consumer focus retail, financial communication, and marketing
In the previous lesson you have studied various elements and organizations. It enables these companies to determine
uses of data mining. In this lesson, I will explain you the relationships among “ internal” factors such as price, product
difference and significance of Data, information and knowledge. positioning, or staff skills and external factors such as economic
Further, you will also study about the need and working of indicators, competition and customer demographics. And it
data mining. corporate profits. Finally, it enables them to “drill down” into
summary information to view detail transactional data.
Data, Information and Knowledge
With data mining, a retailer could use point of sale records of
Data are any facts, numbers, or text that can be processed by a
customer purchase to send targeted promotions based on an
computer. Today, organizations are accumulating vast and
individual’s purchase history. By mining demographic data from
growing amounts of data in different formats and different
comment or warranty cards, the retailer could develop products
databases. This includes
and promotions to appeal to specific customer segments. For
• Operational or transactional data such as sales, cost, example, Blockbuster Entertainment mines its video rental
inventory, payroll, accounting. history database to recommend rentals to individual customers.
• Non-operational data such as industry sales, forecast data, American Express can suggest products to its cardholders based
and macro economic data. on analysis of their monthly expenditures.
• Meta data-data about the data itself, such as logical database Wal-Mart is pioneering massive data mining to transform its
design or data dictionary definitions supplier relationships. Captures point of sale transactions from
Information over 2,900 stores in 6 countries and continuously transmits this
The patterns, associations, or relationships among all this data data to its massive 7.5 terabyte tera data warehouse. Wal-Mart
can provide information. For example, analysis of retail point allows more than 3,500 suppliers, to access data on their
of sale transaction data can yield information on which product products and perform data analyses. These suppliers use this
are selling and when. data to identify customer-buying patterns at the store display
level. They use this information to mange local store inventory
Meta data-data about the data itself, such as logical database
and identify new merchandising opportunities. In 1995, Wal-
design or data dictionary definitions
mart computers processed over 1 million complex queries.
78
The national Basketball Association (NBA) is exploring a data Data storage becomes easier as the availabilities of large amount

mining application that can be used in conjunction with image computing power at low cost i.e., the cost of processing power
recordings of basketball games. The Advanced Scout software and storage is falling, made data cheap. There was also a
analyzes the movement of players to help coaches orchestrate introduction of few machine learning method for statistical
players and strategies. For example and analysis of the play by analysis of data. The new method tend to be computationally
play sheet of the game played between the NEW YORK Knick intensive hence a demand for more processing power.
and the Cleveland Cavaliers on January 6, 1995 reveals that Having concentrated so much attention of the accumulation of
when mark price played the guard position, john Williams data the problem was what to do with this valuable resource? It
attempted four jump shots and made each one! Advanced was recognize that information is at the heart of business
scout not only finds this pattern, but also explains that it is operations and that decision-markets could make use of the
interesting because it differs considerably form the average data stored to gain valuable insight into the business. Database
shooting percentage of 49.30%, for the cavaliers during that Management systems gave access to the data stored but this was
game. only a small part of what would be gained from the data.
By using the NBA universal clock, a coach can automatically Traditional on-line transaction processing system, OLTP, Sare
bring up the video clips showing each of the jump shots goods at putting data into database quickly, safely and efficiently
attempted by Williams with price on the floor, without needing but are not good at delivering meaningful analysis in return.
to comb through hours of video footage. Those clips show Analyzing data can provide further knowledge about a business
very successful picks and roll pay in which price draws the knick’s by going beyond the data explicitly stored to derive knowledge
defense, and then finds Williams for and open jumps shot. about the business. This is where data mining or knowledge
discovery in data base (KDD) has obvious for any enterprise.
How does Data Mining Work?
While large-scale information technology has been evolving The term data mining has been stretched beyond its limits to
separate transaction and analytical systems, data mining apply to any form of data analysis. Some of the numerous
provides the link between the two. Data mining software definitions of data mining, or knowledge discovery in database
analyzes relationships and patterns in stored transaction data are:
based on open-ended user queries. Several type of analytical Data mining or knowledge discovery in database (KDD) as it is
software is available: statistical, machine learning, and neural also known is the nontrivial extraction of implicit, previously
networks. Generally, any of four types of relationships are unknown, and potentially useful information from data. This
sought: encompasses a number of different technical approaches, such
Classes: Stored data is used to locate data in predetermined as clustering, data summarization learning classification rules
groups. For example, a restaurant chain could mine customer finding dependency net works analyzing changes and detecting
purchase data to determine when customers visit and what they anomalies.
typically order. This information could be used to increase traffic Willam J frawley, Gregory piatetsky – Shapiro and christopher j
by having daili specials matheus Data ming is the search of relationships and global
Clusters: Data items are grounded according to logical relation- patterns that exit in large database but are hidden among the
ship or consumer preferences; For example, data can be mined vast amount of data such as a relationship between patient data
to identify market segments or consumer affinities. and their medical diagnosis. These relationships represent
valuable knowledge about the database and the object in the
Association: Data can be mined to identify associations. The
database and if the database is a faithful mirror; of the real
beer diaper example is an example of associative mining.
world registered by the database.
Sequential patterns: Data is mined to anticipate behavior
patterns and trends. For example, an outdoor equipment Data mining analysis tends to work from the data up and the
retailer could predict the likelihood of a backpack being best techniques are those developed with an orientation
purchased based on a consumer’s purchase of sleeping bags and towards large volumes of data, making use of as much of the
hiking shoes. collected data as possible to arrive at reliable conclusions and
decisions. The analysis process starts with a set of data, uses a
Data Mining in a Nutshell methodology to develop an optimal representation of the
The past two decades has seen dramatic increases in the amount structure of the data during which time knowledge id acquired.
of information or data being stored in electronic format. This Once knowledge has been acquired this can be extended to
accumulation of data has taken palace at an explosive rate. It has larger data set has a structure similar to the sample data. Again
been estimated that the amount of information in the world this is analogous to a mining operation where large amounts of
doubles every 20 months and number of databases are low-grade materials are sifted through in order to find some-
increasing even faster. The increase in use of electronic data thing of value.
gathering devices such as point-of-sale or remote sensing
Differences between Data Mining and Machine Learning
devices has contributed to this explosion of available data.
Knowledge Discovery in Databases (KDD) or Data Mining,
Figure 1 form the red brick company illustrates the data
and the part of Machine Learning (ML) dealing with learning
explosion.
from examples overlap in the algorithms used and the prob-
lems addressed.
The main differences are:
79
• KDD is concerned with finding understandable knowledge,
while ML is concerned with improving performance of an

agent. So training a neural network to balance a pole is part
of ML, but not of KDD. However, there are efforts to
extract knowledge from neural networks, which are very
relevant for KDD.
• KDD is concerned with very large, real-world databases,
while ML typically (but not always) looks at smaller data sets.
So efficiency questions are much more important for KDD.
• ML is a broader field, which includes not only learning from
examples, but also reinforcement learning, learning with
teacher, etc.
KDD is that part of ML which is concerned with finding
understandable knowledge in large sets of real-world examples.
When integrating machine-learning techniques into database
systems to implement KDD some of the databases require:
• More efficient learning algorithms because realistic databases
are normally very large and noisy. It is usual that the database
is often designed for purposes different from data mining
and so properties or attributes that would simplify the
learning task are not present nor can they be requested from
the real world. Databases are usually contaminated by errors
so the data-mining algorithm has to cope with noise whereas
ML has laboratory type examples i.e. as near perfect as
possible.
• More expressive representations for both data, e.g. tuples in
relational databases, which represent instances of a problem
domain, and knowledge, e.g. rules in a rule-based system,
which can be used to solve users’ problems in the domain,
and the semantic information contained in the relational
schemata.
Practical KDD systems are expected to include three intercon-
nected phases
• Translation of standard database information into a form
suitable for use by learning facilities;
• Using machine learning techniques to produce knowledge
bases from databases; and
• Interpreting the knowledge produced to solve users’
problems and/or reduce data spaces. Data spaces being the
number of examples.
References
Asia, 1996.
Related Websites
• www.cise.ufl.edu/class/cis6930fa03dm/notes.html
• www.ecestudents.ul.ie/Course_Pages/Btech_ITT/
• http://web.utk.edu/~peilingw/is584/ln.pdf
80
81
LESSON 20
DATA MINING MODELS
Structure Though the terms data mining and KDD are used above
• Objective synonymously, there are debates on the difference and similarity
between data mining and knowledge discovery. In the present
• Introduction book, we shall be suing these two terms synonymously.
• Data mining However, we shall also study the aspects in which these two
• Data Mining Models terms are said to be different.
• Verification Model Data retrieval, in its usual sense in database literature, attempts
• Discovery Model to retrieve data that is stored explicitly in the database and
presents it to the user in a way that the user can understand. It
• Data warehousing
does not attempt to extract implicit information. One may
Objective argue that if we store ‘date-of-birth’ as a field in the database
At the end of this lesson you will be able to and extract ‘age’ from it, the information received from the
• Reviewing the concept of Data mining database is not explicitly available. But all of us would agree that
the information is not ‘non-trivial’. On the other hand, if one
• Study various types of data mining models
attempts to as a sort of non-trivial extraction of implicit
• Understand the difference between Verification and information. Then, can we say that extracting the average age o f
Discovery model. the employees of a department from the employee’s database
Introduction (which stores the date-of-birth of every employee) is a data-
In the previous lesson, I have explained you the difference and mining task? The task is surely ‘non-trivial’ extraction of
significance of Data, Information and Knowledge. You have implicit information. It is needed a type of data mining task,
also studied about the need and working of data mining. In but at a very low level. A higher-level task would, for example,
this lesson, I will explain you various types of Data Mining be to find correlations between the average age and average
Models. income of individuals in an enterprise.
Data Mining 2. Data mining is the search for the relationships and global
Data mining, the extraction of the hidden predictive informa- patterns that exist in large databases but are hidden among
tion from large databases is a powerful new technology with vast amounts of data, such as the relationship between
great potential to analyze important information in the data patient data and their medical diagnosis. This relationship
warehouse. Data mining scours databases for hidden patterns, represents valuable knowledge about the databases, and the
finding predictive information that experts may miss, as it goes objects in the database, if the database is a faithful mirror of
beyond their expectations. When implemented on a high the real world registered by the database.
performance client/server or parallel processing computers, data Consider the employee database and let us assume that we have
mining tolls can analyze massive databases to deliver answers to some tools available with us to determine some relationships
questions such as which clients are most likely to respond to the between fields, say relationship between age and lunch-patterns.
next promotions mailing. There is an increasing desire to use Assume, for example, that we find that most of employees in
this new technology in the new application domain, and a there thirties like to eat pizzas, burgers or Chinese food during
growing perception that these large passive databases can be their lunch break. Employees in there forties prefer to carry a
made into useful actionable information. home-cooked lunch from their homes. And employees in there
The term ‘data mining’ refers to the finding of relevant and fifties take fruits and salads during lunch. If our tool finds this
useful information from databases. Data mining and knowl- pattern from the database which records the lunch activities of
edge discovery in the databases is a new interdisciplinary field, all employees for last few months, then we can term out tool as
merging ideas from statistics, machine learning, databases and a data-mining tool. The daily lunch activity of all employees
parallel computing. Researchers have defined the term ‘ data collected over a reasonable period fo time makes the database
mining’ in many ways. very vast. Just by examining the database, it is impossible to
notice any relationship between age and lunch patterns.
1. Data mining or knowledge discovery in databases, as it is
also known is the non-trivial extraction of implicit, 3. Data mining refers to using a variety of techniques to
previously unknown and potentially useful information identify nuggets of information or decision-making
from the data. This encompasses a number of technical knowledge in the database and extracting these in such a way
approaches, such as clustering, data summarization, that they can be put to use in areas such as decision support,
classification, finding dependency networks, analyzing prediction, forecasting and estimation. The data is often
changes, and detecting anomalies. voluminous, but it has low value and no direct use can be
82
made of it. It is the hidden information in the data that is used to target a mailing campaign. The whole operation can be

useful. refined by ‘drilling down’ so that the hypothesis reduces the
Data mining is a process of finding value from volume. In any ‘set’ returned each time until the required limit is reached.
enterprise, the amount of transactional data generated during The problem with this model is the fact that no new informa-
its day-to-day operations is massive in volume. Although these tion is created in the retrieval process but rather the queries will
transactions record every instance of any activity, it is of little use always return records to verify or negate the hypothesis. The
in decision-making. Data mining attempts to extract smaller search process here is iterative in that the output is reviewed, a
pieces of valuable information from this massive database. new set of questions or hypothesis formulated to refine the
4. Discovering relations that connect variables in a database is search and the whole process repeated. The user is discovering
the subject of data mining. The data mining system self- the facts about the data using a variety of techniques such as
learns from the previous history of the investigated system, queries, multidimensional analysis and visualization to guide
formulating and testing hypothesis about rules which the exploration of the data being inspected.
systems obey. When concise and valuable knowledge about Discovery Model
the system of interest is discovered, it can and should be The discovery model differs in its emphasis in that it is the
interpreted into some decision support system, which helps system automatically discovering important information hidden
the manager to make wise and informed business decision. in the data. The data is sifted in search of frequently occurring
Data mining is essentially a system that learns from the existing patterns, trends and generalizations about the data without
data. One can think of two disciplines, which address such intervention or guidance from the user. The discovery or data
problems- Statistics and Machine Learning. Statistics provide mining tools aim to reveal a large number of facts about the
sufficient tools for data analysis and machine learning deals with data in as short a time as possible.
different learning methodologies. While statistical methods are An example of such a model is a bank database, which is mined
theory-rich-data-poor, data mining is data-rich-theory-poor to discover the many groups of customers to target for a
approach. On the other hand machine-learning deals with mailing campaign. The data is searched with no hypothesis in
whole gamut of learning theory, which most often data mining mind other than for the system to group the customers
is restricted to areas of learning with partially specified data. according to the common characteristics found.
5. Data mining is the process of discovering meaningful, new The typical discovery driven tasks are
correlation patterns and trends by sifting through large
• Discovery of association rules
amount of data stored in repositories, using pattern
recognition techniques as well as statistical and mathematical • Discovery of classification rules
techniques. • Clustering
One important aspect of data mining is that it scans through a These tasks are of an exploratory nature and cannot be directly
large volume of to discover patterns and correlations between handed over to currently available database technology. We shall
attributes. Thus, though there are techniques like clustering, concentrate on these tasks now.
decision trees, etc., existing in different disciplines, are not Discovery of Association Rules
readily applicable to data mining, as they are not designed to An association rule is an expression of the form XÞY, where X
handle Ii amounts of data. Thus, in order to apply statistical and Y are the sets of items. The intuitive meaning of such a
and mathematical tools, we have to modify these techniques to rule is that the transaction of the database which contains X
be able efficiently sift through large amounts of data stored in tends to contain Y. Given a database, the goal is to discover all
the secondary memory. the rules that have the support and confidence greater than or
Data Mining Models equal to the minimum support and confidence, respectively.
IBM have identified two types of model or modes of opera- Let L ={l1,l2,….,lm} be a set of literals called items. Let D, the
tion which may be used to unearth information of interest to database, be a set of transactions, where each transactions,
the user. where each transaction T is a set of items. T is a set of items. T
Verification Model supports an item x, if x is in T. T is said to support a subset of
The Verification model takes a hypothesis from the user and items X, if T supports each item x in X. XÞY holds with
tests the validity of it against the data. The emphasis is with the confidence c, if c% of the transactions in D that support X also
user who is responsible for formulating the hypothesis and support. Y. the rule X Þ Y has support is the transaction set D
issuing the query on the data to affirm or negate the hypothesis. if s% of the transactions in D support XOY. Support means
how often X and Y occur together as a percentage of the total
In a marketing division for example with a limited budget for a
transactions. Confidence measures how much a particular item
mailing campaign to launch a new product it is important to
is dependent on another.
identify the section of the population most likely to buy the
new product. The user formulates an hypothesis to identify Thus, the association with a very high support and confidence is
potential customers and the characteristics they share. Historical a pattern that occurs often in the database that should be
data about customer purchase and demographic information obvious to the end user. Patterns with extremely low support
can then be queried to reveal comparable purchases and the and confidence should be regarded as of no significance. Only
characteristics shared by those purchasers, which in turn can be patterns with a combination of intermediate values of confi-
83
dence and support provide the user with interesting and more specifically, different transactions collected over a period of
previously unknown information. We shall study the tech- time. The clustering methods will help him in identifying
niques to discover association rules in the following chapters. different categories of customers. During the discovery process,
the differences between data sets can be discovered in order to
Discovery of Classification Rules
separate them into different groups, and similarity between data
Classification involves finding rules that partition the data into
sets can be used to group similar data together. We shall discuss
disjoint groups. The input for the classification is the training
in detail about using the clustering algorithm for data mining
data set, whose class labels are already known. Classification
tasks in further lessons.
analyzes the training data set and constructs a model based on
the class label, and aims to assign a class label to the future Data Warehousing
unlabelled records. Since the class field is known, this type of Data mining potential can be enhanced if the appropriate data
classification is known as supervised learning. A set of classifies has been collected and stored in a data warehouse. A data
future data and develops a better understanding of each class in warehouse is a relational database management system
the database. We can term this as supervised learning too. (RDBMS) designed specifically to meet the needs of transaction
There are several classification discovery models. They are: the processing systems. It can be loosely defined as any centralized
decision trees, neural networks, genetic algorithms and the data repository which can be queried for business benefit but
statistical models like linear/ geometric discriminates. The this will be more clearly defined later. Data warehousing is a new
applications include the credit card analysis, banking, medical powerful technique making it possible to extract archived
applications and the like. Consider the following example. operational data and overcome inconsistencies between different
legacy data formats. As well as integrating data throughout an
The domestic flights in our country were at one time only
enterprise, regardless of location, format, or communication
operated by Indian Airlines. Recently, many other private airlines
requirements it is possible to incorporate additional or expert
began their operations for domestic travel. Some of the
information. It is,
customers of Airlines started flying with these private airlines
and, as a result, Indian Airlines lost these customers. Let us The logical link between what the managers see in their
assume that Indian Airlines want to understand why some decision support EIS applications and the company’s
customers remain loyal while others leave. Ultimately, the airline operational activities
wants to predict which customers it is most likely to lose to its John McIntyre of SAS Institute Inc
competitors. Their aim to build a model based on the historical In other words the data warehouse provides data that is already
data of loyal customers versus customers who have left. This transformed and summarized, therefore making it an appropri-
becomes a classifications problem. It is supervised learning task, ate environment for more efficient DSS and EIS applications.
as the historical data becomes the training set, which is used to
trading the model. The decision tree is the most popular Discussion
classification technique. We shall discuss different methods of • Explain the relation between a data warehouse and data
decision tree construction in the forthcoming lessons. mining
Clustering • What are the various kinds of Data mining models?
Clustering is a method of grouping data into different groups, • “Data warehousing is a new powerful technique making it
so that the data in each group share similar trends and patterns. possible to extract archived operational data and overcome
Clustering constitutes a major class of data mining algorithms. inconsistencies between different legacy data formats”.
The algorithm attempts to automatically partition the data space Comment.
into a set of regions or clusters, to which the examples in the • Explain the problems associated with the Verification Model.
table are assigned, either deterministically or probability-wise.
• “Data mining is essentially a system that learns from the
The goal of process is to identify all sets of similar examples in
existing data”. Illustrate with examples.
the data, in some optimal fashion.
Clustering according to similarity is a concept which appears in
many disciplines. If a measure of similarity is available, then
there are a number of techniques for forming clusters. Another
approach is to build set functions that measure some particular
property of groups. This latter approach achieves what is
known as optimal partitioning.
The objectives of clustering are:
• To uncover natural groupings
• To initiate hypothesis about the data
• To find consistent and valid organization of the data.
A retailer may want to know where similarities in his customer
base, so that he can create and understand different groups. He
can use the existing database of the different customers or,
84
85
86
LESSON 21
ISSUES AND CHALLENGES IN DM, DM APPLICATIONS AREAS
Structure which rely on subjective or measurement judgements can give

• Objective rise to errors such that some examples may even be mis-
classified. Error in either the values of attributes or class
• Introduction.
information are known as noise. Obviously where possible it is
• Data mining problems/issues desirable to eliminate noise from the classification information
• Limited Information as this affects the overall accuracy of the generated rules.
• Noise and missing values Missing data can be treated by discovery systems in a number of
• Uncertainty ways such as;
• Size, updates, and irrelevant fields • Simply disregard missing values
• Other mining problems • Omit the corresponding records
• Sequence Mining • Infer missing values from known values
• Web Mining • Treat missing data as a special value to be included
additionally in the attribute domain
• Text Mining
• Spatial Data Mining
• Or average over the missing values using Bayesian
techniques.
• Data mining Application Areas
Noisy data in the sense of being imprecise is characteristic of all
Objective data collection and typically fit a regular statistical distribution
At the end of this lesson you will be able to such as Gaussian while wrong values are data entry errors.
• Understand various issues related to data mining Statistical methods can treat problems of noisy data, and
• Learn difference between Sequence mining, Web mining, Text separate different types of noise.
and Spatial data mining. 3. Uncertainty
• Study in detail about different application areas of Data Uncertainty refers to the severity of the error and the degree of
mining. noise in the data. Data precision is an important consideration
in a discovery system.
Introduction
In the previous lesson you have studied various types of Data 4. Size, updates, and irrelevant fields
Mining Models. In this lesson you will learn about various Databases tend to be large and dynamic in that their contents
issues and problems related to Data mining. Here, you will also are ever-changing as information is added, modified or
study about Sequence mining, Web mining, Text mining and removed. The problem with this from the data mining
spatial data mining. perspective is how to ensure that the rules are up-to-date and
consistent with the most current information. Also the learning
Data Mining Problems/Issues system has to be time-sensitive as some data values vary over
Data mining systems rely on databases to supply the raw data time and the discovery system is affected by the ‘timeliness’ of
for input and this raises problems in that databases tend be the data.
dynamic, incomplete, noisy, and large. Other problems arise as a
result of the adequacy and relevance of the information stored. Another issue is the relevance or irrelevance of the fields in the
database to the current focus of discovery for example post
1. Limited Information codes are fundamental to any studies trying to establish a
A database is often designed for purposes different from data geographical connection to an item of interest such as the sales
mining and sometimes the properties or attributes that would of a product.
simplify the learning task are not present nor can they be
requested from the real world. Inconclusive data causes Other Mining Problems
problems because if some attributes essential to knowledge We observed that a data mining system could either be a
about the application domain are not present in the data it may portion of a data warehousing system or a stand-alone system.
be impossible to discover significant knowledge about a given Data for data mining need not always be enterprise -related data
domain. For example cannot diagnose malaria from a patient residing on a relational database. Data sources are very diverse
database if that database does not contain the patient’s red and appear in varied form. It can be textual data, image data,
blood cell count. CAD data, Map data, ECG data or the much talked about
Genome data. Some data are structured and some are unstruc-
2. Noise and missing values tured. Data mining remains an important tool, irrespective of
Databases are usually contaminated by errors so it cannot be
assumed that the data they contain is entirely correct. Attributes
87
the forms or sources of data. We shall study the Data mining Text Mining
problems for different types of data. The term text mining or KDT (Knowledge Discovery in Text)
was first proposed by Feldman and Dagan in 1996. They
Sequence Mining
suggest that text documents be structured by means of
Sequence mining is concerned with mining sequence data. It
information extraction, text categorization, or applying NLP
may be noted that in the, discovery of association rules, we are
techniques as a preprocessing step before performing any kind
interested in finding associations between items irrespective of
of KDTs. Presently the term text mining, is being used to cover
their order of occurrence. For example, we may be interested in
many applications such as text categorization, exploratory data
the association between the purchase of a particular brand of
analysis, text clustering, finding patterns in text databases,
soft drinks and the occurrence of stomach upsets. But it is
finding sequential patterns in texts, IE (Information Extrac-
more relevant to identify whether there is some pattern in the
tion), empirical computational linguistic tasks, and association
stomach upsets which occurs after the purchase of the soft
discovery.
drink. Then one is inclined to infer that the soft drink causes
stomach upsets. On the other hand, if it is more likely that the Spatial Data Mining
purchase of the soft drink follows the occurrence of the Spatial data mining is the branch of data mining that deals with
stomach upset, then it is probable that the soft drink provides spatial (location) data. The immense explosion in geographi-
some sort of relief to the user. Thus, the discovery of temporal cally-referenced data occasioned by developments in IT, digital
sequences of events concerns causal relationships among the mapping, remote sensing, and the global diffusion of GIS,
events in a sequence. Another application of this domain places demands on developing data driven inductive approaches
concerns drug misuse. Drug misuse can occur unwittingly, to spatial analysis and modeling. Spatial data mining is regarded
when a patient is prescribed two or more interacting drugs as a special type of data mining that seeks to perform similar
within a given time period of each other. Drugs that interact generic functions as conventional data mining tools, but
undesirably are recorded along with the time frame as a pattern modified to take into account the special features of spatial
that can b located within the patient records. The rules that information.
describe such instances of drug misuse are then successfully For example, we may wish to discover some association among
inducted based on medical records. patterns of residential colonies and topographical features. A
Another related area which falls into the larger domain of typical spatial association may look like: ‘The residential land
temporal data mining is trend discovery. One characteristic of pockets are dense in a plain region and rocky areas are thinly
sequence-pattern discovery in comparison with trend discovery populated”; or, “The economically affluent citizens reside in
is the lack of shapes, since the causal impact of a series of hilly, secluded areas whereas the middle income group residents
events, cannot be shaped. prefer having their houses near the market”.
Web Mining Data mining Application Areas
With the huge amount of information available online, the The discipline of data mining is driven in part by new applica-
World Wide Web is a ferti1e area for data mining research. Web tions, which require new capabilities that are not currently being
mining research is at the crossroads of research from several supplied by today’s technology. These new applications can be
research communities, such as database, information retrieval, naturally divided into three broad categories [Grossman, 1999].
and within AI, especially the sub areas of machine learning and
A. Business And E-Commerce Data
natural language processing. Web mining is the use of data
This is a major source category of data for data mining applica-
mining techniques to automatically discover and extract
tions. Back-office front-office, and network applications produce
information from web documents and services. This area of
large amounts of data about business processes. Using this
research is so huge today partly due to the interests of various
data for effective decision-making remains a fundamental
research commuI1ities, the tremendous growth of information
challenge.
sources available on the web and the recent interest in e-
commerce. This phenomenon often creates confusion when we Business Transactions
ask what constitutes web mining. Web mining can be broken Modern business processes are consolidating with millions of
down into following subtasks: customers and billions of their transactions. Business enter-
1. Resource finding: retrieving documents intended for the prises require necessary information for their effective
web. functioning in today’s competitive world. For example, they
would like know: “Is this transaction fraudulent?”; “Which
2. Information selection and preprocessing: automatically customer is likely to migrate?”, and “What product is this
selecting and preprocessing specific information from customer most likely to buy next?’.
resources retrieved from the web.
3. Generalization: to automatically discover general patterns at
Electronic Commerce
Not only does electronic commerce produce large data sets in
individual web site as well as across multiple sites.
which the analysis marketing patterns and risk patterns is critical
4. Analysis: validation and/or interpretation of the mined but, it is also important to do this near-real time, in order to
patterns. meet the demands of online transactions.
88
Data Mining Applications-Case Studies

B. Scientific, Engineering And Health Care Data
Scientific data and metadata tend to be more complex in There is a wide range of well-established business applications
structure than business data. In addition, scientists and for data mining. These include customer attrition, profiling,
engineers are making increasing use of simulation and systems promotion forecasting, product cross-selling, fraud detection,
with application domain knowledge. targeted marketing, propensity analysis, credit scoring, risk
analysis, etc. We shall now discuss a few mock case studies and
Genomic Data
areas of DM applications.
Genomic sequencing and mapping efforts have produced a
number of databases, which are accessible on the web. In Housing Loan Prepayment Prediction
addition, there are also a wide variety of other online databases. A home-finance loan actually has an average life-span of only 7
Finding relationships between these data sources is another to 10 years, due to prepayment. Prepayment means that the
fundamental challenge for data mining. . loan is paid off early, rather than at the end of say, 25 years.
People prepay loans when they refinance or when they sell their
Sensor Data
home. The financial return that a home-finance institution
Remote sensing data is another source of voluminous data.
derives from a loan depends on its life-span. Therefore, it is
Remote sensing satellites and a variety of other sensors produce
necessary for the financial institutions to be able to predict the
large amounts of geo-referenced data. A fundamental challenge
life-spans of their loans. Rule discovery techniques can be used
is to understand the relationships, including causal relation-
to accurately predict the aggregate number of loan prepayments
ships, amongst this data.
in a given quarter (or, in a year), as a function of prevailing
Simulation Data interest rates, borrower characteristics, and account data. This
Simulation is now accepted as an important mode of science, information can be used to fine-tune loan parameters such as
supplementing theory and experiment. Today, not only do interest rates, points, and fees, in order to maximize profits.
experiments produce huge data sets, but so do simulations.
Data mining and, more generally, data intensive computing is Mortgage Loan Delinquency Prediction
proving to be a critical link between theory; simulation, and Loan defaults usually entail expenses and losses for the banks
experiment. and other lending institutions. Data mining techniques can be
used to predict whether or not a loan would go delinquent
Health Care Data within the succeeding 12 months, based on historical data, on
Hospitals, health care organizations, insurance companies, and account information, borrower demographics, and economic
the concerned government agencies accumulate large collections indicators. The rules can be used to estimate and finetune loan
of data about patients and health care-related details. Under- loss reserves and to gain some business insight into the
standing relationships in this data is critical for a Wide variety of characteristics and circumstances of delinquent loans. This will
problems ranging from determining what procedures and also help in deciding the funds that should be kept aside to
clinical protocols are most effective, to how best deliver health handle bad loans.
care to the maximum number of people.
Crime Detection
Web Data Crime detection is another area one might immediately associate
The data on the web is growing not only in volume but also in with data mining. Le us consider a specific case: to find patterns
complexity. Web data now includes not only text, audio and in ‘bogus official’ burglaries.
video material, but also streaming data and numerical data.
A typical example of this kind of crime is when someone turns
Multimedia Documents up at the door pretending to be from the water board, electricity
Today’s technology for retrieving multimedia items on the web board, telephone department or gas company. Whilst they
is far from satisfactory. On the other hand, an increasingly large distract the householder, their partners will search the premises
number of matters are on the web and the number of users is and steal cash and items of value. Victims of this sort of crime
also growing explosively. It is becoming harder to extract tend to be the elderly. These cases have no obvious leads, and
meaningful information from the archives of multimedia data data mining techniques may help in providing some unexpected
as the volume grows. connections to known perpetrators.
Data Web In order to apply data mining techniques, let us assume that
Today, the web is primarily oriented toward documents and each case is filed electronically, and contains descriptive informa-
their multimedia extensions. HTML has proved itself to be a tion about the thieves. It also contains a description of their
simple, yet powerful, language for supporting this. Tomorrow, modus operandi. We can use any of the clustering techniques to
the potential exists for the web to prove equally important for examine a situation where a group of similar physical descrip-
working with data. The Extensible Markup Language (XML) is tions coincide with a group of similar modus operandi. If there is
an emerging language for working with data in networked a good match here, and the perpetrators are known for one or
environments. As this infrastructure grows, data mining is more of the offences, then each of the unsolved cases could
expected to be a critical enabling technology for the emerging have well been committed by the same people.
data web. By matching unsolved cases with known perpetrators, it would
be possible to clear up old cases and determine patterns of
behavior. Alternatively, if the criminal is unknown but a large
89
cluster of cases seem to point to the same offenders, then these • Determine credit card spending by’ customer groups
frequent offenders can be subjected to careful examination. • Finding hidden correlations between different financial
Store-Level Fruits Purchasing Prediction indicators
A super market chain called ‘Fruit World’ sells fruits of different • Identifying stock trading rules from historical market data
types and it purchases these fruits from the wholesale suppliers
Insurance and Health Care
on a day-to-day basis. The problem is to analyze fruit-buying
patterns, using large volumes of data captured at the ‘basket’ • Claims analysis - i.e., which medical procedures are claimed
level. Because fruits have a short shelf life, it is important that together
accurate store-level purchasing predictions should be made to • Predict which customers will buy new policies
ensure optimum freshness and availability. The situation is • Identify behavior patterns of risky customers
inherently complicated by the ‘domino’ effect. For example,
• Identify fraudulent behavior
when one variety of mangoes is sold out, then sales are
transferred to another variety. With help of data mining Transportation
techniques, a thorough understanding of purchasing trends • Determine the distribution schedules among outlets
enables a better availability of fruits and greater customer • Analyze loading patterns
satisfaction.
Medicine
Other Application Area
• Characterize patient behavior to predict office visits
Risk Analysis • Identify successful medical therapies for different illnesses
Given a set of current customers and an assessment of their
risk-worthiness, develop descriptions for various classes. Use Discussion
these descriptions to classify a new customer into one of the • Discuss different data mining tasks.
risk categories. • What is spatial data mining?
Targeted Marketing • What is sequence mining?
Given a database of potential customers and how they have • What is web mining?
responded to a solicitation, develop a model of customers
most likely to respond positively, and use the model for more • What is text mining?
focused new customer solicitation. Other applications are to • Discuss the applications of data mining in the banking
identify buying patterns from customers; to find associations industry.
among customer demographic characteristics, and to predict the • Discuss the applications of data mining in customer
response to mailing campaigns. relationship management.
Retail/Marketing • How is data mining relevant to scientific data?
• Identify buying patterns from customers • How is data mining relevant for web-based computing?
• Find associations among customer demographic • Discuss the application of data mining in science data.
characteristics
Bibliography
• Predict response to mailing campaigns
• Agrawal R, Gupta A., and Sarawagi S., Modeling
• Market basket analysis multidimensional databases. ICDE, 1997.
Customer Retention • Anahory S., and Murray D. Data warehousing in the Real
Given a database of past customers and their behavior prior to World: Apracfical guide for building decision support systems.
attrition, deve1op a model of customers most likely to leave. Addison Wesley Longman, 1997.
Use the model for determining the course of action for these • Barbara D. (ed.) Special Issue on Mining of Large Datasets;
customers. IEEE Data Engineering Bulletin, 21 (1), 1998
Portfolio Management • Brachman R., Khabaza T., Kloesgen W., Shapiro G.P.,
Given a particular financial ‘asset, predict the return on invest- and Simoudis E., Industrial applications of data mining
ment to determine the inclusion of the asset in a folio or not. and knowledge discovery, Communication of ACM,- 1926.
Brand Loyalty • Fayyad U.M., Piatetsky-Shapiro G., Smyth P.,
Given a customer and the product he/she uses, predict whether Uthurusamy R. (Eds.): Advances in Knowledge Discovery and
the customer will switch brands. Data Mining. Menlo Park, CA: AAAI Press/ The MIT Press,
1996
Banking
The application areas in banking are: • Fayyad U.M., Uthurusamy R. (eds.): Special issue on data
mining. CommunicatJon of ACM,1996 .
• Detecting patterns of fraudulent credit card use
• Identifying ‘loyal’ customers • Grossman R., Kasif S.,Moore R., Rocke D. and Ullmann
J. Data Mining Research: Opportunities and Challenges, A Report.
• Predicting customers likely to change their credit card www..ncdni.uic.edu/M3D-final-report.htm., Jan 1999.
affiliation
90
• Heckerman D., Bayesian networks for data, mining. Data

mining and knowledge Discovery, 1997.
• Imielinski T., Virmani A., Association Rules ... and What’s
Next? Towards Second Generation Data Mining Systems; In
Proceedings of the 2nd East=European ADBIS Conference., pp. 6-
25,1998.
• Mannila H., Methods and Problems in Data Mining; In
Proceedings of the 6’h International Ccmference on Database
Theory 1997 ,.Springer LNCS 1186.
• Nestorov S., and Tsur S. Integrating data miniqg with:
relational DBMS: A tightly coupled approach, “www-
db.stanford.edu/people/evitov.html; 1998.
• Piatetsky-Shapiro G., and Frawley W. (ed.): Knowledge
Discovery in Databases, MIT Press, Cambridge, Ma, 1991
• Sarawagi S., et al. Integrating Association Rule Mining with
Relational Database Systems: Alternatives and Implications;
In Proceedings of the ACM SIGMOD International Conference
on Management of Data 1998,343-354.
• Vinnani A. Second Generation Data Mining: Concepts and
implementation ; Ph.D. Thesis, Rutgers University, Apri11998.
Some slides on important Topics:
91
Notes
92
LESSON 22 CHAPTER 5
VARIOUS TECHNIQUES OF DATA DATA MINING TECHNIQUES
MINING NEAREST NEIGHBOR AND
CLUSTERING TECHNIQUES
Structure is likely to buy shoes. (2) An X-ray image maintaining

• Objective characteristics a and b is likely to also exhibit characteristic c.
• Introduction. 2. Classification hierarchies-The goal is to work from an
existing set of even a transaction to create a hierarchy of
• Types of Knowledge Discovered during Data Mining
classes. Examples: (1) A population may be -divided into
• Association Rules five ranges of credit worthiness based on a history of
• Classification hierarchies previous co transactions. (2) A model may be developed for
• Sequential patterns the factors that determine desirability of location of a. store-
on a 1-10 scale. (3) Mutual funds may be classified based on
• Patterns within time series
performance data using characteristics such as growth,
• Categorization and segmentation income, and stability.
• Comparing the Technologies 3. Sequential patterns - A sequence of actions or events is
• Clustering And Nearest-Neighbor Prediction Technique sought. Example: If a patient underwent cardiac bypass
• Where to Use Clustering and Nearest-Neighbor Prediction surgery for blocked arteries and an aneurysm and later
developed high blood urea within a year of surgery, he or she
• Clustering for clarity
is likely to suffer from kidney failure within the next 18
• Nearest neighbor for prediction months. Detection of sequential pat-terns is equivalent to
• There is no best way to cluster detecting association among events with certain temporal
• How are tradeoffs made when determining which records fall relationships.
into which clusters? 4. Patterns within time series - Similarities can be detected
• What is the difference between clustering nearest-neighbor within positions of the time series. Three examples follow
prediction? with the stock market price data as a time series: (1) Stocks of
a utility company ABC Power and a financial company XYZ
• Cluster Analysis: Overview
Securities show the same pattern during 1998 in terms of
Objective closing stock price. (2) Two products show the same selling
At the end of this lesson you will be able to pattern in summer but a different one in win-ter. (3) A
• Understand various techniques used in Data mining pattern in solar magnetic wind may be used to predict
• Study about Nearest Neighbor and clustering techniques changes in earth atmospheric conditions
• Understand about Cluster Analysis. 5. Categorization and segmentation - A given population of

events or items can be partitioned (segmented) into sets of
Introduction “similar” elements. Examples: (1) An entire population of
In this lesson you will study about various techniques used in treatment data on a disease may be divided into groups
Data mining. You will study in detail about Nearest Neighbor based on the similarity of side effects produced. (2) The
and clustering techniques. I will also cover Cluster Analysis in adult popu1ation in the United States may be categorized
this lesson. into five groups from “most likely to buy” to “least likely to
Types of Knowledge Discovered during Data Mining buy” a new product. (3) The web accesses made by a
The term “knowledge” is broadly interpreted as involving some collection of users against a set of documents (say, in a
degree of intelligence. Knowledge is often class inductive and digital library) may be analyzed in. terms of the key-words of
deductive. Data mining addresses inductive knowledge. documents to reveal clusters or categories of Users.
Knowledge can be represented in many forms in an unstruc- Comparing the Technologies
tured sense; it can be represented by rules, or prepositional logic. Most of the data mining technologies that are out there today
In a structured form, it may be represented in decision trees, are relatively new to the business community, and there are a lot
semantic networks, neural works, or hierarchies of classes or of them (each technique usually is accompanied by a plethora of
frames. new companies and products). Given this state of affairs, one
The knowledge discovered during data mining can be described of the most important questions being asked right after “What
in five ways, as follows. is data mining?” is “Which technique(s) do I choose for my
1. Association Rules-These rules correlate the presence of a set particular business problem?” The answer is, of course, not a
of items another range of values for another set of variables. simple one. There are inher-ent strengths and weaknesses of
Examples: (1) When a _ retail shopper buys a handbag, she the different approaches, but most of the weaknesses can be
overcome. How the technology is implemented into the data-
93
mining product can make all the difference in how easy the Where to Use Clustering and Nearest-
product is to use independent of how complex the underlying Neighbor Prediction

technology is. Clustering and nearest-neighbor prediction are used in a wide
The confusion over which data mining technology to use is variety of appli-cations, ranging from personal bankruptcy
further exacerbated by the data mining companies themselves prediction to computer recognition-on of a person’s handwrit-
who will often lead one to believe that their product is deploying. People who may not even realize that they are doing any
ing a brand-new technology that is vastly superior to any other kind of clustering also use these methods every day. For
technology currently developed. Unfortunately this is rarely the instance, we may group certain types of foods or automobiles
case, and as we show in the chapters on modeling and compar- together (e.g., high-fat foods, U.S. manufactured cars).
ing the technologies, it requires a great’ deal of discipline and Clustering for Clarity
good experimental method to fairly compare different data Clustering is a method in which like records are grouped
mining methods. More often than not, this discipline is not together. Usually this is done to give the end user a high-level
used when evaluating many of the newest technologies. Thus view of what is going on in the database.
the claims of improved accuracy that are often made are not
Clustering is a data mining technique that is directed toward the
always defensible.
goals of identification and classification. Clustering to identify a
To appear to be different from the rest, many of the products finite set of categories or clusters to which each data object
that arrive on the market are packaged in a way so as to mask the (tuple) can be mapped. The categories may be disjoint or
inner workings of the data mining algorithm Many data mining overlapping and may sometimes be organized into trees. For
companies emphasize the newness and the black-box nature of example, one might form categories of customers into the
their technology. There will, in fact, be data mining offerings form d” tree and then map each customer to one or more of
that seek to combine every possible new technology into their the categories. A closely related problem is that of estimating
product in the belief that more is better. In fact, more technol- multivariate probability density functions of all variables that -
ogy is usually just more confusing and makes it more difficult could be attributes in a relation or from different relations.
to make a fair comparison between offer-ings. When these
techniques are understood and their similarities researched, one
Clustering for Outlier Analysis
Sometimes clustering is performed not so much to keep records
will find that many techniques that appeared to initially be
together as to make it easier to see when one record sticks out
different when they were not well understood are, in fact, quite
from the rest. For instance
similar. For that reason the data mining technologies that are
introduced in this book are the basics from which the thou- Most wine distributors selling inexpensive wine in Missouri
sands of subtle variations are made. If you can understand and that ship a certain volume of product produce a certain level
these technologies and where they can be used, you will of profit. A cluster of stores can be formed with these character-
probably understand better than 99 percent of all the techniques istics. One store stands out, however, as producing significantly
and products that are currently available. lower profit. On closer examination it turns out that the
distributor was delivering product to but not collecting
To help compare the different technologies and make the
payment from one of its customers.
business user a lit-tle more savvy in how to choose a technol-
ogy, we have introduced a high-level system of scorecards for A sale on men’s suits is being held in all branches of a depart-
each data mining technique described in this book. These ment store for southern California. All stores except one with
scorecards can be used by the reader as a first-pass high-level these characteristics have seen at least a 100 percent jump in
look at what the strengths and weaknesses are for each of the revenue since the start of the sale.
different techniques. Along with the scorecard will be a more
detailed description of how the scores were arrived! It, and if
the score is low, what possible changes or workarounds could
be made in the technique to improve the situation.
Clustering And Nearest-Neighbor
Prediction Technique
Clustering and the nearest-neighbor prediction technique are
among the oldest techniques used in data mining. Most people Nearest Neighbor for Prediction
have an intuition that they understand what clustering is – One essential element underlying the concept of clustering is
namely that like records are grouped or clustered together and that one particu-lar object (whether cars, food, or customers) can
put into the same grouping. Nearest neighbor is a prediction be closer to another object than can some third object. It is
technique that is quite similar to clustering; its essence is that in interesting that most people have an innate sense of ordering
order to determine what a prediction value is in one record, the placed on a variety of different objects.
user should look for records with similar predictor values in the
historical database and use the prediction value from the record Most people would agree that an apple is closer to an orange
that is “nearest” to the unknown record. than it is to a tomato and that a Toyota Corolla is closer to a
Honda Civic than to a Porsche. This sense of ordering on many
different objects helps us place them in time and space and to
94
make sense of the world. It is what allows us to build clus-ters- constructed for no particular purpose except to note similarities

both in databases on computers and in our daily lives. This between some of the records and that the view of the database
definition of nearness that seems to be ubiquitous also allows could be somewhat simplified by using clusters. But even the
us to make predictions. The nearest-neighbor prediction differences that were created by the two different clustering were
algorithm, simply stated, is driven by slightly different motivations (financial vs. romantic).
Objectsthatare“near”to each other will have similar prediction In general, the reasons for clustering are just this ill defined
values as well. Thus, if you know the prediction value of one because clusters are used more often than not for exploration
of the objects, you can predict it for its nearest neighbors. and summarization and nut _s much as for prediction.
One of the classic places where nearest neighbor has been used How are tradeoffs made when determining which
for prediction has been in text retrieval. The problem to be records fall into which clusters?
solved in text retrieval is one in which end users define a Note that for the first clustering example, there was a pretty
document (e.g., a Wall Street Journal article, a techni-cal confer- simple rule by which the records could be broken up into
ence paper) that is interesting to them and they solicit the clusters-namely, by income.
system to “find more documents like this one,” effectively TABLE : A Simple Clustering of the Example Database
defining a target of “this is the interesting document” or “this
is not interesting.” The prediction problem is that only a very
few of the documents in the database actually have values for ID. Name Prediction Age Balance ($) Income Eyes Gender
this prediction field (viz., only the documents that the reader
has had a chance to look at so far). The nearest-neighbor 3 Betty No 47 16,543 High Brown F
technique is used to find other documents that share important 5 Carla Yes 21 2,300 High Blue F
characteristics with those documents that have been marked as 6 Carl No 27 5,400 High Brown M
interesting. AB with almost all prediction algorithms, nearest
8 Don Yes 46 0 High Blue M
neigh-bor can be used for a wide variety of places. Its successful
use depends mostly on the pre-formatting of the data, so that 1 Amy No 62 0 Medium Brown F
nearness can be calculated, and where individual records can be 2 AI No 53 1,800 Medium Green M
defined. In the text-retrieval example this was not too difficult- 4 Bob Yes 32 45 Medium Green M
the objects were documents. This is not always as easy as it is
for text retrieval. Consider what it might be like in a time series 7 Donna Yes 50 165 Low Blue F
problem-say, for pre-dicting the stock market. In this case the 9 Edna Yes 27 . 500 Low Blue F
input data is just a long series of stock prices over time without 10 Ed No 68 1,200 Low Blue M
any particular record that could be considered to be an object.
The value to be predicted is just the next value of the stock
price. What is the difference between clustering nearest-
This problem is solved for both nearest-neighbor techniques neighbor prediction?
and for some other types of prediction algorithms by creating The main distinction between clustering and the nearest-
training records, taking, for instance, 10 consecutive stock prices neighbor technique is that clustering is what is called an
and using the first 9 as predictor values and the 10th as the unsupervised learning technique and near-est neighbor is generally
prediction value. Doing things this way, if you had 100 data used for prediction or a supervised learning technique. Unsuper-
points in your time series, you could create at least 10 different vised learning techniques are unsupervised in the sense that
training records. when they are run, there is no particular reason for the creation
of the models the way there is for supervised learning tech-
You could create even more training records than 10 by creating
niques that are trying to perform predic-tion. In prediction, the
a new record starting at every data point. For instance, you could
patterns that are found in the database and presented in the
take the first 10 Data points in your time series and create a
model are always the most important patterns in the database
record. Then you could take the 10 consecutive data points
for perform-ing some particular prediction. In clustering there is
starting at the second data point, then the 10 consecu-tive data
no particular sense of why certain records are near to each other
points starting at the third data point. Even though some of
or why they all fall into the same cluster. Some of the differ-
the data points would overlap from one record to the next, the
ences between clustering and nearest-neighbor prediction are
prediction value would always be different. In our example of
summarized in Table 20.7.
100 initial data points, 90 different training records could be
created this way, as opposed to the 10 training records created How is the space for clustering and nearest
via the other method. neighbor defined?
For clustering, the n-dimensional space is usually defined by
There is no best way to cluster assigning one predictor to each dimension. For the nearest-
This example, although simple, points up some important
neighbor algorithm, predictors are also mapped to dimensions,
questions about clus-tering. For instance, is it possible to say
but then those dimensions are literally stretched or compressed
whether the first clustering that was per-formed above (by
according to how important the particular predictor is in making
financial status) was better or worse than the second clustering
the prediction. The stretching of a dimension effectively makes
(by age and eye color)? Probably not, since the clusters were
95
that dimension (and hence predictor) more important than the
Clustering and segmentation basically partition the database so
others in calculating the distance. that each partition or group is similar according to some criteria
For instance, ‘if you were a mountain climber and someone or metric. Clustering according to similarity is a concept, which
told you that you were 2 mi from your destination, the distance appears in many disciplines. If a measure of similarity is
would be the same whether it were 1 mi north and’ 1 mi up the available there are a number of techniques for forming clusters.
face of the mountain or 2 mi north on level ground, but clearly Membership of groups can be based on the level of similarity
the former route is much different from the latter. The dis-tance between members and from this the rules of membership can
traveled straight upward is the most important in figuring out be defined. Another approach is to build set functions that
how long it will really take to get to the destination, and you measure some property of partitions ie groups or subsets as
would probably like to con-sider this “dimension” to be more functions of some parameter of the partition. This latter
important than the others. In fact, you, as a mountain climber, approach achieves what is known as optimal partitioning.
could “weight” the importance of the vertical dimension in Many data mining applications make use of clustering according
calculating some new distance by reasoning that every mile to similarity for example to segment a client/customer base.
upward is equiva-lent to 10 mi on level ground. Clustering according to optimization of set functions is used in
If you used this rule of thumb to weight the importance of data analysis e.g. when setting insurance tariffs the customers
one dimension over the other, it would be clear that in one case can be segmented according to a number of parameters and the
you were much “farther away” from your destination (limit) optimal tariff segmentation achieved.
than in the second (2 mi). In the net section we’ll show how the Clustering/segmentation in databases are the processes of
nearest neighbor algorithm uses the distance measure that separating a data set into components that reflect a consistent
similarly weights the important dimensions more heavily when pattern of behavior. Once the patterns have been established
calculating a distance. they can then be used to “deconstruct” data into more under-
standable subsets and also they provide sub-groups of a
population for further analysis or action, which is important
Nearest Neighbor Clustering when dealing with very large databases. For example a database
Used for prediction as Used mostly for consolidating data into could be used for profile generation for target marketing where
well as consolidation. a high-level view and general grouping previous response to mailing campaigns can be used to generate
of records into like behaviors. a profile of people who responded and this can be used to
predict response and filter mailing lists to achieve the best
Space is defined by the Space is defined as default n- response.
problem to be solved dimensional space, or is defined by the Discussion
(Supervised Learning). user, or is a predefined space driven by1. Write short notes on:
past experience (Unsupervised learning).
• Clustering
Generally, only uses Can use other metrics besides distance • Sequential patterns
distance metrics to to determine nearness of two records – • Segmentation
determine nearness. for example, linking points together. • Association rules
• Classification hierarchies
2. Explain Cluster analysis.
3. Correctly contrast the difference between supervised and
Cluster Analysis: Overview unsupervised learning.
In an unsupervised learning environment the system has to 4. Discuss in brief, where Clustering and Nearest-Neighbor
discover its own classes and one way in which it does this is to Prediction are used?
cluster the data in the database as shown in the following
5. How is the space for clustering and nearest neighbor
diagram. The first step is to discover subsets of related objects
defined? Explain.
and then find descriptions e.g., D1, D2, D3 etc. which describe
each of these subsets. 6. What is the difference between clustering nearest-neighbor
prediction?
Figure 5: Discovering clusters and descriptions in a database
7. How are tradeoffs made when determining which records fall
into which clusters?
8. Explain the following:
• Clustering
• Nearest-Neighbor
96
97
Some important slides
LESSON 23
DECISION TREES
Structure What is a Decision Tree?

• Objective A decision tree is a predictive model that, as its name implies,
can be viewed as a tree. Specifically, each branch of the tree is a
• Introduction classification question, and the leaves of the tree are partitions
• What is a Decision Tree? of the data set with their classification. For instance, if we were
• Advantages and Shortcomings of Decision Tree going to classify customers who churn (don’t renew their phone
Classifications: contracts) in the cellular telephone industry, a decision tree
• Where to Use Decision Trees? might look something like that found in following figure.
• Tree Construction Principle
• The Generic Algorithm
• Guillotine Cut
• Overfit
Objective
At the end of this lesson you will be able to understand
Decision trees, as techniques for data mining.
Introduction
The classification of large data sets is an important problem in
data mining. The classification problem can be simply stated as
follows. For a database with a number of records and for a set
of classes such that each record belongs to one of the given
classes, the problem of classification is to decide the class to
which a given record belongs. But there is much more to this
than just simply classifying. The classification problem is also
You may notice some interesting things about the tree.
concerned with generating a description or a model for each class
from the given data set. Here, we are concerned with a type of
classification called supervised classification. In supervised • It divides the data on each branch point without losing any
classification, we have a training data set of records and for each of the data (the number of total records in a given parent
record of this set; the respective class to which it belongs is also node is equal to the sum of the records contained in its two
known. Using the training set, the classification process children).
attempts to generate the descriptions of the classes, and these • The number of churners and no churners is conserved as
descriptions help to classify the unknown records. In addition you move up or down the tree.
to the training set, we can also have a test data set, which is used • It is pretty easy to understand how the model is being built
to determine the effectiveness of a classification. There are (in contrast to
several approaches to supervised classifications. Decision trees
(essentially, Classification trees) are especially attractive in the data- • The models from neural networks or from standard
mining environment as they represent rules. Rules can readily be statistics).
expressed in natural language and are easily comprehensible. • It would also be pretty easy to use this model if you actually
Rules can also be easily mapped to a database access language, had to target
like SQL. • Those customers who are likely to churn with a targeted
This lesson is concerned with decision trees as techniques for marketing offer.
data mining. Though the decision-tree method is a well-known • You may also build some intuitions about your customer
technique in statistics and machine learning, these algorithms are base, for example,
not suitable for data mining purposes. The specific require- • Customers who have been with you for a couple of years
ments that should be taken into consideration while designing and have up-to-date
any decision tree construction algorithms for data mining are
that: • Cellular phones and are pretty loyal
a. The method should be efficient in order to handle a very From a business perspective, decision trees can be viewed as
large-sized database, creating a seg-mentation of the original data set (each segment
would be one of the leaves of the tree). Segmentation of
b. The method should be able to handle categorical attributes.
98
customers, products, and sales regions is something that Table 23.1 Training Data Set

marketing managers have been doing for many years. In the
past this seg-mentation has been performed in order to get a
high-level view of a large amount of data-with no particular
reason for creating the segmentation except that the records OUTLOOK TEMP(F) HUMIDITY(%) WINDY CLASS
within each segmentation were somewhat similar to each other.
In this case the segmentation is done for a particular reason- sunny 79 90 true no play
namely, for the prediction of some important piece of
information. The records that fall within each segment fall there sunny 56 70 false play
because they have similarity with respect to the infor-mation
being predicted-not just that they are similar-without “similar- sunny 79 75 true play
ity” being well defined. These predictive segments that are
derived from the deci-sion tree also come with a description of sunny 60 90 true no play
the characteristics that define the pre-dictive segment. Thus the
decision trees and the algorithms that create them may be overcast 88 88 false no play
complex, but the results can be presented in an easy-to-
understand way that can be quite useful to the business user. overcast 63 75 true play
Decision Trees
overcast 88 95 false play
Decision trees are simple knowledge representation and they
classify examples to a finite number of classes, the nodes are
ram 78 60 false play
labeled with attribute names, the edges are labeled with possible
values for this attribute and the leaves labeled with different
ram 66 70 false no play
classes. Objects are classified by following a path down the tree,
by taking the edges, corresponding to the values of the
ram 68 60 true no play
attributes in an object.
The following is an example of objects that describe the
weather at a given time. The objects contain information on the
outlook, humidity etc. Some objects are positive examples There is a special attribute: the attribute class is the class label.
denote by P and others are negative i.e. N. Classification is in The attributes, temp (temperature) and humidity are numerical
this case the construction of a tree structure, illustrated in the attributes and the other attributes are categorical, that is, they
following diagram, which can be used to classify all the objects cannot be ordered. Based on the training data set, we want to
correctly. find a set of rules to know what values of outlook, temperature,
humidity and wind, determine whether or not to play golf. Figure
23.1 gives a sample decision tree for illustration.
In the above tree (Figure 23.1), we have five leaf nodes. In a
decision’ tree, each leaf node represents a rule. We have the
following rules corresponding to the tree given in Figure 23.1.
RULE 1 If it is sunny and the humidity is not above 75%,
then play.
RULE 2 If it is sunny and the humidity is above 75%, then do
not play.
RULE 3 If it is overcast, then play.
Decision Tree Structure RULE 4 If it is rainy and not windy, then play.
In order to have a clear idea of a decision tree, I have explained RULE 5 If it is rainy and windy. then don’t play.
it with the following examples: Please note that this may not be the best set of rules that can
Example 23.1 _be derived from the given set of training data.
Let us consider the following data sets-the training data set (see
Table 23.1) and the test data set (see Table 23.2). The data set Outlook
has five attributes.

Humidity Play Windy
Play No Play No Play Play
< = 75 > 75 True False

Figure 23.1 A Decision Tree
99
Example 23.2
The classification of an unknown input vector is done by
traversing the tree from the root node to a leaf node. A record At this stage, let us consider another example to illustrate the
enters the tree at the root node. At the root, a test is applied to concept of categorical attributes. Consider the following training
determine which child node the record will encounter next. This data set (Table 23.3). There are three attributes, namely, age, pin
process is repeated until the record arrives at a leaf node. All the code and class. The attribute class is used for class label.
records that end up at a given leaf of the tree are classified in the Table 23.3 Another Example
same way. There is a unique path from the root to each leaf. The
path is a rule, which is used to classify the records.
ID AGE PINCODE CLASS
In the above tree, we can carry out the classification for an
1 30 5600046 Cl
unknown record as follows. Let us assume” for the record, that
2 25 5600046 Cl
we know the values of the first four attributes (but we do not
know the value of class attribute) as 3 21 5600023 C2
4' 43 5600046 Cl
outlook= rain; temp = 70; humidity = 65; and windy= true.
5 18 5600023 C2
We start from the root node to check the value of the attribute
6 33 5600023 Cl
associated at the root node. This attribute is the splitting
7 29 5600023 Cl
attribute at this node. Please note that for a decision tree, at
8 55 5600046 C2
every node there is an attribute: associated with the node called
the splitting attribute. In our example, outlook is the splitting 9 48 5600046 Cl
attribute at root. Since for the given record, outlook = rain, we
move to the right-most child node of the root. At this node, The attribute age is a numeric attribute, whereas pincode is a
the splitting attribute is windy and we find that for the record categorical one. Though the domain of pincode is numeric, no
we want classify, windy = true. Hence, we move to the left child ordering can be defined among pincode values. You cannot
node to conclude that the class label is “no play”. derive any useful information if one pin-code is greater than
Note that every path from root node to leaf nodes represents a another pincode. Figure 23.2 gives a decision tree for this
rule. It may be noted that many different leaves of the tree may training data. The splitting attribute at the root is pincode and
refer to the same class labels, but each leaf refers to a different the splitting criterion here is pincode = 500 046. Similarly, for
rule. the left child node, the splitting criterion is age £ 48 (the
splitting attribute is age). Although the right child node has the
The accuracy of the classifier is determined by the percentage of
same attribute as the splitting attribute, the splitting criterion is
the test data set that is correctly classified. Consider the follow-
different.
ing test data set (Table 23.2).
Most decision tree building algorithms begin by trying to find
Table 23.2 Test Data Set
the test, which does the best job of splitting the records among
the desired categories. At each succeeding level of the tree, the
OUTLOOK TEMP(F) HUMIDITY(%) WINDY CLASS subsets created by the preceding split are themselves split
according to whatever rule works best for them. The tree
sunny 79 90 true play
continues to grow until it is no longer possible to find better
sun)1Y 56 70 false play ways to split up incoming records, or when all the records are in
sunny 79 75 true no play one class.
sunny 50 90 true no play Figure 23.2 A Decision Tree
overcast 88 88 false no play
Pin code = 560046; [1-9]
overcast 63 75 true Play
overcast 88 95 false Play
ram 78 60 false play Age < = 48; Age < = 21;
1,2,4,8,9 [3,5,6,7]
ram 66 70 false no play
rain 68 60 true play
C1; 1,2,4,9 C2; 8 C2; [3,5] C1;[6,7]
We can see that for Rule 1 there are two records of the test data
set satisfying outlook= sunny and humidity s 75, and only one In Figure 23.2, we see that at the root level we have 9 records.
of these is correctly classified as play. Thus, the accuracy of this The associated splitting criterion is pincode = 500 046. As a
rule is 0.5 (or 50%). Similarly, the accuracy of Rule 2 is also 0'.5 result, we split the records into two subsets, Records 1, 2, 4, 8
(or 50%). The accuracy of Rule 3 is 0.66. and 9 are to the left child node and the remaining to the right
node. This process is repeated at every node.
A decision tree construction process is concerned with identify-
ing the splitting attributes and splitting criteria at every level of
the tree. There are several alternatives and the main aim of the
100
decision tree construction process is to generate simple, prediction to time series prediction of the exchange rate of

comprehensible rules with high accuracy. different international currencies. There are also some problems
Some rules are apparently better than others. In the above where decision trees will not do as well. Some very simple
example, we see that Rule 3 is simpler than Rule 1. The measure problems in which the prediction is just a simple multiple of
of simplicity is the number of antecedents of the rule. It may the predictor can be solved much more quickly and easily by
happen that another decision tree may yield a rule like: "if the linear regression. Usually the models to be built and the
temperature lies between 70° F and 80° F, and the humidity is interactions to be detected are much more complex in real-world
between 75% and 90%, and it is not windy, and it is sunny, problems, and this is where decision trees excel.
then play". Naturally, we would prefer a rule like Rule 1 to this Tree Construction Principle
rule. That is why simplicity is sought after. After having understood the basic features of decision trees, we
Sometimes the classification efficiency of the tree can be shall now focus on the methods of building such trees from a
improved by revising the Tree through some processes like given training data set. Based on the foregoing discussion, I
pruning and grafting. These processes are activated after the shall formally define few concepts for your study.
decision tree is built. Definition 23.1: Splitting Attribute
Advantages and Shortcomings of Decision Tree Classifications: With every node of the decision tree, there is an associated
The major strength~ of the decision tree methods are the attribute whose values determine the partitioning of the data
following: set when the node is expanded.
• Decision trees are able to generate understandable rules, Definition 23.2: Splitting Criterion
• They are able to handle both numerical and the categorical The qualifying condition on the splitting attribute for data set
attributes, and splitting at a node, is called the splitting criterion at that node.
For a numeric attribute, the criterion can be an equation or an
• They provide a clear indication of which fields are most
inequality. For a categorical attribute, it is a membership
important for prediction or classification.
condition on a subset of values.
Some of the weaknesses of the decision trees are:
All the decision tree construction methods are based on the
• Some decision trees can only deal with binary-valued target principle of recursively' partitioning the data set till homogene-
classes. Others are able to assign records to an arbitrary ity is achieved. We shall study this common principle later and
number of classes, but are error-prone when the number of discuss in detail the features of different algorithms individu-
training examples per class gets small. This can happen rather ally. The construction of the decision tree involves the following
quickly in a tree with many levels and/or many branches per three main phases.
node.
• Construction phase - The initial decision tree is constructed
• The process of growing a decision tree is computationally in this phase, based on the entire training data set. It requires
expensive. At each node, each candidate splitting field is recursively partitioning the training set, into two, or more,
examined before its best split can be found. sub-partitions using splitting criteria, until a stopping criteria
Where to use Decision Trees? is met.
Decision trees are a form of data mining technology that has • Pruning phase - The tree constructed in the previous phase
been around in a form very similar to the technology of today may not result in the best possible set of rules due to over-
for almost 20 years now, and early versions of the algorithms fitting (explained below). The pruning phase removes some
date back till the 1960s. Often these techniques were originally of the lower branches and nodes to improve its
developed for statisticians to automate the process of determin- performance.
ing which fields in their database were actually useful or • Processing the pruned tree - to improve understandability.
correlated with the particular problem that they were trying to
Though these three phases are common to most of the well-
understand. Partly because of this history, decision tree
known algorithms, some of them attempt to integrate the first
algorithms tend to automate the entire process of hypothesis
two phases into a single process.
generation and then validation much more completely and in a
much more integrated way than any other data mining tech- The Generic Algorithm
niques. They are also particularly adept at handling raw data with Most of the existing algorithms of the construction phase use
little or no preprocessing. Perhaps also because they were Hunt's method as the basic principle in this phase. Let the
originally developed to mimic the way an analyst interactively training data set be T with class-labels {C1, C2, .., Ck}, The tree
performs data mining, they provide a simple-to-understand is built by repeatedly partitioning the training data, using some
predictive model based on rules (such as "90 percent of the time criterion like the goodness of the split. The process is continued
credit card customers of less than 3 months who max out their till all the records in a partition belong to the same class.
credit limits are going to default on their credit card loans"). • T is homogeneous T contains cases all belonging to a single
Because decision trees score so highly on so many of the critical class Cj. The decision tree for T is a leaf identifying class Cj.
features of data mining, they can be used in a wide variety of • T is not homogeneous T contains cases that belong to a
business problems for both exploration and prediction. They mixture of classes. A test is chosen, based on a single
have been used for problems ranging from credit card attrition attribute, that has one or more mutually exclusive outcomes
101
{Ol, O2, ..., On} .T is partitioned into the subsets T1, T2, a. Overfitted models are incorrect
T3, ..., Tn where T1; contains all those cases in T that have b. Overfitted decision trees require more space and more
the outcome OJ of the chosen test. The decision tree for T computational resources.
consists of a decision node identifying the test, and one
c. Overfitted models require the collection of unnecessary
branch for each possible outcome. The same tree building
features
method is applied recursively to each subset of training cases.
Most often, n is chosen to be 2 and hence, the algorithm d. They are more difficult to comprehend.
generates a binary decision tree. The pruning phase helps in handling the overfitting problem.
• T is trivial T contains no cases. The decision tree T is a leaf, The decision tree is pruned back by removing the subtree
but the class to be associated with the leaf must be rooted at a node and replacing it by a leaf node, using some
determined from information other than T. criterion. Several pruning algorithms are reported in literature.
The generic algorithm of decision tree construction outlines the In the next lesson we will study about Decision Tress Construc-
common principle of all algorithms. Nevertheless, the follow- tion Algorithms and the working of decision trees.
ing aspects should be taken into account while studying any Exercises
specific algorithm. In one sense, the following are three major 1. What is a decision tree? Illustrate with an example.
difficulties, which arise when one uses a decision tree in a real-
life situation. 2. Describe the essential features in a decision tree. How is it
useful to classify data?
Guillotine Cut 3. What is a classification problem? What is supervised
Most decision tree algorithms examine only a single attribute at
classification? How is a decision tree useful in classification?
a time. As mentioned in the earlier paragraph, normally the
splitting is done for a single attribute at any stage and if the 4. Explain where to use Decision Trees?
attribute is numeric, then the splitting test is an inequality. 5. What are the disadvantages of the decision tree over other
Geometrically, each splitting can be viewed as a plane parallel to classification techniques?
one of the axes. Thus, splitting one single attribute leads to 6. What are advantages and disadvantages of the decision tree
rectangular classification boxes that may not correspond too approach over other approaches of data mining?
well with the actual distribution of records in the decision space.
7. What are the three phases of construction of a decision tree?
We call this the guillotine cut phenomenon. The test is of the
Describe the importance of each of the phases.
form (X> z) or (X < z), which is called a guillotine cut, since it
creates a guillotine cut subdivision of the Cartesian space of the 8. The 103 generates a
ranges of attributes. a. Binary decision tree
However, the guillotine cut approach has a serious problem if a b. A decision tree with as many branches as there are
pair of attributes are correlated. For example, let us consider distinct values of the attribute
two numeric attributes, height (in meters) and weight (in c. A tree with a variable number of branches, not related
Kilograms). Obviously, these attributes have a strong correla- to the domain of the attributes
tion. Thus, whenever there exists a correlation between d. A tree with an exponential number of branches.
variables, a decision tree with the splitting criteria on a single
attribute is not accurate. Therefore, some researchers propose an Suggested Readings
oblique decision tree that uses a splitting criteria involving more 1. Pieter Adriaans, Dolf Zantinge Data Mining, Pearson
than one attribute. Education, 1996
Over Fit 2. George M. Marakas Modern Data Warehousing, Mining, and
Decision trees are built from the available data. However, the Visualization: Core Concepts, Prentice Hall, 1st edition, 2002
training data set may not be a proper representative of the real- 3. Alex Berson, Stephen J. Smith Data Warehousing, Data
life situation and may also contain noise. In an attempt to build Mining, and OLAP (Data Warehousing/Data Management),
a tree from a noisy training data set, we may grow a decision tree McGraw-Hill, 1997
just deeply enough to perfectly classify the training data set. 4. Margaret H. Dunham Data Mining, Prentice Hall, 1st
Definition 23.3 Overfit edition, 2002
A decision tree T is said to over fit the training data if there 5. David J. Hand Principles of Data Mining (Adaptive
exists some other tree T' which is a simplification of T, such Computation and Machine Learning), Prentice Hall, 1st
that T has smaller error than T' over the training set but T' has a edition, 2002
smaller error than T over the entire distribution of instances. 6. Jiawei Han, Micheline Kamber Data Mining, Prentice Hall,
Overfitting can lead to difficulties when there is noise in the 1st edition, 2002
training data, or when the number of training examples is too
7. Michael J. Corey, Michael Abbey, Ben Taub, Ian Abramson
small. Specifically, if there is no conflicting instances in the
Oracle 8i Data Warehousing McGraw-Hill Osborne Media,
training data set, the error of the fully built tree is zero, while
2nd edition, 2001
the true error is likely to be bigger. There are many disadvan-
tages of an overfitted decision tree:
102
LESSON 24
DECISION TREES - 2
Structure does the best job of separating the records into groups, where a
• Objective single class predominates. To choose the best splitter at a node,
we consider each independent attribute in turn.
• Introduction
Assuming that an attribute takes on multiple values, we sort it
• Best Split and then, using some evaluation function as the measure of
• Decision Tree Construction Algorithms goodness, evaluate each split. We compare the effectiveness of
• CART the split provided by the best splitter from each attribute. The
• ID3 winner is chosen as the splitter for the root node. How does
one know which split is better than the other? We shall discuss
• C4.5
below two different evaluation functions to determine the
• CHAID splitting attributes and the splitting criteria.
• When does the tree stop growing? Decision Tree Construction Algorithms
• Why would a decision tree algorithm prevent the tree from A number of algorithms for inducing decision trees have been
growing if there weren’t enough data? proposed over the years. However, they differ among them-
• Decision trees aren’t necessarily finished after they are fully selves in the methods employed for selecting splitting attributes
grown and splitting conditions. In the following few sections, we shall
• Are the splits at each level of the tree always binary yes/no study some of the major methods of decision tree construc-
splits? tions.
• Picking the best predictors CART

CART (Classification And Regression Tree) is one of the
• How decision trees works?
popular methods of building decision trees in the machine
Objective learning community. CART builds a binary decision tree by
The objective of this lesson is to introduce you with decision splitting the records at each node, according to a function of a
tree construction algorithms along with the working of decision single attribute. CART uses the gini index for determining the
trees. best split. CART follows the above principle of constructing
Introduction the decision tree. We outline the method for the sake of
In this lesson, I will explain you various kinds of decision tree completeness
construction algorithms like CART, ID3, C4.5 and CHAID. The initial split produces two nodes, each of which we now
You will study about the working of decision trees in detail. attempt to split in the same manner as the root node. Once
again, we examine all the input fields to find the candidate
Best Split
splitters. If no split can be found that significantly decreases the
We have noticed that there are several alternatives to choose
diversity of a given node, we label it as a leaf node. Eventually,
from for the splitting attribute and the splitting criterion. But in
only leaf nodes remain and we have grown the full decision
order to build an optimal decision tree, it is necessary to select
tree. The full tree may generally not be the tree that does the
those corresponding to the best possible split. The main
best job of classifying a new set of records, because of
operations during the tree building are
overfitting.
1. Evaluation of splits for each attribute and the selection of
At the end of the tree-growing process, every record of the
the best split; determination of the splitting attribute,
training set has been assigned to some leaf of the full decision
2. Determination of the splitting condition on the selected tree. Each leaf can now be assigned a class and an error rate. The
splitting attribute, and error rate of a leaf node is the percentage of incorrect classifica-
3. Partitioning the data using the best split. tion at that node. The error rate of an entire decision tree is a
The complexity lies in determining the best split for each weighted sum of the error rates of all the leaves. Each leaf ’s
attribute. The splitting also depends on the domain of the contribution to the total is the error rate at that leaf multiplied
attribute being numerical or categorical. The generic algorithm by the probability that a record will end up in there.
for the construction of decision trees assumes that the method ID3
to decide the splitting attribute at a node and the splitting Quinlan introduced the ID3, Iterative Dichotomizer 3, for
criteria are known. The desirable feature of splitting is that it constructing the decision trees from data. In ID3, each node
should do the best job of splitting at the given stage. corresponds to a splitting attribute and each arc is a possible
The first task is to decide which of the independent attributes value of that attribute. At each node the splitting attribute is
makes the best splitter. The best split is defined as one that selected to be the most informative among the attributes not
103
yet considered in the path from the root. Entropy is used to categories) in the partition, and thus does not take into account
measure how informative is a node. This algorithm uses the the fact that different numbers of branches are considered.
criterion of information gain to determine the goodness of a
When does the tree stop growing?
split. The attribute with the greatest information gain is taken as
If the decision tree algorithm just continued like this, it could
the splitting attribute, and the data set is split for all distinct
conceivably cre-ate more and more questions and branches in
values of the attribute.
the tree so that eventually there was only one record in the
C4.5 segment. To let the tree grow to this size is compu-tationally
C4.5 is an extension of ID3 that accounts for unavailable expensive and also unnecessary. Most decision tree algorithms
values, continuous attribute value ranges, pruning of decision stop growing the tree when one of three criteria are met:
trees and rule derivation. In building a decision tree, we can deal 1. The segment contains only one record or some
with training sets that have records with unknown attribute algorithmically defined min-imum number of records,
values by evaluating the gain, or the gain ratio, for an attribute (Clearly, there is no way to break a smile-record segment into
by considering only those records where those attribute values two smaller segments, and segments with very few records
are available. We can classify records that have unknown are not likely to be very helpful in the final prediction since
attribute values by estimating the probability of the various the predictions that they are making won’t be based on
possible results. Unlike CART, which generates a binary decision sufficient historical data.)
tree, C4.5 produces trees with variable branches per node. When
2. The segment is completely organized into just one
a discrete variable is chosen as the splitting attribute in C4.5,
prediction value. There is no reason to continue further
there will be one branch for each value of the attribute.
segmentation since this data is now com-pletely organized
CHAID (the tree has achieved its goal).
CHAID, proposed by Kass in 1980, is a derivative of AID 3. The improvement in organization is not sufficient to
(Automatic Interaction Detection), proposed by Hartigan in warrant making the split. For instance, if the starting
1975. CHAID attempts to stop growing the tree before segment were 90 percent churners and the resulting segments
overfitting occurs, whereas the above algorithms generate a fully from the best possible question were 90.001 percent churners
grown tree and then carry out pruning as post-processing step. and 89.999 percent churners, then not much progress would
In that sense, CHAID avoids the pruning phase. have been or could be made by continuing to build the tree.
In the standard manner, the decision tree is constructed by
Why would a decision tree algorithm prevent the
partitioning the data set into two or more subsets, based on the
tree from growing if there weren’t enough data?
values of one of the non-class attributes. After the data set is
Consider the following example of a segment that we might
partitioned according to the chosen attributes, each subset is
want to split further because it has only two examples. Assume
considered for further partitioning using the same algorithm.
that it has been created out of a much larger customer database
Each subset is partitioned without regard to any other subset.
by selecting only those customers aged 27 with blue eyes and
This process is repeated for each subset until some stopping
with salaries ranging between $80,000 and $81,000.
criterion is met. In CHAID, the number of subsets in a
partition can range from two up to the number of distinct In this case all the possible questions that could be asked about
values of the splitting attribute. In this regard, CHAID differs the two cus-tomers turn out to have the same value (age, eye
from CART, which always forms binary splits, and from ID3 or color, salary) except for name.
C4.5, which form a branch for every distinct value. TABLE: Decision Tree Algorithm Segment>
The splitting attribute is chosen as the one that is most Name Age Eyes Salary ($) Churned?
significantly associated with the dependent attributes according
Steve 27 Blue 80,000 Yes
to a chi-squared test of independence in a contingency table (a
Alex 27 Blue 80,000 No
cross-tabulation of the non-class and class attribute). The main
stopping criterion used by such methods is the p-value from * This segment cannot be split further except by using the
this chi-squared test. A small p-value indicates that the observed predictor “name.”
association between the splitting attribute and the dependent
Decision trees aren’t necessarily finished after they
variable is unlikely to have occurred solely as the result of
are fully grown
sampling variability.
After the tree has been grown to a certain size (depending on
If a splitting attribute has more than two possible values, then the particular stop-ping criteria used in the algorithm), the
there may be a very large number of ways to partition the data CART algorithm has still more work to do. The algorithm then
set based on these values. A combinatorial search algorithm can checks to see if the model has been over fit to the data. It does
be used to find a partition that has a small p-value for the chi- this in several ways using a cross-validation approach or a test
squared test. The p-values for each chi-squared test are adjusted set valida-tion approach-basically using the same mind-
for the multiplicity of partitions. A Bonferroni adjustment is numbingly simple approach it used to find the best questions
used for the p-values computed from the contingency tables, in the first place: trying many different simpler versions of the
relating the predictors to the dependent variable. The adjust- tree on a held-aside test set. The algorithm as the best model
ment is conditional on the number of branches (compound selects the tree that does the best on the held-aside data. The
104
nice thing about CART is that this testing and selection is all an Table - Two Possible Splits for Eight Records with

integral part of the algo-rithm as opposed to the after-the-fact Calculation of Entropy for Each Split Shown*
approach that other techniques use.
Are the splits at each level of
the tree always binary yes/no
splits?
There are several different methods of
building decision trees, some of which
can make splits on multiple values at
time-for instance, eye color: green, blue,
and brown. But recognize that any tree
that can do binary splits can effectively
partition the data in the same way by
to determine which split is chosen should make the correct
just building two levels of the tree: the first, which splits brown
choice in this case and others like it. The metric needs to take
and blue from green; and the second, which splits apart the
into account two main effects:
brown and blue split. Either way, the minimum number of
questions you need to ask is two. • How much has the disorder been lowered in the new
segments?
How the Decision Tree Works
In the late 1970s J. Ross Quinlan introduced a decision tree • How should the disorder in each segment be weighted?
algorithm named ID3. This was one of the first decision tree The entropy measure can easily be applied to each of the new
algorithms though it was built solidly on previous work on segments as eas-ily as it was applied to the parent segment to
inference systems and concept learning systems from that answer the first question, but the second criterion is a bit harder.
decade and the preceding decade. Initially ID3 was used for Should all segments that result from a split be treated equally?
tasks such as learn-ing good game-playing strategies for chess This question needs to be answered in the example above
end games. Since then ID3 has been applied to a wide variety of where the split has produced a perfect new segment but with
problems in both academia and industry and has been modi- little real value because of its size. If we just took the average
fied, improved, and borrowed from many times over. entropy for the new segments, we would choose split B since in
ID3 picks predictors and their splitting values on the basic of that case the average of 0.99 and 0.0 is around 0.5 We can also
the gain in information that the split or splits provide. Gain do this calculation for split A and come up with an average
represents the difference between the amount of information entropy of 0.72 for the new segments.
that is needed to correctly make a predic-tion both before and If, on the other hand, we weighted the contribution of each
after the split has been made (if the amount of informa-tion new segment with respect to the size of the segment (and
required is much lower after the split is made, then that split has consequently how much of the database that segment ex-
decreased the disorder of the original single segment) and is plained), we would get a quite different measure of the
defined as the dif-ference between the entropy of the original disor-der across the two new segments. In this case the
segment and the accumulated entropies of the resulting split weighted entropy of the two segments for split A is the same
segments. Entropy is a well-defined measure of the disorder or as before but the weighted entropy of split B is quite a bit
information found in data. higher. (See the following Table)
The entropies of the child segments are accumulated by Since the name of this game is to reduce entropy to as little as
weighting their con-tribution to the entire entropy of the split possible, we are faced with two different choices of which is the
according to the number of records they contain. For instance, best split. If we average the entropies of the new segments, we
which of the two splits shown in Table 18.7 would you think would pick split B; if we took into account the number of
decreased the entropy the most and thus would provide the records that are covered by each split, we would pick split A.
largest gain? ID3 uses the weighted entropy approach as it has been found,
Split A is actually a much better split than B because it separates in general, to produce better predictions than just averaging the
out more of the data despite the fact that split B creates a new entropy. Part of the reason for this may be that, as .we have
segment that is perfectly homogeneous (0 entropy). The seen from the modeling chapter, that the more data that is used
problem is that this perfect zero-entropy seg-ment has only one in a prediction, the more likely the prediction is to be correct and
record in it and splitting off one record at a time will not cre-ate the more likely the model is to match the true underlying causal
a very useful decision tree. The small number of records in each reasons and processes that are actually at work in forming the
segment (Let 1) is unlikely to provide useful prediction values.
repeatable patterns. The calculation (met-ric)
that we use
105
State of the Industry Use this example to illustrate the working of different algo-
The current offerings in decision tree software emphasize rithms.

different important aspects and use of the algorithm. The 4. Overfitting is an inherent characteristic of decision tree and
different emphases are usually driven because of differences in its occurrence depends on the construction process and not
the targeted user and the types of problems being solved. There on the training data set. True or False?
are four main categories of products:
5. Pruning is essentially to avoid overfitting. True or False?
• Business-those that emphasize ease of use for the business
6. Bootstrapping is carried out in the main memory. True or
users
False?
• Performance-those that emphasize the overall performance
and database size
• C4.5
• Exploratory-those that emphasize ease of use for the
analyst • CHAID
• Research-those tailored specifically for detailed research • ID3
or academic experimentation • CART
Tools such as Pilot Software’s Discovery Server fall into the Suggested Readings
category of busi-ness use. The Pilot Discovery Server (trade- 1. Pieter Adriaans, Dolf Zantinge Data Mining, Pearson
mark) provides easy-to-use graphi-cal tools to help the business Education, 1996
user express their modeling problem and also provides
applications such as the Pilot Segment Viewer and Pilot Profit 2. George M. Marakas Modern Data Warehousing, Mining, and
Chart (both trademarks) to allows business end users to Visualization: Core Concepts, Prentice Hall, 1st edition, 2002
visualize the model and per-form simple profitless models on 3. Alex Berson, Stephen J. Smith Data Warehousing, Data
different targeted marketing applications. A tool that falls into Mining, and OLAP (Data Warehousing/Data Management),
the performance category would be Thinking Machines Cor- McGraw-Hill, 1997
poration’s Star Tree tool, which implements CART on MPP 4. Margaret H. Dunham Data Mining, Prentice Hall, 1st
and SMP computer hardware and has been optimized for large edition, 2002
databases and difficult-to-solve problems. Angoss’ Knowledge 5. David J. Hand Principles of Data Mining (Adaptive
Seeker (trademark) tool, on the other hand, is targeted mostly at Computation and Machine Learning), Prentice Hall, 1st edition,
the PC user but provides more control to the analyst to spec-ify 2002
different parameters that control the underlying CHAID
6. Jiawei Han, Micheline Kamber Data Mining, Prentice Hall,
algorithm, if desired. Salford Systems’ CART product provides
1st edition, 2002
even more control over the underlying algorithms but provides
only limited GUI or applications support; however, it is useful 7. Michael J. Corey, Michael Abbey, Ben Taub, Ian
to researchers and business analysts who want in-depth analysis Abramson Oracle 8i Data Warehousing McGraw-Hill Osborne
and control over their model creation. Media, 2nd edition, 2001
Exercises
1. What are advantages and disadvantages of the decision tree
approach over other approaches of data mining?
2. Describe the ID3 algorithm of the decision tree construction.
Why is it unsuitable for data mining applications?
3. Consider the following examples
M O T O R WHEELS DOORS SIZE TYPE CLASS
NO 2 0 small cycle bicycle
NO 3 0 small cycle tricycle
YES 2 0 small cycle motorcycle
YES 4 2 small automobile Sports car
YES 4 3 medium automobile minivan
YES 4 4 medium automobile sedan
YES 4 4 large automobile sumo
106
LESSON 25
NEURAL NETWORKS
Structure ment, which started off with the premise that machines could
• Objective be made to “think” if scientists found ways to mimic the struc-
ture and functioning of the human brain on the computer.
• Introduction
Thus historically neural networks grew out of the community
• What is a Neural Network? of artificial intelligence rather than from the discipline of
• Learning in NN statistics. Although scientists are still far from understanding
• Unsupervised Learning the human brain, let alone mimicking it, neural networks that
run on computers can do some of the things that people can
• Data Mining using NN: A Case Study
do.
Objective lt is difficult to say exactly when the first “neural network” on a
The aim of this lesson is to introduce you with the concept of computer was built. During World War II a seminal paper was
Neural Networks. It also includes various topics, which explains published by McCulloch and Pitts which first outlined the idea
how the method of neural network is helpful in extracting the that simple processing units (like the individual neurons in the
knowledge from the warehouse. human brain) could be connected together in large \networks
Introduction to create a system that could solve difficult problems and
Data mining is essentially a task of learning from data and display behavior that’ was much more complex than the simple
hence, any known technique which attempts to learn from data pieces that made it up. Since that time much progress has been
can, in principle, be applied for data mining purposes. In made in finding ways to apply artifi cial neural networks to real-
general, data mining algorithms aim at minimizing I/O world prediction problems and improving the per-formance of
operations of disk-resident data, whereas conventional the algorithm in general. In many respects the greatest break-
algorithms are more concerned about time and space complexi- through in neural networks in recent years have been in their
ties, accuracy and convergence. Besides the techniques discussed application to more mundane real-world problems such as
in the earlier lessons, a few other techniques hold promise of customer response prediction or fraud detection rather than the
being suitable for data mining purposes. These are Neural loftier goals that were originally set out for the techniques such
Networks (NN), Genetic Algorithms (GA) and Support Vector as overall human learning and computer speech and image
Machines (SVM). The intention of this chapter is to briefly understanding.
present you the underlying concepts of these subjects and Don’t neural networks learn to make better
demonstrate their applicability to data mining. We envisage that predictions?
in the coming years these techniques are going to be important Because of the origins of the techniques and because of some
areas of data mining techniques. of their early suc-cesses, the techniques have enjoyed a great deal
Neural Networks of interest. To understand how neural networks can detect
The first question that comes to the mind is, what is this patterns in a database, an analogy is often made that they
Neural Network? “learn” to detect these patterns and make better predictions,
When data mining algorithms are discussed these days, people similar to the way human beings do. This view is encouraged by
are usually talking about either decision trees or neural net- the way the historical training data is often supplied the
works. Of the two, neural net-works have probably been of network-one record (example) at a time.
greater interest through the formative stages of data mining Networks do “learn” in a very real sense, but under the hood,
technology. As we will see, neural networks do have disadvan- the algo-rithms and techniques that are being deployed are not
tages that can be limiting in their ease of use and ease of truly different from the techniques found in statistics or other
deployment, but they do also have some significant advantages. data mining algorithms. It is, for instance, unfair to assume that
Foremost among these advantages are their highly accurate neural networks could outperform other tech-niques because
predictive models, which can be applied across a large number they “learn” and improve over time while the other techniques
of different types of problems. remain static. The other techniques, in fact, “learn” from
To be more precise, the term neural network might be defined historical examples in exactly the same way, but often the
as an “artifi-cial” neural network. True neural networks are examples (historical records) to learn from are processed all at
biological systems [also known as (a.k.a.) brains] that detect once in a more efficient manner than are neural networks, which
patterns, make predictions, and learn. The artifi-cial ones are often modify their model one record at a time.
computer programs implementing sophisticated pattern Are neural networks easy to use?
detection and machine learning algorithms on a computer to A common claim for neural networks is that they are automated
build predictive models from large historical databases. Artificial to a degree where the user does not need to know that much
neural networks derive their name from their historical develop- about how they work, or about predictive modeling or even the
107
database in order to use them. The implicit claim is also that The first tactic has seemed to work quite well because when the
most neural networks can be unleashed on your data straight technique is used for a well-defined problem, many of the
out of the box without the need to, rearrange or modify the difficulties in preprocessing the data can be automated (because
data very much to begin with. the data structures have been seen before) and interpretation of
Just the opposite is often true. Many important design the model is less of an issue since entire industries begin to use
decisions need to be made in order to effectively use a neural the technology successfully and a level of trust is created. Several
network, such as ven-dors have deployed this strategy (e.g., HNC’s Falcon system
for credit card fraud prediction and Advanced Software Applica-
• How should the nodes in the network be connected?
tions’ Model MAX package for direct marketing).
• How many neurons like processing units should be used?
Packaging up neural networks with expert consultants is also a
• When should “training” be stopped in order to avoid over viable strat-egy that avoids many of the pitfalls of using neural
fitting? networks, but it can be quite expensive because it is human-
There are also many important steps required for preprocessing intensive. One of the great promises of data mining is, after all,
the data that goes into a neural network-most often there is a the automation of the predictive modeling process. These
requirement to normalize numeric data between 0.0 and 1.0, neural network-consulting teams are little different from the
and categorical predictors may need to be broken up into virtual analytical departments many companies already have in house.
predictors that are 0 or 1 for each value of the original categorical Since there is not a great difference in the overall predictive
predictor. And, as always, understanding what the data in your accuracy of neural networks over standard statistical techniques,
database means and a clear definition of the business problem the main difference becomes the replacement of the sta-tistical
to be solved are essential to ensuring eventual success. The expert with the neural network expert. Either with statistics or
bottom line is that neural networks provide no shortcuts. neural network experts, the value of putting easy-to-use tools
into the hands of the business end user is still not achieved.
Business Scorecard
Neural networks are very powerful predictive modeling Neural networks rate high for accurate mod-els that provide
techniques, but some of the power comes at the expense of good return on investment but rate low in terms of automa-
ease of use and ease of deployment. As we will see in this tion and clarity, making them more difficult to deploy across the
chapter, neural networks create very complex models that are enterprise.
almost always impossible to fully understand, even by experts. Where to use Neural Networks
The model itself is represented by numeric values in a complex Neural networks are used in a wide variety of applications. They
calculation that requires all the predictor values to be in the form have been used in all facets of business from detecting the
of a number. The output of the neural net-work is also fraudulent use of credit cards and credit risk prediction to
numeric and needs to be translated if the actual prediction value increasing the hit rate of targeted mailings. They also have a
is categorical (e.g., predicting the demand for blue, white, or long history of application in other areas such as the military for
black jeans for a clothing manufacturer requires that the the automated driving of an unmanned vehicle at 30 mph on
predictor values blue, black and white for the predictor color be paved roads to biological simulations such as learning the
converted to numbers). Because of the complexity of these correct pronunciation of English words from written text.
techniques, much effort has been expended in trying to increase
the clar-ity with which the model can be understood by the end Neural Networks for Clustering
user. These efforts are still in their infancy but are of tremen- Neural networks of various kinds can be used for clustering and
dous importance since most data mining techniques including prototype cre-ation. The Kohonen network described in this
neural networks are being deployed against real business chapter is probably the most common network used for
problems where significant investments are made on the basis clustering and segmentation of the database. Typi-cally the
of the predic-tions from the models (e.g., consider trusting the networks are used in a unsupervised learning mode to create the
predictive model from a neural network that dictates which one clus-ters. The clusters are created by forcing the system to
million customers will receive a $1 mailing). compress the data by cre-ating prototypes or by algorithms that
steer the system toward creating clus-ters that compete against
These shortcomings in understanding the meaning of the
each other for the records that they contain, thus ensuring that
neural network model have been successfully addressed in two
the clusters overlap as little as possible.
ways:
1. The neural network is packaged up into a complete solution Business Score Card for Neural Networks
such as fraud prediction. This allows the neural network to Data mining measure Description
be carefully crafted for one particular application, and once it Automation Neural networks are often repre
has been proven successful, it can be used over and over sented as automated data mining
again without requiring a deep understanding of how it techniques. While they are very
works. powerful at building predictive
models, they do require significant
2. The neural network is packaged up with expert consulting data preprocessing and a good
services. Here trusted experts who have a track record of understanding and definition of
success deploy the neural network. The experts either are able the prediction target. Usually
to explain the models or trust that the models do work.
108
normalizing predictor values between 0.0 and the picture looks like, but certainly describing it in terms of

1.0 and converting categorical to numeric high-level features requires much less communication of
values is required. The networks themselves information than the “paint by numbers” approach of
also require the setting of numerous describing the color on each square millimeter of the image.
parameters that determine how the neural If we think of features in this way, as an efficient way to
network is to be constructed (e.g., the communicate our data, then neural networks can be used to
number of hidden nodes). There can be automatically extract them. Using just five hidden nodes uses
significant differences in performance due to the neural network shown in Fig. 25.1 to extract features by
small differences in the neural network setup requiring the network to learn to re-create the input data at the
or the way the data is preformatted. output nodes. Consider that if you were allowed 100 hidden
Clarity The bane of neural networks is often the nodes, then re-creating the data for the network would be rather
clarity with which the user can see and trivial-involving simply pass-ing the input node value directly
understand the results that are being through the corresponding hidden node and on to the output
presented. To some degree the complexity of node. But as there are fewer and fewer hidden nodes, that infor-
the neural network models goes hand in mation has to be passed through the hidden layer in a more
hand with their power to create accurate and more efficient manner since there are less hidden nodes to
predictions. This shortcoming in clarity is help pass along the information.
recognized by the neural network vendors, To accomplish this, the neural network tries to have the hidden
and they have tried to provide powerful nodes extract features from the input nodes that efficiently
techniques to better visualize the neural describe the record rep-resented at the input layer. This forced
networks and to possibly “squeezing” of the data through the narrow hidden layer forces
ROI provide understandable rulers prototypes to the neural network to extract only those predictors and combi-
help explain the models. Neural networks nations of predictors that are best at re-creating the input
do provide powerful predictive models and record. The link weights used to create the inputs to the hidden
theoretically are more general than other data nodes are effectively cre-ating features that are combinations of
mining and standard statistical techniques. In the input node values.
practice, however, the gains in accuracy over
Applications Score Card
other techniques are often quite small and
Table 25.1 show the applications scorecard for neural net-works
can be dwarfed by some of the costs because
with respect to how well they perform for a variety of basic
of careless construction or use of the model
underlying applications. Neural networks have been used for
by nonexperts. The models can also be quite
just about every type of supervised and unsupervised learning
time-consuming to build.
application. Because the underlying model is a complex
Neural Networks for Feature Extraction mathematical equation, the generation of rules and the effi-cient
One of the important problems in all data mining is determin- detection of links in the database is a stretch for neural net-
ing which predic-tors are the most relevant and the most works. Also, because of the large number of different words in
important in building models that are most accurate at predic- text-based applications (high dimensionality), neural networks
tion. These predictors may be used by themselves or in are seldom used for text retrieval. They do provide some sense
conjunction with other predictors to form “features.” A simple of confidence in the degree of the prediction so that outliers
example of a fea-ture in problems that neural networks are who do not match the existing model can be detected.
working on is the feature of a verti-cal line in a computer image.
The predictors, or raw input data, are just the colored pixels
(picture elements) that make up the picture. Recognizing that
the predictors (pixels) can be organized in such a way as to create
lines, and then using the line as the input predictor, can prove
to dramatically improve the accuracy of the model and decrease
the time to create it.
Some features such “as lines in computer images are things that
humans are already pretty good at detecting; in other problem
domains it is more difficult to recognize the features. One novel
way that neural networks have been used to detect features is to
exploit the idea that features are a form of a compres-sion of Fig: 25.1
the training database. For instance, you could describe an image

to a friend by rattling off the color and intensity of each pixel
on every point in the picture, or you could describe it at a higher
level in terms of lines and circles or maybe even at a higher level
of features such as trees and mountains. In either case your
friend eventually gets all the information needed to know what
109
TABLE 25.1 Applications Score Card for Neural Networks Text Because of the large number of possible
Problem type Description input nodes (number of different words

used in a given language), neural networks
Clusters Although neural networks were
are seldom used for text retrieval. They have
originally conceived to mimic neural
been used at a higher level to create a network
function in the brain and then used
that learns the relationships between
for a variety of prediction and
documents.
classification tasks, they have also
been found useful for clustering. The General Idea
Almost coincidentally, the self- What does a neural network look like?
organizing nature of the brain A neural network is loosely based on concepts of how the
when mimicked in an artificial human brain is orga-nized and how it learns. There are two
neural network results in the main structures of consequence in the neural network:
clustering of records from a
1. The node-which loosely corresponds to the neuron in the
database.
human brain
Links Neural networks can be used to
2. The link-which loosely corresponds to the connections
determine links and patterns in the
between neurons
database, although to be efficient,
neural architectures very different (Axons, dendrites, and synapses) in the human brain
from the standard single hidden Figure 25.2 is a drawing of a simple neural network. The round
layer need to be used. To do this circles repre-sent the nodes, and the connecting lines represent
efficiently, a network would the links. The neural net-work functions by accepting predictor
generally have as many input nodes values at the left and performing calculations on those values to
as output nodes and each node produce new values in the node at the far right. The value at this
would represent an individual node represents the prediction from the neural network model.
object that could be linked together. In this case the network takes in values for predictors for age
Outliers The general structure of the neural and income and predicts whether the person will default on a
network is not designed for outlier bank loan.
detection in the way that nearest- How does a neural net make a prediction?
neighbor classification techniques
In order to make a prediction, the neural network accepts the
are, but they can be used for outlier
values for the predictors on what are called the input nodes.
detection by simply building the
These become the values for those nodes; these values are then
predictive model and seeing which
multiplied by values that are stored in the links (sometimes
record’s actual values correspond to
called weights and in some ways similar to the weights that are
the predicted values. Any large
applied to predictors in the nearest-neighbor method). These
disparity between the actual and
values are then added together at the node at the far right (the
predicted could well be an outlier.
output node), a special threshold function is applied, and the
Rules Neural networks do not generate resulting number is the prediction. In this case, if the resulting
rules either for classification or number is 0, the record is considered to be a good
explanation. Some new techniques
are now being developed that
would create rules after the fact to
try to help explain the neural
network, but these are additions to
the basic neural network architec
ture.
Sequences Because of their strengths in
performing predictions for numeric
prediction values and regression in
general, neural networks are often
used to do sequence prediction (like
predicting the stock market).
Generally a significant amount of
preprocessing of the data needs to
be performed to convert the time
series data into something useful to
the neural network.
110
Fig. 25.4
How complex can the neural network model

become?
The models shown in Figs. 25.2 and 25.3 have been designed
to be as simple as possible in order to make them understand-
able. In practice no networks are as simple as these. Figure 25.4
shows a network with many more links and many more nodes.
This was the architecture of a neural network system called
NET talk, which learned how to pronounce written English
words. This draw-ing shows only some of the nodes and links.
Each node in the network was con-nected to every node in the
level above it and below it, resulting in 18,629 link weights that
needed to be learned in the network. Note that this network
Figure 25.3, The normalized input values are multiplied by
also now has a row of nodes in between the input nodes and
the link weights and added together at the output.
the output nodes. These are called “hidden nodes” or the
Credit risk (no default); if the number is 1; the record is “hidden layer” because the values of these nodes are not visible
considered to be a bad credit risk (likely default). to the end user in the way that the output nodes are (which
A simplified version of the calculations depicted in Fig. 25.2 contain the prediction) and the input nodes (which just contain
might look like Fig. 25.3. Here the value of age of 47 is the pre-dictor values). There are even more complex neural
normalized to fall between 0.0 and 1.0 and has the value 0.47, network architectures that have more than one hidden layer. In
and the income is normalized to the value 0.65. This simplified practice one hidden layer seems to suffice, however.
neural network makes the prediction of no default for a 47-year-
old making $65,000. The links are weighted at 0.7 and 0.1, and Notes
the resulting value after multiplying the node values by the link
weights is 0.39. The network has been trained to learn that an
output value of 1.0 indicates default and that 0.0 indicate no
default. The output value calculated here (0.39) is closer to 0.0
than to 1.0, so the record is assigned a no default prediction.
How is the neural network model created?
The neural network model is created by presenting it with many
examples of the predictor values from records in the training set
(in this example age and income are used) and the prediction
value from those same records. By comparing the correct answer
obtained from the training record and the pre-dicted answer
from the neural network, it is possible to slowly change the
behavior of the neural network by changing the values of the
link weights. In some ways this is like having a grade school
teacher ask questions of her student (a.k.a. the neural network)
and if the answer is wrong, to verbally correct the student. The
greater the error, the harsher the verbal correction; thus large
errors are given greater attention at correction than are small
errors.
For the actual neural network, it is the weights of the links that
actually con-trol the prediction value for a given record. Thus the
particular model that is being found by the neural network is, in
fact, fully specified by the weights and the architectural structure
of the network. For this reason it is the link weights that are
modified each time an error is made.
111
LESSON 26
NEURAL NETWORKS
Structure The network has 2 binary inputs, I0 and I1 and one binary
• Objective output Y. W0 and W1 are the connection strengths of input 1
and input 2, respectively. Thus, the total input received at the
• What Is A Neural Network?
processing unit is given by
• Hidden nodes are like trusted advisors to the output nodes
W0I0 + W1I1 - Wb,
• Design decisions in architecting a neural network
where
• How does the neural network resemble the human brain?
Wb is the threshold (in another notational convention, it is
• Applications Of -Neural Networks
viewed as the bias). The output Y takes on the value 1, if W0I0
• Data Mining Using NN: A Case Study + W1I1 - Wb, > 0 and, otherwise, it is 0 if
Objective W0I0 + Will - Wb £ O.
The main objective of this lesson is to introduce you the But the model, known as perceptron, was far from a true model
principles of neural computing. of a biological neuron as, for a start, the biological neuron’s
What is a Neural Network? output is a continuous function rather than a step function.
Neural networks are a different paradigm for computing, which This model also has a limited computational capability as it
draws its inspiration from neuroscience. The human brain represents only a linear-separation. For two classes of inputs,
consists of a network of neurons, each of which is made up of which are linearly separable, we can find the weights such that
a number of nerve fibres called dendrites, connected to the cell the network returns 1 as output for one class and 0 for another
body where the cell nucleus is located. The axon is a long, single class.
fibre that originates from the cell body and branches near its end There have been many improvements on this simple model
into a number of strands. At the ends of these strands are the and many architectures have been presented in recently. As a first
transmitting ends of the synapses that connect to other step, the threshold function or the step function is replaced by
biological neurons through the receiving ends of the synapses other more general, continuous functions called activation.
found on the dendrites as well as the cell body of biological
neurons. A single axon typically makes thousands of synapses
with other neurons. The transmission process is a complex
chemical process, which effectively increases or decreases the
electrical potential within the cell body of the receiving neuron.
When this electrical potential reaches a threshold value (action
potential),itentersitsexcitatorystateandissaid tofire. It is the
connectivity of the neuron that gives these simple ‘devices’ their
real power.
Artificial neurons (or processing elements, PE) are highly
simplified models of biological neurons. As in biological
neurons, an artificial neuron has a number of inputs, a cell body Figure 26.1 a simple perceptron
(most often consisting of the summing node and the transfer
function), and an output, which can be connected to a number
of other artificial neurons. Artificial neural networks are densely
interconnected networks of PEs, together with a rule (learning
rule) to adjust the strength of the connections between the
units in response to externally supplied data.
The evolution of neural networks as a new computational
model originates from the pioneering work of McCulloch and
Pitts in 1943. They suggested a simple model of a neuron that
connoted the weighted sum of the inputs to the neuron and an
output of 1 or 0, according to whether the sum was over a
threshold value or not. A 0 output would correspond to the
inhibitory state of the neuron, while a 1 output would Figure 26.2 A Typical Artificial Neuron with Activation
correspond to the excitatory state of the neuron. Consider a Function
simple example illustrated below.
112
functions. Figure 26.2 illustrates the structure of a node (PE) 1. It is difficult to trust the prediction of the neural network if
with an activation function. For this particular node, n weighted the meaning of these nodes is not well understood.
inputs (denoted W;, i = 1,..., n) are combined via a combination 2. Since the prediction is made at the output layer and the
function that often consists of a simple summation. A transfer difference between the prediction and the actual value is
function then calculates a corresponding value, the result calculated there, how is this error cor-rection fed back
yielding a single output, usually between 0 and 1. Together, the through the hidden layers to modify the link weights. Those
combination function and the transfer function make up the connect them?
activation function of the node. The meaning of these hidden nodes is not necessarily well
Three common transfer functions are the sigmoid, linear and understood but sometimes after the fact they can be studied to
hyperbolic functions. The sigmoid function (also known as the see when they are active (have larger numeric values) and when
logistic function) is very widely used and it produces values they are not and derive some meaning from them. In some of
between 0 and 1 for any input from the combination function. the earlier neural networks were used to learn the family trees of
The sigmoid function is given by (the subscript n identifies a two different families-one was Italian and one was English and
PE): the network was trained to take as inputs either two people and
return their rela-tionship (father, aunt, sister, etc.) or given one
person and a relationship to return the. Other person. After
training, the units in one of the hidden layers were exam med
Y= to see If there was any discernible explanation as to their role in
the prediction. Several of the nodes did seem to have specific
and under-standable purposes. One, for instance, seemed to
Note that the function is strictly positive and defined for all break up the input records (people) into either Italian or
values of the input. When plotted, the graph takes on a English descent, another unit encoded for which generation a
sigmoid shape, with an inflection point at (0, .5) in the Carte- person belonged to, and another encoded for the branch of the
sian plane. The graph (Figure 26.3) plots the different values of family that the person came from. The neutral network to aid in
S as the input varies from -10 to 10. predictor automatically extracted each of these nodes.
Individual nodes are linked together in different ways to create Any interpretation of the meaning of the hidden nodes needs
neural networks. In a feed-forward network, the connections to be done after the fact-after the network has been trained, and
between layers are unidirectional from input to output. We it is not always possible to determine a logical description for
discuss below two different architectures of the feed-forward the particular function for the hidden nodes. The second
network, Multi-Layer Perceptron and Radial-Basis Function. problem with the hidden nodes is perhaps more serious (if it
hadn’t been solved, neural networks wouldn’t work). Luckily it
has been solved.
The learning procedure for the neural network has been defined
to work for the weights in the links connecting the hidden layer.
A good analogy of how this works would be a military
operation in some war where there are many layers of com-
mand with a general ultimately responsible for making the
decisions on where to advance and where to retreat. Sev-eral
lieutenant generals probably advise the general, and several
major generals, in turn, probably advise each lieutenant general.
This hierarchy continues downward through colonels and
privates at the bottom of the hierarchy.
This is not too far from the structure of a neural network with
several hid-den layers and one output node. You can think of
the inputs coming from the hidden nodes as advice. The link
weight corresponds to the trust that generals have in their
Figure 26.3 Sigmoid Functions advisors. Some trusted advisors have very high weights and
some advisors may not be trusted and, in fact, have negative
Hidden nodes are like trusted advisors to the output
weights. The other part of the advice from the advisors has to
nodes
do with how competent the particular
The meanings of the input nodes and the output nodes are
usually pretty well understood-and are usually defined by the Advisor is for a given situation. The general may have a trusted
end user with respect to the par-ticular problem to be solved advisor, but if that advisor has no expertise in aerial invasion
and the nature and structure of the database. The hidden and the situation in question involves the air force, this advisor
nodes, however, do not have a predefined meaning and are may be very well trusted but the advisor per-sonally may not
determined by the neural network as it trains. This poses two have any strong opinion one way or another.
problems:
In this analogy the link weight of a neural network to an
output unit is like the trust or confidence that commanders
have in their advisors and the actual node value represents how
strong an opinion this particular advisor has about this TABLE 26.1 Neural Network Nodes.
particular situation. To make a decision, the general considers
Advisor's Advisor's Change to
how trustworthy and valuable the advice is and how knowl- General's trust
edgeable and confident all the advisors are in making their Recommendation Confidence General’s trust
suggestions; then, taking all this into account, the general makes
the decision to advance or retreat. High Good High Great increase
In the same way, the output node will make a decision (a
High Good Low Increase
prediction) by tak-ing into account all the input from its
advisors (the nodes connected to it). In the case of the neural
High Bad High Great decrease
network multiplying the link weight by the output value of the
node and summing these values across all nodes reach this
High Bad Low Decrease
decision. If the prediction is incorrect, the nodes that had the
most influence on making the decision have their weights Low Good High Increase
modified so that the wrong prediction is less likely to be made
the next time. Low Good Low Small increase
This learning in the neural network is very similar to what
happens when the general makes the wrong decision. The Low Bad High Decrease
confidence that the general has in all those advisors who gave
the wrong recommendation is decreased-and all the more so for Low Bad Low Small decrease
those advisors who were very confident and vocal in they’re rec-
ommendations. On the other hand, any advisors who were
making the correct recommendation but whose input was not
taken as seriously would be taken more seriously the next time. *The link weights in a neural network are analogous to the
Likewise, any advisors who were reprimanded for giving the confidence that gen-erals might have. In there trusted advisors.
wrong advice to the general would then go back to their own diction from the neural network) through the hidden layers and
advisors and determine which of them should have been to the input layers. At each level the link weights between the
trusted less and whom should have been listened to more layers are updated so as to decrease the chance of making the
closely in rendering the advice or recommendation to the same mistake again.
general. The changes generals should make in listening to their
Design decisions in architecting a neural network
advisors to avoid the same bad decision in the future are shown
Neural networks are often touted as self-learning automated
in Table 26.1.
techniques that simplify the analysis process. The truth is that
This feedback can continue in this way down throughout the there still are many decisions to be made by the end user in
organizational each level, giving increased emphasis to those designing the neural network even before training begins. If
advisors who had advised cor-rectly and decreased emphasis to these decisions are not made wisely, the neural network will
those who had advised incorrectly. In this way the entire likely come up with a sub optimal model. Some of the
organization becomes better and better at supporting the gen- decisions that need to be made include
eral in making the correct decision more of the time. • How will predictor values be transformed for the input
A very similar method of training takes place in the neural nodes? Will normal-ization be sufficient? How will
network. It is called back propagation and refers to the propaga- categoricals be entered?
tion of the error backward from the output nodes (where the • How will the output of the neural network be interpreted?
error is easy to determine as the difference between the actual
prediction value from the training database and the pre- • How many hidden layers will there be?
• How will the nodes be connected? Will every node be
connected to every other node, or will nodes just be
connected between layers?
• How many nodes will there be in the hidden layer? (This can
have an impor-tant influence on whether the predictive
model is over fit to the training database.)
• How long should the network be trained for? (This also has
an impact on whether the model over fits the data.)
Depending on the tool that is being used, these decisions may
be (explicit, where the user must set some parameter value, or
they may be decided for the user because the particular neural
• Distant neurons seemed to inhibit each other

network is being used for a specific type of prob-lem (like fraud
detection). • The tasks assigned to other neurons.
Different types of Neural Networks Much of this early research came from the desire to simulate the
There are literally hundreds of variations on the back propaga- way that vision worked in the brain. For instance, some of the
tion feed forward neural networks that have been briefly early physiological showed that surgically rotating a section of a
described here. One involves changing the architecture of the frog’s eye ball so that it was upside down would result in the
neural network to include recurrent connections where the frog jumping up for food that was al below the frog’s body.
output from the output layer is connected back as input into This led to the belief that the neurons had certainty enable roles
the hidden layer. These recurrent nets are sometimes used for that were dependent on the physical location of the neuron.
sequence prediction, in which the previous outputs from the Koho-nen networks were developed to accommodate these
network need to be stored someplace and then fed back into the physiological features by a very simple learning algorithm:
network to provide context for the current prediction. Recurrent 1. Layout the output nodes of the network on a two-
networks have also been used for decreasing the amount of dimensional grid with no hidden layer.
time that it takes to train the neural network. Another twist on 2. Fully connect the input nodes to the output nodes.
the neural net theme is to change the way that the network
learns. Back propagation effec-tively utilizes a search technique 3. Connect the output nodes so that nearby nodes would
called gradient descent to search for possible improvement in strengthen each other and distant nodes would weaken each
the link weights to reduce the error. There ever, many other other.
ways of doing search in a high-dimensional space effectively 4. Start with random weights on the links.
utilizes a search technique called gradient descent to search for 5. Train by determining which output node responded most
the best possible improvement in the link weights to reduce the strongly to the Current record being input.
error. There are, how-ever, many other ways of doing search in a 6. Change the weights to that highest-responding node to
high-dimensional space (each link weight corresponds to a enable it to respond even more strongly in the future. This is
dimension), including Newton’s methods and conju-gate also known as Hebbian Learning.
gradient as well as simulating the physics of cooling metals in a
7. Normalize the link weights so that they add up to some
process called simulated annealing or in simulating the search
constant amount; thus, increasing one weight decreases
process that goes on in biological evolution and using genetic
some other.
algorithms to optimize the weights of the
8. Continue training until some form of global organization is
Neural networks. It has even been suggested that creating a
formed on the two-dimensional output grid (where there are
large number of neural networks with randomly weighted links
clear winning nodes for each input and, in general, local
and picking the one with the lowest error rate would be the best
neighborhoods of nodes are activated).
learning procedure.
When these networks were run, in order to simulate the real-
Despite all these choices, the back propagation learning proce-
world visual system it became obvious that the organization
dure is the most commonly used. It is well understood, is
that was automatically being con-structed on the data was also
relatively simple, and seems to work in a large number of
very useful for ‘3egmenting and clustering the train-ing
problem domains. There are, however, two other neural net-
database: Each output node represented a cluster and nearby
work architectures that are used relatively often. Kohonen
clusters were nearby in the two-dimensional output layer. Each
feature maps are often used for unsupervised learning and
record in the database would fall into one and only one cluster
clustering, and radial-basis-function networks are used for
(the most active output node), but the other clusters in which it
supervised learning and in some ways represent a hybrid
might also fit would be shown and likely to be next to the best
between nearest-neighbor and neural network classifications.
matching cluster. Figure 26.4 shows the general form of a
Kohonen feature Maps Kohonen network.
Kohonen feature maps were developed in the 1970s and were
created late certain human brain functions. Today they are used
mostly to unsupervised learning and clustering.
Kohonen networks are feed forward neural networks generally
with den layer. The networks contain only an input layer and an
output h the nodes in the output layer compete among
themselves to display the strongest activation to a given record,
what is sometimes called a “win all” strategy.
Behaviors 01 the real neurons were ‘taken into effect-namely,
that physical locality of the neurons seems to play an important
role in the behavior learning of neurons. The specific features Fig:26.4
of real neurons were:
• Nearby neurons seem to compound the activation of each
other.
115
How does the neural network resemble the human • Marketing:
brain? Neural networks have been used to improve marketing

Since the inception of the idea of neural networks, the ultimate mailshots. One technique is to run a test mailshot, and look
goal for these techniques has been to have them recreated at the pattern of returns from this. The idea is to find a
human thought and learning. This has once again proved to be predictive mapping from the data known about the clients to
a difficult task-despite the power of these new techniques and how they have responded. This mapping is then used to
the similarities of their architecture to that of the human brain. direct further mailshots.
Many of the things that people take for granted are difficult for
neural networks-such as avoiding over fitting and working with
Data Mining Using NN: A Case Study
In this section, I will outline a case study to you to illustrate the
real-world data without a lot of preprocessing required. There
potential application of NN for Data mining. This case study is
have also been some exciting successes.
taken from [Shalvi, 1996].
The human brain is still much more
powerful Knowledge Extraction Through Data
With successes like NET talk and ALVINN and some of the
Mining
Kohonen, self-organizing maps (SOMs) are used to cluster a
commercial suc-cesses of neural networks for fraud prediction
specific medical data set containing information about the
and targeted marketing, it 1.1; tempting to claim that neural
patients’ drugs, topographies (body locations) and morpholo-
networks are making progress toward “think-ing,” but it is
gies (physiological abnormalities); these categories can be
difficult to judge just how close we are. Some real facts that we
identified as the three input subspaces. Data mining techniques
can look at are to contrast the human brain as a computer to the
are used to collapse the subspaces into a form suitable for
neural net-work implemented on the computer.
network classification. The goal is to acquire medical knowledge
Today it would not be possible to create an artificial neural which may lead to tool formation, automation and to assist
network that even had as many neurons in it as the human medical decisions regarding population. The data is organized
brain, let alone all the process-ing required for the complex as three hierarchical trees, identified as Drugs, Topography and
calculations that go on inside the brain. The cur-rent estimates Morphology. The most significant portion of the morphology
are that there are 100 billion neurons in the average person tree is displayed in Figure 26.5. Before presenting the data to the
(roughly 20 times the number of people on earth). Each single neural network, certain preprocessing should be done. Standard
neuron can receive input from up to as many as 100,000 techniques can be employed to clean erroneous and redundant
synapses or connections to other neurons, and overall there are data.
10,000 trillion synapses.
To get an idea of the size of these numbers, consider that if
you had a 10-Tbyte data warehouse (the largest warehouse in
existence today), and you were able to store all of the complex-
ity of a synapse in only a single byte of data within that
warehouse, you would still require 1000 of these warehouses
just to store the synapse information. This doesn’t include the
data required for the neu-rons or all the computer processing
power required to actually run this simu-lated brain. The
bottom line is that we’re still a factor of 1000 away from even
storing the required data to simulate a brain. If storage densities
on disks and other media keep increasing and the prices
continue to decrease, this problem may well be solved. None-
theless, there is much more work to be done in under-standing
how real brains function.
Applications of Neural Networks
These days, neural networks are used in a very large number of
applications. We list here some of those relevant to our study.
Neural networks are being used in
• Investment analysis:
To predict the movement of stocks, currencies etc., from
previous data. There, they are replacing earlier simpler linear
models.
• Monitoring:
Fig:26.5
Networks have been used to monitor the state of aircraft

engines. By monitoring vibration levels and sound, an early
warning of engine problems can be given.
116
differentiation presented to the SOM’s input. For example,

The data is processed at the root level of each tree-fourteen root
level drugs, sixteen root level topographies and ten root level every one of the tuples in square (1,1) contains root level data
morphologies. By constraining all the data to the root level, the only for Drug 6, Topography 6 and Morphology 5. The tuple at
degree of differentiation has been greatly reduced from square (2,1) contains these three root level nodes as well as Drug
thousands to just 40. Each tuple is converted into a bipolar 7, a difference slight enough for the network to distinguish the
format. Thus, each tuple is a 40-dimensional bipolar array-a tuple by classifying it one square away from (1,1). All 34 tuples
value of either 1 or -1 depending on whether any data existed in square (3,1) contain Drug 6, Topography D and Morphology
for the leaves of that root node. 5; but only 29 of the 34 tuples contain Topography 6. Clearly,
the difference between square (3,1) and square (1,1) is greater
The Kohonen self-organizing map (SOM) was chosen to
than the difference between square (2,1) and square (1,1).
organize the data, in order to make use of a spatially ordered, 2-
dimensional map of arbitrary granularity. An n x n SOM was Exercises
implemented for several values ofn. We describe below only the
1. Describe the principle of neural computing and discuss its
case with n= 1 O. The input layer consists of 40 input nodes;
suitability to data mining.
the training set consists of 2081 tuples; and the training period
was of 30 epochs. The learning coefficient S is initialized to 0.06. 2. Discuss various application areas of Neural Networks.
After approximately 7Y2 epochs, S is halved to 0.03. After 3. Explain in brief different types of neural networks.
another 7Y2 epochs, it is halved again to 0.015. For the final set 4. “Hidden nodes are like trusted advisors to the output
of7Y2 epochs, it is halved again to become 0.0075. nodes”. Discuss.
5. Explain in brief Kohonen feature maps.
Notes
After the network is trained it is used for one final pass through
the input data set, in which the weights are not adjusted. This
provides the final classification of each input data tuple into a
single node in the l0 x 10 grid. The output is taken from the
coordinate layer as an (x, y) pair. The output of the SOM is a
population distribution of tuples with spatial significance
(Figure 26.6). This grid displays the number of tuples that were
classified into each Kohorien layer node (square) during testing;
for example, Square (1,1) contains 180 tuples.
Upon examination of the raw data within these clusters, one
finds similarities between the tuples which are indicative of
medical relationships or dependencies. Numerous hypotheses
can be made regarding these relationships, many of which were
not known a priori. The SOM groups together tuples in each
square according to their similarity. The only level at which the
SOM can detect similarities between tuples is at the root level of
each of the three subspace trees, since this is the level of
117
LESSON 27
ASSOCIATION RULES AND GENETIC ALGORITHM
Structure Þ Juice has only 25% support. Another term for support is
preva-lence of the rule.
• Objective
• Association Rules To compute confidence we consider all transactions that include
items in LHS. The confidence for the association rule LHS
• Basic Algorithms for Finding Association Rules
ÞRHS is the percentage (fraction) of such transactions that also
• Association Rules among Hierarchies include RHS. Another term for confidence is strength of the
• Negative Associations rule.
• Additional Considerations for Association Rules For MilkÞJuice, the confidence is 66.7% (meaning that, of three
• Genetic Algorithms (GA) transactions in which milk occurs, two contain juice) and
• Crossover breadÞjuice has 50% confidence (meaning that one of two
transactions containing bread also contains juice.)
• Mutation
As we can see, support and confidence do not necessarily go
• Problem-Dependent Parameters
hand in hand. The goal of mining association rules, then, is to
• Encoding generate all possible rules that exceed some mini-mum user-
• The Evaluation Step specified support and confidence thresholds. The problem is
thus decomposed into two subproblems:
• Data Mining Using GA
1. Generate all item sets that have a support that exceeds the
Objective threshold. These sets of items are called large itemsets. Note
The objective of this lesson is to introduce you with data that large here means large support.
mining techniques like association rules and genetic algorithm. 2. For each large item set, all the rules that have a minimum
Association Rules confidence are gener-ated as follows: for a large itemset X and
One of the major technologies in data mining involves the Y C X, let Z = X – Y; then if support (X) /support (Z) Þ
discovery of association rules. The database is regarded as a minimum confidence, the rule Z ÞY (Le., X - Y Þ Y) is a
collection of transactions, each involving. Set of items. A valid rule. [Note: In the previous sentence, Y C X reads, “Y
common example is that of market-basket data. Here the is a subset of X.”]
market basket corresponds to what a consumer buys in a
supermarket during one visit. Consider four such transactions Generating rules by using all large itemsets and their supports is
in a random sample: relatively straightforward. However, discovering all large
Transaction-id Time Items- Brought itemsets together with the value for their support is a major
101 6:35 milk, bread, juice problem if the cardinality of the set of items is very high. A
typical supermarket has thousands of items. The number of
792 7:38 milk, juice distinct itemsets is 2m, where m is the number c£ items, and
1130 8:05 milk, eggs counting support for all possible itemsets becomes very
1735 8:40 bread, cookies, coffee computation-intensive.
An association rule is of the form X=>Y, where X = {x1, x2... To reduce the combinatorial search space, algorithms for finding
xn}, and Y = {y1, y2...ym} are sets, of items, with xi and Yj being association rule have the following properties:
distinct items for all i and j. This association states that if a • A subset of a large itemset must also be large (i.e., each
customer buys X, he or she is also likely to buy Y. In general, subset of a large itemset: L exceeds the minimum required
any association rule has the form LHS (left-hand side) Þ RHS support).
(right-hand side), where LHS and RHS are sets of items. • Conversely, an extension of a small itemset is also small
Association rules should supply both support and confidence. (implying that it does 00: have enough support).
The support for the rule LHSÞRHS is the percentage of The second property helps in discarding an itemset from further
transactions that hold all of the items in the union, the set LHS consideration J extension, if it is found to be small.
U RHS. If the support is low, it implies that there is no
overwhelming evidence that items in LHS U RHS occur Basic Algorithms for Finding Association
together, because the union happens in only a small fraction of Rules
transactions. The rule MilkÞJuice has 50% support, while Bread The current algorithms that find large itemsets are designed to
work as follows:
118
1. Test the support for itemsets of length 1, called l-itemsets,

by scanning the database. Discard those that do not meet
minimum required support.
2. Extend the large l-itemsets into 2-itemsets by appending one
item each time, to generate all candidate itemsets of length
two. Test the support for all candidate itemsets by scanning
the database and eliminate those 2-itemsets that do not meet
the minimum support.
3. Repeat the above steps at step k; the previously found (k - 1)
itemsets are extended into k-itemsets and tested for
minimum support.
The process is repeated until no large item sets can be found.
However, the naive ver-sion of this algorithm is a combinato-
rial nightmare. Several algorithms have been pro-posed to mine
the association rules. They vary mainly in terms of how the However, associations of the type Healthy-brand frozen yogurt
candidate itemsets are generated, and how the supports for the Þ bottled water, or Richcream-brand ice cream Þ wine cooler may
candidate item sets are counted. produce enough confidence and sup-port to be valid association
Some algorithms use such data structures as bitmaps and has rules of interest.
trees to keep informa-tion about item sets. Several algorithms Therefore, if the application area has a natural classification of
have been proposed that use multiple scans of the database the itemsets into hier-archies, discovering associations within
because the potential number of item sets, 2m, can be too large the hierarchies is of no particular interest. The ones of specific
to set up counters during a single scan. We have proposed an interest are associations across hierarchies. They may occur
algorithm called the Partition algorithm Summarized below. among item groupings at-different levels.
If we are given a database with a small number of potential Negative Associations
large item sets, say, a few thousand, then the support for all of The problem of discovering a negative association is harder
them can be tested in one scan by using a partition-ing tech- than that of discovering a positive association. A negative
nique. Partitioning divides the database into no overlapping association is of the following type: “60% of customers who
partitions; these are individually considered as separate data- buy potato chips do not buy bottled water.” (Here, the 60%
bases and all large item sets for that partition are generated in refers to the confidence for the negative association rule.) In a
one pass. At the end of pass one, we thus generate a list of database with 10,000 items, there are 210000 possible combina-
large item sets from each partition. When these lists are merged, tions of items, a majority of which do not appear even once in
they contain some false positives. That is, some of the itemsets the database. If the absence of a certain item combination is
that are large in one partition may not qualify in several other taken to mean a negative association, then we potentially have
par-titions and hence may not exceed the minimum support millions and millions of negative ass0ci-ation rules with RHSs
when the original database is considered. Note that there are no that are of no interest at all. The problem, then, is to find only
false negatives, i.e., no large itemsets will be missed. The union interesting negative rules. In general, we are interested in cases in
of all large item sets identified in pass one is input to pass two which two specific sea of items appear very rarely in the same
as the candidate itemsets, and their actual support is measured transaction.
for the entire database. At the end of phase two, all actual large
This poses two problems: For a total item inventory of 10,000
itemsets are identified. Partitions are chosen in such a way that
items, the probability of any two being bough together is (1/
each partition can be accommodated in main memory and a
10,000) * (1/10,000) = 10-8. If we find the actual support for
partition is read only once in each phase. The Partition algo-
these two occurring together to be zero, that does not’ represent
rithm lends itself to parallel implementation, for efficiency.
a significant departure from expectation and hence is not an
Fur-ther improvements to this algorithm have been suggested. interesting (negative) association.
Association Rules among Hierarchies The other problem is more serious. We are looking for item
There are certain types of associations that are particularly combinations with very low support, and there are millions and
interesting for a special reason. These associations occur among millions with low or even zero support. For example, a data set
hierarchies of items. Typically, it is possible to divide items of 10 million transactions has most of the 2.5 billion pairwise
among disjoint hierarchies based on the nature of the domain. combinations of 10,000 items missing. This would generate
For example, foods in a supermarket, items in a department billions of useless rules.
store, or articles in a sports shop can be categorized into classes Therefore, to make negative association rules interesting we
and subclasses that give rise to hierarchies. Which shows the must use prior knowledge about the itemsets. One approach is
taxonomy of items in a supermarket. The figure shows two to use hierarchies. Suppose we use the hierar-chies of soft
hierarchies—beverages and desserts, respectively. The entire drinks and chips shown in Fig 27.1. A strong positive associa-
groups may not produce associations of the form beverages p tion has been shown between soft drinks and chips. If we find
Þ desserts, or dessertsÞbeverages. a large support for the fact that when customers buy pays chips
119
they predominantly buy Topsy and not Joke and not Wakeup Association rules can be generalized for data mining purposes
that would be interesting. This is so because we would Nor- although the notion of itemsets was used above to discover
mally expect that if there is a strong association between Days association rules, almost any data in the standard relational
and Topsy, there should also be such a strong association form with a number of attributes can be used. For example,
between Days and Joke or Days and Wakeup. consider blood-test data with attributes like hemoglobin, red
In the frozen yogurt and bottled water groupings in Fig 27.1, blood cell count, white blood cell count, blood--sugar, urea, age
suppose the Reduce versus Healthy brand division is 80-20 and of patient, and so on. Each of the attributes can be divided
the Plain and Clear brands division 60-40 among respective into ranges, and the presence of an attribute with a value can be
categories. This would give a joint probability of Reduce frozen considered equivalent to an item. Thus, if the hemoglobin
yogurt. attribute is divided into ranges 0-5, 6-7, 8-9, 10-12, 13-14, and
above 14, then we can. Consider them as items HI, H2... H7.
Being purchased with Plain bottled water as 48% among the
Then a specific hemoglo-bin value for a patient corresponds to
transactions containing a frozen yogurt and bottled water. If
one of these seven items being present. The mutual exclusion
this support, however, is found to be only 20%, that would
among these hemoglobin items can be used to some advantage
indicate a significant negative association among Reduce yogurt
in the scanning for large itemsets. This way of dividing variable
and Plain bottled water; again, that would be interesting.
values into ranges allows us to apply the association-rule
The problem of finding negative association is important in the machinery to any database for mining purposes. The ranges
above situations given the domain knowledge in the form of have to be determined from domain knowledge such as the
item generalization hierarchies (that is, the beverage given and relative importance of each of the hemo-globin values.
desserts hierarchies shown in Fig 27.1), the existing positive
associations (such as between the frozen-yogurt and bottled Genetic Algorithm
water group’s), and the distribution of items (such as the name Genetic algorithms (GA), first proposed by Holland in 1975,
brands within related groups). Recent work has been reported are a class of computational models that mimic natural
by the database group at Georgia Tech in this context (see evolution to solve problems in a wide variety of domains.
bibliographic notes). The Scope of dis-covery of negative Genetic algorithms are particularly suitable for solving complex
associations is limited in terms of knowing the item hierarchies optimization problems and for applications that require
and dis-tributions. Exponential growth of negative associa- adaptive problem-solving strategies. Genetic algorithms are
tions remains a challenge. search algorithms based on the mechanics of natural genetics,
i.e., operations existing in nature. They combine a Darwinian
Additional Considerations for ‘survival of the fittest approach’ with a structured, yet random-
Association Rules ized, information exchange. The advantage is that they can
For very large datasets, one way to improve efficiency is by search complex and large amount of spaces efficiently and locate
sampling. If a representative sample can be found that truly near-optimal solutions pretty rapidly. GAs was developed in the
repre-sents the properties of the original data, then most of the early 1970s by John Holland at the University of Michigan
rules can be found. The problem then reduces to one of
(Adaptation in Natural and Artificial Systems, 1975).
devising a proper sampling procedure. This process has the
potential danger of discovering some false positives (large item A genetic algorithm operates on a set of individual elements
sets that are not truly large) as well as hawing false negatives by (the population) and there is a set of biologically inspired
missing some large itemsets and corresponding association operators that can change these individuals. According to the
rules. evolutionary theory_ only the more suited individuals in the
population are likely to survive and to generate offspring, thus
transmitting their biological heredity to new generations.
In computing terms, genetic algorithms map strings of
numbers to each potential solution. Each solution becomes an
Fig: 27.1 “individual in the population, and each string becomes a
representation of an individual. There should be a way to derive
Mining association rules in real-life databases is further compli- each individual from its string representation. The genetic
cated by the following factors. algorithm then manipulates the most promising strings in its
The cardinality of itemsets in most situations is extremely large, search for an improved solution. The algorithm operates
and the volume of transactions is very high as well. Some through a simple cycle:
operational databases in retailing and commu-nication indus- • Creation of a population of strings.
tries collect tens of millions of transactions per day. • Evaluation of each string.
Transactions show variability in such factors as geographic
• Selection of the best strings.
location and seasons, making sampling difficult. Item classifica-
tions exist along multiple dimensions. Hence, driving the • Genetic manipulation to create a new population of strings.
discovery process with domain knowledge, particularly for Figure 27.2 shows how these four stages interconnect. Each
negative rules, is extremely difficult. Quality of data is variable; cycle produces a new generation of possible solutions (indi-
significant problems exist with missing, erroneous, con-flicting, viduals) for a given problem. At the first stage, a population of
as well as redundant data in many industries. possible solutions is created as a starting point. Each individual
120
in this population is enc6ded into a string (the chromosome) to operation of cutting and combining strings from a father and a

be manipulated by the genetic operators. In the next stage, the mother. An initial population of well-varied population is
individuals are evaluated, first the individual is created from its provided, and a game of evolution is played in which muta-
string description (its chromosome), then its performance in tions occur among strings. They combine to produce a new
relation to the target response is evaluated. This determines generation of individuals the fittest indi-viduals survive and
how fit this individual is in relation to the others. in the mutate until a family of successful solutions develops.
population. Based on each individual’s fitness, a selection
The solutions produced by genetic algorithms (GAs) are
mechanism chooses the best pairs for the genetic manipulation
distinguished from most other search techniques by the
process. The selection policy is responsible to ensure the
following characteristics:
survival of the fittest individuals.
• A GA search uses a set of solutions during each generation
The manipulation process enables the genetic operators to rather than a single solu-tion.
produce a new population of individuals, the offspring, by
manipulating the genetic information possessed by the pairs
• The search in the string-space represents a much larger
parallel search in the space of encoded solutions. .
chosen to reproduce. This information is stored in the strings
(Chromosomes) that describe the individuals. Two operators • The memory of the search done is represented solely by the
are used: Crossover and Mutation. The offspring generated by this set of solutions available for a generation.
process take the place of the older population and the cycle is • A genetic algorithm is a randomized algorithm since search
repeated until a desired level of fitness is attained, or a deter- mechanisms use ‘probabi-listic operators.
mined number of cycles is reached. • While progressing from one generation to next, a GA finds
near-optimal balance between knowledge acquisition and
exploitation by manipulating encoded solutions.
Genetic algorithms are used for problem solving and clustering
problems. Their ability to solve problems in parallel provides a
powerful tool for data mining. The draw-backs of GAs include
the large overproduction, of individual solutions, the random
char-acter of the searching process, and the high demand on
computer processing. In general, substantial computing power
is required to achieve anything of significance with genetic
algorithms.
Crossover
Crossover is one of the genetic operators used to recombine the
population’s genetic material. It takes two chromosomes and
swaps part of their genetic information to produce new
chromosomes. As Figure 27.3 shows, after the crossover point
has been randomly chosen, portions of the parent’s chromo-
some (strings). Parent 1 and Parent 2 are combined to produce Figure 27.3 Crossover
the new offspring, Son.
Genetic Algorithms In detail The selection process associated with the recombination made
Genetic algorithms (GAs) are a class of randomized search by the crossover, assures that special genetic structures, called
processes capable of adaptive and robust search over a wide building blocks, are retained for future generations. These
range of search space to pologies. Modeled after the adaptive building blocks represent the fittest genetic structures in the
emergence of biological species from evolutionary mechanisms, population.
and introduced by Holland GAs have been successfully applied
in such diverse. Fields such as image analysis, scheduling, and Mutation
engineering design. The mutation operator introduces new genetic structures in the
population by randomly changing some of its building blocks.
Genetic algorithms extend the idea from human genetics of the Since the modification is totally random and thus not related to
four-letter alphabet loosed on the A, C, T, G nucleotides) of any previous genetic structures present in the population, it
the human DNA code. The construction of a genetic algorithm creates different structures related to other sections of the search
involves devising an alphabet that encodes the solutions to the space. As shown in Figure 27.4, the mutation is implemented
deci-sion problem in terms of strings of that alphabet. Strings by occasionally altering a random bit from a chromosome
are equivalent to individuals. A fitness function defines which (string). The figure shows the operator being applied to the
solutions, can survive and which cannot. The ways in which fifth element of the chromosome.
solutions can be combined are patterned after the crossover
121
A number of other operators, apart from crossover and Data Mining using GA
mutation, have been introduced since the basic model was The application of the genetic algorithm in the context of data
proposed. They are usually versions of the recombination and mining is generally for the tasks of hypothesis testing and
genetic alterations processes adapted to the constraints of a refinement, where the user poses some hypothesis and the
particular problem. Examples of other operators are: inversion, system first evaluates the hypothesis and then seeks to refine it.
dominance and genetic edge recombination. Hypothesis refinement is achieved by “seeding” the system with
the hypothesis and then allowing some or all parts of it to vary.
One can use a variety of evaluation functions to determine the
fitness of a candidate refinement. The important aspect of the
GA application is the encoding of the hypothesis and the
evaluation function for fitness.
Another way to use GA for data mining is to design hybrid
techniques by blending one of the known techniques with GA.
For example, it is possible to use the genetic algorithm for
optimal decision tree induction. By randomly generating
different samples, we can build many decision trees using any
Figure 27.4 Mutation of the traditional techniques. But we are not sure of the
optimal tree. At this stage, the GA is very useful in deciding on
the optimal tree and optimal splitting attributes. The genetic
Problem-Dependent Parameters algorithm evolves a population of biases for the decision tree
This description of the GA’ s computational model reviews the induction algorithm. We can use a two-tiered search strategy. On
steps needed to create the algorithm. However, a real implemen- the bottom tier, the traditional greedy strategy is performed
tation takes into account a number of problem-dependent through the space of the decision trees. On the top tier, one can
parameters. For instance, the offspring produced by genetic have a genetic search in a space of biases. The attribute selection
manipulation (the next population to be evaluated) can either parameters are used as biases, which are used to modify the
replace the whole population (generational approach) or just its behavior of the first tier search. In other words, the GA
less fit members (steady-state approach). The problem con- controls the preference for one type of decision tree over
straints will dictate the best option. Other parameters to be another.
adjusted are the population size, crossover and mutation rates, An individual (a bit string) represents a bias and is evaluated by
evaluation method, and convergence criteria. using testing data subsets. The “fitness” of the individual is the
average cost of classification of the decision tree. In the next
Encoding
generation, the population is replaced with new individuals.
Critical to the algorithm’s performance is the choice of underly-
The new individuals are generated from the previous genera-
ing encoding for the solution of the optimization problem (the
tion, using mutation and crossover. The fittest individuals in
individuals or the population). Traditionally, binary encoding
the first generation have the most offspring in the second
has being used because they are easy to implement. The
generation. After a fixed number of generations, the algorithm
crossover and mutation operators described earlier are specific to
halts and its output is the decision tree the determined by the
binary encoding. When symbols other than 1 or 0 are used, the
fittest individual.
crossover and mutation operators must be tailored accordingly.
Exercises
The Evaluation Step
1. Write short notes on
The evaluation step in the cycle, shown in Figure 27.2, is more
closely related to the actual application the algorithm is trying to a. Mutation
optimize. It takes the strings representing the individuals of the b. Negative Associations
population and, from them, creates the actual individuals to be c. Partition algorithm
tested the way the individuals are coded as strings will depend
2. Discuss the importance of association rules.
on what parameters one is tying to optimize and the actual
structure of possible solutions (individuals). After the actual 3. Explain the basic Algorithms for Finding Association Rules.
individuals have been created, they have to be tested and scored. 4. Discuss the importance of crossover in Genetic algorithm.
These two tasks again are closely related to the actual system 5. Explain Association Rules among Hierarchies with example.
being optimized. The testing depends on what characteristics
6. Describe the principle of Genetic algorithm and discuss its
should be optimized and the scoring. The production of a
suitability to data mining.
single value representing the fitness of an individual depends
on the relative importance of each different characteristic value 7. Discuss the salient features of the genetic algorithm. How
can a data-mining problem be an optimization problem?
obtained during testing.
How do you use GA for such cases?
122
Reference

• Goldberg D.E., Genetic Algorithm in Search, Optimization and
Machine Learning, Addison-Wesley Publishing Company,
1989.
• Holland J.H. Adaptation in Natural and Artificial System (2nd
ed.), Prentice Hall, 1992.
• Marmelstein R., and Lamont G. Pattern Classification using a
Hybrid Genetic Program-Decision Tree Approach. In
Proceedings of the Jed Annual Genetic Programming Conference,
223-231, 1998.
• McCallum R., and Spackman K. Using genetic algorithms to
learn disjunctive rules from examples. In Proceedings of the
7th International Conference on Machine Learning, 149-152,
1990.
• Ryan M.D., and Rayward-Smith V.J. The evolution of
decision trees. In Proceedings of the Third Annual Genetic
Programming Conference, 350-358, 1998.
• Syswerda G. In First Workshop on the Foundations of Genetic
Algorithms and Classification Systems, Morgan Kaufmann,
1990.
Notes
123
LESSON 28 CHAPTER 6
ONLINE ANALYTICAL PROCESSING, OLAP
NEED FOR OLAP MULTIDIMENSIONAL
DATA MODEL
Structure Codd has developed rules or requirements for an OLAP

• Objective system;
• Introduction • Multidimensional conceptual view
• On-line Analytical processing • Transparency
• What is Multidimensional (MD) data and when does it • Accessibility
become OLAP? • Consistent reporting performance
• OLAP Example • Client/server architecture
• What is OLAP? • Generic dimensionality
• Who uses OLAP and WHY? • Dynamic sparse matrix handling
• Multi-Dimensional Views • Multi-user support
• Complex Calculation capabilities • Unrestricted cross dimensional operations
• Time intelligence • Intuitative data manipulation
Objective • Flexible reporting
At the end of this lesson you will be able to • Unlimited dimensions and aggregation levels
• Understand the significance of OLAP in Data mining An alternative definition of OLAP has been supplied by Nigel
• Study about Multi-Dimensional Views, Complex Calculation Pendse who unlike Codd does not mix technology prescrip-
capabilities, and Time intelligence tions with application requirements. Pendse defines OLAP as,
Fast Analysis of Shared Multidimensional Information,
Introduction which means:
This lesson focuses on the need of Online Analytical Process- Fast in that users should get a response in seconds and so
ing. Solving modern business problems such as market analysis doesn’t lose their chain of thought;
and financial forecasting requires query-centric database schemas
that are array-oriented and multidimensional in nature. These Analysis in that the system can provide analysis functions in an
business problems are characterized by the need to retrieve large intuitative manner and that the functions should supply
numbers of records from very large data sets and summarize business logic and statistical analysis relevant to the users
them on the fly. The multidimensional nature of the problems application;
it is designed to address is the key driver for OLAP. Shared from the point of view of supporting multiple users
In this lesson i will cover all the important aspects of OLAP. concurrently;
Multidimensional as a main requirement so that the system
On Line Analytical Processing
supplies a multidimensional conceptual view of the data
A major issue in information processing is how to process including support for multiple hierarchies;
larger and larger databases, containing increasingly complex data,
Information is the data and the derived information required
without sacrificing response time. The client/server architecture
by the user application.
gives organizations the opportunity to deploy specialized
servers, which are optimized for handling specific data manage- One question that arises is,
ment problems. Until recently, organizations have tried to target What is Multidimensional (MD) data and when does
relational database management systems (RDBMSs) for the it become OLAP?
complete spectrum of database applications. It is however It is essentially a way to build associations between dissimilar
apparent that there are major categories of database applications pieces of information using predefined business rules about
which are not suitably serviced by relational database systems. the information you are using. Kirk Cruikshank of Arbor
Oracle, for example, has built a totally new Media Server for Software has identified three components to OLAP, in an issue
handling multimedia applications. Sybase uses an object- of UNIX News on data warehousing;
oriented DBMS (OODBMS) in its Gain Momentum product, • A multidimensional database must be able to express
which is designed to handle complex data such as images and complex business calculations very easily. The data must be
audio. Another category of applications is that of on-line referenced and mathematics defined. In a relational system
analytical processing (OLAP). OLAP was a term coined by E F there is no relation between line items, which makes it very
Codd (1993) and was defined by him as; difficult to express business mathematics.
The dynamic synthesis, analysis and consolidation of large
volumes of multidimensional data
124
• Intuitive navigation in order to ‘roam around’ data, which • In addition to answering who and what questions OLAPs

requires mining hierarchies. can answer “what if “ and “why”.
• Instant response i.e. the need to give the user the • Thus OLAP enables strategic decision-making.
information as quickly as possible. • OLAP calculations are more complex than simply summing
Dimensional databases are not without problem as they are not data.
suited to storing all types of data such as lists for example • However, OLAP and Data Warehouses are complementary
customer addresses and purchase orders etc. Relational systems
• The data warehouse stores and manages data while the
are also superior in security, backup and replication services as
OLAP transforms this data into strategic information.
these tend not to be available at the same level in dimensional
systems. The advantages of a dimensional system are the Who uses OLAP and WHY?
freedom they offer in that the user is free to explore the data • OLAP applications are used by a variety of the functions of
and receive the type of report they want without being restricted an organisation.
to a set format. • Finance and accounting:
OLAP Example • Budgeting
An example OLAP database may be comprised of sales data
• Activity-based costing
which has been aggregated by region, product type, and sales
channel. A typical OLAP query might access a multi-gigabyte/ • Financial performance analysis
multi-year sales database in order to find all product sales in • And financial modelling
each region for each product type. After reviewing the results, an • Sales and Marketing
analyst might further refine the query to find sales volume for
• Sales analysis and forecasting
each sales channel within region/product classifications. As a
last step the analyst might want to perform year-to-year or • Market research analysis
quarter-to-quarter comparisons for each sales channel. This • Promotion analysis
whole process must be carried out on-line with rapid response • Customer analysis
time so that the analysis process is undisturbed. OLAP queries • Market and customer segmentation
can be characterized as on-line transactions which:
• Production
• Access very large amounts of data, e.g. several years of sales
data. • Production planning
• Analyze the relationships between many types of business

• Defect analysis
Thus, OLAP must provide managers with the information they
elements e.g. sales, products, regions, and channels.
need for effective decision-making. The KPI (key performance
• Involve aggregated data e.g. sales volumes, budgeted dollars indicator) of an OLAP application is to provide just-in-time
and dollars spent. (JIT) information for effective decision-making. JIT informa-
• Compare aggregated data over hierarchical time periods e.g. tion reflects complex data relationships and is calculated on the
monthly, quarterly, and yearly. fly. Such an approach is only practical if the response times are
• Present data in different perspectives e.g. sales by region vs. always short The data model must be flexible and respond to
sales by channels by product within each region. changing business requirements as needed for effective decision
• Involve complex calculations between data elements e.g. making.
expected profit as calculated as a function of sales revenue for In order to achieve this in widely divergent functional areas
each type of sales channel in a particular region. OLAP applications all require:
• Are able to respond quickly to user requests so that users can • MD views of data
pursue an analytical thought process without being stymied • Complex calculation capabilities
by the system. • Time intelligence
What is OLAP? Multi-Dimensional Views
• Relational databases are used in the areas of operations and • MD views inherently represent actual business models,
control with emphasis on transaction processing. which normally have more than three dimensions e.g., Sales
• Recently relational databases are used for building data data is looked at by product, geography, channel and time.
warehouses, which stores tactical information (< 1 year into • MD views provide the foundation for analytical processing
the future) that answers who and what questions. through flexible access to information.
• In contrast OLAP uses Multi-Dimensional (MD) views of • MD views must be able to analyse data across any dimension
aggregate data to provide access strategic information. at any level of aggregation with equal functionality and ease
• OLAP enables users to gain insight to a wide variety of and insulate users from the complex query syntax
possible views of information and transforms raw data to • What ever the query is they must have consistent response
reflect the enterprise as understood by the user e.g. Analysts, times.
managers and executives.
125
• Users queries should not be inhibited by the complex to The OLAP performance benchmark contains how time is used
form a query or receive an answer to a query. in OLAP applications. Eg the forecast calculation uses this year’s
• The benchmark for OLAP performance investigates a servers vs. last year’s knowledge, year-to-date knowledge factors.
ability to provide views based on queries of varying
complexity and scope.
• Basic aggregation on some dimensions
• More complex calculations are performed on other
dimensions
• Ratios and averages
• Variances on sceneries
• A complex model to compute forecasts
• Consistently quick response times to these queries are
imperative to establish a server’s ability to provide MD views
of information.
Complex Calculations
• The ability to perform complex calculations is a critical test
for an OLAP database.
• Complex calculations involve more than aggregation along a
hierarchy or simple data roll-ups, they also include percentage
of the total share calculations and allocations utilising
hierarchies from a top-down perspective.
• Further calculations include:
• Algebraic equations for KPI
• Trend algorithms for sales forecasting
• Modelling complex relationships to represent real
world situations
• OLAP software must provide powerful yet concise
computational methods.
• The method for implementing computational methods
must be clear and non-procedural
• Its obvious why such methods must be clear but they must
also be non-procedural otherwise changes can not be done in
a timely manner and thus eliminate access to JIT
information.
• In essence OLTP systems are judged on their ability to collect
and manage data while OLAP systems are judged on their
ability to make information from data. Such abilities involves
the use of both simple and complex calculations
Time Intelligence
• Time is important for most analytical applications and is a
unique dimension in that it is sequential in character. Thus
true OLAP systems understand the sequential nature of
time.
• The time hierarchy can be used in a different way to other
hierarchies eg sales for june or sales for the first 5 months of
2000.
• Concepts such as year to date must be easily defined
• OLAP must also understand the concept of balances over
time. Eg in some cases, for employees, an average is used
while in other cases an ending balance is used.
126
127
Exercise
• Multidimensional Views
• Time Intelligence
• Complex Calculations
2. What do you understand by Online Analytical Processing
(OLAP)? Explain the need of OLAP.
3. Who uses OLAP and why?
4. Correctly contrast the difference between OLAP and Data
warehouse.
5. Discuss various applications of OLAP.
Notes
128
LESSON 29
OLAP VS. OLTP, CHARACTERISTICS OF OLAP
Structure each table contains a customer identification number, which is

• Objective used to relate the rows from the different tables. The relation-
ships between the records are simple and only a few records are
• Definitions of OLAP
actually retrieved or updated by a single transaction.
• Comparison of OLAP and OLTP
The difference between OLAP and OLTP has been summarized
• Characteristics of OLAP: FASMI as, OLTP servers handle mission-critical production data
• Basic Features of OLAP accessed through simple queries; while OLAP servers handle
• Special features management-critical data accessed through an iterative analytical
investigation. Both OLAP and OLTP have specialized require-
Objective ments and therefore require special optimized servers for the
At the end of this lesson you will be able to two types of processing.
• Understand the significance of OLAP. OLAP database servers use multidimensional structures to
• Compare between OLAP and OLTP store data and relationships between data. Multidimensional
• Learn about various characteristics of OLAP. structures can be best visualized as cubes of data, and cubes
within cubes of data. Each side of the cube is considered a
Definitions of OLAP dimension.
In a white paper entitled ‘Providing OLAP (On-line Analytical
Processing) to User-Analysts: An IT Mandate’, E.F. Codd Each dimension represents a different category such as product
established 12 rules to define an OLAP system. In the same type, region, sales channel, and time. Each cell within the
paper he listed three characteristics of an OLAP system. Dr. multidimensional structure contains aggregated data relating
Codd later added 6 additional features of an OLAP system to elements along each of the dimensions. For example, a single
his original twelve rules. cell may contain the total sales for a given product in a region for
a specific sales channel in a single month. Multidimensional
Three significant characteristics of an OLAP system
databases are a compact and easy to understand vehicle for
• Dynamic Data Analysis visualizing and manipulating data elements that have many
This refers to time series analysis of data as opposed to static inter relationships.
data analysis, which does not allow for manipulation across OLAP database servers support common analytical operations
time. In an OLAP system historical data must be able to be including: consolidation, drill-down, and “slicing and dicing”.
manipulated over multiple data dimensions. This allows the
• Consolidation - involves the aggregation of data such as
analysts to identify trends in the business.
simple roll-ups or complex expressions involving inter-
• Four Enterprise Data Models related data. For example, sales offices can be rolled-up to
• The Categorical data model describes what has gone on districts and districts rolled-up to regions.
before by comparing historical values stored in the relational • Drill-Down - OLAP data servers can also go in the reverse
database. The Exegetical data model reflects what has direction and automatically display detail data, which
previously occurred to bring about the state, which the comprises consolidated data. This is called drill-downs.
categorical model reflects. The Contemplative data model Consolidation and drill-down are an inherent property of
supports exploration of ‘what-if ’ scenarios. The Formulaic OLAP servers.
data model indicates which values or behaviors across
• “Slicing and Dicing” - Slicing and dicing refers to the ability
multiple dimensions must be introduced into the model to
to look at the database from different viewpoints. One slice
affect a specific outcome.
of the sales database might show all sales of product type
Comparison of OLAP and OLTP within regions. Another slice might show all sales by sales
OLAP applications are quite different from On-line Transaction channel within each product type. Slicing and dicing is often
Processing (OLTP) applications, which consist of a large performed along a time axis in order to analyse trends and
number of relatively simple transactions. The transactions find patterns.
usually retrieve and update a small number of records that are OLAP servers have the means for storing multidimensional
contained in several distinct tables. The relationships between data in a compressed form. This is accomplished by dynamically
the tables are generally simple. selecting physical storage arrangements and compression
A typical customer order entry OLTP transaction might retrieve techniques that maximize space utilization. Dense data (i.e., data
all of the data relating to a specific customer and then insert a exists for a high percentage of dimension cells) are stored
new order for the customer. Information is selected from the separately from sparse data (i.e., a significant percentage of cells
customer, customer order, and detail line tables. Each row in are empty). For example, a given sales channel may only sell a
129
few products, so the cells that relate sales channels to products • Accessibility: OLAP as a mediator, Dr. Codd essentially
will be mostly empty and therefore sparse. By optimizing space describes OLAP engines as middleware, sitting
utilization, OLAP servers can minimize physical storage heterogeneous data sources & OLAP front-end.
requirements, thus making it possible to analyse exceptionally • Batch Extraction vs. Interpretive: this rule effectively requires
large amounts of data. It also makes it possible to load more that products offer both their own staging database for
data into computer memory, which helps to significantly OLAP data as well as offering live access to external data.
improve performance by minimizing physical disk I/O.
• OLAP analysis models: Dr. Codd requires that OLAP
In conclusion OLAP servers logically organize data in multiple products should support all four-analysis models that
dimensions, which allows users to quickly, and easily analyze describes in his white paper model
complex data relationships. The database itself is physically
• Client server architecture: Dr. Codd requires not only that the
organized in such a way that related data can be rapidly retrieved
product should be client/server but that the server
across multiple dimensions. OLAP servers are very efficient
component of an OLAP product should be sufficiently
when storing and processing multidimensional data. RDBMSs
intelligent that various client can be attached with minimum
have been developed and optimized to handle OLTP applica-
effort and programming for integration.
tions. Relational database designs concentrate on reliability and
transaction processing speed, instead of decision support need. • Transparency: full compliance means that a user should be
The different types of server can therefore benefit a broad range able to get full value from an OLAP engine and not even be
of data management applications. aware of where the data ultimately comes from. To do this
products must allow live excess to heterogeneous data
Characteristics of OLAP: FASMI sources from a full function spreadsheet add-in, with the
Fast – means that the system targeted to deliver most re- OLAP server in between.
sponses to user within about five second, with the simplest • Multi-user support: Dr. Codd recognizes that OLAP
analysis taking no more than one second and very few taking applications are not all read-only, & says that, to be regarded
more than 20 seconds. as strategic, OLAP tools must provide concurrent access,
Analysis – means that the system can cope with any business integrity & security.
logic and statistical analysis that it relevant for the application
Special features
and the user, keep it easy enough for the target user. Although
some pre programming may be needed we do not think it • Treatment of non-normalize data: this refers to the
acceptable if all application definitions have to be allow the user integration between an OLAP engine and denormalized
to define new adhoc calculations as part of the analysis and to source data.
report on the data in any desired way, without having to • Storing OLAP Result: keeping them separate from source
program so we exclude products (like Oracle Discoverer) that do data. This is really an implementation rather than a product
not allow the user to define new adhoc calculation as part of the issue. Dr. Codd is endorsing the widely held view that read-
analysis and to report on the data in any desired product that do write OLAP applications should not be implemented directly
not allow adequate end user oriented calculation flexibility. on live transaction data, and OLAP data changes should be
Share – means that the system implements all the security kept distinct from transaction data.
requirements for confidentiality and, if multiple write access is • Extraction of missing value: all missing value are to be
needed, concurrent update location at an appropriated level not distinguished from zero values.
all applications need users to write data back, but for the • Treatment of Missing values: all missing values to be
growing number that do, the system should be able to handle ignored by the OLAP analyzer regardless of their source.
multiple updates in a timely, secure manner.
Multidimensional – is the key requirement. OLAP system
must provide a multidimensional conceptual view of the data,
including full support for hierarchies, as this is certainly the
most logical way to analyze business and organizations.
Information – are all of the data and derived information
needed? Wherever it is and however much is relevant for the
application. We are measuring the capacity of various products
in terms of how much input data they can handle, not how
many gigabytes they take to store it.
Basic Features of OLAP
• Multidimensional Conceptual view: We believe this to be the
central core of OLAP
• Initiative data manipulation: Dr. Codd prefers data
manipulation to be done through direct action on cells in the
view w/o resource to menus of multiple actions.
130
Exercise
1. Write short notes on
• Client Server Architecture
• Slicing and Dicing
• Drill down
2. Correctly contrast and Compare OLAP and OLTP with
example.
3. What is FASMI? Explain in brief.
4. Explain various Basic Features of OLAP
5. Discuss the importance of Multidimensional View in OLAP.
Explain with an example.
131
LESSON 30
MULTIDIMENSIONAL VERSES MULTIRELATIONAL OLAP,
FEATURES OF OLAP
Structure typically require extensions to ANSI SQL rarely found in

• Objective commercial products.
• Introduction Response time and SQL functionality are not the only prob-
lems. OLAP is a continuous, iterative, and preferably interactive
• Multidimensional Data Model
process. An analyst may drill down into the data to see, for
• Multidimensional versus Multirelational OLAP example, how an individual salesperson’s per-formance affects
• OLAP Guidelines monthly revenue numbers. At the same time, the drill-down
Objective procedure may help the analyst discover certain patterns in sales
At the end of this lesson you will be able to of given products. This discovery can force another set of
questions of similar or greater complexity. Technically, all these
• Study in detail about Multidimensional Data Model analytical questions can be answered by a large number of rather
• Understand the difference between Multidimensional verses complex queries against a set of detailed and presum-marized
Multirelational OLAP. data views. In reality, however, even if the analyst could quickly
• Identify various OLAP Guidelines and accurately formulate SQL statements of this complexity, the
response time and resource consumption problems would still
Introduction
persist, and the analyst’s produc-tivity would be seriously
OLAP is an application architecture, not intrinsically a data
impacted.
warehouse or a database management system (DBMS).
Whether it uti-lizes a data warehouse or not, OLAP is becom- Multidimensional Data Model
ing an architecture that an increasing number of enterprises are The multidimensional nature of business questions is reflected
implementing to support analytical applications. The majority in the fact that, for example, marketing managers are no longer
of OLAP applications are deployed in a “stovepipe” fashion, satisfied by asking simple one-dimensional questions such as
using specialized MDDBMS technology, a narrow set of data, “How much revenue did the new product gener-ate?” Instead,
and, often, a prefabricated application-user interface. AB we look they ask questions such as “How much revenue did the new
at OLAP trends, we can see that the architectures have clearly product generate by month, in the northeastern division,
defined layers and that delin-eation exists between the applica- broken down by user demographic, by sales office, relative to
tion and the DBMS. the previous version of the product, compared the with plan?”-
Solving modern business problems such as market analysis and A six-dimensional question. One way to look at the
financial fore-casting requires query-centric database schemas that multidimensional data model is to view it as a cube (see Fig.
are array-oriented and multidimensional in nature. These 30.1). The table on the left contains detailed sales data by
business problems are characterized by the need to retrieve large product, market, and time. The cube on the right associates
numbers of records from very large data sets (hundreds of sales numbers (units sold) with dimensions-product type,
gigabytes and even terabytes) and summarize them on the fly. market, and time-with the UNIT variables organized as cells in
The multi-dimensional nature of the problems it is designed to an array. This cube can be expanded to include another array-
address is the key driver for OLAP. price-which can be associated with all or only some dimensions
The result set may look like a multidimensional spreadsheet (for example, the unit price of a product mayor may not change
(hence. the term multidimensional). Although all the necessary with time, or from city to city). The cube sup-ports matrix
data can be represented in a relational database and accessed via arithmetic that allows the cube to present the dollar sales array
SQL, the two-dimensional relational model of data and the simply by performing a single matrix operation on all cells of
Structured Query Language (SQL) have some serious limita- the array {dollar sales = units * price}.
tions for such complex real-world problems. For example, a The response time of the multidimensional query still depends
query may translate into a number of complex SQL statements, on how many cells have to be added on the fly. The caveat here
each of which may involve full table scan, multiple joins, is that, as the number of dimensions increases, the number of
aggregations and sorting, and large temporary tables for storing the cube’s cells increases exponentially. On the other hand, the
intermediate results. The resulting query may require signif-icant majority of multidimensional queries deal with sum-marized,
computing resources that may not be available at all times and high-level data. Therefore, the solution to building an efficient
even then may take a long time to complete. Another drawback multi-dimensional database is to preaggregate (consolidate) all
of SQL is its weakness in handling time series data and logical subtotals and totals along all dimensions. This
complex mathematical functions. Time series calculations such preaggregation is especially valuable since typical dimensions are
as a 3-month moving average or net present value calcula-tions hierarchical in nature. For example, the TIME dimen-sion may
contain hierarchies for years, quarters, months, weeks, and days;
132
GEOGRAPHY may contain country, state, city, etc. Having the (e.g., find account balance, find grade in course)

predefined hier-archy within dimensions allows for logical pre • Updates are frequent
aggregation and, conversely, allows for a logical drill-down-from
(e.g., concert tickets, seat reservations, shopping carts)
the product group to individual products, from annual sales to
weekly sales, and so on. 2 OLAP: On-Line Analytical Processing
Another way to reduce the size of the cube is to properly handle • Long transactions, usually complex queries
sparse data. Often, not every cell has a meaning across all (e.g., all statistics about all sales, grouped by dept and
dimensions (many marketing databases may have more than 95 month)
percent of all cells empty or containing 0). Another kind of • “Data mining” operations
sparse data is created when many cells contain duplicate data
• Infrequent updates
(Le., if the cube contains a PRICE dimension, the same price
may apply to all markets and all quarters for the year). The ability OLAP Guidelines
of a multidimensional data-base to skip empty or repetitive The data that is presented through any OLAP access route
cells can greatly reduce the size of the cube and the amount of should be identical to that used in operational systems. The
processing. values achieved through ‘drilling down’ on the OLAP side
should match the data accessed through an OLTP system.
Dimensional hierarchy, sparse data management, and pre
aggregation are the keys, since they can significantly reduce the 12 Rules satisfied by an OLAP system
size of the database and the need to calculate values. Such a 1. Multi-Dimensional Conceptual View
design obviates the need for multi table joins and provides This is a key feature of OLAP. OLAP databases should
quick and direct access to the arrays of answers, thus signifi- support multi-dimensional view of the data allowing for
cantly speeding up execution of the multidimensional queries. ‘slice and dice’ operations as well as pivoting and rotating the
cube of data. This is achieved by limiting the values of
dimensions and by changing the orders of the dimensions
when viewing the data.
2. Transparency
Users should have no need to know they are looking at an
OLAP database. The users should be focused only upon the
tool used to analysis the data, not the data storage.
3. Accessibility
OLAP engines should act like middleware, sitting between
data sources and an OLAP front end. This is usually achieved
by keeping summary data in an OLAP database and detailed
Fig :30.1
data in a relational database.
4. Consistent Reporting Performance
Multidimensional versus Multirelational Changing the number of dimensions or the number of
OLAP aggregation levels should not significantly change reporting
These relational implementations of multidimensional performance.
database systems are sometimes referred to as multirelational 5. Client-Server Architecture
database systems. To achieve the required speed, these products OLAP tools should be capable of being deployed in a client-
use the star or snowflake schemas-specially optimized and server environment. Multiple clients should be able to access
denormalized data models that involve data restructuring and the server with minimum effort.
aggregation. (The snowflake schema is an extension of the star
schema that supports multiple fact tables and joins between 6. Generic Dimensionality
them.) Each dimension must be equivalent in both its structure and
operational capabilities. Data structures, formulae, and
One benefit of the star schema approach is reduced complexity
reporting formats should not be biased toward any data
in the data model, which increases data “legibility,” making it
dimension.
eas-ier for users to pose business questions of OLAP nature.
Data warehouse queries can be answered up to 10 times faster 7. Dynamic Sparse Matrix Handling
because of improved navigations. A multi-dimensional database may have many cells that have
Two types of database activity: no appropriate data. These null values should be stored in a
way that does not adversely affect performance and
1. OLTP: On-Line Transaction Processing
minimizes space used.
• Short transactions, both queries and updates 8. Multi-User Support
(e.g., update account balance, enroll in course) OLAP applications should support concurrent access while
• Queries are simple maintaining data integrity.
133
9. Unrestricted Cross-Dimensional Operations • Shared
All forms of calculations should be allowed across all The system implements all the security requirements for
dimensions. confidentiality. Also, if multiple write access is needed, the
10.Intuitive Data Manipulation system provides concurrent update locking at an appropriate
level.
The users should be able to directly manipulate the data
without interference from the user interface. • Multidimensional
11.Flexible Reporting This is the key requirement. The system should provide a
multidimensional conceptual view of the data, including
The user should be able to retrieve any view of data required
support for hierarchies and multiple hierarchies.
and present it in any way that they require.
12.Unlimited Dimensions and Aggregation Levels
• Information
The system should be able to hold all data needed by the
There should be no limit to the number of dimensions or
applications. Data sparsity should be handled in an efficient
aggregation levels.
manner.
Six additional features of an OLAP system
1. Batch Extraction vs. Interpretive
OLAP systems should offer both their own multi-
dimensional database as well as live access to external data.
This describes a hybrid system where users can transparently
reach through to detail data.
2. OLAP Analysis Models
OLAP products should support all four data analysis models
described above (Categorical, Exegetical, Contemplative, and
Formulaic)
3. Treatment of Non-Normalized Data
OLAP systems should not allow the user to alter de-
normalized data stored in feeder systems. Another
interpretation is that the user should not be allowed to alter
data in calculated cells within the OLAP database.
4. Storing OLAP Results: keeping them separate from Source
Data
Read-write OLAP applications should not be implemented
directly on live transaction data and OLAP data changes
should be kept distinct from transaction data.
5. Extraction of Missing Values
Missing values should be treated as Null values by the OLAP
database instead of zeros.
6. Treatment of Missing Values
An OLAP analyzer should ignore missing values.
Many people take issue with the rules put forth by Dr. Codd.
Unlike his rules for relational databases, these rules are not
based upon mathematical principles. Because a software
company, Arbor Software Corporation, sponsored his paper,
some members of the OLAP community feel that his rules are
too subjective. Nigel Pendise of the OLAP Report has offered
an alternate definition of OLAP. This definition is based upon
the phrase Fast Analysis of Shared Multidimensional Informa-
tion (FASMI).
• Fast
The system should deliver most responses to users within a
few seconds. Long delays may interfere with the ad hoc
analysis.
• Analysis
The system should be able to cope with any business logic
and statistical analysis that is relevant for the application.
134
Exercise

• OLTP
• Consistent Reporting Performance
• OLAP
• FASMI
2. Illustrate with the help of a diagram the Client-Server
Architecture in brief.
3. Explain the importance of Multidimensional Data Model in
OLAP.
4. Correctly contrast the difference between Multidimensional
versus Multirelational OLAP.
5. Discuss in brief OLAP guidelines suggested by C.J.Codd.
Notes
135
LESSON 31
OLAP OPERATIONS
Structure
• Objective
cool mild hot
• Introduction
day 1 0 0 0
• OLAP Operations day 2 0 0 0
• Lattice of cubes, slice and dice operations day 3 0 0 1
• Relational representation of the data cube day 4 0 1 0
• Database management systems (DBMS), Online Analytical day 5 1 0 0
Processing (OLAP) and Data Mining day 6 0 0 0
• Example of DBMS, OLAP and Data Mining: Weather data day 7 1 0 0
day 8 0 0 0
Objective
day 9 1 0 0
day 10 0 1 0
various OLAP Operations
day 11 0 1 0
Introduction day 12 0 1 0
In today’s fast-paced, information-driven economy, organiza-
day 13 0 0 1
tions heavily rely on real-time business information to make
day 14 0 0 0
accurate decisions. The number of individuals within an
enterprise who have a need to perform more sophisticated
analysis is growing. With their ever-increasing requirements for
data manipulating tools, end users cannot be already satisfied
with flat grids and a fixed set of parameters for query execution. Lattice of Cubes, Slice and Dice
Operations
OLAP is the best technology that empowers users with
The number of dimensions defines the total number of data
complete ease in manipulating their data. The very moment you
cubes that can be created. Actually this is the number of
replace your common grid with an OLAP interface users will be
elements in the power set of the set of attributes. Generally if
able independently to perform various ad-hoc queries, arbitrarily
we have a set of N attributes, the power set of this set will
filter data, rotate a table, drill down, get desired summaries, and
have 2N elements. The elements of the power set form a
rank. From users’ standpoint, the information system equipped
lattice. This is an algebraic structure that can be generated by
with OLAP-tool gains a new quality; helps not only get
applying intersection to all subsets of the given set. It has a
information but also summarize and analyze it.
bottom element - the set itself and a top element - the empty
From the developer’s point of view, OLAP is an elegant way to set. Here is a part of the lattice of cubes for the weather data
avoid thankless and tedious programming of multiple on-line cube.
and printed reports.
OLAP Operations {}
Assume we want to change the level that we selected for the _____|______
| |
temperature hierarchy to the intermediate level (hot, mild, cool). ... {outlook} {temperature} ...
To do this we have to group columns and add up the values ___________|________
according to the concept hierarchy. This operation is called roll- | |
up, and in this particular case it produces the following cube. ... {temperature,humidity} {outlook,temperature} ...
| |
... ...
cool mild hot ...
week 1 2 1 1 |
week 2 1 3 1 {outlook,temperature,humidity,windy} {time,temperature,humidity,windy}
|____________________________________|
|
{time,outlook,temperature,humidity,windy}
In other words, climbing up the concept hierarchy produces
roll-up’s. Inversely, climbing down the concept hierarchy
expands the table and is called drill-down. For example, the
In the above terms the selection of dimensions actually means
drill down of the above data cube over the time dimension
selection of a cube, i.e. an element of the above lattice.
produces the following:
136
There are two other OLAP operations that are related to the Using this technique the whole data cube can be represented as a

selection of a cube - slice and dice. Slice performs a selection on single relational table as follows (we use higher levels in the
one dimension of the given cube, thus resulting in a subcube. concept hierarchies and omit some rows for brevity):
For example, if we make the selection (temperature=cool) we
will reduce the dimensions of the cube from two to one,
resulting in just a single column from the table above.
Time Outlook Temperature Humidity Windy Play
week 1 sunny cool normal true 0
cool week 1 sunny cool normal false 0
day 1 0 week 1 sunny cool normal ALL 0
day 2 0
week 1 sunny cool high true 0
day 3 0
week 1 sunny cool high false 0
day 4 0
week 1 sunny cool high ALL 0
day 5 1
week 1 sunny cool ALL true 0
day 6 0
week 1 sunny cool ALL false 0
day 7 1
day 8 0 week 1 sunny cool ALL ALL 0
day 9 1 week 1 sunny mild normal true 0
day 10 0 ... ... ... ... ... ...
day 11 0 week 1 overcast ALL ALL ALL 2
day 12 0 week 1 ALL ALL ALL ALL 4
day 13 0 week 2 sunny cool normal true 0
day 14 0 week 2 sunny cool normal false 1
week 2 sunny cool normal ALL 1
week 2 sunny cool high true 0
... ... ... ... ... ...
The dice operation works similarly and performs a selection on
ALL ALL ALL high ALL 3
two or more dimensions. For example, applying the selection
(time = day 3 OR time = day 4) AND (temperature = cool ALL ALL ALL ALL true 3
OR temperature = hot) to the original cube we get the ALL ALL ALL ALL false 6
following subcube (still two-dimensional): ALL ALL ALL ALL ALL 9
cool hot
day 3 0 1
day 4 0 0
The above table allows us to use an unified approach to
implement all OLAP operations - they all can me implemented
Relational Representation of the Data just by selecting proper rows. For example, the following cube,
Cube can be extracted from the table by selecting the rows that match
The use of the lattice of cubes and concept hierarchies gives us a the pattern (*, ALL, *, ALL, ALL), where * matches all legiti-
great flexibility to represent and manipulate data cubes. mate values for the corresponding dimension except for ALL.
However, a still open question is how to implement all this. An
interesting approach to this based on a simple extension of cool mild hot
standard relational representation used in DBMS is proposed week 1 2 1 1
by Jim Gray and collaborators. The basic idea is to use the value week 2 1 3 1
ALL as a legitimate value in the relational tables. Thus, ALL
will represent the set of all values aggregated over the corre-
sponding dimension. By using ALL we can also represent the
lattice of cubes, where instead of dropping a dimension when
intersecting two subsets, we will replace it with ALL. Then all
cubes will have the same number of dimensions, where their
values will be extended with the val,ue ALL. For example, a
part of the above shown lattice will now look like this:
{ALL,ALL,temperature,ALL,ALL}
__________________|_________________
| |
{ALL,ALL,temperature,humidity,ALL} {ALL,outlook,temperature,ALL,ALL}
137
Database Management Systems (DBMS), Online By querying a DBMS containing the above table we may answer
Analytical Processing (OLAP) and Data Mining questions like:

• What was the temperature in the sunny days? {85, 80, 72, 69,
75}
Area DBMS OLAP Data Mining • Which days the humidity was less than 75? {6, 7, 9, 11}
• Which days the temperature was greater than 70? {1, 2, 3, 8,
Knowledge
Extraction of 10, 11, 12, 13, 14}
Summaries, trends discovery of
Task detailed and • Which days the temperature was greater than 70 and the
and forecasts hidden patterns
summary data
and insights humidity was less than 75? The intersection of the above
Type of Insight and two: {11}
Information Analysis
result Prediction
OLAP
Deduction Induction (Build Using OLAP we can create a Multidimensional Model of our
Multidimensional data data (Data Cube). For example using the dimensions: time,
(Ask the the model, apply
Method modeling,
question, verify it to new data, outlook and play we can create the following model.
Aggregation, Statistics
with data) get the result)
9 / 5 sunny rainy overcast
Week 1 0 / 2 2 / 1 2 / 0
Who Who will buy a
What is the average Week 2 2 / 1 1 / 1 2 / 0
purchased mutual fund in
Example income of mutual
mutual funds the next 6
question fund buyers by region Obviously here time represents the days grouped in weeks
in the last 3 months and
by year? (week 1 - days 1, 2, 3, 4, 5, 6, 7; week 2 - days 8, 9, 10, 11, 12, 13,
years? why?
14) over the vertical axis. The outlook is shown along the
horizontal axis and the third dimension play is shown in each
individual cell as a pair of values corresponding to the two
values along this dimension - yes / no. Thus in the upper left
Example of DBMS, OLAP and Data Mining: Weather corner of the cube we have the total over all weeks and all
Data outlook values.
Assume we have made a record of the weather conditions
By observing the data cube we can easily identify some impor-
during a two-week period, along with the decisions of a tennis
tant properties of the data, find regularities or patterns. For
player whether or not to play tennis on each particular day. Thus
example, the third column clearly shows that if the outlook is
we have generated tuples (or examples, instances) consisting of
overcast the play attribute is always yes. This may be put as a
values of four independent variables (outlook, temperature,
rule:
humidity, windy) and one dependent variable (play). See the
textbook for a detailed description. if outlook = overcast then play = yes
We may now apply “Drill-down” to our data cube over the time
DBMS
dimension. This assumes the existence of a concept hierarchy
Consider our data stored in a relational table as follows:
for this attribute. We can show this as a horizontal tree as
follows:
Day Outlook Temperature Humidity Windy Play • time
1 Sunny 85 85 false no • week 1
2 Sunny 80 90 true no • day 1
3 overcast 83 86 false yes • day 2
4 Rainy 70 96 false yes • day 3
5 Rainy 68 80 false yes • day 4
6 Rainy 65 70 true no • day 5
7 overcast 64 65 true yes
• day 6
8 Sunny 72 95 false no
• day 7
9 Sunny 69 70 false yes
10 Rainy 75 80 false yes
• week 2
11 Sunny 75 70 true yes • day 8

12 overcast 72 90 true yes • day 9
13 overcast 81 75 false yes • day 10
14 Rainy 71 91 true no • day 11
• day 12
• day 13
138
• day 14 These rules show some attribute values sets (the so called item

• day 15 sets) that appear frequently in the data. The numbers after each
rule show the support (the number of occurrences of the item
The drill-down operation is based on climbing down the
set in the data) and the confidence (accuracy) of the rule.
concept hierarchy, so that we get the following data cube:
Interestingly, rule 3 is the same as the one that we produced by
observing the data cube.
9/5 Sunny Rainy Overcast Classification by Decision Trees and
1 0/1 0/0 0/0 Rules
2 0/1 0/0 0/0 Using the ID3 algorithm we can produce the following decision
3 0/0 0/0 1/0 tree (shown as a horizontal tree):
4 0/0 1/0 0/0 • outlook = sunny
5 0/0 1/0 0/0 • humidity = high: no
6 0/0 0/1 0/0 • humidity = normal: yes
7 0/0 0/0 1/0
• outlook = overcast: yes
8 0/1 0/0 0/0
9 1/0 0/0 0/0
• outlook = rainy
10 0/0 1/0 0/0 • windy = true: no
11 1/0 0/0 0/0 • windy = false: yes
12 0/0 0/0 1/0 The decision tree consists of decision nodes that test the values
13 0/0 0/0 1/0 of their corresponding attribute. Each value of this attribute
14 0/0 0/1 0/0 leads to a subtree and so on, until the leaves of the tree are
reached. They determine the value of the dependent variable.
Using a decision tree we can classify new tuples (not used to
generate the tree). For example, according to the above tree the
The reverse of drill-down (called roll-up) applied to this data tuple {sunny, mild, normal, false} will be classified under
cube results in the previous cube with two values (week 1 and play=yes.
week 2) along the time dimension. A decision trees can be represented as a set of rules, where each
Data Mining rule represents a path through the tree from the root to a leaf.
By applying various Data Mining techniques we can find Other Data Mining techniques can produce rules directly. For
associations and regularities in our data, extract knowledge in example the Prism algorithm available in Weka generates the
the forms of rules, decision trees etc., or just predict the value following rules.
of the dependent variable (play) in new situations (tuples). Here If outlook = overcast then yes
are some examples (all produced by Weka): If humidity = normal and windy = false then yes
Mining Association Rules If temperature = mild and humidity = normal then yes
To find associations in our data we first discretize the nu- If outlook = rainy and windy = false then yes
meric attributes (a part of the data pre-processing stage in If outlook = sunny and humidity = high then no
data mining). Thus we group the temperature values in three If outlook = rainy and windy = true then no
intervals (hot, mild, cool) and humidity values in two (high, Prediction Methods
normal) and substitute the values in data with the correspond- Data Mining offers techniques to predict the value of the
ing names. Then we apply the Apriori algorithm and get the dependent variable directly without first generating a model.
following association rules: One of the most popular approaches for this purpose is based
1. Humidity=normal windy=false 4 ==> play=yes (4, 1) of statistical methods. It uses the Bayes rule to predict the
2. Temperature=cool 4 ==> humidity=normal (4, 1) probability of each value of the dependent variable given the
values of the independent variables. For example, applying
3. Outlook=overcast 4 ==> play=yes (4, 1) Bayes to the new tuple discussed above we get:
4. Temperature=cool play=yes 3 ==> humidity=normal(3, 1) P(play=yes | outlook=sunny, temperature=mild,
5. Outlook=rainy windy=false 3 ==> play=yes (3, 1) humidity=normal, windy=false) = 0.8
6. Outlook=rainy play=yes 3 ==> windy=false (3, 1) P(play=no | outlook=sunny, temperature=mild,
7. Outlook=sunny humidity=high 3 ==> play=no (3, 1) humidity=normal, windy=false) = 0.2
8. Outlook=sunny play=no 3 ==> humidity=high (3, 1) Then obviously the predicted value must be “yes”.
9. Temperature=cool windy=false 2 ==> humidity=normal
play=yes (2, 1)
10. Temperature=cool humidity=normal windy=false 2 ==>
play=yes (2, 1)
139
Exercise
o Relational representation of the data cube
o Mining Association Rules
o Slice and dice operations
2. Explain in brief various OLAP Operations.
3. Differentiate between Database management systems
(DBMS), Online Analytical Processing (OLAP) and Data
Mining
4. Explain the difference between DBMS, OLAP and Data
Mining with related example.
Notes
140
LESSON 32
CATEGORIZATION OF OLAP TOOLS CONCEPTS USED IN MOLAP/ ROLAP
Summary (MDDBMSs)] to organize, navi-gate, and analyze data, typically

• Objective in an aggregated form, and traditionally required a tight
coupling with the application layer and presentation layer. There
• Categorization of OLAP Tools
recently has been a quick movement by MOLAP vendors to
• MOLAP segregate the OLAP through the use of published application
• ROLAP programming interfaces (APIs). Still, there remains the need to
• Managed query environment (MQE) store the data in a way similar to the way in which it will be
utilized, to enhance the performance and provide a degree of
• Cognos PowerPlay
predictability for complex analysis queries. Data structures use
• Pilot Software array technology and, in most cases, provide improved storage
• OLAP Tools and the Internet techniques to minimize the disk space requirements through
Objective sparse data management. This architecture enables excellent
The objective of this lesson is to introduce you with various performance when the data is utilized as designed, and predict-
OLAP Tools able application response times for applications address-ing a
narrow breadth of data for a specific DSS requirement. In
Categorization of OLAP Tools addition, some products treat time as a special dimension (e.g.,
On-line analytical processing (OLAP) tools are based on the Pilot Software’s Analysis Server), enhancing their ability to
concepts of multi-dimensional databases and allow a sophisti- perform time series analysis. Other products provide strong
cated user to analyze the data using elaborate, analytical capabilities (e.g., Oracle’s Express Server) built into the
multidimensional, complex views. Typical business applica- database.
tions for these tools include product performance and
Applications requiring iterative and comprehensive time series
profitability, effective-ness of a sales program or a marketing
analysis of trends are well suited for MOLAP technology (e.g.,
campaign, sales forecasting, and capacity planning. These tools
financial analysis and bud-geting). Examples include Arbor
assume that the data is organized in a multidi-mensional
Software’s Essbase, Oracle’s Express Server, Pilot Software’s
model, which is supported by a special multidimensional
Lightship Server, Sinper’s TM/l, Planning Sciences’ Gentium,
database (MDDB) or by a relational database designed to enable
multidimensional prop-erties (e.g., star schema,). A chart and Kenan Technology’s Multiway.
comparing capabilities of these two classes of OLAP tools is Several challenges face users considering the implementation of
shown in Fig. 32.1. applications with MOLAP products. First, there are limitations
in the ability of data struc-tures to support multiple subject
areas of data (a common trait of many strate-gic DSS applica-
tions) and the detail data required by many analysis applications.
This has begun to be addressed in some products, utilizing
rudi-mentary “reach through” mechanisms that enable the
MOLAP tools to access detail data maintained in an RDBMS (as
shown in Fig. 32.2). There are also limitations in the way data
can be navigated and analyzed, because the data is structured
around the navigation and analysis requirements known at the
time the data structures are built. When the navigation or
dimension requirements change, the data structures may need
to be physically reorganized to optimally support the new
requirements. This problem is similar in nature to the older
hierarchical and network DBMSs (e.g., IMS, IDMS), where
different sets of data had to be created for each application that
used the data in a manner different from the way the data was
originally maintained. Finally, MOLAP products require a
Fig 32.1
different set of skills and tools for the database administrator
to build and maintain the database, thus increasing the cost and
complexity of support.
MOLAP To address this particular issue, some vendors significantly
Traditionally, these products utilized specialized data structures enhanced their reach-through capabilities. These hybrid
[i.e., multi-dimensional database management systems solutions have as their primary char-acteristic the integration of
141
specialized multidimensional data storage with RDBMS
technology, providing users with a facility that tightly “couples”

the multidimensional data structures (MDDSs) with data
maintained in an RDBMS (see Fig. 32.2, left). This allows the
MDDSs to dynamically obtain detail data maintained in an
RDBMS, when the application reaches the bottom of the
multidimensional cells during drill-down analysis.
Fig.32.3
The ROLAP tools are undergoing some technology realign-

ment. This shift in technology emphasis is coming in two
forms. First is the movement toward pure middleware
technology that provides facilities to simplify development of
multidimensional applications. Second, there continues further
Fig. 32.2 blurring of the lines that delineate ROLAP and hybrid-OLAP
products. Vendors of ROLAP tools and RDBMS products
look to provide an option to create multi-dimensional,
This may deliver the best of both worlds, MOLAP and
persistent structures, with facilities to assist in the administra-
ROLAP. This approach can be very useful for organizations
tion of these structures. Examples include Information
with performance-sensitive multidimensional analysis require-
Advantage (Axsys), MicroStrategy (DSS AgentIDSS Server),
ments and that have built, or are in the process of building, a
Platinum/Prodea Software (Beacon), Informix/Stanford
data warehouse architecture that contains multiple subject areas.
Technology Group (Metacube), and Sybase (HighGate Project).
An example would be the creation of sales data measured by
several dimensions (e.g., product and sales region) to be stored Managed Query Environment (MQE)
and maintained in a persistent structure. This structure would This style of OLAP, which is beginning to see increased activity,
be provided to reduce the application overhead of performing provides users with the ability to perform limited analysis
calcula-tions and building aggregations during application capability, either directly against RDBMS products, or by
initialization. These struc-tures can be automatically refreshed at leveraging an intermediate MOLAP server (see Fig. 32.4). Some
predetermined intervals established by an administrator. products (e.g., Andyne’sPablo) that have a heritage in ad hoc
ROLAP query have developed features to provide “datacube” and “slice
This segment constitutes the fastest-growing style of OLAP and dice” analysis capabilities. This is achieved by first develop-
technology, with new vendors (e.g., Sagent Technology) entering ing a query to select data from the DBMS, which then delivers
the market at an accelerating pace. Products in this group have the requested data to the desktop, where it is placed into a
been engineered from the beginning to support RDBMS datacube. This datacube can be stored and maintained locally, to
products directly through a dictionary layer of metadata, reduce the overhead required to create the struCture each time
the query is executed. Once the data is in the datacube, users can
bypassing any requirement for creating a static multidimensional
perform multidimen-sional analysis (i.e., slice, dice, and pivot
data structure (see Fig. 32.3). This enables multiple multidimen- operations) against it. Alternatively, these tools can work with
sional views of the two-dimensional relational tables to be MOLAP servers, and the data from the relational DBMS can be
created without the need to structure the data around the delivered to the MOLAP server, and from there to the desktop.
desired view. Finally, some of the products in this segment have The simplicity of the installation and administration of such
developed strong SQL- generation engines to support the products makes them particularly attractive to organizations
looking to provide seasoned users with more sophisticated
complexity of multidimensional analysis. This includes the
analysis capabilities, without the significant cost and mainte-
creation of multiple SQL statements to handle user requests, nance of more complex products. With all the ease of
being “RDBMS-aware,” and providing the capability to generate installation and administration that accompanies the desktop
the SQL based on the optimizer of the DBMS engine. While OLAP products, most of these tools require the datacube to be
flexibility is an attractive feature of ROLAP products, there are built and maintained on the desktop or a sep-arate server.
products in this segment that recommend, or require, the use
of highly denormalized database designs (e.g., star schema).
142
investment in the relational database technology to provide

multidimensional access to enterprise data, at the same time
proving robustness, scalability, and administrative control.
Cognos PowerPlay is an open OLAP solution that can
interoperate with a wide variety of third-party software tools,
databases, and applications. The analytical data used by
PowerPlay is stored in multidimensional data sets called
PowerCubes. Cognos’ client/server architecture allows for the
Power-Cubes to be stored on the Cognos universal client or on
a server. PowerPlay offers a single universal client for OLAP
Fig. 32.4
servers that supports PowerCubes located locally, on the LAN,
or (optionally) inside popular relational databases. In addition
to the fast installation and deployment capabilities, PowerPlay
With metadata definitions that assist users in retrieving the cor-
pro-vides a high level of usability with a familiar Windows
rect set of data that makes up the datacube, this method causes
interface, high perfor-mance, scalability, and relatively low cost
a plethora of data redundancy and strain to most network
of ownership.
infrastructures that support many users. Although this
mechanism allows for the flexibility of each user to build a Specifically, starting with version 5, Cognos PowerPlay client
custom datacube, the lack of data consistency among users, and offers
the rel-atively small amount of data that can be efficiently • Support for enterprise-size data sets (PowerCubes) of 20+
maintained are significant challenges facing tool administrators. million records,100,000 categories, and 100 measures
Examples include Cognos Software’s PowerPlay, Andyne • A drill-through capability for queries from Cognos
Software’s Pablo, Business Objects’ Mercury Project, Dimen- Impromptu
sional Insight’s CrossTarget, and Speedware’s Media. • Powerful 3-D charting capabilities with background and
OLAP tools provide an intuitive way to view corporate data. rotation control for advanced users
These tools aggre-gate data along common business subjects or • Scatter charts that let users show data across two measures,
dimensions and then let users navigate through the hierarchies, allowing easy comparison of budget to actual values
and dimensions with the click of a mouse but-ton. Users can
• Linked displays that give users multiple views of the same
drill down, across, or up levels in each dimension or pivot and
data in a report
swap out dimensions to change their view of the data.
• Full support for OLE2 Automation, as both a client and a
Some tools, such as Arbor Software Corp.’s Essbase and
server
Oracle’s Express, pre aggregate data in special multidimensional
databases. Other tools work directly against relational data and • Formatting features for financial reports: brackets for
aggregate data on the fly, such as Micro-Strategy, Inc.’s DSS negative numbers,
Agent or Information Advantage, Inc.’s DecisionSuite. Some • Single and double underlining, and reverse sign for expenses
tools process OLAP data on the desktop instead of a server. • Faster and easier ranking of data
Desktop OLAP tools include Cognos’ PowerPlay, Brio
• A “home” button that automatically resets the dimension
Technology, Inc.’s HrioQuery, Plan-ning Sciences, Inc.’s Gen-
line to the top level
tium, and Andyne’s Pablo. Many of the differences between
OLAP tools are fading. Vendors are rearchitecting their products • Unlimited undo levels and customizable toolbars
to give users greater control over the tradeoff between flexibility • An enhanced PowerPlay Portfolio that lets users build
and perfor-mance that is inherent in OLAP tools. Many graphical, interactive, EIS-type briefing books from
vendors are rewriting pieces of their products in Java. PowerPlay reports; Impromptu reports; word processing,
Database vendors eventually might be the largest OLAP spreadsheet, or presentation documents; or any other
providers. The lead-ing database vendors plan to incorporate documents or reports
OLAP functionality in their database kernels. Oracle, Informix • A 32-bit architecture for Windows NT, Windows 95, and
Software, Inc., and-most recently-Microsoft have taken the first Windows 3.1
step toward this end by acquiring OLAP vendors (lRI Software, • Access to third-party OLAP tools including direct native
Stanford Technology Group, and Panorama, respectively.) access to Arbor’s Essbase and Oracle’s Express
Red Brick Systellls’ Red Brick Warehouse has always supported multidimensional databases
SQL exten-sions that perform simple OLAP functions, such as • PowerCube creation and access within in existing relational
rank, sorts, and moving averages. Red Brick Warehouse 5.0 also databases such as ORACLE, SYBASE, or Microsoft SQL
supports data mining algorithms. Server right inside the data warehouse.
Cognos PowerPlay • PowerCube creation scheduled for off-peak processing, or
PowerPlay from Cognos is a mature and popular software tool sequential to other processes
for multidimen-sional analysis of corporate data. PowerPlay can • Advanced security control by dimension, category, and
be characterized as an MQE tool that can leverage corporate measure-on the client, the server, or both
143
• Remote analysis where users pull subsets of information marketing professionals. The following products are at the core
from the server down to the client of Pilot Software’s offering:

• Complete integration with relational database security and Pilot Analysis Server. A full-function multidimensional
data manage-ment features database with high-speed consolidation, graphical user interface
• An open API through OLE Automation, allowing both (Pilot Model Builder), and expert-level interface. .The latest
server- and client--based PowerCubes to be accessed by version includes relational integration of the multidimensional
Visual Basic applications, spreadsheets, and other third-party model with relational data stores, thus allowing the user the
tools and applications choice between high-speed access of a multidimensional
database or on-the-fly (ROLAP) access of detail data stored
PowerPlay Administrator. As was mentioned above, Power
directly in the data ware-house or data mart.
Play’s capabilities include drill-to-detail using Impromptu. Also,
cubes can be built using data from multiple data sources. For Pilot Link. A database connectivity tool that includes ODBC
the administrators who are responsible for cre-ating multidi- connectivity and high-speed connectivity via specialized drivers
mensional cubes, new capabilities allow them to populate these to the most popular rela-tional database platforms. A graphical
PowerCubes inside popular relational databases, and to do the user interface allows the user seam-less and easy access to a wide
processing off the desktop and on UNIX servers. To provide a variety of distributed databases.
robust administration capabil-ities, Cognos offers a companion Pilot Designer. An application design environment specifically
tool-PowerPlay Administrator, which is available in Database created to enable rapid development of OLAP applications.
and Server editions. Pilot Desktop. A collection of applications that allow the end
In Power Play Administrator Database edition, the administra- user easy nav-igation and visualization of the multidimensional
tor would con-tinue to model the cube and run the population database.
of the cube process (called Transform) on the client platform. Pilot Sales and Marketing Analysis; Library. A collection of
The advantage is that data from multiple sources can now be sophisticated applications designed for the Sales and Marketing
used to generate a PointerCube for the client, and the actual business end user (includ-ing 80/20 Pareto analysis, time-based
PowerCube can be inside a relational database. This means that ranking, BCG quadrant analysis, trendline, and statistical
exist-ing database management tools and the database adminis- forecasting). The applications can be modified and tailored to
trator can be used to manage the business data, and a single meet individual needs fGr particular customers.
delivery mechanism can be employed for both application and
Pilot Discovery Server. A predictive data mining tool that
OLAP processing. A sophisticated security model is provided
embeds directly into the relational database and does not require
which in effect creates a “master” cube to service a variety of
the user to copy or trans-form the data. The data mining results
users. This is defined and controlled through the Authenticator,
are stored with metadata into the data warehouse as a predictive
also included with PowerPlay.
segmentation and are embedded into the multidimensional
The Administrator Server edition of PowerPlay lets users model as a dimension. The discovery server contains a graphical
process the popu-lation of the cube on a UNIX platform. An user interface called Pilot Discovery Server Launch, which eases
administrator uses client Trans-former to create a model, and building data mining models.
moves it to the UNIX server using the supplied software
Pilot Marketing Intelligence Library. Currently there are two
component called PowerGrid. The server Transformer, once
applications for exposing the value of the data mining results.
triggered, will create the PowerCube, and the only prerequisite is
One allows the user to graphically view the predictive segmenta-
that all data sources be accessible. Once completed, the resulting
tion and rules that describe the prediction; the other allows for
PowerCube (or PointerCube if the multidimensional database
profi1lloss and return on investment analysis for the data
is placed inside an RDBMS) is copied or trans-ferred to the
mining results.
client platform for subsequent PowerPlay analysis by the user.
The Authenticator can be used to establish user classes and Pilot Internet Publisher. A tool that easily allows users to
access security, and can also be used to redirect cube access since access their Pilot multidimensional database via browsers on the
all database passwords and locations can be known to the Internet or intranets.
Authenticator. Some of the distinguishing features of Pilot’s product offering
PowerPlay supports clients on Windows 3.1, Windows 95, and include the over-all complete solution from powerful OLAP
Windows NT. Administrator Database and Server editions engine and data mining engine to their customizable business
execute on HP/UX, IBM AIX, and Sun Solaris, and support applications. Within their OLAP offering, these are some of the
PowerCubes in ORACLE 7, SYBASE SQL Server, and key features:
Microsoft SQL Server. Time intelligence. The Pilot Analysis Server has a number of
Pilot Software features to support time as a special dimension. Among these
Pilot Software offers the Pilot Decision Support Suite of tools are the ability to process data on the fly to convert from the
from a high speed multidimensional database (MOLAP), Data native periodicity (e.g., the data was col-lected weekly) to the
Warehouse integration (ROLAP), data mining, and a diverse set periodicity preferred by the customer viewing the data (e.g., view
of customizable business applications targeted after sales and the data monthly). This feature is accomplished via special opti-
mized structures within the multidimensional database.
144
Embedded data mining. The Pilot Analysis Server is the first or a script written to a proprietary Web-server API (Le.,

product to integrate predictive data mining (as it is described in Netscape Server API, or NSAPI). The gateway submits SQL
this book) with the multidimensional database model. This queries to the database, receives the results, translates them into
allows the user to benefit from not only the predictive power of HTML, and sends the pages to the requester (see Fig. 32.5).
data mining but also the descriptive and ana-lytical power of Requests for the unstructured data (e.g., images, other HTML
multidimensional navigation. documents, etc.) can be sent directly to the unstructured data
Multidimensional database compression. In addition to the store.
compression of sparsity (the removal of cells in the multidi- The emerging third-generation Web sites replace HTML
mensional database which have no value), Pilot Analysis Server gateways with Web--based application servers. These servers can
also has special code for compressing data values over time. A download Java applets or ActiveX applications that execute on
new feature called a “dynamic dimension” allows some clients, or interact with corresponding applets running on
dimensions to be calculated on the fly when they are attributes servers-servlets. The third-generation Web servers pro-vide
of an exist-ing dimension. This allows the database to be much users with all the capabilities of existing decision-support
smaller and still provide fast access. Dynamic variables, which are applications without requiring them to load any client software
also calculated on the fly, are also available to further decrease the except a Web browser.
total size of the database and thus also decrease the time for
consolidation of the database.
Relational Integration. Pilot allows for a seamless integration
of both MOLAP and ROLAP to provide the user with either
the speed of MOLAP or the more space-efficient ROLAP. The
users interface with the system by defining the multidimen-
sional model or view that they prefer, and the sys-tem
self-optimizes the queries into precalculated MOLAP storage or
directly in the relation data store depending on the usage pattern
and the time/space tradeoff preferences of the end user.
OLAP Tools and the Internet Fig. 34.5
The two most pervasive themes in computing have been the

Internet/WWW and data warehousing. From a marketing Not surprisingly, vendors of decision support applications,
perspective, a marriage of these two giant technologies is a especially query, reporting, and OLAF tools, are rapidly convert-
natural and unavoidable event. The reason for this trend is ing their tools to work on the Web. Vendor approaches for
simple; the compelling advantages in using the Web for access deploying tools on the Web include
are _magnified even further in a data warehouse. Indeed:
HTML publishing. This approach involves transforming an
The Internet is a virtually free resource, which provides a output of a query into the HTML page that can be downloaded
universal connec-tivity within and between companies. The Web into a browser. This approach does not support interactive
eases complex administrative tasks of managing distributed access to data or reports.
envi-ronments.
Helper applications. In this approach, a tool is configured as a
The Web allows companies to store and manage both data and helper appli-cation that resides within a browser. This is the case
applications on servers that can be centrally managed, main- of a “fat” client, in which, once the data is downloaded, users
tained, and updated, thus eliminating problems with software can take advantage of all capabil-ities of the tool to analyze data.
and data currency. However, maintaining these helper applica-tions becomes
For these and other reasons, the Web is a perfect medium for another task for system administrators.
decision support. Let’s look at the general features of the Web-
enabled data access. Plug-ins. A variation on the previous approach, plug-ins are
The first-generation Web sites used a static distribution model, helper appli-cations that are downloaded from the Web server
in which clients access static HTML pages via Web browsers. In prior to their initial use. Since the plug-ins are downloaded
this model, the deci-sion support reports were stored as HTML from the server, their normal administra-tion and installation
documents and delivered to users on request. Clearly, this tasks are significantly reduced. However, typically plug-ins are
model has some serious deficiencies, including inability to browser-specific, and may not run on all platforms or with all
provide Web clients with interactive analytical capabilities such as browsers. Also, as browsers get updated, this plug-ins may have
drill-down. to be upgraded as well, creating additional administration
The second-generation Web sites support interactive database workload.
queries by uti-lizing a multitiered architecture in which a Web Server-centric components. In this approach the vendor
client submits a query in the form of HTML-encoded request rebuilds a desktop tool as a server component, or creates a new
to a Web server, which in turn transforms the request for server component that can be integrated with the Web via a
structured data into a CGI (Common Gateway Interface) script, Web gateway (e.g., CGI or NSAFI scripts).
145
Java and ActiveX applications. This approach is for a vendor
to redevelop all or portions of its tool in Java or ActiveX. The

result is a true “thin” client model. There are advantages and
disadvantages to both, but this approach appears to be one of
the most promising and flexible.
The remainder of this section looks at several OLAF tools from
a perspective of Internet/Web implementations.
146
Exercise
• Managed Query environment
• MDDB
2. Explain the following:
• ROLAP
• MOLAP
3. Discuss the difference between Relational OLAP and
Multidimensional OLAP.
4. Explain the basic architecture of OLAP with the help of a
diagram.
5. What are the various OLAP tools available? Explain any one
of them.
Notes
147
“The lesson content has been compiled from various sources in public domain including but not limited to the
internet for the convenience of the users. The university has no proprietary right on the same.”
9, Km Milestone, NH-65, Kaithal - 136027, Haryana

Website: www.niilmuniversity.in

Data Warehousing & Mining

Uploaded by

Copyright:

Available Formats

Data Warehousing & Mining

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Data Warehousing & Mining

Uploaded by

Copyright:

Available Formats

MEDIA

DATA WAREHOUSING AND DATA

Basic Concepts of Data Warehousing

Building a Data Warehouse Project

Managing and Implementing a Data Warehouse Project

Techniques of Data Mining

1. Pieter Adriaans, Dolf Zantinge Data Mining, Pearson Education

information age. employees, most organizations process an enormous number

OLAP vs. OLTP, Multidimensional data model, Multidimen-

DATA WAREHOUSING AND DATA MINING

Lesson 3 OnLine Transaction Processing 9

Lesson 8 Building Data Warehousing - 2 36

Lesson 10 Technical Consideration, Implementation Consideration 42

Structure communication requirements it is possible to incorporate

DATA WAREHOUSING AND DATA MINING

DATA WAREHOUSING AND DATA MINING

DATA WAREHOUSING AND DATA MINING

Structure Data warehousing evolved with the integration of a number of

• Data warehousing Data Warehousing

DATA WAREHOUSING AND DATA MINING

Structure Similarities and Differences in OLTP and Data

warehouse. aggregate data to provide access strategic information.

What is OLAP? Multi-Dimensional Views

DATA WAREHOUSING AND DATA MINING

Figure 3: An example of levels of summarization of data

DATA WAREHOUSING AND DATA MINING

Two common multidimensional schemas are the star schema

Structure them a better understanding of their data and a better under-

Data Warehouse Physical Architectures

DATA WAREHOUSING AND DATA MINING

DATA WAREHOUSING AND DATA MINING

DATA WAREHOUSING AND DATA MINING

In short, the separation of operational data from the analysis

provide some attributes such as social security number, address,

Enterprise Data Warehouse

Available Inventory Warehouse

may begin with one or two operational applications that Product

illustrates the extensible architecture of the data warehouse. Product Price

structure Marketing programs

A data warehouse logical model aligns with the business

DATA WAREHOUSING AND DATA MINING

Extensible data warehouse

are completely independent of each other. Available Inventory Warehouse

cantly de-normalized. De-normalization reduces the need for

Static Relationships in Historical Data Physical Transformation of Operational

DATA WAREHOUSING AND DATA MINING

Structure companies of a large corporation may have different fiscal

DATA WAREHOUSING AND DATA MINING

DATA WAREHOUSING AND DATA MINING

DATA WAREHOUSING AND DATA MINING

Structure • The amount of data is doubling every 18 months, which

DATA WAREHOUSING AND DATA MINING

Structure • The communications infrastructure that connects the

• Balanced approach • The systems management framework that enables centralized

DATA WAREHOUSING AND DATA MINING

and clean it up. • Write short notes on:

Structure at your organization is not striking it rich, you might want to

For example, the telecom industry uses data warehouses to

DATA WAREHOUSING AND DATA MINING