UNIT -2

Hadoop
Defini on: Hadoop is an open-source framework designed for storing and processing large volumes
of data in a distributed compu ng environment. It enables scalable and efficient data handling by
u lizing clusters of commodity hardware. The Hadoop framework allows parallel processing and fault
tolerance, making it a powerful tool for managing big data.

Introduc on: In today’s digital world, the volume of data generated is enormous, requiring systems
that can handle, store, and analyze large datasets efficiently. Tradi onal database management
systems (DBMS) struggle with scalability and performance when dealing with massive amounts of
data. Hadoop was developed as a solu on to these challenges, providing a distributed compu ng
model that processes large datasets across mul ple nodes simultaneously. By leveraging the Hadoop
Distributed File System (HDFS) and the MapReduce processing paradigm, Hadoop ensures high
availability, fault tolerance, and efficient handling of structured and unstructured data.

Features of Hadoop: Hadoop is known for its unique capabili es that make it an essen al tool for big
data analy cs. The key features include:

1. Scalability: Hadoop can easily scale horizontally by adding more nodes to a cluster, allowing
it to handle increasing amounts of data.

2. Fault Tolerance: It ensures data reliability by replica ng data across mul ple nodes. If a node
fails, another node takes over its tasks.

3. Distributed Storage: Hadoop uses HDFS to distribute and store data across mul ple
machines, improving accessibility and efficiency.

4. Parallel Processing: The framework processes data in parallel using the MapReduce
programming model, boos ng speed and performance.

5. Flexibility: It supports structured, semi-structured, and unstructured data, including text,

6. Cost-Effec veness: Hadoop operates on inexpensive commodity hardware, reducing

7. High Availability: Even if some nodes fail, the system con nues func oning without loss of
data.

8. Security and Authen ca on: Hadoop includes authen ca on mechanisms such as Kerberos
for secure data access and processing.

Key Advantages of Hadoop: Hadoop provides numerous benefits, making it one of the most widely
used big data technologies:

1. Efficient Handling of Large Datasets: It can process petabytes of data efficiently by

3. Support for Diverse Data Types: Hadoop can handle data from various sources, including
logs, social media, and IoT devices.

4. Integra on with Other Technologies: It works seamlessly with Spark, Hive, Pig, and other big
data tools.

5. Automa c Data Replica on: Hadoop ensures data redundancy, making it reliable in the
event of hardware failures.

Versions of Hadoop: Hadoop has evolved over me, with mul ple versions improving its efficiency
and capabili es.

 Hadoop 1.x: The first version introduced MapReduce for distributed data processing but had
limita ons in resource management.

 Hadoop 2.x: Introduced YARN (Yet Another Resource Nego ator) for be er resource
management and scalability.

 Hadoop 3.x: Brought enhancements such as erasure coding, improved memory

Overview of the Hadoop Ecosystem: The Hadoop ecosystem consists of various tools that enhance
its capabili es. Some of the major components include:

1. HDFS (Hadoop Distributed File System): The storage layer that distributes data across
clusters.

2. MapReduce: A programming model for parallel processing of large datasets.

3. YARN (Yet Another Resource Nego ator): Manages compu ng resources in Hadoop clusters.

4. Hive: A data warehouse infrastructure that supports SQL-like queries for Hadoop data.

5. Pig: A scrip ng language used for processing large datasets.

6. HBase: A NoSQL database that provides real- me read/write access to large data.

7. Sqoop: A tool for transferring data between Hadoop and rela onal databases.

8. Flume: Used for collec ng and aggrega ng log data.

9. Oozie: A workflow scheduler for managing Hadoop jobs.

10. Spark: A fast and general-purpose engine for large-scale data processing.
Hadoop Distribu ons: Various companies provide Hadoop distribu ons with addi onal features and
enterprise support:

 Apache Hadoop: The official open-source version maintained by the Apache So ware
Founda on.

 Cloudera Distribu on (CDH): Includes enterprise security, real- me analy cs, and a user-
friendly interface.

 Hortonworks Data Pla orm (HDP): Provides a fully open-source Hadoop ecosystem with
enhanced security.

 MapR Hadoop Distribu on: Offers real- me processing, distributed file systems, and NoSQL
database integra on.

Need for Hadoop: Hadoop is essen al due to the increasing volume, velocity, and variety of data
being generated. Businesses require efficient solu ons to process, analyze, and derive insights from
large datasets. Tradi onal databases fail to scale efficiently, whereas Hadoop provides an open-
source, cost-effec ve, and scalable solu on for managing big data.

RDBMS vs. Hadoop: The following table highlights the key differences between tradi onal Rela onal
Database Management Systems (RDBMS) and Hadoop:

Feature RDBMS Hadoop

Data Structure Structured (Tables) Structured, Semi-structured, Unstructured

Processing Transac onal (OLTP) Batch Processing (MapReduce)

Schema Fixed Schema Schema-on-Read

Scalability Ver cal (Limited) Horizontal (Unlimited)

Cost Expensive Cost-effec ve (Commodity Hardware)

Fault Tolerance Low High (Data Replica on)

Querying SQL NoSQL (Hive, Pig, Spark)

Speed Fast for Small Data Op mized for Large Data

1. Data Consistency: Ensuring all nodes have synchronized data.

2. Fault Tolerance Management: Handling failures while maintaining system integrity.

3. Network Latency: Communica on delays between nodes can impact performance.

4. Load Balancing: Distribu ng data and computa on evenly across nodes.

5. Security Risks: Implemen ng strong authen ca on and encryp on to prevent data

History of Hadoop: Hadoop was inspired by Google's MapReduce and Google File System (GFS). It
was developed by Doug Cu ng and Mike Cafarella in 2006 under the Apache So ware Founda on.
Since its incep on, Hadoop has evolved into a comprehensive ecosystem that powers modern big
data applica ons.

HDFS (Hadoop Distributed File System): HDFS is the core storage component of Hadoop, designed to
store and manage vast amounts of data efficiently. It follows a Master-Slave Architecture:

1. NameNode (Master): Manages metadata and file system namespace.

2. DataNodes (Slaves): Store actual data in blocks and perform read/write opera ons.

HDFS provides fault tolerance, scalability, and high throughput, making it an integral part of the
Hadoop ecosystem.