Scribd Logo
Upload

Welcome to Scribd!

  • 2 views

    Lambda Archi

    Uploaded by

    Arul John Bosco Susairaj

    Copyright:

    © All Rights Reserved
    Download as TXT, PDF, TXT or read online from Scribd

    Lambda Archi

    Uploaded by

    Arul John Bosco Susairaj
    2 views2 pages

    Copyright

    © © All Rights Reserved

    Available Formats

    TXT, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Copyright:

    © All Rights Reserved
    Download as TXT, PDF, TXT or read online from Scribd
    Download as txt, pdf, or txt
    2 views2 pages

    Lambda Archi

    Uploaded by

    Arul John Bosco Susairaj

    Copyright:

    © All Rights Reserved
    Download as TXT, PDF, TXT or read online from Scribd
    Download as txt, pdf, or txt
    of 2

    What is Lambda Architecture?

    Lambda architecture is a data processing architecture designed to handle massive


    amounts of data efficiently and in a fault-tolerant manner. It achieves this by
    combining batch processing for historical data and stream processing for real-time
    data, offering the best of both worlds: low-latency data processing and accurate
    analytics.

    Key Components of Lambda Architecture


    The architecture is typically divided into three layers:

    Batch Layer

    Stores the raw, immutable data (e.g., in a Data Lake or distributed file system
    like Hadoop).
    Processes the data in bulk at regular intervals using batch jobs.
    Produces a batch view, which contains precomputed results for accurate querying.
    Tools: Hadoop, Apache Spark, Azure Data Lake, etc.
    Speed Layer

    Processes data in real-time as it arrives (e.g., events, transactions).


    Provides low-latency, approximate results immediately.
    Complements the batch layer by covering only the most recent data.
    Tools: Apache Kafka, Apache Flink, Azure Event Hub, etc.
    Serving Layer

    Combines the batch and real-time outputs to provide a unified, queryable view of
    the data.
    Delivers results to end-users or applications via APIs or dashboards.
    Tools: Databases (e.g., Cassandra, Elasticsearch), Power BI, etc.
    How it Works:
    Data Ingestion: Raw data flows into both the batch and speed layers simultaneously.
    Processing:
    The batch layer processes the entire dataset at regular intervals to ensure
    accuracy.
    The speed layer processes incoming data in real-time for low-latency responses.
    Serving:
    The serving layer combines outputs from both layers, prioritizing real-time data
    for immediacy but relying on the batch layer for historical and accurate results.
    Example: Social Media Analytics
    Imagine a social media platform tracking user interactions like likes, shares, and
    comments.

    Batch Layer:
    Historical data of all user interactions is stored in a data lake and processed
    nightly to generate accurate metrics like monthly active users (MAU) or engagement
    trends.

    Speed Layer:
    Real-time interactions are processed as they happen to display the latest trending
    topics or live user counts.

    Serving Layer:
    A dashboard shows a combination of real-time stats (current active users, live
    trends) and historical data (engagement over the last month).

    Underlying Architecture
    Data Sources: Events, logs, sensors, transactions, etc.
    Ingestion Layer: Tools like Apache Kafka, Azure Event Hubs, or Amazon Kinesis bring
    data into the system.
    Batch Layer Storage: Data is stored in distributed file systems (HDFS, Azure Data
    Lake) for processing.
    Batch Layer Processing: Engines like Apache Spark or Hadoop process the data in
    large-scale jobs.
    Stream Layer Processing: Stream processing tools (Flink, Storm) handle real-time
    events.
    Serving Layer: Combines and serves data using databases or visualization tools
    (e.g., Power BI, Tableau).
    Benefits of Lambda Architecture
    Scalability: Handles vast amounts of data.
    Fault Tolerance: Each layer ensures resilience in case of failures.
    Flexibility: Can process both real-time and historical data.
    Limitations
    Complexity: Maintaining separate batch and speed layers requires more effort.
    Data Duplication: Raw data is processed in both layers, leading to redundancy.
    Latency in Batch Layer: Accurate batch results are delayed until the job completes.
    Would you like to explore a practical implementation of Lambda Architecture?

    You might also like

    pFad - Phonifier reborn

    Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

    Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


    Alternative Proxies:

    Alternative Proxy

    pFad Proxy

    pFad v3 Proxy

    pFad v4 Proxy