Scribd Logo
Upload

Welcome to Scribd!

  • 201 views

    Hadoop Python MapReduce Tutorial For Beginners

    Uploaded by

    Ahmed Mohamed
    The document provides a tutorial on using Python for MapReduce jobs on Hadoop. It introduces the Hadoop streaming library, shows how to write a simple word count MapReduce program in Python, and a more complex example that counts stadiums by their playing surface type from input data. The tutorial explains key aspects of running Python MapReduce jobs on Hadoop using the streaming library.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd

    Hadoop Python MapReduce Tutorial For Beginners

    Uploaded by

    Ahmed Mohamed
    201 views15 pages
    The document provides a tutorial on using Python for MapReduce jobs on Hadoop. It introduces the Hadoop streaming library, shows how to write a simple word count MapReduce program in Python, and a more complex example that counts stadiums by their playing surface type from input data. The tutorial explains key aspects of running Python MapReduce jobs on Hadoop using the streaming library.

    Original Title

    Hadoop Python MapReduce Tutorial for Beginners

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    The document provides a tutorial on using Python for MapReduce jobs on Hadoop. It introduces the Hadoop streaming library, shows how to write a simple word count MapReduce program in Python, and a more complex example that counts stadiums by their playing surface type from input data. The tutorial explains key aspects of running Python MapReduce jobs on Hadoop using the streaming library.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    201 views15 pages

    Hadoop Python MapReduce Tutorial For Beginners

    Uploaded by

    Ahmed Mohamed
    The document provides a tutorial on using Python for MapReduce jobs on Hadoop. It introduces the Hadoop streaming library, shows how to write a simple word count MapReduce program in Python, and a more complex example that counts stadiums by their playing surface type from input data. The tutorial explains key aspects of running Python MapReduce jobs on Hadoop using the streaming library.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    of 15

    Home (/) About Me (/about.

    html) Archive (/archive)

    Hire Me  (https://rathbonelabs.com)

    Beekeeper Studio  (https://beekeeperstudio.io)

    MATTHEW RATHBONE (/)

    Topics (/archive/topics.html) / Hadoop (/archive/topics.html#hadoop) /


    Hadoop Python MapReduce Tutorial for Beginners

    Hadoop Python MapReduce Tutorial for Beginners

    By Matthew Rathbone on November 17 2013

     Share (http://www.facebook.com/sharer/sharer.php?
    u=https://blog.matthewrathbone.com/2013/11/17/python-map-reduce-on-hadoop-a-
    beginners-tutorial.html)

     Tweet (https://twitter.com/intent/tweet?text=Hadoop Python MapReduce Tutorial for


    Beginners&url=https://blog.matthewrathbone.com/2013/11/17/python-map-reduce-on-
    hadoop-a-beginners-tutorial.html&via=rathboma)

     Post (http://www.linkedin.com/shareArticle?
    mini=true&url=https://blog.matthewrathbone.com/2013/11/17/python-map-reduce-on-
    hadoop-a-beginners-tutorial.html&title=Hadoop Python MapReduce Tutorial for
    Beginners&summary=A step-by-step tutorial for writing your rst map reduce with Python
    and Hadoop Streaming.&source=rathboma)
    This article originally accompanied my tutorial session at the Big
    Data Madison Meetup, November 2013
    (http://www.meetup.com/BigDataMadison/events/149122882/).

    The goal of this article is to:

    introduce you to the hadoop streaming library (the mechanism


    which allows us to run non-jvm code on hadoop)
    teach you how to write a simple map reduce pipeline in Python
    (single input, single output).
    teach you how to write a more complex pipeline in Python
    (multiple inputs, single output).
    There are other good resouces online about Hadoop streaming, so
    I’m going over old ground a little. Here are some good links:

    1. Hadoop Streaming of cial Documentation


    (http://hadoop.apache.org/docs/r1.1.2/streaming.html)
    2. Michael Knoll’s Python Streaming Tutorial (http://www.michael-
    noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-
    python/)
    3. An Amazon EMR Python streaming tutorial
    (http://aws.amazon.com/articles/2294)
    If you are new to Hadoop, you might want to check out my beginners
    guide to Hadoop (/2013/04/17/what-is-hadoop.html) before digging
    in to any code (it’s a quick read I promise!).
    Setup

    I’m going to use the Cloudera Quickstart VM


    (http://www.cloudera.com/content/support/en/downloads/download-
    components/download-products.html?productID=F6mO278Rvo) to
    run these examples.

    Once you’re booted into the quickstart VM we’re going to get our
    dataset. I’m going to use the play-by-play n data by Brian Burke
    (http://www.advancedn stats.com/2010/04/play-by-play-data.html).
    To start with we’re only going to use the data in his Git repository
    (https://github.com/eljefe6a/n data).

    Once you’re in the cloudera VM, clone the repo:

    cd ~/workspace
    git clone https://github.com/eljefe6a/nfldata.git

    To start we’re going to use stadiums.csv . However this data was


    encoded in Windows (grr) so has ^M line separators instead of new
    lines \n . We need to change the encoding before we can play with it:

    cd workspace/nfldata
    cat stadiums.csv # BAH! Everything is a single line
    dos2unix -l -n stadiums.csv unixstadiums.csv
    cat unixstadiums.csv # Hooray! One stadium per line

    Hadoop Streaming Intro

    The way you ordinarily run a map-reduce is to write a java program


    with at least three parts.
    1. A Main method which con gures the job, and lauches it
    set # reducers
    set mapper and reducer classes
    set partitioner
    set other hadoop con gurations

    2. A Mapper Class
    takes K,V inputs, writes K,V outputs

    3. A Reducer Class
    takes K, Iterator[V] inputs, and writes K,V outputs

    Hadoop Streaming is actually just a java library that implements


    these things, but instead of actually doing anything, it pipes data to
    scripts. By doing so, it provides an API for other languages:

    read from STDIN


    write to STDOUT
    Streaming has some (con gurable) conventions that allow it to
    understand the data returned. Most importantly, it assumes that
    Keys and Values are separated by a \t . This is important for the rest
    of the map reduce pipeline to work properly (partitioning and
    sorting). To understand why check out my intro to Hadoop
    (/2013/04/17/what-is-hadoop.html), where I discuss the pipeline in
    detail.

    RUNNING A BASIC STREAMING JOB

    It’s just like running a normal mapreduce job, except that you need to
    provide some information about what scripts you want to use.

    Hadoop comes with the streaming jar in it’s lib directory, so just nd
    that to use it. The job below counts the number of lines in our
    stadiums le. (This is really overkill, because there are only 32
    records)

    hadoop fs -mkdir nfldata/stadiums


    hadoop fs -put ~/workspace/nfldata/unixstadiums.csv nfldata/stad

    hadoop jar /usr/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop


    -Dmapred.reduce.tasks=1 \
    -input nfldata/stadiums \
    -output nfldata/output1 \
    -mapper cat \
    -reducer "wc -l"

    # now we check our results:


    hadoop fs -ls nfldata/output1

    # looks like files are there, lets get the result:


    hadoop fs -text nfldata/output1/part*
    # => 32

    A good way to make sure your job has run properly is to look at the
    jobtracker dashboard. In the quickstart VM there is a link in the
    bookmarks bar.

    You should see your job in the running/completed sections, clicking


    on it brings up a bunch of information. The most useful data on this
    page is under the Map-Reduce Framework section, in particular look for
    stuff like:

    Map Input Records


    Map Output Records
    Reduce Output Records
    In our example, input records are 32 and output records is 1:
    A Simple Example in Python

    Looking in columns.txt we can see that the stadium le has the


    following elds:

    Stadium (String) - The name of the stadium


    Capacity (Int) - The capacity of the stadium
    ExpandedCapacity (Int) - The expanded capacity of the stadium
    Location (String) - The location of the stadium
    PlayingSurface (String) - The type of grass, etc that the stadium
    IsArtificial (Boolean) - Is the playing surface artificial
    Team (String) - The name of the team that plays at the stadium
    Opened (Int) - The year the stadium opened
    WeatherStation (String) - The name of the weather station closest
    RoofType (Possible Values:None,Retractable,Dome) - The type of roo
    Elevation - The elevation of the stadium

    Lets use map reduce to nd the number of stadiums with arti cial
    and natrual playing surfaces.

    The pseudo-code looks like this:


    def map(line):
    fields = line.split(",")
    print(fields.isArtificial, 1)

    def reduce(isArtificial, totals):


    print(isArtificial, sum(totals))

    You can nd the nished code in my Hadoop framework examples


    repository (https://github.com/rathboma/hadoop-framework-
    examples).

    IMPORTANT GOTCHA!

    The reducer interface for streaming is actually different than in Java.


    Instead of receiving reduce(k, Iterator[V]) , your script is actually sent
    one line per value, including the key.

    So for example, instead of receiving:

    reduce('TRUE', Iterator(1, 1, 1, 1))


    reduce('FALSE', Iterator(1, 1, 1))

    It will receive:

    TRUE 1
    TRUE 1
    TRUE 1
    TRUE 1
    FALSE 1
    FALSE 1
    FALSE 1

    This means you have to do a little state tracking in your reducer. This
    will be demonstrated in the code below.
    To follow along, check out my git repository
    (https://github.com/rathboma/hadoop-framework-examples) (on the
    virtual machine):

    cd ~/workspace
    git clone https://github.com/rathboma/hadoop-framework-examples.g
    cd hadoop-framework-examples

    MAPPER

    import sys

    for line in sys.stdin:


    line = line.strip()
    unpacked = line.split(",")
    stadium, capacity, expanded, location, surface, turf, team, op
    results = [turf, "1"]
    print("\t".join(results))

    REDUCER
    import sys

    # Example input (ordered by key)


    # FALSE 1
    # FALSE 1
    # TRUE 1
    # TRUE 1
    # UNKNOWN 1
    # UNKNOWN 1

    # keys come grouped together


    # so we need to keep track of state a little bit
    # thus when the key changes (turf), we need to reset
    # our counter, and write out the count we've accumulated

    last_turf = None
    turf_count = 0

    for line in sys.stdin:

    line = line.strip()
    turf, count = line.split("\t")

    count = int(count)
    # if this is the first iteration
    if not last_turf:
    last_turf = turf

    # if they're the same, log it


    if turf == last_turf:
    turf_count += count
    else:
    # state change (previous line was k=x, this line is k=y)
    result = [last_turf, turf_count]
    print("\t".join(str(v) for v in result))
    last_turf = turf
    turf_count = 1

    # this is to catch the final counts after all records have been re
    print("\t".join(str(v) for v in [last_turf, turf_count]))

    You might notice that the reducer is signi cantly more complex then
    the pseudocode. That is because the streaming interface is limited
    and cannot really provide a way to implement the standard API.
    As noted, each line read contains both the KEY and the VALUE , so it’s up
    to our reducer to keep track of Key changes and act accordingly.

    Don’t forget to make your scripts executable:

    chmod +x simple/mapper.py
    chmod +x simple/reducer.py

    TESTING

    Because our example is so simple, we can actually test it without


    using hadoop at all.

    cd streaming-python
    cat ~/workspace/nfldata/unixstadiums.csv | simple/mapper.py | sort
    # FALSE 15
    # TRUE 17

    Looking good so far!

    Running with Hadoop should produce the same output.

    hadoop jar /usr/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop


    -mapper mapper.py \
    -reducer reducer.py \
    -input nfldata/stadiums \
    -output nfldata/pythonoutput \
    -file simple/mapper.py \
    -file simple/reducer.py
    # ...twiddle thumbs for a while

    hadoop fs -text nfldata/pythonoutput/part-*


    FALSE 15
    TRUE 17
    A Complex Example in Python

    Check out my advanced python MapReduce guide


    (/hadoop/2016/02/09/python-tutorial.html) to see how to join two
    datasets together using python.

    Python MapReduce Book

    While there are no books speci c to Python MapReduce


    development the following book has some pretty good examples:

    (http://amzn.to/2hVekf0)

    MASTERING PYTHON FOR DATA SCIENCE

    (HTTP://AMZN.TO/2HVEKF0)

    While not speci c to MapReduce, this book gives some examples of


    using the Python 'HadoopPy' framework to write some MapReduce
    code. It's also an excellent book in it's own right.

     Share (http://www.facebook.com/sharer/sharer.php?
    u=https://blog.matthewrathbone.com/2013/11/17/python-map-reduce-on-hadoop-a-
    beginners-tutorial.html)

     Tweet (https://twitter.com/intent/tweet?text=Hadoop Python MapReduce Tutorial for


    Beginners&url=https://blog.matthewrathbone.com/2013/11/17/python-map-reduce-on-
    hadoop-a-beginners-tutorial.html&via=rathboma)

     Post (http://www.linkedin.com/shareArticle?
    mini=true&url=https://blog.matthewrathbone.com/2013/11/17/python-map-reduce-on-
    hadoop-a-beginners-tutorial.html&title=Hadoop Python MapReduce Tutorial for
    Beginners&summary=A step-by-step tutorial for writing your rst map reduce with Python
    and Hadoop Streaming.&source=rathboma)

    Matthew Rathbone
    Consultant Big Data Infrastructure Engineer at Rathbone Labs
    (https://www.rathbonelabs.com). British. Data Nerd. Lucky husband and
    father.

    (https://click.linksynergy.com/deeplink?

    id=KuenSouhzgQ&mid=40328&murl=https%3A%2F%2Fwww.courser
    data)
    PREVIOUS

    « Hadoop MapReduce Scoobi Tutorial with Examples (/2013/11/03/real-world-


    hadoop-implementing-a-left-outer-join-with-scoobi.html)

    NEXT

    Reading data from HDFS programatically using java (and scala) »


    (/2013/12/28/Reading-data-from-HDFS-even-if-it-is-compressed.html)

    Related Hadoop Articles


    Should you use Parquet? (/2019/12/20/parquet-or-bust.html)

    Beginners Guide to Columnar File Formats in Spark and Hadoop


    (/2019/11/21/guide-to-columnar- le-formats.html)

    Join the discussion


    Show Comments

    Join my newsletter
    Learn more about Hadoop and Big Data.

    email address

    Subscribe
    Beekeeper Studio
    I maintain an open source SQL editor and database manager with a focus on usability.
    It is cross-platform and really nice to use.

    Check it Out (https://beekeeperstudio.io)

    Related Articles
    Should you use Parquet? (/2019/12/20/parquet-or-bust.html)

    Beginners Guide to Columnar File Formats in Spark and Hadoop


    (/2019/11/21/guide-to-columnar- le-formats.html)
    A Quick Guide to Concurrency in Scala (/2017/03/28/scala-concurrency-
    options.html)

    4 Fun and Useful Things to Know about Scala's apply() functions (/2017/03/06/scala-
    object-apply-functions.html)

    10+ Great Books and Resources for Learning and Perfecting Scala
    (/2017/02/14/scala-books.html)

    Links
    Twitter (https://twitter.com/rathboma)
    GitHub (https://github.com/rathboma)
    Beekeeper Studio (https://beekeeperstudio.io)

    Resources
    Hadoop HDFS Cheatsheet (/pages/hdfs-cheatsheet.html)

    Copyright Matthew Rathbone 2020, All Rights Reserved. Background image from Subtle Patterns
    (http://subtlepatterns.com/)

    You might also like

    pFad - Phonifier reborn

    Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

    Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


    Alternative Proxies:

    Alternative Proxy

    pFad Proxy

    pFad v3 Proxy

    pFad v4 Proxy