Everand Logo

Welcome to Everand!

  • Discover millions of ebooks, audiobooks, and so much more with a free trial

    From $11.99/month after trial. Cancel anytime.

    Building Python Real-Time Applications with Storm
    Building Python Real-Time Applications with Storm
    Building Python Real-Time Applications with Storm
    Ebook230 pages2 hours

    Building Python Real-Time Applications with Storm

    By Bhatnagar Kartik and Barry Hart

    ()

    Read preview

    About this ebook

    Learn to process massive real-time data streams using Storm and Python—no Java required!

    About This Book

    - Learn to use Apache Storm and the Python Petrel library to build distributed applications that process large streams of data
    - Explore sample applications in real-time and analyze them in the popular NoSQL databases MongoDB and Redis
    - Discover how to apply software development best practices to improve performance, productivity, and quality in your Storm projects

    Who This Book Is For

    This book is intended for Python developers who want to benefit from Storm’s real-time data processing capabilities. If you are new to Python, you’ll benefit from the attention to key supporting tools and techniques such as automated testing, virtual environments, and logging. If you’re an experienced Python developer, you’ll appreciate the thorough and detailed examples

    What You Will Learn

    - Install Storm and learn about the prerequisites
    - Get to know the components of a Storm topology and how to control the flow of data between them
    - Ingest Twitter data directly into Storm
    - Use Storm with MongoDB and Redis
    - Build topologies and run them in Storm
    - Use an interactive graphical debugger to debug your topology as it’s running in Storm
    - Test your topology components outside of Storm
    - Configure your topology using YAML

    In Detail

    Big data is a trending concept that everyone wants to learn about. With its ability to process all kinds of data in real time, Storm is an important addition to your big data “bag of tricks.”
    At the same time, Python is one of the fastest-growing programming languages today. It has become a top choice for both data science and everyday application development. Together, Storm and Python enable you to build and deploy real-time big data applications quickly and easily.
    You will begin with some basic command tutorials to set up storm and learn about its configurations in detail. You will then go through the requirement scenarios to create a Storm cluster. Next, you’ll be provided with an overview of Petrel, followed by an example of Twitter topology and persistence using Redis and MongoDB. Finally, you will build a production-quality Storm topology using development best practices.

    Style and approach

    This book takes an easy-to-follow and a practical approach to help you understand all the concepts related to Storm and Python.
    LanguageEnglish
    PublisherPackt Publishing
    Release dateDec 2, 2015
    ISBN9781784392871
    Building Python Real-Time Applications with Storm

    Related to Building Python Real-Time Applications with Storm

    Related ebooks

    Programming For You

    View More
    View More

    Related podcast episodes

    Related articles

    • UNLIMITED

      DJANGO Create A Database-driven Website

      Jun 4, 2019

      The Django web framework was named after the famous guitarist Django Reinhardt and was first created by web developers at a small newspaper in Kansas. The main goals of Django is to enable fast development of complex websites with database needs. It

      7 min read

    • UNLIMITED

      Usability

      Oct 19, 2021

      Usability
      3 min read

    • UNLIMITED

      Metrics & Visuals In Go

      Nov 17, 2020

      Mihalis Tsoukalos is a DataOps engineer and a technical writer. He’s the author of Go Systems Programming and Mastering Go, 2nd edition. The subject of this tutorial is two-fold. First, it’s about creating a Go application that exports metrics to P

      7 min read

    • UNLIMITED

      Access Your Mac Anywhere

      Nov 8, 2022

      Access Your Mac Anywhere
      2 min read

    • UNLIMITED

      What’s A Switch?

      Jan 11, 2021

      What’s A Switch?
      1 min read

    • UNLIMITED

      Why Python?

      Apr 7, 2020

      Python is an interpreted, high-level, general-purpose programming language that was first released in 1991 by its creator, Guido van Rossum. Very similar in programming construct to how BASIC (Beginners All-purpose Sybollic Instruction Code) was used

      1 min read

    • UNLIMITED

      Mucking About With AI

      May 22, 2023

      Mucking About With AI
      2 min read

    • UNLIMITED

      Basic Concepts

      Jul 2, 2019

      A messaging system such as Kafka enables you to send messages between processes, applications and servers. Applications connect to Kafka to send or get data. Strictly speaking, a Kafka ‘topic’ is a unit of storage in Kafka: data in Kafka is stored in

      1 min read

    • UNLIMITED

      Create A RESTful Server In Go

      Oct 19, 2021

      Create A RESTful Server In Go
      8 min read

    • UNLIMITED

      In Brief

      Jun 1, 2021

      Mu is a code editor for many forms of Python. We can write standard Python 3 code, create web apps and write code for microcontrollers such as the new Raspberry Pi Pico. Mu is designed for new users and does away with complicated IDEs in favour of a

      1 min read

    • UNLIMITED

      Common Errors

      Aug 27, 2019

      If you receive a ‘Script not found’ error, this probably means that you don’t have the mod scripts installed in your Minecraft directory. Check that you’ve replaced .minecraft with the one from McPiFoMo; this should include mcpipy, which will be full

      1 min read

    • UNLIMITED

      LXF’s NEW $HOME

      Jun 1, 2021

      LXF’s NEW $HOME
      7 min read

    • UNLIMITED

      Use EBPF To Keep Tabs On Your CPU

      Oct 18, 2022

      Did you miss part one? Get hold of it on page 60 Mihalis Tsoukalos is a systems engineer and a technical writer. You can reach him at @mactsouk. We’re continuing our dive into the notoriously complex Extended Berkeley Packet Filter (eBPF) feature of

      9 min read

    • UNLIMITED

      Grafana, Telegraf And Influxdb

      Jun 30, 2020

      If you don’t like Netdata or if you want to try something else, you can give Grafana (https://grafana.com), Telegraf (www.influxdata.com/time-series-platform/telegraf) and InfluxDB (www.influxdata.com/products/influxdb-overview) a try. Grafana can’t

      1 min read

    • UNLIMITED

      An Introduction To Rabbitmq

      Jun 29, 2021

      RabbitMQ is a Message Broker, which means that it can safely hold messages generated by applications and make them available to other applications. The main advantages are reliability, support for clustering and high-availability queues, tracing capa

      1 min read

    • UNLIMITED

      Perl at 34

      Feb 8, 2022

      Perl at 34
      7 min read

    • UNLIMITED

      How To Develop A RESTful Client In Go

      Nov 16, 2021

      Mihalis Tsoukalos is a systems engineer and technical writer. He’s the author of Go Systems Programming and Mastering Go. You can reach him at @mactsouk. The subject of this month’s tutorial is RESTful services. In particular, you’re going to learn h

      9 min read

    Related categories

    Reviews for Building Python Real-Time Applications with Storm

    0 ratings

    0 ratings0 reviews

    What did you think?

    Tap to rate

    Review must be at least 10 words

      Book preview

      Building Python Real-Time Applications with Storm - Bhatnagar Kartik

      Table of Contents

      Building Python Real-Time Applications with Storm

      Credits

      About the Authors

      About the Reviewers

      www.PacktPub.com

      Support files, eBooks, discount offers, and more

      Why subscribe?

      Free access for Packt account holders

      Preface

      What this book covers

      What you need for this book

      Who this book is for

      Conventions

      Reader feedback

      Customer support

      Downloading the example code

      Errata

      Piracy

      Questions

      1. Getting Acquainted with Storm

      Overview of Storm

      Before the Storm era

      Key features of Storm

      Storm cluster modes

      Developer mode

      Single-machine Storm cluster

      Multimachine Storm cluster

      The Storm client

      Prerequisites for a Storm installation

      Zookeeper installation

      Storm installation

      Enabling native (Netty only) dependency

      Netty configuration

      Starting daemons

      Playing with optional configurations

      Summary

      2. The Storm Anatomy

      Storm processes

      Supervisor

      Zookeeper

      The Storm UI

      Storm-topology-specific terminologies

      The worker process, executor, and task

      Worker processes

      Executors

      Tasks

      Interprocess communication

      A physical view of a Storm cluster

      Stream grouping

      Fault tolerance in Storm

      Guaranteed tuple processing in Storm

      XOR magic in acking

      Tuning parallelism in Storm – scaling a distributed computation

      Summary

      3. Introducing Petrel

      What is Petrel?

      Building a topology

      Packaging a topology

      Logging events and errors

      Managing third-party dependencies

      Installing Petrel

      Creating your first topology

      Sentence spout

      Splitter bolt

      Word Counting Bolt

      Defining a topology

      Running the topology

      Troubleshooting

      Productivity tips with Petrel

      Improving startup performance

      Enabling and using logging

      Automatic logging of fatal errors

      Summary

      4. Example Topology – Twitter

      Twitter analysis

      Twitter's Streaming API

      Creating a Twitter app to use the Streaming API

      The topology configuration file

      The Twitter stream spout

      Splitter bolt

      Rolling word count bolt

      The intermediate rankings bolt

      The total rankings bolt

      Defining the topology

      Running the topology

      Summary

      5. Persistence Using Redis and MongoDB

      Finding the top n ranked topics using Redis

      The topology configuration file – the Redis case

      Rolling word count bolt – the Redis case

      Total rankings bolt – the Redis case

      Defining the topology – the Redis case

      Running the topology – the Redis case

      Finding the hourly count of tweets by city name using MongoDB

      Defining the topology – the MongoDB case

      Running the topology – the MongoDB case

      Summary

      6. Petrel in Practice

      Testing a bolt

      Example – testing SplitSentenceBolt

      Example – testing SplitSentenceBolt with WordCountBolt

      Debugging

      Installing Winpdb

      Add Winpdb breakpoint

      Launching and attaching the debugger

      Profiling your topology's performance

      Split sentence bolt log

      Word count bolt log

      Summary

      A. Managing Storm Using Supervisord

      Storm administration over a cluster

      Introducing supervisord

      Supervisord components

      Supervisord installation

      Configuration of supervisord.conf

      Configuration of supervisord.conf on 172-31-19-62

      Summary

      Index

      Building Python Real-Time Applications with Storm

      Building Python Real-Time Applications with Storm

      Copyright © 2015 Packt Publishing

      All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

      Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

      Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

      First published: November 2015

      Production reference: 1261115

      Published by Packt Publishing Ltd.

      Livery Place

      35 Livery Street

      Birmingham B3 2PB, UK.

      ISBN 978-1-78439-285-7

      www.packtpub.com

      Credits

      Authors

      Kartik Bhatnagar

      Barry Hart

      Reviewers

      Oscar Campos

      Pavan Narayanan

      Commissioning Editor

      Usha Iyer

      Acquisition Editor

      Larissa Pinto

      Content Development Editor

      Anish Sukumaran

      Technical Editor

      Tanmayee Patil

      Copy Editor

      Vikrant Phadke

      Project Coordinator

      Izzat Contractor

      Proofreader

      Safis Editing

      Indexer

      Rekha Nair

      Production Coordinator

      Aparna Bhagat

      Cover Work

      Aparna Bhagat

      About the Authors

      Kartik Bhatnagar loves nature and likes to visit picturesque places. He is a technical architect in the big data analytics unit of Infosys. He is passionate about new technologies. He is leading the development work of Apache Storm and MarkLogic NoSQL for a leading bank. Kartik has a total 10 years of experience in software development for Fortune 500 companies in many countries. His expertise also includes the full Amazon Web Services (AWS) stack and modern open source libraries. He is active on the StackOverflow platform and is always eager to help young developers with new technologies. Kartik has also worked as a reviewer of a book called Elasticsearch Blueprints, Packt Publishing. In the future, he wants to work on predictive analytics.

      Barry Hart began using Storm in 2012 at AirSage. He quickly saw the potential of Storm while suffering from the limitations of the basic storm.py that it provides. In response, he developed Petrel, the first open source library for developing Storm applications in pure Python. He also contributed some bug fixes to the core Storm project.

      When it comes to development, Barry has worked on a little of everything: Windows printer drivers, logistics planning frameworks, OLAP engines for the retail industry, database engines, and big data workflows.

      Barry is currently an architect and senior Python/C++ developer at Pindrop Security, helping fight phone fraud in banking, insurance, investment, and other industries.

      I want to thank my wonderful wife, Beth, for all her love and support. I would also like to thank my two little boys, who keep me young and make every day special.

      About the Reviewers

      Oscar Campos has been working with Python since early 2007. He is the author of the famous Anaconda Python IDE package for Sublime Text 3, available as free software at http://github.com/DamnWidget/anaconda.

      He currently works as a senior software engineer on EXADS, programming high-concurrency backend system applications in Golang.

      Oscar has also reviewed PySide GUI Application Development, Packt Publishing.

      I want to thank my wife, Lydia, for all her support in every aspect of my life—without you, nothing could be possible.

      Pavan Narayanan is a blogger at DataScience Hacks (https://datasciencehacks.wordpress.com), experienced in developing mathematical programming and data analytics solutions. He has utilized Apache Storm for developing real-time analytics prototype and his interests are exploring problem solving techniques, from industrial mathematics to machine learning. He can be reached at .

      Pavan has also reviewed Apache Mahout Essentials, Learning Apache Mahout Classification, and Mastering Machine Learning with R, all by Packt Publishing.

      I would like to thank my family and God almighty for all the strength and endurance, and the folks at Packt Publishing for the opportunity to work on this book.

      www.PacktPub.com

      Support files, eBooks, discount offers, and more

      For support files and downloads related to your book, please visit www.PacktPub.com.

      Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and

      Enjoying the preview?
      Page 1 of 1
      pFad - Phonifier reborn

      Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

      Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


      Alternative Proxies:

      Alternative Proxy

      pFad Proxy

      pFad v3 Proxy

      pFad v4 Proxy