4220 6 (DataFormat)

RDD, pandas dataframe, numpy
Instructor: Li Yang
RDD (resilient distributed dataset) or Dataframe ?
• Dataframe
• New trend in spark
• MLlib is on a shift to dataframe based API
• Spark streaming is moving towards structured streaming that is heavily based on dataframe API
• RDD
• Out of date? No!
• RDD is the basic building block of spark
• Dataframe is build on top of RDD

When to use RDD or Dataframe ?
• RDD
• Low-level transformation, action and control on datasets
• Unstructured data, i.e., media streams or streams of text
• need to manipulate data with complicated functional programming
• Dataframe
• Simple processing, i.e., sum, average, SQL queries
• High-level optimization, i.e., MLlib

RDD (resilient distributed datasets)
• RDD examples: {‘a’,’b’,’c’,’d’}, {0,-1.5,2.5,3}, {(‘a’,1), (‘c’,-2),(‘d’,10)}
• RDD can contain any types of objects, including user-defined one
• RDD is a capsulation of a large dataset. In Spark all the work is expressed

as manipulations on RDD (create a new RDD; transform a RDD; call
operations on RDD)
• A special RDD is pair RDD. It has a key-value pair: (key, value)
• For example: {(‘Lily’, (18,1.65,100)), (‘John’, (16,1.85,150)), (‘Ann’,

(22,1.55,90))}. Here name is key, (age, height, weight) is value.
Remark: spark distributes the data contained in RDD across clusters

Operations on RDD
• Transformations
• Apply some functions to data in RDD to create a new RDD
• Actions
• Compute a result based on RDD

Create RDD
• Create RDD directly from external dataset file
• (python code): sc = SparkContext(“local”, “texfile”)
rdd_new = sc.textFile(“/mnt/S3data/sample.text”)
• Create RDD from existing Dataframe
• (python code) rdd_new = df.rdd
• Conversion from rdd to dataframe: rdd_new.toDF()

Transforms on RDD
• Transformations
• Apply some functions to data in RDD to create a new RDD
• Two common transformations on single RDD (for example, our RDD = {(‘Lily’, (18,1.65,100)), (‘John’, (16,1.85,150)), (‘Ann’, (22,1.55,90))}):
• Filter : select some data in RDD to create a new RDD (similar to select in SQL)
• rdd.filter(lambda x: x[0]==`John’) : choose John’s data
• rdd.filter(lambda x: x[1][0]<=20) : choose students’ data whose age <=20
• Map: apply a function to each data in RDD (very useful)
• rdd.map(lambda x: (x[0], x[1][1]*0.393)) : a new RDD = {(‘Lily’, 64.9), (‘John’, 72.8), (‘Ann’, 61)}, name+height in inches
• Transformations on two RDD:
• union, intersection
• Transformations on two pair RDD:
• join: merge two RDD by key values to generate a new RDD

Actions on RDD
• Action
• return a final value based on RDD
• Several common actions on single RDD:
• collect: return a list of all the data in RDD (shouldn’t be used for large dataset)
• take(n) : return the n rows of data in RDD
• count : return the number of rows
• reduce : apply a function to any two elements of RDD to create a new element and
continue until there is only one element in RDD
Python Function
• Definition 1:
• (python): def fun_name(input) :
return output
• Example: def Addition(x) :
return x+2
• Defintion 2:
• (python): lambda input: output
• Example: lambda x: x+2

Pandas Dataframe
• Pandas dataframe is a 2-dimensional labeled data structure with columns

of potentially different types
• Difference between dataframe and pandas dataframe?
• Pandas dataframe is build on top of dataframe. It is a higher level.
• Pandas dataframe is easier to be manipulated
• Dataframe → pandas dataframe: df.toPandas()
• Pandas dataframe → dataframe: spark.createDataFrame(df)

Pandas Dataframe
• Call pandas module (python): import pandas as pd
• Construct pandas dataframe (python): pd_new = df.toPandas()
• Display pandas dataframe (python): pd.head(2), df.tail(2)
• Basic manipulations of pandas dataframe (python): pd.columns, pd.describe()
• Slicing dataframe:
• Columns(python): pd[[‘column1’, ‘column2’]]
• Rows(python): pd[0:3]
• Accessing dataframe:
• (python): pd.loc[0,’column’]
• (python): pd.iloc[0,1]
Pandas Dataframe
• Apply function on columns (python): pd[‘colum1’,’colum2’].apply(fun)
• Merging two pandas dataframes (python): pd.merge(pd1, pd2, on=‘column’)
• Concatenate two pandas dataframe (python): pd.concat([pd1,pd2])
• Other basic manipulations of pandas dataframe (python):
• pd.dropna(axis=0, inplace=True)
• pd.drop_duplicates(subset=‘column’, inplace=True)
Numpy
• A numpy array is a grid of values, all of the same type
• A simpler data format of dataframe. Operations on numpy are similar to Matlab matrix operation
• Call a numpy (python): import numpy as np
• Construct a numpy (python):
• np.array([[1,2,3],[4,5,6]])
• np1 = pd.to_numpy(), (conversion from pandas dataframe)
• np.zeros((2,2)), np.ones((2,2))
• np.arange(0,3,0.1)
• Access data (python): np1[0,1], np1[0,:], np1[:,0]
• Matrix operations (elementwise) (python): np1+np2, np1-np2, np1*np2, np1/np2
• Real Matrix operations (python): np1.dot(np2), np1.T
• Conversion between pandas dataframe and numpy (python):
• Pandas → numpy: pd.to_numpy()
• numpy → pandas: pd.Dataframe(N1) (here N1 is a numpy)

Comparison data formats
Dataframe RDD Pandas Dataframe Numpy
Advantages Easy I/O; easy to transfer Very powerful to Easy to access and Easy to work on matrix
to other data formats ; easy manipulate the data manipulate data manipulations
for sql query
Disadvantage Hard to access and Hard to master it since you Something wrong with I/O Not fit for string type data
manipulate the data need to be very familiar on databricks
with it
• My suggestion:
1. Construct dataframe from dataset; Transform it to pandas dataframe for further operations
2. If you need simple matrix operations, transfer it to numpy; if you need complicated data operations, transfer it to rdd
Other modules
• re: module (package) to work on strings
• Call re (python): import re
• Substitute a string in a text(python): re.sub(‘<br>’, text)
• Search a string in a text(python): re.search(‘<br>’, text)
• matplotlib: plotting module similar to Matlab plot
• Call it (python): import matplotlib.pyplot as plt
• Plot a function with x as horizontal axis and y as vertical axis: plt.plot(x,y)
• imageio: basic operations on images
• Call it (python): from imageio import imread, imsave
• Read an image (python): img = imread(‘input image file.jpeg’)
• Show an image (python): plt.imshow(img)
• Save an array as image (python): imsave(‘output image file.jpeg’, img)

4220 6 (DataFormat)

Uploaded by

Copyright:

Available Formats

4220 6 (DataFormat)

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

4220 6 (DataFormat)

Uploaded by

Copyright:

Available Formats

RDD, pandas dataframe, numpy

• New trend in spark

• MLlib is on a shift to dataframe based API

• Out of date? No!

• RDD is the basic building block of spark

• Dataframe is build on top of RDD

• Low-level transformation, action and control on datasets

• Unstructured data, i.e., media streams or streams of text

• need to manipulate data with complicated functional programming

• Simple processing, i.e., sum, average, SQL queries

• High-level optimization, i.e., MLlib

• RDD examples: {‘a’,’b’,’c’,’d’}, {0,-1.5,2.5,3}, {(‘a’,1), (‘c’,-2),(‘d’,10)}

• RDD can contain any types of objects, including user-defined one

• RDD is a capsulation of a large dataset. In Spark all the work is expressed

• A special RDD is pair RDD. It has a key-value pair: (key, value)

• For example: {(‘Lily’, (18,1.65,100)), (‘John’, (16,1.85,150)), (‘Ann’,

Remark: spark distributes the data contained in RDD across clusters

• Apply some functions to data in RDD to create a new RDD

• Compute a result based on RDD

• Create RDD directly from external dataset file

• (python code): sc = SparkContext(“local”, “texfile”)

• Create RDD from existing Dataframe

• (python code) rdd_new = df.rdd

• Conversion from rdd to dataframe: rdd_new.toDF()

• Apply some functions to data in RDD to create a new RDD

• rdd.filter(lambda x: x[0]==`John’) : choose John’s data

• rdd.filter(lambda x: x[1][0]<=20) : choose students’ data whose age <=20

• Map: apply a function to each data in RDD (very useful)

• Transformations on two RDD:

• Transformations on two pair RDD:

• join: merge two RDD by key values to generate a new RDD

• return a final value based on RDD

• Several common actions on single RDD:

• take(n) : return the n rows of data in RDD

• count : return the number of rows

• (python): def fun_name(input) :

• Example: def Addition(x) :

• (python): lambda input: output

• Example: lambda x: x+2

• Pandas dataframe is a 2-dimensional labeled data structure with columns

• Difference between dataframe and pandas dataframe?

• Pandas dataframe is build on top of dataframe. It is a higher level.

• Pandas dataframe is easier to be manipulated

• Dataframe → pandas dataframe: df.toPandas()

• Pandas dataframe → dataframe: spark.createDataFrame(df)

• Construct pandas dataframe (python): pd_new = df.toPandas()

• Display pandas dataframe (python): pd.head(2), df.tail(2)

• Basic manipulations of pandas dataframe (python): pd.columns, pd.describe()

• Columns(python): pd[[‘column1’, ‘column2’]]

• Apply function on columns (python): pd[‘colum1’,’colum2’].apply(fun)

• Merging two pandas dataframes (python): pd.merge(pd1, pd2, on=‘column’)

• Concatenate two pandas dataframe (python): pd.concat([pd1,pd2])

• Other basic manipulations of pandas dataframe (python):

• Call a numpy (python): import numpy as np

• Construct a numpy (python):

• np1 = pd.to_numpy(), (conversion from pandas dataframe)

• Access data (python): np1[0,1], np1[0,:], np1[:,0]

• Matrix operations (elementwise) (python): np1+np2, np1-np2, np1*np2, np1/np2