04/05/2023, 10:44 pandas - Jupyter Notebook

pandas
pandas stands for panel data and is the core library for data manipulation,data analysis.

it consists of single and multidimentional ds for data manipulation.

pandas is a python library used for working with data sets.

high performence data analysis tool

working with large data set

represents in tabular way

working on missing data

three ds in pands

1. series- one dimensional

2. dataframe-two dimentional
3. panel- multidimentional (data,major axis,minor axis)

create Pandas Series

In [1]:

import pandas as pd
import numpy as np

In [2]:

arr = np.array([1,2,3,4])
print(arr)

[1 2 3 4]

In [3]:

s = pd.Series(arr)
print(s)
print(type(s))

0 1
1 2
2 3
3 4
dtype: int64
<class 'pandas.core.series.Series'>

localhost:8888/notebooks/anaconda3/Python/pandas.ipynb 1/10
04/05/2023, 10:44 pandas - Jupyter Notebook

In [4]:

print(s[0:5])

0 1
1 2
2 3
3 4
dtype: int64

In [5]:

a[2]

--------------------------------------------------------------------
-------
NameError Traceback (most recent cal
l last)
/tmp/ipykernel_8943/4164697690.py in <module>
----> 1 a[2]

NameError: name 'a' is not defined

In [ ]:

a = pd.Series(['a','b','c'])

In [ ]:

In [ ]:

a = pd.date_range(start = '2023-03-01', end = '2023-03-28')

In [ ]:

In [ ]:

type(a)

Pandas dataframe
In [ ]:

arr = np.array([[1,2,3],[4,5,6]])
print(arr)

localhost:8888/notebooks/anaconda3/Python/pandas.ipynb 2/10
04/05/2023, 10:44 pandas - Jupyter Notebook

In [ ]:

df = pd.DataFrame(arr)
print(df)

In [ ]:

temp = np.random.randint(low = 20, high =100, size = [20,])

name = np.random.choice(['Abhay','Teclov','Geekshub','Ankit'],20)
random = np.random.choice([10,11,13,12,14],20)

In [ ]:

df = pd.DataFrame({"Temp":temp,"Name":name,"Random":random})
df

In [ ]:

a = list(zip(temp, name, random))

print(a)

In [ ]:

df = pd.DataFrame(data = a, columns=['Temp','Name','Random'])

In [ ]:

df

In [ ]:

type(df)

In [ ]:

temp = np.random.randint(low = 20, high =100, size = [20,])

name = np.random.choice(['Abhay','Teclov','Geekshub','Ankit'],20)
random = np.random.choice([10,11,13,12,14],20)

In [ ]:

df = pd.DataFrame({'temp':temp, 'name':name, 'random':random})

In [ ]:

type(df)

In [ ]:

df.head()

In [ ]:

df.tail()

localhost:8888/notebooks/anaconda3/Python/pandas.ipynb 3/10
04/05/2023, 10:44 pandas - Jupyter Notebook

In [ ]:

df.shape

In [ ]:

df.columns

In [ ]:

df.name

In [ ]:

df['name']

In [ ]:

df['temp'].describe()

In [ ]:

df.info()

In [ ]:

df.values

In [ ]:

df

In [ ]:

df.set_index('temp', inplace = True)

In [ ]:

df

In [ ]:

df.sort_index(axis =0, ascending=False)

In [ ]:

df.sort_values(by ='random', ascending = False)

In [ ]:

df.drop(['random'], axis =1)

localhost:8888/notebooks/anaconda3/Python/pandas.ipynb 4/10
04/05/2023, 10:44 pandas - Jupyter Notebook

In [ ]:

df.head()

In [ ]:

df.iloc[[0,1]]

In [ ]:

df.iloc[1:3,1]

In [ ]:

df.iloc[[True,True,False]]

In [ ]:

df.head()

In [ ]:

df.loc[:,:]

In [ ]:

df.loc[[39,84,34]]

In [ ]:

df.loc[[39,84],'name':'random']

In [ ]:

df.loc[[True, True, False, True]]

In [ ]:

df.loc[df.random > 13]

In [ ]:

df.loc[(df.random > 13) | (df.random == 10),:]

In [ ]:

# Merging & concat

d1 = pd.DataFrame([['a', 1], ['b', 2]],columns=['col1', 'number'])
d2 = pd.DataFrame([['c', 3, 'lion'], ['d', 4, 'tiger']],columns=['letter', 'numbe

In [ ]:

d1

localhost:8888/notebooks/anaconda3/Python/pandas.ipynb 5/10
04/05/2023, 10:44 pandas - Jupyter Notebook

In [ ]:

d2

In [ ]:

pd.concat([d1,d2],axis =0)

In [ ]:

pd.concat([d1,d2], axis =0, ignore_index=True)

In [ ]:

pd.concat([d1,d2], axis = 1)

In [ ]:

d1 = pd.DataFrame({
"city" : ["lucknow","kanpur","agra","delhi"],
"temperature" : [32,45,30,40]
})

In [ ]:

d1

In [ ]:

d2 = pd.DataFrame({
"city" : ["delhi","lucknow","kanpur"],
"humidity" : [68,65,75]
})

In [ ]:

d2

In [ ]:

df = pd.merge(d1,d2, on='city')

In [ ]:

df

In [ ]:

pd.merge(d1,d2, on=['city'], how ='outer')

In [ ]:

pd.merge(d1, d2, on =['city'], how='left')

localhost:8888/notebooks/anaconda3/Python/pandas.ipynb 6/10
04/05/2023, 10:44 pandas - Jupyter Notebook

In [ ]:

# dataset from https://github.com/codebasics/py/blob/master/pandas/6_handling_miss

In [ ]:

df1 = pd.read_csv("weather_data.csv")

In [ ]:

df1

In [ ]:

# pip3 install openpyxl

df1.to_excel('df_xl.xlsx', sheet_name = 'weather_data')

In [ ]:

# pip3 install xlrd

df2 = pd.read_excel('df_xl.xlsx')

In [ ]:

df2

In [ ]:

df2.to_csv('file.csv')

In [ ]:

df2.to_csv('file_noindex.csv', index = False)

In [ ]:

df_group = df2.groupby("event")

In [ ]:

df_group

In [ ]:

for temperature in df_group:

print(temperature)

In [ ]:

df_group.get_group('Rain')

In [ ]:

df_group.describe()

localhost:8888/notebooks/anaconda3/Python/pandas.ipynb 7/10
04/05/2023, 10:44 pandas - Jupyter Notebook

In [ ]:

def hot_temp(x):
return x > 30

In [ ]:

df2['hot_temp'] = df2['temperature'].apply(hot_temp)

In [ ]:

df2

In [ ]:

df2['hot_temp'] = df2['temperature'].apply(lambda x: x > 30)

In [ ]:

df2

In [ ]:

#pivot table

In [ ]:

df2.pivot_table(values = 'temperature', index = 'event', aggfunc = 'mean')

In [ ]:

df2.pivot_table(columns = 'temperature')

In [ ]:

help(pd.DataFrame.pivot_table)

In [ ]:

df3.to_csv("/home/apiiit-rkv/Desktop/dsp unit-3")

In [ ]:

import pandas as pd

In [ ]:

d=pd.read_excel("//home//apiiit-rkv//Desktop//marks.xlsx")
df=pd.DataFrame(d)
df

localhost:8888/notebooks/anaconda3/Python/pandas.ipynb 8/10
04/05/2023, 10:44 pandas - Jupyter Notebook

In [ ]:

#correlation
Correlation coefficients quantify the association between variables or features o
These statistics are of high importance for science and technology, and Python ha
tools that you can use to calculate them. SciPy, NumPy, and pandas correlation met
fast, comprehensive, and well-documented.

What Pearson, Spearman, and Kendall correlation coefficients are

How to use SciPy, NumPy, and pandas correlation functions
How to visualize data, regression lines, and correlation matrices with Matplot

1. Negative correlation (red dots): In the plot on the left, the y values tend
as the x values increase. This shows strong negative correlation, which o
large values of one feature correspond to small values of the other, and v

2.Weak or no correlation (green dots): The plot in the middle shows no obv
trend. This is a form of weak correlation, which occurs when an assoc
between two features is not obvious or is hardly observable.

Positive correlation (blue dots): In the plot on the right, the y val
to increase as the x values increase. This illustrates strong pos
correlation, which occurs when large values of one feature corres
large values of the other, and vice versa.

In [ ]:

import pandas as pd
x = pd.Series(range(10, 20))
x

In [ ]:

y = pd.Series([2, 1, 4, 5, 8, 12, 18, 25, 96, 48])

y

In [ ]:

x.corr(y) # Pearson's r

In [ ]:

y.corr(x)

In [ ]:

x.corr(y, method='spearman') # Spearman's rh

In [ ]:

x.corr(y, method='kendall')

In [ ]:

localhost:8888/notebooks/anaconda3/Python/pandas.ipynb 9/10
04/05/2023, 10:44 pandas - Jupyter Notebook

localhost:8888/notebooks/anaconda3/Python/pandas.ipynb 10/10