Ram ---10 Years

eclasess----

SQL Server
MSBI
Power Bi
Azure Data engineering

-----------------------------------------
Azure Developer
Java,.Net...APP Development

Azure Data Enginerr---store,process and manage---basic SQL

Azure Admin
Azure Devops

---------------------
what is cloud
what is on premises
what is microsot azure
what are different deployment mdes
what are different cateories services
what advantages of coud

6 months-----
2000 GB---after 1 month 500 More Gb
SQL Server---1 Lakh

IAAS---Infra structer related service---

networking,hard,memory,cpu,ram,security.........
PAAS----platform AS A SERVICE ---operating system ,sql server
SAAS----SOFTWARE AS A SERVICE---gmail,offce365.com(outlook,skype)

AZURE DATA ENGINEER

--------------------
AZURE STORAGE ACCOUNT
AZURE SQL
AZURE SYNAPSE DW
AZURE ANALYSIS SERVICES
AZURE DATA LAKE

AZURE DATABRICKS
AZURE DATA FACTORY----70%

7997457228

ONLINE----OFFLINE
CLASS NOTE ---RECORDING

TWO OFFICE

40 hOURS---DAILY 1 HOUR---mONDAY---SATRDAY---EVERY SESSION IS PRACTICAL

TRAINING---PLACEMENT---WORKSUPPORT

27-jULY-2021

CERTIFICTION-DP-203

---------------------------------
Name
Email
Watsapp Number

--------------------
Class Notes
Class recording video Access
Assignement Document
-----------------------------
7 am IST to 8 AM IST

Monday to Satrday

Practical Session

Softwares ---Required we will Provide

Credit Card ---nothing

Windows Operating System 10---
Email---Microsoft Email

-----------------------------------------------------------------------------

Azure Storage Account

Azure SQL
Azure Synapse dw-----second priority
Azure Data Lake
Azure Analysis Services
Azure Data Bricks
Azure Data Factory-----high priority
Azure cosmose DB

JJJJAAA
-----------------------------------------------------------------------------
what is one premises
what is Cloud computing
What are different kind of categories of services
IAAS---INFRA STRUCTER AS A SERVICE---NETWORK,SECURITY,HARD,CPU ,MEMORY...........
PAAS---PLATFORM AS A SERVICE---OPERATING SYSTEM,SQL SERVER
SAAS---SOFTWARE AS A SERVICE---OFFICE365.COM

what are different ways of cloud deployment

PUBLIC
PRIVATE
HYBRID

what are advanatges of cloud computing

what are different cloud providers

Microsoft Azure---www.portal.azure.com
Amazon--AWS--Amazon Web Services
Google--GCP--Google Cloud provider

what is microsoft azure

what is Azure Data Engineer roles and responsbilites
what are pre requisites to learn Azure Data Engineer

500 GB---1000 GB more

sql server 2019------2020 version

Intial capital Investment

Manitaince cost
Imporving Development Time

Azure Storage Account

Azure SQL
Azure Synapse dw-----second priority
Azure Data Lake
Azure Analysis Services
Azure cosmose DB

Azure Data Engineerring-dp-203

Azure Data Bricks

Azure Data Factory-----high priority

www.portal.azure.com

Subscription----charge amount---for services

Email---Gmail,yahooh,oraganization

onmicrosoft.com

Credit Card----Free subscription--28 days dont charge anything---after 28 days

account will be disabled---upgrad paid account
2 or 3 services---30 min or 1 hour after one need to dleete
resources----10 to 20 rupees

Free Subscription
-----------------------------------------------------------------------------------
-------------
100 + Services
Azure Data Engineering---15+ Services

25+ Services Life Long Free

25+ more services 1 Year Free
50+ needs to pay

Free subscription---14500 rupees--with 28 days you need to consume these amount

for above specified 50 + services

with in 10 days i have spent all amount what ever given 14500---immidiuatly 11th
day account will be disable

-----------------------------------------
20 people training

All 20 subsceriptions ---2 persons all other people can use same account
other 2 persons can create

cant use same credit card again for creating new free subscription

onmicrosoft.com

Credit Card

ram@eclasessdevazure.onmicrosoft.com

Microsoft Azure Account---Subscription----Resource Group Managers--Resources

Storage Account--Service Available in Microosft Azure.

Free Subscription--- eclasessproject1---Location---name of service(storage

Account).

PAAS--Platforma As services
IAAS--Infra structer as as serices

Storage Account
-----------------

standard--HDD---Magnetic Drives--hard disk space---500 GB--read or querying

performence is not much faster--storage cost is less.
premium--SSD---Solid State Drives--100 GB--read or querying performence is very
faster--storage cost very high.

provisioning of services---data center--sub---azure admin---

Access Tiers---
Cool
Hot

Storage Account contains 4 Different kind of Storage Modes:

---------------------------------------------------------------
Container-----folder-----files(blobs)
Block----multiple block--audio,video,images,files...
Page----virtual machine---Needs some storage---storing of disk images and operating
system files
Append---cotinuosly appending data...log files generating by applications or
virtual machines needs to append to files

Table
File Share
Quese

-------------------------------------------------------------
Security mechanisams of storage account access

1. Access keys

vt5B6uad71O4hHOlUrNlsVcCeLIA1McELrqu9TWmuuxBrZ+R9ZGkSVlrCIdfC/
Ht98XFTLJcW2Wf+FcFiUPl1w==

2. Connection Strings

DefaultEndpointsProtocol=https;AccountName=eclasessmemory321;
AccountKey=vt5B6uad71O4hHOlUrNlsVcCeLIA1McELrqu9TWmuuxBrZ+R9ZGkSVlrCIdfC/
Ht98XFTLJcW2Wf+FcFiUPl1w==;EndpointSuffix=core.windows.net

3. Shared Access Signatures

Restricting Access on Storage mode level
Restricting permissions like read write.....
setting expiration time
allowing only from specific ip address

4. RBAC--Role Based Access control--IAM(Identity Access management)

5. Service Principal Identity

---------------------------------------Azure
SQL----------------------------------------

Azure SQL Database

DTU---Data Transaction Units

vcore--processors
Elastic Pool
Database
Security
Integrationcatelog

Creating new application need simple database service

always it will install latest version and latest edition of server.

------------------------------------Azure SQL in Virtual

Machine------------------------

every thing will be taken care by our self

about SQL Maintaince

when ever database service required with full control of infrastructer

any kind of version any kind of edition can be installed

------------------------------------Azure SQL managed

Instance--------------------------

Creating Logical Servrver---

DTU
Vcore

will get all features what ever available on Azure SQL inside vm

migrating Existing On prem Application to Cloud

always it will install latest version and latest edition of server.

-----------------------------------------------------------------------------------
-----

Move -----7:30 am to 8:30 am ISt

or

8 am ISt to 9 am IST

--------------------------------------------
Azure Data Factory
Azure databaricks
Azure Synapse DW--Polybase

Azure data ENgineering Book as along with interview question.....

------------------------------------------------------------------

E--Extraction
T-Transformation
L-Loading
SSIS---MSBI(SQL Server Integration Services)--On premises ETL

Azure Cloud

Azure Data Factory

Cloud based etl tool

Extracting data from Different source systems

Transforming Data(Converting Currency)
Loading Data into Destination

Main Components of Azure Data Factory or Buliding blocks of ADF

--------------------------------------------------------------------
Pipeline--Desigining of ETL process by using different activites and defining
execution flow.
Activity---to performe a specific operation like copy data ex: Copy Data Activity
to delete file existing in location Ex: Delete Activity
Dataset---Preperation of Data to move and transform and load
Linked Service---Connection Information to any data source
Integration RunTimes---compute infrastructer to execute activites and performing
data movements from different networks
Triggers---schedule of pielines on specific time automatically, we need triggers
Parameters---if u want to pass any input value to pipeline while executing from
outside.
Variables---if u wnat to store or assign value temporarly in pipeline execution

Scenario---Loading Data From Blob Storage to SQL Database

---------------------------------------------------------
Pipeline
----------------------------------------------------
Linked Services:
----------------------
1. Blob
2.SQL

Datasets:
----------------------
1.Blob Dataset
2.SQL Database table

Activites:
---------------------
Copy Data Activity

Integration Runtimes

Triggers

Q1.What is difference between ADF V1 and V2.

v1 doesnt conatin azure devops git repository integration

V1 doesnt conatin many of activites
v1 doesnt conatins CICD methodoly of deployment
How many Views
--------------------
Home
Author----
Monitor
Manage

-------------------------------------------------------
Files from Blob to SQL Server
Files from Data Lake to SQL Server
Data From Azure SQL DAtabase to SQL Server
Data From SQL Server On premises to SQL Server

How to perform incremental Loading

How to perform Slowly changing Dimension mechanisam
How to performLooping of Files

How to secure data connection infrmation using Azure keyvaluts

How to send email notification using logic apps
How to lift and shifting of SSIS Packages into adf
How to integrate with Git Reposotiory
How to Schedule
How to Monitor
How to deploye CICD
What are different kind of integration runtimes , how to setup.
what are different kind of activites.

Integration runtimes providing compute infrstaruer to execute activites to perform

certain operations.
Auto resolve intergration runtime---to connect cloud services.
Self Hosted Integration runtime---to connect to private and virtual networks.
SSIS integration runtimes---lift and shifting of ssis packages into azure data
factory

SQL SErver Integration Services---ETL(extraction , transformation, loading)---on

premises

MSBI(SSIS,SSRS,SSAS)

HydebaraD---DIFFERENT
BANGALORE---DIFFERENT

Copy Data Activity

GetMetadata-----for getting metadat information about
files....count,exist,filename,filetype,all filenames in container,structer
For Each Loop---loop and execute activites kept inside for each loop based on no of
items out put passed from prior activity.
filter activity ---filtering input based on condition and returns output to next
activity
Lookup Activity
Stored Procedure
---------------------------------------------incremental
loading-------------------------------
create table data_source_table
(
PersonID int,
Name varchar(255),
LastModifytime datetime
);

INSERT INTO data_source_table

(PersonID, Name, LastModifytime)
VALUES
(1, 'aaaa','9/1/2017 12:56:00 AM'),
(2, 'bbbb','9/2/2017 5:23:00 AM'),
(3, 'cccc','9/3/2017 2:36:00 AM'),
(4, 'dddd','9/4/2017 3:21:00 AM'),
(5, 'eeee','9/5/2017 8:06:00 AM');

create table data_target_table

(
PersonID int,
Name varchar(255),
LastModifytime datetime
);

create table watermarktable

(

TableName varchar(255),
WatermarkValue datetime,
);

-----5500----microsoft---

--------------Microsoftot Azure---7997457228-----dp 203

--------------Power BI -----da 100

------------------------------
Copy Data Activity---copying data or files or any thing from one location to
another
GetMetadata-----for getting metadat information about
files....count,exist,filename,filetype,all filenames in container,structer
For Each Loop---loop and execute activites kept inside for each loop based on no of
items out put passed from prior activity.
filter activity ---filtering input based on condition and returns output to next
activity
stored procedure
Lookup

lookup ---lastwatermarkvalue

Set variable---Assiging value to variable with data type string or boolen

Append Variable---Assigining value to variable with data type Array--means more
then one value
{'hyd','bang','chennai'}
Wait
Delete
validation

Web Activity----to send email notification---Logic Apps

Execute Pipeline---executing another pipeline from one pipeline
Execute SSIS Package---lift and shfting of ssis packages from on premises to cloud
if condition
switch
until
NoteBook
U_SQL
Data Flows---slowly changing dimension incremntal loading by using different
transformations

Azure key valuts

triggers
monotor
git repository
deployment arm with cicd release piplines

Diff between parameters and Variable

Parameters are passing values from out side at the time executing pipleine
variables internally storing of values

SSIS--ETL

Pipeline--Packages
Activity---Task

Linked Services---Connection Managers

Create SSIS Packages in On premises using SQLserver data tools for visual studio..

Lift and shifting of SSIS packages from on prem to Cloud

deploye into Azure SQL database integration Service catelog---ssis db----SSIS

Integration RunTime

Execute package activity in adf--------

-----------------------------------------------------------------------------------
------------------

Fully Loading
-----------------------------

source--customer --100+20 new+30 old but modified Rows

Target --dimcustomer--100+20
Copy data activity
slowly changing dimension
--------------------------------------
inserting new record
updating modified record

Data Flows---Source---Transformations---Sink
spark----distributing data parlelly on multi[ple clusters ---in memory clusters
\\\\

Source
------------------
Customer

Target
------------------
DimCustomer

Different kind of Transformations inside Data Flows

------------------------------------------------------
Lookup : Comparing two Datasets based on condition by matching common key column
and returning matched and unmatched records.

Derived Column : If you want to add new column or replace existing column by
deriving new expression inside derived column Transformation.

Conditional Split : Splitting input Data Based on Conditions Ex: All Hyd Data Load
into One destionation.
All Bang Data Load
into Another Destination.
Other Locations
data needs to load into different destination.

Aggregate : If you want to perform any aggrgate operations like

SUM,MIN,MAX,COUNT,AVG by grouping on input data.

Union : If want to append or comboining more then two datasets data into one
dataset.

Join : Mergeing or joining two tables by using common key columns by performing
different kind of joins
Inner join
Left outer Join
Right Outer Join
Full Outer join
Custome Cross Join

New branch : It will take one input and giving multiple output with same set of
data.
Ex: When Same data Wants to Load into Multiple destinations

Exist : It is identifying what are all records from source1 available in source2
It is identifying what are all records from source1 doesn't exist in
source2
Select : This Transformation can use to skip columns and keeping what ever column
you want to pass to next transformation.

Alter Row : It is identifying and preparing what are records needs to be

insert update and deleted in destination based on input stream.
it is marking each row based on operation which records needs to be
inserted
Updated
Deleted
Upsert is combination of Insert + Update

Filter : If you want to apply any filter conditions on input stream data before
passing to next transformation.

Sort : If you want to sort input source data based on column in ascending or
descending order.

parse
Flattern
Pivot
Unpivot
Window
Rank

Data Flow is a new feature of Azure Data Factory (ADF) that allows you to develop
graphical data transformation.
It contains several transformations to perform ETL process.