Real-Time Streaming Data in Databricks [End to End Project]

Name: Real-Time Streaming Data in Databricks [End to End Project] | Open Video
Uploaded: 2025-08-06T12:03:54+00:00
Duration: 29 min 14 s

0:00
realtime data streaming in date braks is
0:03
realtime data streaming in date braks is
0:03
realtime data streaming in date braks is a powerful capability which allows
0:05
a powerful capability which allows
0:05
a powerful capability which allows organizations to process and analyze
0:07
organizations to process and analyze
0:07
organizations to process and analyze data as it flows in efficiently and
0:10
data as it flows in efficiently and
0:10
data as it flows in efficiently and seamlessly by leveraging aart SP
0:13
seamlessly by leveraging aart SP
0:13
seamlessly by leveraging aart SP structure streaming within date braks
0:15
structure streaming within date braks
0:15
structure streaming within date braks users can build robust scalable and
0:17
users can build robust scalable and
0:17
users can build robust scalable and fault tolerant data pipelines to handle
0:20
fault tolerant data pipelines to handle
0:20
fault tolerant data pipelines to handle a continuous stream of data from various
0:22
a continuous stream of data from various
0:22
a continuous stream of data from various sour such as a event apps iot devices
0:26
sour such as a event apps iot devices
0:26
sour such as a event apps iot devices social media logs and more in this new
0:28
social media logs and more in this new
0:28
social media logs and more in this new endtoend video I'm going to work you
0:30
endtoend video I'm going to work you
0:30
endtoend video I'm going to work you through how we can architect real time
0:32
through how we can architect real time
0:32
through how we can architect real time data stream using the vs code python
0:34
data stream using the vs code python
0:34
data stream using the vs code python generated data a event ups and data
0:37
generated data a event ups and data
0:37
generated data a event ups and data bricks notebook we're going to see how
0:38
bricks notebook we're going to see how
0:38
bricks notebook we're going to see how we can use the multi op which is
0:40
we can use the multi op which is
0:40
we can use the multi op which is popularly known as the medallon
0:42
popularly known as the medallon
0:42
popularly known as the medallon architecture combined with unity catalog
0:44
architecture combined with unity catalog
0:44
architecture combined with unity catalog in for seamless real time dat to
0:46
in for seamless real time dat to
0:46
in for seamless real time dat to streaming so let's get started let's go
0:49
streaming so let's get started let's go
0:49
streaming so let's get started let's go ahead and create event or and share
0:51
ahead and create event or and share
0:51
ahead and create event or and share access signature to send and receive
0:53
access signature to send and receive
0:53
access signature to send and receive event from the source so I'm going to
0:55
event from the source so I'm going to
0:55
event from the source so I'm going to come to the portal. ao.com I can see
0:59
come to the portal. ao.com I can see
0:59
come to the portal. ao.com I can see I've got this Cornerstone analytics
1:01
I've got this Cornerstone analytics
1:01
I've got this Cornerstone analytics event up already created so click on
1:03
event up already created so click on
1:03
event up already created so click on that and then I'm going to come to under
1:06
that and then I'm going to come to under
1:06
that and then I'm going to come to under entities click on event UPS now for now
1:09
entities click on event UPS now for now
1:09
entities click on event UPS now for now I've got no event up created so I'm
1:11
I've got no event up created so I'm
1:11
I've got no event up created so I'm going to click on this plus sign to
1:12
going to click on this plus sign to
1:12
going to click on this plus sign to create a new event up and then I'm going
1:15
create a new event up and then I'm going
1:15
create a new event up and then I'm going to call this one weather streaming data
1:18
to call this one weather streaming data
1:18
to call this one weather streaming data and then for the partition count I'm
1:20
and then for the partition count I'm
1:20
and then for the partition count I'm just going to leave it at the default
1:22
just going to leave it at the default
1:22
just going to leave it at the default one which is enough for this
1:23
one which is enough for this
1:23
one which is enough for this demonstration so for the retention
1:25
demonstration so for the retention
1:25
demonstration so for the retention policy this going to be just for 1 hour
1:27
policy this going to be just for 1 hour
1:27
policy this going to be just for 1 hour retention time and this can be up to 24
1:30
retention time and this can be up to 24
1:30
retention time and this can be up to 24 hours that is a single day so this is
1:32
hours that is a single day so this is
1:32
hours that is a single day so this is going to be a delete clean up policy so
1:35
going to be a delete clean up policy so
1:35
going to be a delete clean up policy so all this is fine I'm going to click on
1:37
all this is fine I'm going to click on
1:37
all this is fine I'm going to click on reval plus create and this is going to
1:39
reval plus create and this is going to
1:39
reval plus create and this is going to be validated and I'm going to click on
1:41
be validated and I'm going to click on
1:41
be validated and I'm going to click on create so the event up is now going to
1:44
create so the event up is now going to
1:44
create so the event up is now going to be created in just a matter of few
1:46
be created in just a matter of few
1:46
be created in just a matter of few seconds so we can see we have the
1:48
seconds so we can see we have the
1:48
seconds so we can see we have the weather streaming data created now I'm
1:51
weather streaming data created now I'm
1:51
weather streaming data created now I'm going to click on that and I'm going to
1:53
going to click on that and I'm going to
1:53
going to click on that and I'm going to go to the shared access policies so I'm
1:56
go to the shared access policies so I'm
1:56
go to the shared access policies so I'm going to create a new policy so I'm
1:58
going to create a new policy so I'm
1:58
going to create a new policy so I'm going to call this one where
2:00
going to call this one where
2:00
going to call this one where SAS key and then I'm going to click on
2:03
SAS key and then I'm going to click on
2:03
SAS key and then I'm going to click on manage now the manage allows you to send
2:05
manage now the manage allows you to send
2:05
manage now the manage allows you to send and listen to events so I'm going to
2:07
and listen to events so I'm going to
2:07
and listen to events so I'm going to click on create so this is now created
2:10
click on create so this is now created
2:10
click on create so this is now created now I'm going to click on that and then
2:12
now I'm going to click on that and then
2:12
now I'm going to click on that and then we can see we've got the primary key
2:14
we can see we've got the primary key
2:14
we can see we've got the primary key secondary key the primary connection
2:16
secondary key the primary connection
2:16
secondary key the primary connection string and the secondary connection
2:18
string and the secondary connection
2:18
string and the secondary connection string I'm going to need the combination
2:20
string I'm going to need the combination
2:20
string I'm going to need the combination of the primary connection string and the
2:23
of the primary connection string and the
2:23
of the primary connection string and the weather streaming data that is the event
2:26
weather streaming data that is the event
2:26
weather streaming data that is the event name in order to authenticate and be
2:28
name in order to authenticate and be
2:28
name in order to authenticate and be able to send and listen to events now
2:30
able to send and listen to events now
2:30
able to send and listen to events now this is sorted I'm going to come to my
2:32
this is sorted I'm going to come to my
2:32
this is sorted I'm going to come to my vs code in order to generate sample data
2:35
vs code in order to generate sample data
2:35
vs code in order to generate sample data so I'm going to stick with this Azure
2:37
so I'm going to stick with this Azure
2:37
so I'm going to stick with this Azure event or folder which is on my local
2:40
event or folder which is on my local
2:40
event or folder which is on my local laptop and I want to create a file so
2:42
laptop and I want to create a file so
2:42
laptop and I want to create a file so I'm going to call this on
2:44
I'm going to call this on
2:44
I'm going to call this on weather dat. py now this is going to be
2:47
weather dat. py now this is going to be
2:47
weather dat. py now this is going to be a python code that going to be using in
2:49
a python code that going to be using in
2:49
a python code that going to be using in a now I'm going to paste the code now
2:52
a now I'm going to paste the code now
2:52
a now I'm going to paste the code now before I do the pasting of the code I'm
2:53
before I do the pasting of the code I'm
2:53
before I do the pasting of the code I'm going to just quickly come me and this
2:55
going to just quickly come me and this
2:55
going to just quickly come me and this is my GitHub so I've got this real time
2:57
is my GitHub so I've got this real time
2:57
is my GitHub so I've got this real time streaming data in Azure data breaks
2:59
streaming data in Azure data breaks
2:59
streaming data in Azure data breaks notbook book reposit so I'm going to put
3:01
notbook book reposit so I'm going to put
3:01
notbook book reposit so I'm going to put the link of this repo in order for you
3:03
the link of this repo in order for you
3:03
the link of this repo in order for you to probably play with the sample data
3:05
to probably play with the sample data
3:05
to probably play with the sample data set and the python that is The Notebook
3:07
set and the python that is The Notebook
3:07
set and the python that is The Notebook code so I'm going to go to this weather
3:09
code so I'm going to go to this weather
3:09
code so I'm going to go to this weather data python code generator in vs code
3:12
data python code generator in vs code
3:12
data python code generator in vs code and of course I can see all the code in
3:13
and of course I can see all the code in
3:13
and of course I can see all the code in it so I'm going to click on copy raw
3:17
it so I'm going to click on copy raw
3:17
it so I'm going to click on copy raw file as soon as I click on that I can
3:19
file as soon as I click on that I can
3:19
file as soon as I click on that I can come back to the vs code editor So
3:21
come back to the vs code editor So
3:21
come back to the vs code editor So within this weather data. py file I'm
3:24
within this weather data. py file I'm
3:24
within this weather data. py file I'm going to paste the code so contrl V to
3:27
going to paste the code so contrl V to
3:27
going to paste the code so contrl V to paste it's really important first you
3:29
paste it's really important first you
3:29
paste it's really important first you install the PIP install of the a event
3:33
install the PIP install of the a event
3:33
install the PIP install of the a event up so which is already done I'm going to
3:34
up so which is already done I'm going to
3:34
up so which is already done I'm going to put a link of video on how to pip
3:36
put a link of video on how to pip
3:36
put a link of video on how to pip install the a event or in the vs code
3:39
install the a event or in the vs code
3:39
install the a event or in the vs code editor so after you install the aure
3:41
editor so after you install the aure
3:41
editor so after you install the aure event up now it's really important to
3:43
event up now it's really important to
3:44
event up now it's really important to import the necessary Library so we've
3:45
import the necessary Library so we've
3:45
import the necessary Library so we've got all the libraries and I'm going to
3:47
got all the libraries and I'm going to
3:47
got all the libraries and I'm going to also provide the event up connection
3:49
also provide the event up connection
3:49
also provide the event up connection string and event up name so got we've
3:51
string and event up name so got we've
3:51
string and event up name so got we've got this connection string which is
3:53
got this connection string which is
3:53
got this connection string which is equal to the primary connection stram
3:55
equal to the primary connection stram
3:55
equal to the primary connection stram that we're going to fetch in a moment
3:56
that we're going to fetch in a moment
3:56
that we're going to fetch in a moment and then we also have the event up name
3:58
and then we also have the event up name
3:58
and then we also have the event up name which is going to be the weather
3:59
which is going to be the weather
4:00
which is going to be the weather streaming data and then we're going to
4:01
streaming data and then we're going to
4:01
streaming data and then we're going to create a producer client to send message
4:04
create a producer client to send message
4:04
create a producer client to send message to the event up in AEL so we have this
4:06
to the event up in AEL so we have this
4:06
to the event up in AEL so we have this producer variable equals to event up
4:09
producer variable equals to event up
4:09
producer variable equals to event up producer client do from connection
4:11
producer client do from connection
4:11
producer client do from connection string and inside this open and close
4:13
string and inside this open and close
4:13
string and inside this open and close bracket we have this concore St equal to
4:17
bracket we have this concore St equal to
4:17
bracket we have this concore St equal to whatever value is that we pass into this
4:19
whatever value is that we pass into this
4:19
whatever value is that we pass into this connection string and then for the event
4:21
connection string and then for the event
4:21
connection string and then for the event opening it's going to equal to whatever
4:22
opening it's going to equal to whatever
4:22
opening it's going to equal to whatever is the name of the event as we Define in
4:25
is the name of the event as we Define in
4:25
is the name of the event as we Define in this variable and then we have this list
4:28
this variable and then we have this list
4:28
this variable and then we have this list of City so let me just move this down a
4:30
of City so let me just move this down a
4:30
of City so let me just move this down a little bit so that we can see what's
4:31
little bit so that we can see what's
4:31
little bit so that we can see what's going on so we want to generate some
4:33
going on so we want to generate some
4:33
going on so we want to generate some random data so we going to have this
4:35
random data so we going to have this
4:35
random data so we going to have this city such as New York Los Angeles
4:37
city such as New York Los Angeles
4:37
city such as New York Los Angeles Chicago Houston and so on and so forth
4:40
Chicago Houston and so on and so forth
4:40
Chicago Houston and so on and so forth and I'm going to generate the latitude
4:42
and I'm going to generate the latitude
4:42
and I'm going to generate the latitude and the longitude and then at the bottom
4:44
and the longitude and then at the bottom
4:44
and the longitude and then at the bottom I'm going to generate randomly Peak the
4:46
I'm going to generate randomly Peak the
4:46
I'm going to generate randomly Peak the cities including the latitude and
4:48
cities including the latitude and
4:48
cities including the latitude and longitude and then we're going to return
4:50
longitude and then we're going to return
4:50
longitude and then we're going to return inside this dictionary we have the city
4:53
inside this dictionary we have the city
4:53
inside this dictionary we have the city latitude longitude temperature humidity
4:56
latitude longitude temperature humidity
4:56
latitude longitude temperature humidity wind speed pressure precipitation CL
4:59
wind speed pressure precipitation CL
4:59
wind speed pressure precipitation CL cover weather condition now these are
5:01
cover weather condition now these are
5:01
cover weather condition now these are similar to columns in a tabular format
5:04
similar to columns in a tabular format
5:04
similar to columns in a tabular format and then at the bottom here we're going
5:06
and then at the bottom here we're going
5:06
and then at the bottom here we're going to create a batch of events so we have
5:08
to create a batch of events so we have
5:08
to create a batch of events so we have this variable which is equal to
5:09
this variable which is equal to
5:09
this variable which is equal to producer. create batch and then we're
5:12
producer. create batch and then we're
5:12
producer. create batch and then we're going to also generate a sample weather
5:14
going to also generate a sample weather
5:14
going to also generate a sample weather data so this going to be a variable
5:15
data so this going to be a variable
5:15
data so this going to be a variable weather data and it's going to be
5:17
weather data and it's going to be
5:17
weather data and it's going to be looking at the generatore weather data
5:21
looking at the generatore weather data
5:21
looking at the generatore weather data and then we're going to store the data
5:22
and then we're going to store the data
5:22
and then we're going to store the data as a Json so we can see we have the
5:24
as a Json so we can see we have the
5:24
as a Json so we can see we have the json. dumps and it's going to be feding
5:26
json. dumps and it's going to be feding
5:26
json. dumps and it's going to be feding into this intermediate um variable name
5:29
into this intermediate um variable name
5:29
into this intermediate um variable name and I'm going to add Json format data to
5:31
and I'm going to add Json format data to
5:32
and I'm going to add Json format data to the batch so we have this eventor dataor
5:34
the batch so we have this eventor dataor
5:34
the batch so we have this eventor dataor batch. add method and then we have this
5:37
batch. add method and then we have this
5:37
batch. add method and then we have this event data that we imported equal to the
5:41
event data that we imported equal to the
5:41
event data that we imported equal to the messages and we're going to send and
5:43
messages and we're going to send and
5:43
messages and we're going to send and this going to be going on every 5
5:46
this going to be going on every 5
5:46
this going to be going on every 5 Seconds now this cannot run except we
5:49
Seconds now this cannot run except we
5:49
Seconds now this cannot run except we provide the connection string and the
5:52
provide the connection string and the
5:52
provide the connection string and the event name so let's quickly come to the
5:54
event name so let's quickly come to the
5:54
event name so let's quickly come to the a portal and I want to fetch the
5:57
a portal and I want to fetch the
5:57
a portal and I want to fetch the connection string so I can copy
6:00
connection string so I can copy
6:00
connection string so I can copy the primary um connection string copy
6:03
the primary um connection string copy
6:03
the primary um connection string copy into the cboard come back to the vs code
6:04
into the cboard come back to the vs code
6:04
into the cboard come back to the vs code editor and I'm going to select this
6:06
editor and I'm going to select this
6:06
editor and I'm going to select this place holder value and I'm going to
6:08
place holder value and I'm going to
6:08
place holder value and I'm going to paste that string in there and then for
6:12
paste that string in there and then for
6:12
paste that string in there and then for the name of our event that is the
6:15
the name of our event that is the
6:15
the name of our event that is the weather streaming data I'm going to just
6:16
weather streaming data I'm going to just
6:16
weather streaming data I'm going to just copy this and control C and I'm going to
6:19
copy this and control C and I'm going to
6:19
copy this and control C and I'm going to come carefully into this placeholder so
6:21
come carefully into this placeholder so
6:21
come carefully into this placeholder so this going to be our event or name so
6:24
this going to be our event or name so
6:24
this going to be our event or name so select this place orderer value and I'm
6:26
select this place orderer value and I'm
6:26
select this place orderer value and I'm going to press contrl V to paste so
6:28
going to press contrl V to paste so
6:28
going to press contrl V to paste so that's all we need to do now I can click
6:30
that's all we need to do now I can click
6:30
that's all we need to do now I can click on this play button and let's see what
6:33
on this play button and let's see what
6:33
on this play button and let's see what happened in the terminal window at the
6:36
happened in the terminal window at the
6:36
happened in the terminal window at the bottom so we can see we have data being
6:39
bottom so we can see we have data being
6:39
bottom so we can see we have data being sent so this is the very first data that
6:41
sent so this is the very first data that
6:41
sent so this is the very first data that is being sent to our event or in aure
6:44
is being sent to our event or in aure
6:44
is being sent to our event or in aure platform which is absolutely cool now
6:46
platform which is absolutely cool now
6:46
platform which is absolutely cool now this is sorted now I'm going to come
6:48
this is sorted now I'm going to come
6:48
this is sorted now I'm going to come back to the portal and I want to check
6:50
back to the portal and I want to check
6:50
back to the portal and I want to check the data I'm going to come to the
6:52
the data I'm going to come to the
6:52
the data I'm going to come to the overview in the overview I'm going to
6:54
overview in the overview I'm going to
6:54
overview in the overview I'm going to scroll down and you can see we have
6:56
scroll down and you can see we have
6:56
scroll down and you can see we have evidence of live data coming into this
6:59
evidence of live data coming into this
6:59
evidence of live data coming into this event up which is absolutely fantastic
7:02
event up which is absolutely fantastic
7:02
event up which is absolutely fantastic next we want to create the databas
7:04
next we want to create the databas
7:04
next we want to create the databas cluster and install the Marvin library
7:06
cluster and install the Marvin library
7:06
cluster and install the Marvin library that allows us to connect to the a event
7:09
that allows us to connect to the a event
7:09
that allows us to connect to the a event or and begin to stream the data Into The
7:12
or and begin to stream the data Into The
7:12
or and begin to stream the data Into The datab Brak platform so I'm going to come
7:14
datab Brak platform so I'm going to come
7:14
datab Brak platform so I'm going to come here and then I'm going to go to the
7:16
here and then I'm going to go to the
7:16
here and then I'm going to go to the compute and then I going to create a new
7:18
compute and then I going to create a new
7:18
compute and then I going to create a new compute and I'm going to call this one
7:21
compute and I'm going to call this one
7:21
compute and I'm going to call this one um weather streaming compute you can use
7:24
um weather streaming compute you can use
7:24
um weather streaming compute you can use whatever name you like as the cluster
7:26
whatever name you like as the cluster
7:26
whatever name you like as the cluster name so compute and once I'm done with
7:29
name so compute and once I'm done with
7:29
name so compute and once I'm done with the name name press enter to commit and
7:31
the name name press enter to commit and
7:31
the name name press enter to commit and then for the policy I'm going to choose
7:32
then for the policy I'm going to choose
7:32
then for the policy I'm going to choose a single n and then for the performance
7:35
a single n and then for the performance
7:35
a single n and then for the performance now I'm going to come here and I'm going
7:37
now I'm going to come here and I'm going
7:37
now I'm going to come here and I'm going to point to the 12.2 LTS which is what
7:40
to point to the 12.2 LTS which is what
7:40
to point to the 12.2 LTS which is what is available to access the aure event up
7:43
is available to access the aure event up
7:43
is available to access the aure event up for now so we have this runtime 12.2 and
7:46
for now so we have this runtime 12.2 and
7:46
for now so we have this runtime 12.2 and the scalar is 2.2 and the spark is
7:49
the scalar is 2.2 and the spark is
7:49
the scalar is 2.2 and the spark is 3.3.2 so I'm going to just leave this
7:51
3.3.2 so I'm going to just leave this
7:51
3.3.2 so I'm going to just leave this full th acceleration there on for the N
7:54
full th acceleration there on for the N
7:54
full th acceleration there on for the N TP I'm going to just leave this standard
7:56
TP I'm going to just leave this standard
7:56
TP I'm going to just leave this standard D4 dsor V5 and then this going terminate
8:00
D4 dsor V5 and then this going terminate
8:00
D4 dsor V5 and then this going terminate let's say every 30 minutes in order to
8:01
let's say every 30 minutes in order to
8:01
let's say every 30 minutes in order to save some money now it's really
8:03
save some money now it's really
8:03
save some money now it's really recommended that if your computer is not
8:04
recommended that if your computer is not
8:05
recommended that if your computer is not being used please always terminate it in
8:06
being used please always terminate it in
8:07
being used please always terminate it in order not to incure unnecessary cost so
8:09
order not to incure unnecessary cost so
8:09
order not to incure unnecessary cost so having done all of this we can see the
8:11
having done all of this we can see the
8:11
having done all of this we can see the summary so we can see we have just a
8:12
summary so we can see we have just a
8:12
summary so we can see we have just a single driver and the run time is going
8:14
single driver and the run time is going
8:14
single driver and the run time is going to be this 12.2 and then we have the
8:17
to be this 12.2 and then we have the
8:17
to be this 12.2 and then we have the unity catalog enable which is really
8:20
unity catalog enable which is really
8:20
unity catalog enable which is really important now I'm going to click on
8:22
important now I'm going to click on
8:22
important now I'm going to click on create
8:23
create
8:23
create compute and then this is going to start
8:25
compute and then this is going to start
8:26
compute and then this is going to start spinning I'm going to come to the
8:27
spinning I'm going to come to the
8:27
spinning I'm going to come to the libraries as that is going on I want to
8:30
libraries as that is going on I want to
8:30
libraries as that is going on I want to install the Marvin Library so I'm going
8:32
install the Marvin Library so I'm going
8:32
install the Marvin Library so I'm going to click on install new and then I want
8:34
to click on install new and then I want
8:34
to click on install new and then I want to pick the Marvin Library space and
8:36
to pick the Marvin Library space and
8:36
to pick the Marvin Library space and then we need to provide the coordinate
8:38
then we need to provide the coordinate
8:38
then we need to provide the coordinate that allows us to establish connection
8:39
that allows us to establish connection
8:39
that allows us to establish connection to the Azure event of so if you know the
8:42
to the Azure event of so if you know the
8:42
to the Azure event of so if you know the coordinate you can just copy paste
8:43
coordinate you can just copy paste
8:43
coordinate you can just copy paste directly but you can always set for the
8:45
directly but you can always set for the
8:45
directly but you can always set for the packages sock here and then I want to
8:47
packages sock here and then I want to
8:47
packages sock here and then I want to set for Azure I'm sorry event Ops event
8:52
set for Azure I'm sorry event Ops event
8:52
set for Azure I'm sorry event Ops event Ops and then Das um spark so we're going
8:55
Ops and then Das um spark so we're going
8:55
Ops and then Das um spark so we're going to say this list has been narrowed down
8:57
to say this list has been narrowed down
8:57
to say this list has been narrowed down so we can say we have a event of spar
8:59
so we can say we have a event of spar
8:59
so we can say we have a event of spar parent and we have this EV up spark 2.12
9:03
parent and we have this EV up spark 2.12
9:03
parent and we have this EV up spark 2.12 so I'm going to pick this latest so
9:05
so I'm going to pick this latest so
9:05
so I'm going to pick this latest so click on select and then we have the
9:07
click on select and then we have the
9:07
click on select and then we have the coordinate already provisioned or just
9:10
coordinate already provisioned or just
9:10
coordinate already provisioned or just um produced a now I'm just going to
9:12
um produced a now I'm just going to
9:12
um produced a now I'm just going to click on install and I'm going to wait
9:15
click on install and I'm going to wait
9:16
click on install and I'm going to wait for that library to be installed and
9:18
for that library to be installed and
9:18
for that library to be installed and then I'm going to wait for the weather
9:20
then I'm going to wait for the weather
9:20
then I'm going to wait for the weather streaming Compu to be ready to proceed
9:24
streaming Compu to be ready to proceed
9:24
streaming Compu to be ready to proceed all right so the weather streaming
9:26
all right so the weather streaming
9:26
all right so the weather streaming compute is up and running and then we
9:28
compute is up and running and then we
9:28
compute is up and running and then we have the library in stored from Marvin
9:31
have the library in stored from Marvin
9:31
have the library in stored from Marvin so this simply means that we've
9:32
so this simply means that we've
9:32
so this simply means that we've establish connection to our aure event
9:34
establish connection to our aure event
9:34
establish connection to our aure event up and we can start our streaming data
9:36
up and we can start our streaming data
9:36
up and we can start our streaming data by the way if you yet to subcribe to the
9:37
by the way if you yet to subcribe to the
9:38
by the way if you yet to subcribe to the Channel please click on the Subscribe
9:39
Channel please click on the Subscribe
9:39
Channel please click on the Subscribe button and tole on Bell icon to be
9:41
button and tole on Bell icon to be
9:41
button and tole on Bell icon to be inform of new videos thank you so let's
9:44
inform of new videos thank you so let's
9:44
inform of new videos thank you so let's go ahead and input our notebook from the
9:46
go ahead and input our notebook from the
9:46
go ahead and input our notebook from the G up so I'm going to come here and then
9:48
G up so I'm going to come here and then
9:48
G up so I'm going to come here and then I want to go to this data breaks real
9:51
I want to go to this data breaks real
9:51
I want to go to this data breaks real time streaming and then you can see the
9:53
time streaming and then you can see the
9:53
time streaming and then you can see the notebook in here and I'm just going to
9:55
notebook in here and I'm just going to
9:55
notebook in here and I'm just going to click on raw and then I want to copy the
9:58
click on raw and then I want to copy the
9:58
click on raw and then I want to copy the URL for the so contrl C and I can return
10:01
URL for the so contrl C and I can return
10:01
URL for the so contrl C and I can return back here so I'm going to go to
10:03
back here so I'm going to go to
10:03
back here so I'm going to go to workspace and then I want to go to
10:05
workspace and then I want to go to
10:05
workspace and then I want to go to workspaces and then I want to go to user
10:08
workspaces and then I want to go to user
10:08
workspaces and then I want to go to user and I'm going to select my enter ID and
10:10
and I'm going to select my enter ID and
10:10
and I'm going to select my enter ID and then I'm going to click on this kebab
10:11
then I'm going to click on this kebab
10:11
then I'm going to click on this kebab and then I can say the input
10:12
and then I can say the input
10:12
and then I can say the input functionality I can see the URL so I'm
10:15
functionality I can see the URL so I'm
10:15
functionality I can see the URL so I'm going to paste the URL and then click on
10:17
going to paste the URL and then click on
10:17
going to paste the URL and then click on import all right so we have this data
10:20
import all right so we have this data
10:20
import all right so we have this data brakes real time streaming data notebook
10:23
brakes real time streaming data notebook
10:23
brakes real time streaming data notebook and I'm going to use the Python language
10:25
and I'm going to use the Python language
10:25
and I'm going to use the Python language for our data processing so I'm going to
10:27
for our data processing so I'm going to
10:27
for our data processing so I'm going to follow the multi op that that is The
10:29
follow the multi op that that is The
10:29
follow the multi op that that is The Medallion architecture that consists of
10:31
Medallion architecture that consists of
10:31
Medallion architecture that consists of the bronze silver and the gold layers so
10:35
the bronze silver and the gold layers so
10:35
the bronze silver and the gold layers so we're going to start by using the r
10:36
we're going to start by using the r
10:36
we're going to start by using the r stream to read the data from the event
10:38
stream to read the data from the event
10:38
stream to read the data from the event up and then we're going to write into
10:40
up and then we're going to write into
10:40
up and then we're going to write into this Bron layer which is going to be the
10:42
this Bron layer which is going to be the
10:42
this Bron layer which is going to be the raw data and then we're going to move
10:44
raw data and then we're going to move
10:44
raw data and then we're going to move the data from the Bron layer into the
10:46
the data from the Bron layer into the
10:46
the data from the Bron layer into the Civil layer and then we're going to
10:47
Civil layer and then we're going to
10:47
Civil layer and then we're going to transform the data by cleaning the data
10:49
transform the data by cleaning the data
10:50
transform the data by cleaning the data and then we're going to perform some
10:52
and then we're going to perform some
10:52
and then we're going to perform some aggregation that some calculation in the
10:54
aggregation that some calculation in the
10:54
aggregation that some calculation in the gold layer and afterwards we're going to
10:56
gold layer and afterwards we're going to
10:56
gold layer and afterwards we're going to create a Power report using a direct
10:58
create a Power report using a direct
10:58
create a Power report using a direct query so I'm going to just move that a
11:01
query so I'm going to just move that a
11:01
query so I'm going to just move that a little bit and then I've got some notes
11:02
little bit and then I've got some notes
11:02
little bit and then I've got some notes at the bottom here so you can take your
11:03
at the bottom here so you can take your
11:03
at the bottom here so you can take your time and look at them now it's really
11:05
time and look at them now it's really
11:05
time and look at them now it's really important we attach the cluster to the
11:08
important we attach the cluster to the
11:08
important we attach the cluster to the specific book which I've already done
11:10
specific book which I've already done
11:10
specific book which I've already done and then we're going to start by
11:12
and then we're going to start by
11:12
and then we're going to start by importing some of the required libraries
11:14
importing some of the required libraries
11:14
importing some of the required libraries so we've got all of this so from ppar
11:16
so we've got all of this so from ppar
11:16
so we've got all of this so from ppar seo. functions so we're going to import
11:18
seo. functions so we're going to import
11:19
seo. functions so we're going to import Star and then we're going to also import
11:21
Star and then we're going to also import
11:21
Star and then we're going to also import the call from Json struct type struct
11:24
the call from Json struct type struct
11:24
the call from Json struct type struct field string type double type and so on
11:26
field string type double type and so on
11:26
field string type double type and so on and so forth so I'm just going to press
11:28
and so forth so I'm just going to press
11:28
and so forth so I'm just going to press control enter to run this code and this
11:31
control enter to run this code and this
11:31
control enter to run this code and this executed nicely now I'm going to screw
11:33
executed nicely now I'm going to screw
11:33
executed nicely now I'm going to screw down so we're going to set up the Azure
11:35
down so we're going to set up the Azure
11:35
down so we're going to set up the Azure event of connection so we can see we
11:37
event of connection so we can see we
11:37
event of connection so we can see we have the connection string and I'm going
11:38
have the connection string and I'm going
11:38
have the connection string and I'm going to pass in the primary connection string
11:40
to pass in the primary connection string
11:40
to pass in the primary connection string and then the event of name and I'm going
11:43
and then the event of name and I'm going
11:43
and then the event of name and I'm going to quickly fetch that so I'm going to
11:44
to quickly fetch that so I'm going to
11:44
to quickly fetch that so I'm going to come back here and then I want to go to
11:46
come back here and then I want to go to
11:46
come back here and then I want to go to the Shar access policies and I want to
11:48
the Shar access policies and I want to
11:48
the Shar access policies and I want to go to the weather SAS key and I'm going
11:51
go to the weather SAS key and I'm going
11:51
go to the weather SAS key and I'm going to copy the primary connection string
11:53
to copy the primary connection string
11:53
to copy the primary connection string similar to what we use in the vs code
11:55
similar to what we use in the vs code
11:55
similar to what we use in the vs code editor so come back in here so I'm going
11:57
editor so come back in here so I'm going
11:57
editor so come back in here so I'm going to select this placeholder value
11:59
to select this placeholder value
11:59
to select this placeholder value carefully and I'm going to paste the
12:01
carefully and I'm going to paste the
12:01
carefully and I'm going to paste the string and then for the event up name
12:04
string and then for the event up name
12:04
string and then for the event up name I'm going to just come back here and I'm
12:05
I'm going to just come back here and I'm
12:05
I'm going to just come back here and I'm going to copy this weather streaming
12:07
going to copy this weather streaming
12:07
going to copy this weather streaming data come back here I'm going to get rid
12:10
data come back here I'm going to get rid
12:10
data come back here I'm going to get rid of this placeholder value and inside the
12:13
of this placeholder value and inside the
12:13
of this placeholder value and inside the double quote I'm going to paste the
12:15
double quote I'm going to paste the
12:15
double quote I'm going to paste the event up name and then we have the
12:17
event up name and then we have the
12:17
event up name and then we have the connection string dictionary now in the
12:19
connection string dictionary now in the
12:19
connection string dictionary now in the dictionary we have this event ops.
12:20
dictionary we have this event ops.
12:21
dictionary we have this event ops. connection string and I'm going to be
12:22
connection string and I'm going to be
12:22
connection string and I'm going to be using the event UPS util to encrypt the
12:25
using the event UPS util to encrypt the
12:25
using the event UPS util to encrypt the connection string and then we're going
12:27
connection string and then we're going
12:27
connection string and then we're going to also pass in the event up name which
12:29
to also pass in the event up name which
12:29
to also pass in the event up name which is the same thing as what we have here
12:31
is the same thing as what we have here
12:31
is the same thing as what we have here and similar to the connection string
12:32
and similar to the connection string
12:32
and similar to the connection string here we just pass it inside this encrypt
12:35
here we just pass it inside this encrypt
12:35
here we just pass it inside this encrypt and I'm going to go ahead and run this
12:37
and I'm going to go ahead and run this
12:37
and I'm going to go ahead and run this code so press control enter and this
12:39
code so press control enter and this
12:39
code so press control enter and this executed nicely so let's move to how we
12:42
executed nicely so let's move to how we
12:42
executed nicely so let's move to how we can create the medon architecture want
12:44
can create the medon architecture want
12:44
can create the medon architecture want to create catalog schemas and volumes so
12:47
to create catalog schemas and volumes so
12:47
to create catalog schemas and volumes so first we're going to define the storage
12:49
first we're going to define the storage
12:49
first we're going to define the storage location for our catalog so we have this
12:51
location for our catalog so we have this
12:51
location for our catalog so we have this storage location and we're going to
12:53
storage location and we're going to
12:53
storage location and we're going to points to the storage location AB FSS
12:56
points to the storage location AB FSS
12:56
points to the storage location AB FSS path so how do we get this I'm going to
12:58
path so how do we get this I'm going to
12:58
path so how do we get this I'm going to come to this tab and then I want to go
13:00
come to this tab and then I want to go
13:00
come to this tab and then I want to go to the catalog and in the catalog click
13:02
to the catalog and in the catalog click
13:02
to the catalog and in the catalog click on this settings and I want to go to you
13:05
on this settings and I want to go to you
13:05
on this settings and I want to go to you know create external locations so we
13:06
know create external locations so we
13:06
know create external locations so we have this URL so I'm going to move this
13:08
have this URL so I'm going to move this
13:08
have this URL so I'm going to move this expand a little bit and this is our
13:10
expand a little bit and this is our
13:10
expand a little bit and this is our storage location so I'm going to copy
13:12
storage location so I'm going to copy
13:12
storage location so I'm going to copy that come back in here get rid of this
13:15
that come back in here get rid of this
13:15
that come back in here get rid of this place hold value inside this single
13:17
place hold value inside this single
13:17
place hold value inside this single quote control V and oops okay let me
13:20
quote control V and oops okay let me
13:20
quote control V and oops okay let me just move this line properly delete okay
13:23
just move this line properly delete okay
13:23
just move this line properly delete okay so this is the historic um location and
13:25
so this is the historic um location and
13:25
so this is the historic um location and we're going to create a unit catalog
13:27
we're going to create a unit catalog
13:27
we're going to create a unit catalog named weather and this going to be a
13:29
named weather and this going to be a
13:29
named weather and this going to be a managed location so we can see we have
13:31
managed location so we can see we have
13:31
managed location so we can see we have this store location pass inside this
13:33
this store location pass inside this
13:33
this store location pass inside this call bracket in it and this is going to
13:35
call bracket in it and this is going to
13:35
call bracket in it and this is going to be where the unit catalog will be
13:37
be where the unit catalog will be
13:37
be where the unit catalog will be managed and then we're going to create
13:39
managed and then we're going to create
13:39
managed and then we're going to create bronze layer schema so this is named
13:42
bronze layer schema so this is named
13:42
bronze layer schema so this is named bronze so this is going to be within the
13:44
bronze so this is going to be within the
13:44
bronze so this is going to be within the weather unit catalog so we're going to
13:46
weather unit catalog so we're going to
13:46
weather unit catalog so we're going to create the silver and the gold layer
13:48
create the silver and the gold layer
13:48
create the silver and the gold layer within the same weather unit catalog and
13:50
within the same weather unit catalog and
13:50
within the same weather unit catalog and then we're going to create the bronze
13:52
then we're going to create the bronze
13:52
then we're going to create the bronze layer for checkpoints now the checkpoint
13:55
layer for checkpoints now the checkpoint
13:55
layer for checkpoints now the checkpoint is going to be used for processing our
13:56
is going to be used for processing our
13:56
is going to be used for processing our data coming from the a event up to
13:58
data coming from the a event up to
13:58
data coming from the a event up to ensure for tolerance and data recovery
14:01
ensure for tolerance and data recovery
14:01
ensure for tolerance and data recovery in case of failures so we're going to
14:03
in case of failures so we're going to
14:03
in case of failures so we're going to create the volume for each of the layers
14:06
create the volume for each of the layers
14:06
create the volume for each of the layers so we have this create volume which is
14:09
so we have this create volume which is
14:09
so we have this create volume which is within the weather catalog bronze.
14:12
within the weather catalog bronze.
14:12
within the weather catalog bronze. bronze volume so we can see we have a
14:14
bronze volume so we can see we have a
14:14
bronze volume so we can see we have a spark SQ so the same thing for the
14:16
spark SQ so the same thing for the
14:16
spark SQ so the same thing for the silver so create volume weather. silver.
14:19
silver so create volume weather. silver.
14:19
silver so create volume weather. silver. silver volume and then going to also
14:21
silver volume and then going to also
14:21
silver volume and then going to also create volume weather go good volume so
14:23
create volume weather go good volume so
14:24
create volume weather go good volume so I'm going to just go ahead and run this
14:25
I'm going to just go ahead and run this
14:25
I'm going to just go ahead and run this code so this should work nicely
14:30
code so this should work nicely
14:30
code so this should work nicely okay so it's executed nicely now we can
14:33
okay so it's executed nicely now we can
14:33
okay so it's executed nicely now we can see that catalog in it now going to
14:35
see that catalog in it now going to
14:35
see that catalog in it now going to click on this refresh and then we should
14:37
click on this refresh and then we should
14:37
click on this refresh and then we should see the weather catalog so I'm going to
14:40
see the weather catalog so I'm going to
14:40
see the weather catalog so I'm going to expand and we can see we have the bronze
14:42
expand and we can see we have the bronze
14:42
expand and we can see we have the bronze can expand and I'm going to see the
14:44
can expand and I'm going to see the
14:44
can expand and I'm going to see the volume we just created for the BRS and
14:46
volume we just created for the BRS and
14:46
volume we just created for the BRS and then when I expand the go I'm going to
14:47
then when I expand the go I'm going to
14:47
then when I expand the go I'm going to see the volume and so on and so forth
14:49
see the volume and so on and so forth
14:49
see the volume and so on and so forth that is absolutely cool now let's go to
14:52
that is absolutely cool now let's go to
14:52
that is absolutely cool now let's go to the data processing that is the bronze
14:54
the data processing that is the bronze
14:54
the data processing that is the bronze layer so for the bronze layer we're
14:55
layer so for the bronze layer we're
14:55
layer so for the bronze layer we're going to use the spark. rstream
14:59
going to use the spark. rstream
14:59
going to use the spark. rstream format now the format is going to be the
15:01
format now the format is going to be the
15:01
format now the format is going to be the event up now because we are reading the
15:03
event up now because we are reading the
15:03
event up now because we are reading the data from the event of that you know we
15:05
data from the event of that you know we
15:05
data from the event of that you know we sending data from the vs code in so we
15:07
sending data from the vs code in so we
15:07
sending data from the vs code in so we have the event up as the format and the
15:09
have the event up as the format and the
15:09
have the event up as the format and the option is going to be the connection
15:11
option is going to be the connection
15:11
option is going to be the connection configuration dictionary that we
15:13
configuration dictionary that we
15:13
configuration dictionary that we specified in the Azure event connection
15:16
specified in the Azure event connection
15:16
specified in the Azure event connection here this value connection config and
15:18
here this value connection config and
15:18
here this value connection config and then we're going to move to the load so
15:20
then we're going to move to the load so
15:20
then we're going to move to the load so we're going to load and after loading
15:23
we're going to load and after loading
15:23
we're going to load and after loading we're going to just display the DF so
15:25
we're going to just display the DF so
15:25
we're going to just display the DF so after reading the stream we're going to
15:27
after reading the stream we're going to
15:27
after reading the stream we're going to write the stream into a checkpoint so
15:29
write the stream into a checkpoint so
15:30
write the stream into a checkpoint so the checkpoint is going to be the bronze
15:32
the checkpoint is going to be the bronze
15:32
the checkpoint is going to be the bronze volume so I'm going to provide the
15:35
volume so I'm going to provide the
15:35
volume so I'm going to provide the volume path for the bronze as the
15:37
volume path for the bronze as the
15:37
volume path for the bronze as the checkpoint so I'm going to click on this
15:38
checkpoint so I'm going to click on this
15:38
checkpoint so I'm going to click on this to expand and I can see the bronze
15:40
to expand and I can see the bronze
15:40
to expand and I can see the bronze volume now this is the icon for volume
15:42
volume now this is the icon for volume
15:42
volume now this is the icon for volume in dat breaks environment if you don't
15:43
in dat breaks environment if you don't
15:43
in dat breaks environment if you don't know and I'm going to click on the skab
15:46
know and I'm going to click on the skab
15:46
know and I'm going to click on the skab and then I want to copy the volume path
15:48
and then I want to copy the volume path
15:48
and then I want to copy the volume path so I'm going to come inside this place
15:50
so I'm going to come inside this place
15:50
so I'm going to come inside this place holder value carefully control V so
15:52
holder value carefully control V so
15:52
holder value carefully control V so that's going to be the checkpoint in
15:54
that's going to be the checkpoint in
15:54
that's going to be the checkpoint in order to ensure we have F tolerance and
15:57
order to ensure we have F tolerance and
15:57
order to ensure we have F tolerance and data recovery in case of failure and
15:59
data recovery in case of failure and
15:59
data recovery in case of failure and then the output is going to be a pain
16:01
then the output is going to be a pain
16:01
then the output is going to be a pain and then the format going to be in form
16:03
and then the format going to be in form
16:03
and then the format going to be in form of Delta and then to table so this going
16:05
of Delta and then to table so this going
16:05
of Delta and then to table so this going to be written into the weather. bronze.
16:10
to be written into the weather. bronze.
16:10
to be written into the weather. bronze. bronze data now this is going to be the
16:12
bronze data now this is going to be the
16:12
bronze data now this is going to be the name of the table the name of the schema
16:14
name of the table the name of the schema
16:14
name of the table the name of the schema and the name of the catalog so we have
16:16
and the name of the catalog so we have
16:16
and the name of the catalog so we have the three name space so I can go on and
16:18
the three name space so I can go on and
16:18
the three name space so I can go on and control enter to run the code and let's
16:21
control enter to run the code and let's
16:21
control enter to run the code and let's see what happens okay so we can see
16:24
see what happens okay so we can see
16:24
see what happens okay so we can see stream initializing so now I'm just
16:27
stream initializing so now I'm just
16:27
stream initializing so now I'm just going to wait for some couple of moments
16:29
going to wait for some couple of moments
16:29
going to wait for some couple of moments and I'm going to expand this display
16:30
and I'm going to expand this display
16:30
and I'm going to expand this display query one or let me just expand this one
16:32
query one or let me just expand this one
16:32
query one or let me just expand this one first now we're going to see that we
16:34
first now we're going to see that we
16:34
first now we're going to see that we have the input F processing rate and
16:37
have the input F processing rate and
16:37
have the input F processing rate and then we have the records per seconds and
16:39
then we have the records per seconds and
16:39
then we have the records per seconds and then we have the batch duration in
16:40
then we have the batch duration in
16:40
then we have the batch duration in seconds so we can see data is now coming
16:43
seconds so we can see data is now coming
16:43
seconds so we can see data is now coming into our bronze um volume of the bronze
16:46
into our bronze um volume of the bronze
16:46
into our bronze um volume of the bronze um layer and then we can see we have
16:48
um layer and then we can see we have
16:48
um layer and then we can see we have this data in form of binary format now
16:50
this data in form of binary format now
16:50
this data in form of binary format now this actually difficult to say so for
16:52
this actually difficult to say so for
16:52
this actually difficult to say so for example we can see we have the body
16:54
example we can see we have the body
16:54
example we can see we have the body column we have the paration and then we
16:56
column we have the paration and then we
16:56
column we have the paration and then we have the offset we have the sequence
16:58
have the offset we have the sequence
16:58
have the offset we have the sequence number the ened time the publisher
17:01
number the ened time the publisher
17:01
number the ened time the publisher Partition by properties and so on and so
17:03
Partition by properties and so on and so
17:03
Partition by properties and so on and so forth now we're going to conver the
17:05
forth now we're going to conver the
17:05
forth now we're going to conver the event scent which is in binary format to
17:07
event scent which is in binary format to
17:07
event scent which is in binary format to a tabular format and that's what we're
17:09
a tabular format and that's what we're
17:09
a tabular format and that's what we're going to do in the silver layer so I'm
17:12
going to do in the silver layer so I'm
17:12
going to do in the silver layer so I'm going to scroll down and as you can see
17:14
going to scroll down and as you can see
17:14
going to scroll down and as you can see even before we go on we can see this
17:16
even before we go on we can see this
17:16
even before we go on we can see this actually growing so this is actually a
17:18
actually growing so this is actually a
17:18
actually growing so this is actually a real time data processing so this going
17:19
real time data processing so this going
17:19
real time data processing so this going to be increasing from 13 rows to 14 and
17:22
to be increasing from 13 rows to 14 and
17:22
to be increasing from 13 rows to 14 and so on and so forth that is cool so let's
17:24
so on and so forth that is cool so let's
17:24
so on and so forth that is cool so let's go to the silver layer now in the silver
17:26
go to the silver layer now in the silver
17:26
go to the silver layer now in the silver layer we want to define the Json object
17:29
layer we want to define the Json object
17:29
layer we want to define the Json object using the struct type and struct field
17:31
using the struct type and struct field
17:31
using the struct type and struct field so we're going to call each of the um
17:34
so we're going to call each of the um
17:34
so we're going to call each of the um the keys so we have the um City latitude
17:37
the keys so we have the um City latitude
17:37
the keys so we have the um City latitude longitude and so on and so forth and
17:38
longitude and so on and so forth and
17:38
longitude and so on and so forth and then we have the appropriate data types
17:40
then we have the appropriate data types
17:40
then we have the appropriate data types so I'm going to run this code to define
17:42
so I'm going to run this code to define
17:42
so I'm going to run this code to define the Jon object and this executed nicely
17:45
the Jon object and this executed nicely
17:45
the Jon object and this executed nicely and then we want to move to reading the
17:47
and then we want to move to reading the
17:47
and then we want to move to reading the data from the bronze layer so you can
17:50
data from the bronze layer so you can
17:50
data from the bronze layer so you can see we are reading from this weather.
17:52
see we are reading from this weather.
17:53
see we are reading from this weather. bronze. bronze data table so this is the
17:55
bronze. bronze data table so this is the
17:55
bronze. bronze data table so this is the r stream and then we're going to use the
17:58
r stream and then we're going to use the
17:58
r stream and then we're going to use the width column so want to use this body
18:00
width column so want to use this body
18:00
width column so want to use this body and then we're going to use the call
18:01
and then we're going to use the call
18:02
and then we're going to use the call method to extract the body do cast we're
18:04
method to extract the body do cast we're
18:04
method to extract the body do cast we're going to basically cast the Json object
18:07
going to basically cast the Json object
18:07
going to basically cast the Json object as a string values so we can see we have
18:10
as a string values so we can see we have
18:10
as a string values so we can see we have this this um section here we're using
18:11
this this um section here we're using
18:11
this this um section here we're using the DF do select so we are selecting um
18:14
the DF do select so we are selecting um
18:14
the DF do select so we are selecting um this body do City keys and then we're
18:17
this body do City keys and then we're
18:17
this body do City keys and then we're casting as a string and then we also
18:19
casting as a string and then we also
18:19
casting as a string and then we also aliasing or alas as a city so this is
18:22
aliasing or alas as a city so this is
18:22
aliasing or alas as a city so this is going to be the name of the columns in
18:23
going to be the name of the columns in
18:24
going to be the name of the columns in the table of format so this is how we
18:25
the table of format so this is how we
18:25
the table of format so this is how we can transform the binary format of the
18:28
can transform the binary format of the
18:28
can transform the binary format of the Json data into a table format and I'm
18:31
Json data into a table format and I'm
18:31
Json data into a table format and I'm going to be displaying the DF string
18:33
going to be displaying the DF string
18:34
going to be displaying the DF string which is exactly this variable we
18:35
which is exactly this variable we
18:35
which is exactly this variable we declared here so after doing that we're
18:38
declared here so after doing that we're
18:38
declared here so after doing that we're going to also create a right Stream So
18:41
going to also create a right Stream So
18:41
going to also create a right Stream So for the right stream we have the option
18:43
for the right stream we have the option
18:43
for the right stream we have the option so we want want to create another
18:44
so we want want to create another
18:44
so we want want to create another checkpoint and the checkpoint is going
18:46
checkpoint and the checkpoint is going
18:46
checkpoint and the checkpoint is going to be to the silver layer so I'm going
18:49
to be to the silver layer so I'm going
18:49
to be to the silver layer so I'm going to come to the in the catalog here I'm
18:51
to come to the in the catalog here I'm
18:51
to come to the in the catalog here I'm going to expand the silver layer and I
18:53
going to expand the silver layer and I
18:53
going to expand the silver layer and I can see the silver volume I'm going to
18:55
can see the silver volume I'm going to
18:55
can see the silver volume I'm going to click on this and I want to copy the
18:57
click on this and I want to copy the
18:57
click on this and I want to copy the path so I'm going to come in here inside
18:59
path so I'm going to come in here inside
18:59
path so I'm going to come in here inside the double quote and delete and paste
19:02
the double quote and delete and paste
19:02
the double quote and delete and paste the path so we can see similar to the
19:04
the path so we can see similar to the
19:04
the path so we can see similar to the Bron we have the for SL volumes for SL
19:07
Bron we have the for SL volumes for SL
19:07
Bron we have the for SL volumes for SL weather for/ Silver for/ silver volume
19:10
weather for/ Silver for/ silver volume
19:10
weather for/ Silver for/ silver volume so this is going to be the checkpoints
19:12
so this is going to be the checkpoints
19:12
so this is going to be the checkpoints and then the output is going to be
19:13
and then the output is going to be
19:13
and then the output is going to be appended and the form is going to be a
19:15
appended and the form is going to be a
19:15
appended and the form is going to be a Dela and then to table is going to be
19:17
Dela and then to table is going to be
19:17
Dela and then to table is going to be written to the silver do weather.
19:21
written to the silver do weather.
19:21
written to the silver do weather. silver. silver data so this is going to
19:22
silver. silver data so this is going to
19:22
silver. silver data so this is going to be the three name space again so I'm
19:24
be the three name space again so I'm
19:24
be the three name space again so I'm going to press control enter to run the
19:27
going to press control enter to run the
19:27
going to press control enter to run the code and then let's see what happens and
19:29
code and then let's see what happens and
19:29
code and then let's see what happens and let's see the transform or the cleanse
19:31
let's see the transform or the cleanse
19:31
let's see the transform or the cleanse data in a tabular format so I'm going to
19:34
data in a tabular format so I'm going to
19:34
data in a tabular format so I'm going to screw a little bit and there we go so we
19:37
screw a little bit and there we go so we
19:37
screw a little bit and there we go so we can see our data is now returned in a
19:39
can see our data is now returned in a
19:39
can see our data is now returned in a tabular columnal format so we can see we
19:42
tabular columnal format so we can see we
19:42
tabular columnal format so we can see we have the city column the latitude the
19:44
have the city column the latitude the
19:44
have the city column the latitude the longitude temperature humidity wind
19:47
longitude temperature humidity wind
19:47
longitude temperature humidity wind speed um we have the pressure
19:50
speed um we have the pressure
19:50
speed um we have the pressure precipitation and so on and so forth to
19:52
precipitation and so on and so forth to
19:52
precipitation and so on and so forth to the time stamp now I'm going to expand
19:54
the time stamp now I'm going to expand
19:54
the time stamp now I'm going to expand this and I'm going to see also the input
19:56
this and I'm going to see also the input
19:56
this and I'm going to see also the input and processing rate and so on and so so
19:58
and processing rate and so on and so so
19:58
and processing rate and so on and so so for which is absolutely fantastic now
20:02
for which is absolutely fantastic now
20:02
for which is absolutely fantastic now I'm going to go to the last one which is
20:03
I'm going to go to the last one which is
20:03
I'm going to go to the last one which is the gold layer so for the gold layer
20:05
the gold layer so for the gold layer
20:05
the gold layer so for the gold layer again we're going to be using the spark.
20:08
again we're going to be using the spark.
20:08
again we're going to be using the spark. rstream and the format going to be a
20:10
rstream and the format going to be a
20:10
rstream and the format going to be a Delta and then we're going to call the
20:12
Delta and then we're going to call the
20:12
Delta and then we're going to call the table which is weather. silver. silver
20:14
table which is weather. silver. silver
20:14
table which is weather. silver. silver data table the three name space applied
20:17
data table the three name space applied
20:17
data table the three name space applied and I'm going to just perform this
20:18
and I'm going to just perform this
20:18
and I'm going to just perform this aggregation so we just want to see the
20:20
aggregation so we just want to see the
20:20
aggregation so we just want to see the average temperature average humidity
20:22
average temperature average humidity
20:22
average temperature average humidity average wind speed Group by the city
20:25
average wind speed Group by the city
20:25
average wind speed Group by the city column and this going to be a
20:27
column and this going to be a
20:27
column and this going to be a aggregation a calculation for us and
20:29
aggregation a calculation for us and
20:29
aggregation a calculation for us and then we're going to also display the
20:30
then we're going to also display the
20:30
then we're going to also display the aggregation and we're going to also
20:32
aggregation and we're going to also
20:32
aggregation and we're going to also write the data into this checkpoint so
20:35
write the data into this checkpoint so
20:35
write the data into this checkpoint so I'm going to carefully come back to this
20:37
I'm going to carefully come back to this
20:37
I'm going to carefully come back to this gold expand I can copy the part of the
20:41
gold expand I can copy the part of the
20:42
gold expand I can copy the part of the um the volume I'm going to come here
20:43
um the volume I'm going to come here
20:43
um the volume I'm going to come here carefully select delete and I'm going to
20:46
carefully select delete and I'm going to
20:46
carefully select delete and I'm going to paste that volume path so we're going to
20:48
paste that volume path so we're going to
20:48
paste that volume path so we're going to see we have this part and then for the
20:50
see we have this part and then for the
20:50
see we have this part and then for the output is going to be complete okay but
20:52
output is going to be complete okay but
20:52
output is going to be complete okay but that's the last layer and then this
20:53
that's the last layer and then this
20:53
that's the last layer and then this going to be a Delta format and then this
20:55
going to be a Delta format and then this
20:55
going to be a Delta format and then this going to be written into the weather
20:58
going to be written into the weather
20:58
going to be written into the weather catalog and then the gold schema and the
21:00
catalog and then the gold schema and the
21:00
catalog and then the gold schema and the gold data table so I can go on and
21:03
gold data table so I can go on and
21:03
gold data table so I can go on and execute control enter to run the gold
21:06
execute control enter to run the gold
21:06
execute control enter to run the gold layer and let's see what happens so we
21:10
layer and let's see what happens so we
21:10
layer and let's see what happens so we can see we have the read stream
21:12
can see we have the read stream
21:12
can see we have the read stream initializing and we're going to see our
21:15
initializing and we're going to see our
21:15
initializing and we're going to see our results this um average humidity average
21:18
results this um average humidity average
21:18
results this um average humidity average wind temperature and so on and so for by
21:20
wind temperature and so on and so for by
21:20
wind temperature and so on and so for by city so we going to see that in a moment
21:22
city so we going to see that in a moment
21:22
city so we going to see that in a moment let me just expand so we can see the
21:23
let me just expand so we can see the
21:23
let me just expand so we can see the silver layout so this are the number of
21:26
silver layout so this are the number of
21:26
silver layout so this are the number of the um columns that is the p to. datf
21:29
the um columns that is the p to. datf
21:29
the um columns that is the p to. datf frame. datf frame so let's just wait for
21:31
frame. datf frame so let's just wait for
21:31
frame. datf frame so let's just wait for a moment and we can see this is going on
21:33
a moment and we can see this is going on
21:34
a moment and we can see this is going on and we should see the final results
21:37
and we should see the final results
21:37
and we should see the final results amazing so we can see we have the city
21:40
amazing so we can see we have the city
21:40
amazing so we can see we have the city group by the average temperature average
21:43
group by the average temperature average
21:43
group by the average temperature average humidity average wind speed and the
21:46
humidity average wind speed and the
21:46
humidity average wind speed and the record count which is absolutely
21:48
record count which is absolutely
21:48
record count which is absolutely fantastic I can expand this and we can
21:50
fantastic I can expand this and we can
21:50
fantastic I can expand this and we can also say all of this and dashboard we
21:52
also say all of this and dashboard we
21:52
also say all of this and dashboard we can even come here and see the raw data
21:55
can even come here and see the raw data
21:55
can even come here and see the raw data in form of the Json object that is cool
21:58
in form of the Json object that is cool
21:58
in form of the Json object that is cool now everything is working fine now let's
22:00
now everything is working fine now let's
22:00
now everything is working fine now let's quickly come to this tab and by the way
22:02
quickly come to this tab and by the way
22:02
quickly come to this tab and by the way you can see we have this icon this tells
22:04
you can see we have this icon this tells
22:04
you can see we have this icon this tells you that we have a streaming data going
22:06
you that we have a streaming data going
22:06
you that we have a streaming data going on so I'm not going to interrupt or
22:08
on so I'm not going to interrupt or
22:08
on so I'm not going to interrupt or terminate um this job so I'm going to
22:11
terminate um this job so I'm going to
22:11
terminate um this job so I'm going to come to this tab and then I want to go
22:12
come to this tab and then I want to go
22:12
come to this tab and then I want to go to the compute and I'm sorry I'm going
22:14
to the compute and I'm sorry I'm going
22:14
to the compute and I'm sorry I'm going to go to the catalog and in the catalog
22:16
to go to the catalog and in the catalog
22:16
to go to the catalog and in the catalog I'm going to expand the data now let's
22:18
I'm going to expand the data now let's
22:18
I'm going to expand the data now let's just view the data in the Bron so I'm
22:20
just view the data in the Bron so I'm
22:21
just view the data in the Bron so I'm going to expand now we can see we have
22:22
going to expand now we can see we have
22:22
going to expand now we can see we have tables one and then we have volumes one
22:25
tables one and then we have volumes one
22:25
tables one and then we have volumes one so these are different things so I'm
22:26
so these are different things so I'm
22:26
so these are different things so I'm going to expand the table and when I
22:28
going to expand the table and when I
22:28
going to expand the table and when I click on the I can see the overview so
22:30
click on the I can see the overview so
22:30
click on the I can see the overview so we can see the columns now this is the
22:33
we can see the columns now this is the
22:33
we can see the columns now this is the binary format and let me go to the
22:36
binary format and let me go to the
22:36
binary format and let me go to the silver layer now in the silver layer
22:38
silver layer now in the silver layer
22:38
silver layer now in the silver layer table we're going to see that we have
22:41
table we're going to see that we have
22:41
table we're going to see that we have the names of the columns in a columnal
22:43
the names of the columns in a columnal
22:44
the names of the columns in a columnal method so you have the city latitude and
22:45
method so you have the city latitude and
22:45
method so you have the city latitude and so on and so forth now I'm going to
22:47
so on and so forth now I'm going to
22:47
so on and so forth now I'm going to click on Sample data when I CLI on the
22:49
click on Sample data when I CLI on the
22:49
click on Sample data when I CLI on the sample data I should be able to see at
22:51
sample data I should be able to see at
22:51
sample data I should be able to see at least some of the data and then we can
22:54
least some of the data and then we can
22:54
least some of the data and then we can move on so there we go so this is
22:56
move on so there we go so this is
22:56
move on so there we go so this is everything is looking absolutely
22:57
everything is looking absolutely
22:57
everything is looking absolutely fantastic now we can also go to the
22:59
fantastic now we can also go to the
22:59
fantastic now we can also go to the volume now again so when I go to the
23:01
volume now again so when I go to the
23:01
volume now again so when I go to the volume we can see the details so we have
23:04
volume we can see the details so we have
23:04
volume we can see the details so we have the temp path directory and then we have
23:06
the temp path directory and then we have
23:06
the temp path directory and then we have the commit and then we have all this
23:08
the commit and then we have all this
23:08
the commit and then we have all this metadata and the offset so when I go to
23:10
metadata and the offset so when I go to
23:11
metadata and the offset so when I go to even the gold layer we have similar
23:12
even the gold layer we have similar
23:12
even the gold layer we have similar things let's go to the gold layer let's
23:14
things let's go to the gold layer let's
23:14
things let's go to the gold layer let's see the table for us before we go to the
23:15
see the table for us before we go to the
23:15
see the table for us before we go to the volume so for the table so this is our
23:18
volume so for the table so this is our
23:18
volume so for the table so this is our calculation that we generated and then
23:19
calculation that we generated and then
23:19
calculation that we generated and then we can click on the sample data and then
23:22
we can click on the sample data and then
23:22
we can click on the sample data and then we going to see that um city by average
23:25
we going to see that um city by average
23:25
we going to see that um city by average temperature humidity and so on and so
23:26
temperature humidity and so on and so
23:26
temperature humidity and so on and so forth and I'm want to go to the volumes
23:28
forth and I'm want to go to the volumes
23:29
forth and I'm want to go to the volumes we can come here and we can see all the
23:30
we can come here and we can see all the
23:30
we can come here and we can see all the metadata which is absolutely fantastic
23:34
metadata which is absolutely fantastic
23:34
metadata which is absolutely fantastic now let's go ahead and create a power
23:36
now let's go ahead and create a power
23:36
now let's go ahead and create a power bear report on top of this now to do
23:38
bear report on top of this now to do
23:38
bear report on top of this now to do that is so easy I'm going to come to the
23:41
that is so easy I'm going to come to the
23:41
that is so easy I'm going to come to the marketplace and we have the partner
23:42
marketplace and we have the partner
23:42
marketplace and we have the partner connet which is now in this environment
23:44
connet which is now in this environment
23:44
connet which is now in this environment and you can see we have the DBT Cloud F
23:47
and you can see we have the DBT Cloud F
23:47
and you can see we have the DBT Cloud F Tran tablet desktop and so on and so
23:49
Tran tablet desktop and so on and so
23:49
Tran tablet desktop and so on and so forth and then I'm going to choose the
23:51
forth and then I'm going to choose the
23:51
forth and then I'm going to choose the powerbi click on that now this going to
23:53
powerbi click on that now this going to
23:53
powerbi click on that now this going to ask me to choose the compute that I want
23:55
ask me to choose the compute that I want
23:55
ask me to choose the compute that I want to connect to so I can see the connect
23:58
to connect to so I can see the connect
23:58
to connect to so I can see the connect function fun ality and then I'm going to
24:00
function fun ality and then I'm going to
24:00
function fun ality and then I'm going to browse and choose the weather streaming
24:02
browse and choose the weather streaming
24:02
browse and choose the weather streaming computer that we going to be use for
24:03
computer that we going to be use for
24:04
computer that we going to be use for this job so I'm going to click on
24:05
this job so I'm going to click on
24:05
this job so I'm going to click on download collection file and then it
24:07
download collection file and then it
24:07
download collection file and then it copies down downloaded to my download so
24:09
copies down downloaded to my download so
24:09
copies down downloaded to my download so I can click on that to open so when I
24:11
I can click on that to open so when I
24:11
I can click on that to open so when I click on that this going to launch the
24:13
click on that this going to launch the
24:13
click on that this going to launch the powerbi desktop okay so the first thing
24:16
powerbi desktop okay so the first thing
24:16
powerbi desktop okay so the first thing we're going to see is this environment
24:17
we're going to see is this environment
24:17
we're going to see is this environment which allows us to authenticate in order
24:20
which allows us to authenticate in order
24:20
which allows us to authenticate in order to access the arual data braks now I'm
24:22
to access the arual data braks now I'm
24:22
to access the arual data braks now I'm going to use the active aure active
24:24
going to use the active aure active
24:24
going to use the active aure active directory which is now known as the
24:25
directory which is now known as the
24:26
directory which is now known as the enter ID so I want to make sure I click
24:27
enter ID so I want to make sure I click
24:27
enter ID so I want to make sure I click on the sign I'm going to pick my
24:30
on the sign I'm going to pick my
24:30
on the sign I'm going to pick my account and then I'm going to click on
24:32
account and then I'm going to click on
24:32
account and then I'm going to click on connect so this is going to
24:33
connect so this is going to
24:34
connect so this is going to automatically connect to my data break
24:36
automatically connect to my data break
24:36
automatically connect to my data break cluster and then I can bring the data
24:38
cluster and then I can bring the data
24:38
cluster and then I can bring the data into the powerbi for further um
24:42
into the powerbi for further um
24:42
into the powerbi for further um visualization so we can see we we have
24:44
visualization so we can see we we have
24:44
visualization so we can see we we have in this Navigator we have all of this U
24:47
in this Navigator we have all of this U
24:47
in this Navigator we have all of this U analytics I met store sales I'm going to
24:50
analytics I met store sales I'm going to
24:50
analytics I met store sales I'm going to click on the weather and when I click on
24:52
click on the weather and when I click on
24:52
click on the weather and when I click on the we I'm going to see all the three
24:54
the we I'm going to see all the three
24:54
the we I'm going to see all the three layers including the default so we can
24:56
layers including the default so we can
24:56
layers including the default so we can see the bronse gold silver and the the
24:59
see the bronse gold silver and the the
24:59
see the bronse gold silver and the the default now I'm going to use the sver
25:02
default now I'm going to use the sver
25:02
default now I'm going to use the sver layer in order to create power be inside
25:05
layer in order to create power be inside
25:05
layer in order to create power be inside so I'm going to expand I can see the
25:07
so I'm going to expand I can see the
25:07
so I'm going to expand I can see the table icon so when I click on this I'm
25:08
table icon so when I click on this I'm
25:08
table icon so when I click on this I'm going to see the preview now this might
25:10
going to see the preview now this might
25:10
going to see the preview now this might take up to some maybe one minute
25:13
take up to some maybe one minute
25:13
take up to some maybe one minute depending on the strength of your
25:15
depending on the strength of your
25:15
depending on the strength of your computer so I'm just going to wait and
25:17
computer so I'm just going to wait and
25:17
computer so I'm just going to wait and after the data is up and running I'm
25:19
after the data is up and running I'm
25:19
after the data is up and running I'm going to click on transform data now the
25:21
going to click on transform data now the
25:21
going to click on transform data now the reason why I'm going to trans data is
25:23
reason why I'm going to trans data is
25:23
reason why I'm going to trans data is because all the columns in our table are
25:27
because all the columns in our table are
25:27
because all the columns in our table are being treated as as a string it can say
25:29
being treated as as a string it can say
25:29
being treated as as a string it can say everything is now a string so I'm going
25:31
everything is now a string so I'm going
25:31
everything is now a string so I'm going to actually just perform a little
25:33
to actually just perform a little
25:33
to actually just perform a little transformation in powerbi to transform
25:35
transformation in powerbi to transform
25:35
transformation in powerbi to transform the numerical columns as a whole number
25:37
the numerical columns as a whole number
25:37
the numerical columns as a whole number data types so I'm going to click on this
25:39
data types so I'm going to click on this
25:39
data types so I'm going to click on this and I can go to transform data and this
25:41
and I can go to transform data and this
25:41
and I can go to transform data and this going to automatically launch the data
25:43
going to automatically launch the data
25:43
going to automatically launch the data in the powerbi power query which allows
25:46
in the powerbi power query which allows
25:46
in the powerbi power query which allows us to apply further transformation okay
25:49
us to apply further transformation okay
25:49
us to apply further transformation okay so the data is now in the power query so
25:53
so the data is now in the power query so
25:53
so the data is now in the power query so we can say we have the string Text data
25:55
we can say we have the string Text data
25:55
we can say we have the string Text data types all through now I'm going to S the
25:56
types all through now I'm going to S the
25:56
types all through now I'm going to S the latitude and then I'm going to scroll to
25:58
latitude and then I'm going to scroll to
25:58
latitude and then I'm going to scroll to the right and I'm going to order on the
26:00
the right and I'm going to order on the
26:00
the right and I'm going to order on the shift key and then I want to select the
26:02
shift key and then I want to select the
26:02
shift key and then I want to select the cloud cover so I'm going to right click
26:05
cloud cover so I'm going to right click
26:05
cloud cover so I'm going to right click and I can go to the transform data and
26:07
and I can go to the transform data and
26:07
and I can go to the transform data and I'm going to choose a whole number data
26:09
I'm going to choose a whole number data
26:09
I'm going to choose a whole number data type so this going to automatically
26:11
type so this going to automatically
26:11
type so this going to automatically convert them to a whole number for me
26:12
convert them to a whole number for me
26:12
convert them to a whole number for me which is absolutely cool for um for the
26:15
which is absolutely cool for um for the
26:15
which is absolutely cool for um for the analysis so I'm going to just click on
26:16
analysis so I'm going to just click on
26:16
analysis so I'm going to just click on close and apply so this going to close
26:18
close and apply so this going to close
26:18
close and apply so this going to close the power query and apply the
26:20
the power query and apply the
26:20
the power query and apply the transformation into the powerbi data
26:22
transformation into the powerbi data
26:22
transformation into the powerbi data model and we can start building Insight
26:25
model and we can start building Insight
26:25
model and we can start building Insight around the data that is coming now this
26:28
around the data that is coming now this
26:28
around the data that is coming now this a near real time you know dashboard in
26:31
a near real time you know dashboard in
26:31
a near real time you know dashboard in power we're going to create so there we
26:33
power we're going to create so there we
26:33
power we're going to create so there we go we have the silver data table and can
26:36
go we have the silver data table and can
26:36
go we have the silver data table and can see at the bottom you have the storage
26:37
see at the bottom you have the storage
26:37
see at the bottom you have the storage mode which is the direct query now this
26:39
mode which is the direct query now this
26:39
mode which is the direct query now this simply means for every visual that we
26:41
simply means for every visual that we
26:41
simply means for every visual that we build in powerbi the powerbi was going
26:43
build in powerbi the powerbi was going
26:43
build in powerbi the powerbi was going to send the queries to the data brakes
26:45
to send the queries to the data brakes
26:45
to send the queries to the data brakes and then the data break will release the
26:47
and then the data break will release the
26:47
and then the data break will release the query back to the power and then we have
26:48
query back to the power and then we have
26:48
query back to the power and then we have the visualization so I'm going to expand
26:50
the visualization so I'm going to expand
26:50
the visualization so I'm going to expand this and then we can see we have this um
26:53
this and then we can see we have this um
26:54
this and then we can see we have this um a categorical field like the city and
26:56
a categorical field like the city and
26:56
a categorical field like the city and the weather condition and let's want to
26:58
the weather condition and let's want to
26:58
the weather condition and let's want to see the City by temperature I'm going to
27:01
see the City by temperature I'm going to
27:01
see the City by temperature I'm going to just drag the temperature in there and
27:02
just drag the temperature in there and
27:02
just drag the temperature in there and this going to create an implicit measure
27:04
this going to create an implicit measure
27:04
this going to create an implicit measure no problems now I'm going to just as
27:06
no problems now I'm going to just as
27:07
no problems now I'm going to just as that's going on I'm going to copy and
27:09
that's going on I'm going to copy and
27:09
that's going on I'm going to copy and paste just duplicate and let's I want to
27:11
paste just duplicate and let's I want to
27:11
paste just duplicate and let's I want to see the um weather condition by the wind
27:17
see the um weather condition by the wind
27:17
see the um weather condition by the wind speed and I'm just going to wait now
27:19
speed and I'm just going to wait now
27:19
speed and I'm just going to wait now this might take a while because the
27:21
this might take a while because the
27:21
this might take a while because the power there will have to send the query
27:23
power there will have to send the query
27:23
power there will have to send the query to data Brakes in order to generate the
27:25
to data Brakes in order to generate the
27:25
to data Brakes in order to generate the inside I'm just going to wait and see
27:27
inside I'm just going to wait and see
27:27
inside I'm just going to wait and see okay so we have the result back so we
27:30
okay so we have the result back so we
27:30
okay so we have the result back so we can see we have the sum of temperature
27:32
can see we have the sum of temperature
27:32
can see we have the sum of temperature by city so we can say that MOS is at the
27:34
by city so we can say that MOS is at the
27:34
by city so we can say that MOS is at the top of the list and follow by eou
27:37
top of the list and follow by eou
27:37
top of the list and follow by eou Seattle Chicago and so on and so forth
27:39
Seattle Chicago and so on and so forth
27:39
Seattle Chicago and so on and so forth now I can turn this Su of wind speed by
27:42
now I can turn this Su of wind speed by
27:42
now I can turn this Su of wind speed by weather condition to let's say um a
27:45
weather condition to let's say um a
27:45
weather condition to let's say um a donut chart so you can you know use
27:47
donut chart so you can you know use
27:47
donut chart so you can you know use whatever visualization you want to use
27:48
whatever visualization you want to use
27:48
whatever visualization you want to use in power and then we can see we have the
27:51
in power and then we can see we have the
27:51
in power and then we can see we have the weather condition such as storming foggy
27:54
weather condition such as storming foggy
27:54
weather condition such as storming foggy cloudy and so on and so forth now this
27:56
cloudy and so on and so forth now this
27:56
cloudy and so on and so forth now this is a m real now when I click or I can
28:00
is a m real now when I click or I can
28:00
is a m real now when I click or I can see now we have this value 162 for mosco
28:02
see now we have this value 162 for mosco
28:02
see now we have this value 162 for mosco now I can click on refresh now when I
28:04
now I can click on refresh now when I
28:04
now I can click on refresh now when I click on refresh I'm just going to wait
28:06
click on refresh I'm just going to wait
28:06
click on refresh I'm just going to wait for some couple of moments and the data
28:08
for some couple of moments and the data
28:08
for some couple of moments and the data will automatically you know update based
28:10
will automatically you know update based
28:11
will automatically you know update based on the data coming into the um events
28:14
on the data coming into the um events
28:15
on the data coming into the um events ofs and then into our data brakes
28:18
ofs and then into our data brakes
28:18
ofs and then into our data brakes notebook so I'm just going to wait and
28:20
notebook so I'm just going to wait and
28:20
notebook so I'm just going to wait and let's see what happens this should
28:22
let's see what happens this should
28:22
let's see what happens this should change from 162 to something else okay
28:25
change from 162 to something else okay
28:25
change from 162 to something else okay you can see it has changed so in is now
28:27
you can see it has changed so in is now
28:27
you can see it has changed so in is now at the top of the list so this is 182
28:30
at the top of the list so this is 182
28:30
at the top of the list so this is 182 187 now this is how we can build a Power
28:34
187 now this is how we can build a Power
28:34
187 now this is how we can build a Power report on top of the streaming data from
28:38
report on top of the streaming data from
28:38
report on top of the streaming data from data breakes so we can say we've gone
28:40
data breakes so we can say we've gone
28:40
data breakes so we can say we've gone through all of these steps we started
28:41
through all of these steps we started
28:41
through all of these steps we started off by going to the vs code start by
28:43
off by going to the vs code start by
28:43
off by going to the vs code start by generating data and then we send the
28:45
generating data and then we send the
28:45
generating data and then we send the data into the event H that we created
28:48
data into the event H that we created
28:48
data into the event H that we created and then we establish connection to the
28:50
and then we establish connection to the
28:50
and then we establish connection to the event up and then we implement or
28:52
event up and then we implement or
28:52
event up and then we implement or architect the medon architecture so
28:54
architect the medon architecture so
28:54
architect the medon architecture so these are all the code you can go
28:56
these are all the code you can go
28:56
these are all the code you can go through it now if you need um this all
28:59
through it now if you need um this all
28:59
through it now if you need um this all this theour data that I use I'm going to
29:01
this theour data that I use I'm going to
29:01
this theour data that I use I'm going to put the link of my repo this um um
29:04
put the link of my repo this um um
29:04
put the link of my repo this um um repository into this video so you can
29:06
repository into this video so you can
29:06
repository into this video so you can watch and by the way please subscribe to
29:09
watch and by the way please subscribe to
29:09
watch and by the way please subscribe to this channel if enjoy this video and
29:11
this channel if enjoy this video and
29:11
this channel if enjoy this video and then you've learned from it so thank you
29:12
then you've learned from it so thank you
29:12
then you've learned from it so thank you for watching bye for now

Real-Time Streaming Data in Databricks [End to End Project]

c-sharpcorner_com

What is New in GPT 5

Top 10 Vibe Coding Tools to Try Now

The Cloud Show with Magnus Mårtensson ft. Mahesh Chand - Ep: 73

Best practices in building Cloud distributed systems | Software Architecture Conference

The Cloud Show with Magnus Mårtensson ft. Barbara Forbes - Ep: 67

Crafting Resilient Integration Solutions | Software Architecture Conference

Integration of Blob Storage with D365 CRM Using Azure Data Factory - MVP Show ft. Venkata Subbarao

Learn DynamoDB Data Modeling

Connecting the dots with PowerAutomate, MS Graph and other cool stuff!

Azure DevOps Pipelines with Docker - MVP Show ft. Thiago Vivas

Boosting Productivity with .NET Source Generators

The RIGHT Way to Write Unit Tests for Domain-Driven Design applications

Up next in 10

Real-Time Streaming Data in Databricks [End to End Project]

c-sharpcorner_com