![Video thumbnail for Real-Time Streaming Data in Databricks [End to End Project]](https://video-meta.open.video/resized-posters/vNJ7kI0BdI2/res640.webp?cb=1)
Up next in 10
🚀In this comprehensive end-to-end data engineering video, I demonstrated how to architect real-time streaming data solutions in Databricks, starting from weather data captured in VS Code, seamlessly integrated with Azure Event Hubs. The setup leverages the robust capabilities of the Medallion Architecture and the advanced features of Unity Catalog to ensure efficient data management and processing.In this video, you will learn how to:
👉 Create Event Hubs and Share Access Policies
👉 Generate weather streaming data using Python in VS Code
👉 Send the events to Azure Event Hubs
👉 Create Databricks Cluster and Install Maven Library to authenticate access Azure EventHub
👉 Create unity catalog, schemas and volumes using medallion architecture
👉 Read and write streams across the bronze, silver and gold layers
👉Access Power BI Desktop in Marketplace Partners Connect tab
👉 Build near real-time Power BI report using DirectQuery
Show More Show Less View Video Transcript
0:00
realtime data streaming in date braks is
0:03
realtime data streaming in date braks is
0:03
realtime data streaming in date braks is a powerful capability which allows
0:05
a powerful capability which allows
0:05
a powerful capability which allows organizations to process and analyze
0:07
organizations to process and analyze
0:07
organizations to process and analyze data as it flows in efficiently and
0:10
data as it flows in efficiently and
0:10
data as it flows in efficiently and seamlessly by leveraging aart SP
0:13
seamlessly by leveraging aart SP
0:13
seamlessly by leveraging aart SP structure streaming within date braks
0:15
structure streaming within date braks
0:15
structure streaming within date braks users can build robust scalable and
0:17
users can build robust scalable and
0:17
users can build robust scalable and fault tolerant data pipelines to handle
0:20
fault tolerant data pipelines to handle
0:20
fault tolerant data pipelines to handle a continuous stream of data from various
0:22
a continuous stream of data from various
0:22
a continuous stream of data from various sour such as a event apps iot devices
0:26
sour such as a event apps iot devices
0:26
sour such as a event apps iot devices social media logs and more in this new
0:28
social media logs and more in this new
0:28
social media logs and more in this new endtoend video I'm going to work you
0:30
endtoend video I'm going to work you
0:30
endtoend video I'm going to work you through how we can architect real time
0:32
through how we can architect real time
0:32
through how we can architect real time data stream using the vs code python
0:34
data stream using the vs code python
0:34
data stream using the vs code python generated data a event ups and data
0:37
generated data a event ups and data
0:37
generated data a event ups and data bricks notebook we're going to see how
0:38
bricks notebook we're going to see how
0:38
bricks notebook we're going to see how we can use the multi op which is
0:40
we can use the multi op which is
0:40
we can use the multi op which is popularly known as the medallon
0:42
popularly known as the medallon
0:42
popularly known as the medallon architecture combined with unity catalog
0:44
architecture combined with unity catalog
0:44
architecture combined with unity catalog in for seamless real time dat to
0:46
in for seamless real time dat to
0:46
in for seamless real time dat to streaming so let's get started let's go
0:49
streaming so let's get started let's go
0:49
streaming so let's get started let's go ahead and create event or and share
0:51
ahead and create event or and share
0:51
ahead and create event or and share access signature to send and receive
0:53
access signature to send and receive
0:53
access signature to send and receive event from the source so I'm going to
0:55
event from the source so I'm going to
0:55
event from the source so I'm going to come to the portal. ao.com I can see
0:59
come to the portal. ao.com I can see
0:59
come to the portal. ao.com I can see I've got this Cornerstone analytics
1:01
I've got this Cornerstone analytics
1:01
I've got this Cornerstone analytics event up already created so click on
1:03
event up already created so click on
1:03
event up already created so click on that and then I'm going to come to under
1:06
that and then I'm going to come to under
1:06
that and then I'm going to come to under entities click on event UPS now for now
1:09
entities click on event UPS now for now
1:09
entities click on event UPS now for now I've got no event up created so I'm
1:11
I've got no event up created so I'm
1:11
I've got no event up created so I'm going to click on this plus sign to
1:12
going to click on this plus sign to
1:12
going to click on this plus sign to create a new event up and then I'm going
1:15
create a new event up and then I'm going
1:15
create a new event up and then I'm going to call this one weather streaming data
1:18
to call this one weather streaming data
1:18
to call this one weather streaming data and then for the partition count I'm
1:20
and then for the partition count I'm
1:20
and then for the partition count I'm just going to leave it at the default
1:22
just going to leave it at the default
1:22
just going to leave it at the default one which is enough for this
1:23
one which is enough for this
1:23
one which is enough for this demonstration so for the retention
1:25
demonstration so for the retention
1:25
demonstration so for the retention policy this going to be just for 1 hour
1:27
policy this going to be just for 1 hour
1:27
policy this going to be just for 1 hour retention time and this can be up to 24
1:30
retention time and this can be up to 24
1:30
retention time and this can be up to 24 hours that is a single day so this is
1:32
hours that is a single day so this is
1:32
hours that is a single day so this is going to be a delete clean up policy so
1:35
going to be a delete clean up policy so
1:35
going to be a delete clean up policy so all this is fine I'm going to click on
1:37
all this is fine I'm going to click on
1:37
all this is fine I'm going to click on reval plus create and this is going to
1:39
reval plus create and this is going to
1:39
reval plus create and this is going to be validated and I'm going to click on
1:41
be validated and I'm going to click on
1:41
be validated and I'm going to click on create so the event up is now going to
1:44
create so the event up is now going to
1:44
create so the event up is now going to be created in just a matter of few
1:46
be created in just a matter of few
1:46
be created in just a matter of few seconds so we can see we have the
1:48
seconds so we can see we have the
1:48
seconds so we can see we have the weather streaming data created now I'm
1:51
weather streaming data created now I'm
1:51
weather streaming data created now I'm going to click on that and I'm going to
1:53
going to click on that and I'm going to
1:53
going to click on that and I'm going to go to the shared access policies so I'm
1:56
go to the shared access policies so I'm
1:56
go to the shared access policies so I'm going to create a new policy so I'm
1:58
going to create a new policy so I'm
1:58
going to create a new policy so I'm going to call this one where
2:00
going to call this one where
2:00
going to call this one where SAS key and then I'm going to click on
2:03
SAS key and then I'm going to click on
2:03
SAS key and then I'm going to click on manage now the manage allows you to send
2:05
manage now the manage allows you to send
2:05
manage now the manage allows you to send and listen to events so I'm going to
2:07
and listen to events so I'm going to
2:07
and listen to events so I'm going to click on create so this is now created
2:10
click on create so this is now created
2:10
click on create so this is now created now I'm going to click on that and then
2:12
now I'm going to click on that and then
2:12
now I'm going to click on that and then we can see we've got the primary key
2:14
we can see we've got the primary key
2:14
we can see we've got the primary key secondary key the primary connection
2:16
secondary key the primary connection
2:16
secondary key the primary connection string and the secondary connection
2:18
string and the secondary connection
2:18
string and the secondary connection string I'm going to need the combination
2:20
string I'm going to need the combination
2:20
string I'm going to need the combination of the primary connection string and the
2:23
of the primary connection string and the
2:23
of the primary connection string and the weather streaming data that is the event
2:26
weather streaming data that is the event
2:26
weather streaming data that is the event name in order to authenticate and be
2:28
name in order to authenticate and be
2:28
name in order to authenticate and be able to send and listen to events now
2:30
able to send and listen to events now
2:30
able to send and listen to events now this is sorted I'm going to come to my
2:32
this is sorted I'm going to come to my
2:32
this is sorted I'm going to come to my vs code in order to generate sample data
2:35
vs code in order to generate sample data
2:35
vs code in order to generate sample data so I'm going to stick with this Azure
2:37
so I'm going to stick with this Azure
2:37
so I'm going to stick with this Azure event or folder which is on my local
2:40
event or folder which is on my local
2:40
event or folder which is on my local laptop and I want to create a file so
2:42
laptop and I want to create a file so
2:42
laptop and I want to create a file so I'm going to call this on
2:44
I'm going to call this on
2:44
I'm going to call this on weather dat. py now this is going to be
2:47
weather dat. py now this is going to be
2:47
weather dat. py now this is going to be a python code that going to be using in
2:49
a python code that going to be using in
2:49
a python code that going to be using in a now I'm going to paste the code now
2:52
a now I'm going to paste the code now
2:52
a now I'm going to paste the code now before I do the pasting of the code I'm
2:53
before I do the pasting of the code I'm
2:53
before I do the pasting of the code I'm going to just quickly come me and this
2:55
going to just quickly come me and this
2:55
going to just quickly come me and this is my GitHub so I've got this real time
2:57
is my GitHub so I've got this real time
2:57
is my GitHub so I've got this real time streaming data in Azure data breaks
2:59
streaming data in Azure data breaks
2:59
streaming data in Azure data breaks notbook book reposit so I'm going to put
3:01
notbook book reposit so I'm going to put
3:01
notbook book reposit so I'm going to put the link of this repo in order for you
3:03
the link of this repo in order for you
3:03
the link of this repo in order for you to probably play with the sample data
3:05
to probably play with the sample data
3:05
to probably play with the sample data set and the python that is The Notebook
3:07
set and the python that is The Notebook
3:07
set and the python that is The Notebook code so I'm going to go to this weather
3:09
code so I'm going to go to this weather
3:09
code so I'm going to go to this weather data python code generator in vs code
3:12
data python code generator in vs code
3:12
data python code generator in vs code and of course I can see all the code in
3:13
and of course I can see all the code in
3:13
and of course I can see all the code in it so I'm going to click on copy raw
3:17
it so I'm going to click on copy raw
3:17
it so I'm going to click on copy raw file as soon as I click on that I can
3:19
file as soon as I click on that I can
3:19
file as soon as I click on that I can come back to the vs code editor So
3:21
come back to the vs code editor So
3:21
come back to the vs code editor So within this weather data. py file I'm
3:24
within this weather data. py file I'm
3:24
within this weather data. py file I'm going to paste the code so contrl V to
3:27
going to paste the code so contrl V to
3:27
going to paste the code so contrl V to paste it's really important first you
3:29
paste it's really important first you
3:29
paste it's really important first you install the PIP install of the a event
3:33
install the PIP install of the a event
3:33
install the PIP install of the a event up so which is already done I'm going to
3:34
up so which is already done I'm going to
3:34
up so which is already done I'm going to put a link of video on how to pip
3:36
put a link of video on how to pip
3:36
put a link of video on how to pip install the a event or in the vs code
3:39
install the a event or in the vs code
3:39
install the a event or in the vs code editor so after you install the aure
3:41
editor so after you install the aure
3:41
editor so after you install the aure event up now it's really important to
3:43
event up now it's really important to
3:44
event up now it's really important to import the necessary Library so we've
3:45
import the necessary Library so we've
3:45
import the necessary Library so we've got all the libraries and I'm going to
3:47
got all the libraries and I'm going to
3:47
got all the libraries and I'm going to also provide the event up connection
3:49
also provide the event up connection
3:49
also provide the event up connection string and event up name so got we've
3:51
string and event up name so got we've
3:51
string and event up name so got we've got this connection string which is
3:53
got this connection string which is
3:53
got this connection string which is equal to the primary connection stram
3:55
equal to the primary connection stram
3:55
equal to the primary connection stram that we're going to fetch in a moment
3:56
that we're going to fetch in a moment
3:56
that we're going to fetch in a moment and then we also have the event up name
3:58
and then we also have the event up name
3:58
and then we also have the event up name which is going to be the weather
3:59
which is going to be the weather
4:00
which is going to be the weather streaming data and then we're going to
4:01
streaming data and then we're going to
4:01
streaming data and then we're going to create a producer client to send message
4:04
create a producer client to send message
4:04
create a producer client to send message to the event up in AEL so we have this
4:06
to the event up in AEL so we have this
4:06
to the event up in AEL so we have this producer variable equals to event up
4:09
producer variable equals to event up
4:09
producer variable equals to event up producer client do from connection
4:11
producer client do from connection
4:11
producer client do from connection string and inside this open and close
4:13
string and inside this open and close
4:13
string and inside this open and close bracket we have this concore St equal to
4:17
bracket we have this concore St equal to
4:17
bracket we have this concore St equal to whatever value is that we pass into this
4:19
whatever value is that we pass into this
4:19
whatever value is that we pass into this connection string and then for the event
4:21
connection string and then for the event
4:21
connection string and then for the event opening it's going to equal to whatever
4:22
opening it's going to equal to whatever
4:22
opening it's going to equal to whatever is the name of the event as we Define in
4:25
is the name of the event as we Define in
4:25
is the name of the event as we Define in this variable and then we have this list
4:28
this variable and then we have this list
4:28
this variable and then we have this list of City so let me just move this down a
4:30
of City so let me just move this down a
4:30
of City so let me just move this down a little bit so that we can see what's
4:31
little bit so that we can see what's
4:31
little bit so that we can see what's going on so we want to generate some
4:33
going on so we want to generate some
4:33
going on so we want to generate some random data so we going to have this
4:35
random data so we going to have this
4:35
random data so we going to have this city such as New York Los Angeles
4:37
city such as New York Los Angeles
4:37
city such as New York Los Angeles Chicago Houston and so on and so forth
4:40
Chicago Houston and so on and so forth
4:40
Chicago Houston and so on and so forth and I'm going to generate the latitude
4:42
and I'm going to generate the latitude
4:42
and I'm going to generate the latitude and the longitude and then at the bottom
4:44
and the longitude and then at the bottom
4:44
and the longitude and then at the bottom I'm going to generate randomly Peak the
4:46
I'm going to generate randomly Peak the
4:46
I'm going to generate randomly Peak the cities including the latitude and
4:48
cities including the latitude and
4:48
cities including the latitude and longitude and then we're going to return
4:50
longitude and then we're going to return
4:50
longitude and then we're going to return inside this dictionary we have the city
4:53
inside this dictionary we have the city
4:53
inside this dictionary we have the city latitude longitude temperature humidity
4:56
latitude longitude temperature humidity
4:56
latitude longitude temperature humidity wind speed pressure precipitation CL
4:59
wind speed pressure precipitation CL
4:59
wind speed pressure precipitation CL cover weather condition now these are
5:01
cover weather condition now these are
5:01
cover weather condition now these are similar to columns in a tabular format
5:04
similar to columns in a tabular format
5:04
similar to columns in a tabular format and then at the bottom here we're going
5:06
and then at the bottom here we're going
5:06
and then at the bottom here we're going to create a batch of events so we have
5:08
to create a batch of events so we have
5:08
to create a batch of events so we have this variable which is equal to
5:09
this variable which is equal to
5:09
this variable which is equal to producer. create batch and then we're
5:12
producer. create batch and then we're
5:12
producer. create batch and then we're going to also generate a sample weather
5:14
going to also generate a sample weather
5:14
going to also generate a sample weather data so this going to be a variable
5:15
data so this going to be a variable
5:15
data so this going to be a variable weather data and it's going to be
5:17
weather data and it's going to be
5:17
weather data and it's going to be looking at the generatore weather data
5:21
looking at the generatore weather data
5:21
looking at the generatore weather data and then we're going to store the data
5:22
and then we're going to store the data
5:22
and then we're going to store the data as a Json so we can see we have the
5:24
as a Json so we can see we have the
5:24
as a Json so we can see we have the json. dumps and it's going to be feding
5:26
json. dumps and it's going to be feding
5:26
json. dumps and it's going to be feding into this intermediate um variable name
5:29
into this intermediate um variable name
5:29
into this intermediate um variable name and I'm going to add Json format data to
5:31
and I'm going to add Json format data to
5:32
and I'm going to add Json format data to the batch so we have this eventor dataor
5:34
the batch so we have this eventor dataor
5:34
the batch so we have this eventor dataor batch. add method and then we have this
5:37
batch. add method and then we have this
5:37
batch. add method and then we have this event data that we imported equal to the
5:41
event data that we imported equal to the
5:41
event data that we imported equal to the messages and we're going to send and
5:43
messages and we're going to send and
5:43
messages and we're going to send and this going to be going on every 5
5:46
this going to be going on every 5
5:46
this going to be going on every 5 Seconds now this cannot run except we
5:49
Seconds now this cannot run except we
5:49
Seconds now this cannot run except we provide the connection string and the
5:52
provide the connection string and the
5:52
provide the connection string and the event name so let's quickly come to the
5:54
event name so let's quickly come to the
5:54
event name so let's quickly come to the a portal and I want to fetch the
5:57
a portal and I want to fetch the
5:57
a portal and I want to fetch the connection string so I can copy
6:00
connection string so I can copy
6:00
connection string so I can copy the primary um connection string copy
6:03
the primary um connection string copy
6:03
the primary um connection string copy into the cboard come back to the vs code
6:04
into the cboard come back to the vs code
6:04
into the cboard come back to the vs code editor and I'm going to select this
6:06
editor and I'm going to select this
6:06
editor and I'm going to select this place holder value and I'm going to
6:08
place holder value and I'm going to
6:08
place holder value and I'm going to paste that string in there and then for
6:12
paste that string in there and then for
6:12
paste that string in there and then for the name of our event that is the
6:15
the name of our event that is the
6:15
the name of our event that is the weather streaming data I'm going to just
6:16
weather streaming data I'm going to just
6:16
weather streaming data I'm going to just copy this and control C and I'm going to
6:19
copy this and control C and I'm going to
6:19
copy this and control C and I'm going to come carefully into this placeholder so
6:21
come carefully into this placeholder so
6:21
come carefully into this placeholder so this going to be our event or name so
6:24
this going to be our event or name so
6:24
this going to be our event or name so select this place orderer value and I'm
6:26
select this place orderer value and I'm
6:26
select this place orderer value and I'm going to press contrl V to paste so
6:28
going to press contrl V to paste so
6:28
going to press contrl V to paste so that's all we need to do now I can click
6:30
that's all we need to do now I can click
6:30
that's all we need to do now I can click on this play button and let's see what
6:33
on this play button and let's see what
6:33
on this play button and let's see what happened in the terminal window at the
6:36
happened in the terminal window at the
6:36
happened in the terminal window at the bottom so we can see we have data being
6:39
bottom so we can see we have data being
6:39
bottom so we can see we have data being sent so this is the very first data that
6:41
sent so this is the very first data that
6:41
sent so this is the very first data that is being sent to our event or in aure
6:44
is being sent to our event or in aure
6:44
is being sent to our event or in aure platform which is absolutely cool now
6:46
platform which is absolutely cool now
6:46
platform which is absolutely cool now this is sorted now I'm going to come
6:48
this is sorted now I'm going to come
6:48
this is sorted now I'm going to come back to the portal and I want to check
6:50
back to the portal and I want to check
6:50
back to the portal and I want to check the data I'm going to come to the
6:52
the data I'm going to come to the
6:52
the data I'm going to come to the overview in the overview I'm going to
6:54
overview in the overview I'm going to
6:54
overview in the overview I'm going to scroll down and you can see we have
6:56
scroll down and you can see we have
6:56
scroll down and you can see we have evidence of live data coming into this
6:59
evidence of live data coming into this
6:59
evidence of live data coming into this event up which is absolutely fantastic
7:02
event up which is absolutely fantastic
7:02
event up which is absolutely fantastic next we want to create the databas
7:04
next we want to create the databas
7:04
next we want to create the databas cluster and install the Marvin library
7:06
cluster and install the Marvin library
7:06
cluster and install the Marvin library that allows us to connect to the a event
7:09
that allows us to connect to the a event
7:09
that allows us to connect to the a event or and begin to stream the data Into The
7:12
or and begin to stream the data Into The
7:12
or and begin to stream the data Into The datab Brak platform so I'm going to come
7:14
datab Brak platform so I'm going to come
7:14
datab Brak platform so I'm going to come here and then I'm going to go to the
7:16
here and then I'm going to go to the
7:16
here and then I'm going to go to the compute and then I going to create a new
7:18
compute and then I going to create a new
7:18
compute and then I going to create a new compute and I'm going to call this one
7:21
compute and I'm going to call this one
7:21
compute and I'm going to call this one um weather streaming compute you can use
7:24
um weather streaming compute you can use
7:24
um weather streaming compute you can use whatever name you like as the cluster
7:26
whatever name you like as the cluster
7:26
whatever name you like as the cluster name so compute and once I'm done with
7:29
name so compute and once I'm done with
7:29
name so compute and once I'm done with the name name press enter to commit and
7:31
the name name press enter to commit and
7:31
the name name press enter to commit and then for the policy I'm going to choose
7:32
then for the policy I'm going to choose
7:32
then for the policy I'm going to choose a single n and then for the performance
7:35
a single n and then for the performance
7:35
a single n and then for the performance now I'm going to come here and I'm going
7:37
now I'm going to come here and I'm going
7:37
now I'm going to come here and I'm going to point to the 12.2 LTS which is what
7:40
to point to the 12.2 LTS which is what
7:40
to point to the 12.2 LTS which is what is available to access the aure event up
7:43
is available to access the aure event up
7:43
is available to access the aure event up for now so we have this runtime 12.2 and
7:46
for now so we have this runtime 12.2 and
7:46
for now so we have this runtime 12.2 and the scalar is 2.2 and the spark is
7:49
the scalar is 2.2 and the spark is
7:49
the scalar is 2.2 and the spark is 3.3.2 so I'm going to just leave this
7:51
3.3.2 so I'm going to just leave this
7:51
3.3.2 so I'm going to just leave this full th acceleration there on for the N
7:54
full th acceleration there on for the N
7:54
full th acceleration there on for the N TP I'm going to just leave this standard
7:56
TP I'm going to just leave this standard
7:56
TP I'm going to just leave this standard D4 dsor V5 and then this going terminate
8:00
D4 dsor V5 and then this going terminate
8:00
D4 dsor V5 and then this going terminate let's say every 30 minutes in order to
8:01
let's say every 30 minutes in order to
8:01
let's say every 30 minutes in order to save some money now it's really
8:03
save some money now it's really
8:03
save some money now it's really recommended that if your computer is not
8:04
recommended that if your computer is not
8:05
recommended that if your computer is not being used please always terminate it in
8:06
being used please always terminate it in
8:07
being used please always terminate it in order not to incure unnecessary cost so
8:09
order not to incure unnecessary cost so
8:09
order not to incure unnecessary cost so having done all of this we can see the
8:11
having done all of this we can see the
8:11
having done all of this we can see the summary so we can see we have just a
8:12
summary so we can see we have just a
8:12
summary so we can see we have just a single driver and the run time is going
8:14
single driver and the run time is going
8:14
single driver and the run time is going to be this 12.2 and then we have the
8:17
to be this 12.2 and then we have the
8:17
to be this 12.2 and then we have the unity catalog enable which is really
8:20
unity catalog enable which is really
8:20
unity catalog enable which is really important now I'm going to click on
8:22
important now I'm going to click on
8:22
important now I'm going to click on create
8:23
create
8:23
create compute and then this is going to start
8:25
compute and then this is going to start
8:26
compute and then this is going to start spinning I'm going to come to the
8:27
spinning I'm going to come to the
8:27
spinning I'm going to come to the libraries as that is going on I want to
8:30
libraries as that is going on I want to
8:30
libraries as that is going on I want to install the Marvin Library so I'm going
8:32
install the Marvin Library so I'm going
8:32
install the Marvin Library so I'm going to click on install new and then I want
8:34
to click on install new and then I want
8:34
to click on install new and then I want to pick the Marvin Library space and
8:36
to pick the Marvin Library space and
8:36
to pick the Marvin Library space and then we need to provide the coordinate
8:38
then we need to provide the coordinate
8:38
then we need to provide the coordinate that allows us to establish connection
8:39
that allows us to establish connection
8:39
that allows us to establish connection to the Azure event of so if you know the
8:42
to the Azure event of so if you know the
8:42
to the Azure event of so if you know the coordinate you can just copy paste
8:43
coordinate you can just copy paste
8:43
coordinate you can just copy paste directly but you can always set for the
8:45
directly but you can always set for the
8:45
directly but you can always set for the packages sock here and then I want to
8:47
packages sock here and then I want to
8:47
packages sock here and then I want to set for Azure I'm sorry event Ops event
8:52
set for Azure I'm sorry event Ops event
8:52
set for Azure I'm sorry event Ops event Ops and then Das um spark so we're going
8:55
Ops and then Das um spark so we're going
8:55
Ops and then Das um spark so we're going to say this list has been narrowed down
8:57
to say this list has been narrowed down
8:57
to say this list has been narrowed down so we can say we have a event of spar
8:59
so we can say we have a event of spar
8:59
so we can say we have a event of spar parent and we have this EV up spark 2.12
9:03
parent and we have this EV up spark 2.12
9:03
parent and we have this EV up spark 2.12 so I'm going to pick this latest so
9:05
so I'm going to pick this latest so
9:05
so I'm going to pick this latest so click on select and then we have the
9:07
click on select and then we have the
9:07
click on select and then we have the coordinate already provisioned or just
9:10
coordinate already provisioned or just
9:10
coordinate already provisioned or just um produced a now I'm just going to
9:12
um produced a now I'm just going to
9:12
um produced a now I'm just going to click on install and I'm going to wait
9:15
click on install and I'm going to wait
9:16
click on install and I'm going to wait for that library to be installed and
9:18
for that library to be installed and
9:18
for that library to be installed and then I'm going to wait for the weather
9:20
then I'm going to wait for the weather
9:20
then I'm going to wait for the weather streaming Compu to be ready to proceed
9:24
streaming Compu to be ready to proceed
9:24
streaming Compu to be ready to proceed all right so the weather streaming
9:26
all right so the weather streaming
9:26
all right so the weather streaming compute is up and running and then we
9:28
compute is up and running and then we
9:28
compute is up and running and then we have the library in stored from Marvin
9:31
have the library in stored from Marvin
9:31
have the library in stored from Marvin so this simply means that we've
9:32
so this simply means that we've
9:32
so this simply means that we've establish connection to our aure event
9:34
establish connection to our aure event
9:34
establish connection to our aure event up and we can start our streaming data
9:36
up and we can start our streaming data
9:36
up and we can start our streaming data by the way if you yet to subcribe to the
9:37
by the way if you yet to subcribe to the
9:38
by the way if you yet to subcribe to the Channel please click on the Subscribe
9:39
Channel please click on the Subscribe
9:39
Channel please click on the Subscribe button and tole on Bell icon to be
9:41
button and tole on Bell icon to be
9:41
button and tole on Bell icon to be inform of new videos thank you so let's
9:44
inform of new videos thank you so let's
9:44
inform of new videos thank you so let's go ahead and input our notebook from the
9:46
go ahead and input our notebook from the
9:46
go ahead and input our notebook from the G up so I'm going to come here and then
9:48
G up so I'm going to come here and then
9:48
G up so I'm going to come here and then I want to go to this data breaks real
9:51
I want to go to this data breaks real
9:51
I want to go to this data breaks real time streaming and then you can see the
9:53
time streaming and then you can see the
9:53
time streaming and then you can see the notebook in here and I'm just going to
9:55
notebook in here and I'm just going to
9:55
notebook in here and I'm just going to click on raw and then I want to copy the
9:58
click on raw and then I want to copy the
9:58
click on raw and then I want to copy the URL for the so contrl C and I can return
10:01
URL for the so contrl C and I can return
10:01
URL for the so contrl C and I can return back here so I'm going to go to
10:03
back here so I'm going to go to
10:03
back here so I'm going to go to workspace and then I want to go to
10:05
workspace and then I want to go to
10:05
workspace and then I want to go to workspaces and then I want to go to user
10:08
workspaces and then I want to go to user
10:08
workspaces and then I want to go to user and I'm going to select my enter ID and
10:10
and I'm going to select my enter ID and
10:10
and I'm going to select my enter ID and then I'm going to click on this kebab
10:11
then I'm going to click on this kebab
10:11
then I'm going to click on this kebab and then I can say the input
10:12
and then I can say the input
10:12
and then I can say the input functionality I can see the URL so I'm
10:15
functionality I can see the URL so I'm
10:15
functionality I can see the URL so I'm going to paste the URL and then click on
10:17
going to paste the URL and then click on
10:17
going to paste the URL and then click on import all right so we have this data
10:20
import all right so we have this data
10:20
import all right so we have this data brakes real time streaming data notebook
10:23
brakes real time streaming data notebook
10:23
brakes real time streaming data notebook and I'm going to use the Python language
10:25
and I'm going to use the Python language
10:25
and I'm going to use the Python language for our data processing so I'm going to
10:27
for our data processing so I'm going to
10:27
for our data processing so I'm going to follow the multi op that that is The
10:29
follow the multi op that that is The
10:29
follow the multi op that that is The Medallion architecture that consists of
10:31
Medallion architecture that consists of
10:31
Medallion architecture that consists of the bronze silver and the gold layers so
10:35
the bronze silver and the gold layers so
10:35
the bronze silver and the gold layers so we're going to start by using the r
10:36
we're going to start by using the r
10:36
we're going to start by using the r stream to read the data from the event
10:38
stream to read the data from the event
10:38
stream to read the data from the event up and then we're going to write into
10:40
up and then we're going to write into
10:40
up and then we're going to write into this Bron layer which is going to be the
10:42
this Bron layer which is going to be the
10:42
this Bron layer which is going to be the raw data and then we're going to move
10:44
raw data and then we're going to move
10:44
raw data and then we're going to move the data from the Bron layer into the
10:46
the data from the Bron layer into the
10:46
the data from the Bron layer into the Civil layer and then we're going to
10:47
Civil layer and then we're going to
10:47
Civil layer and then we're going to transform the data by cleaning the data
10:49
transform the data by cleaning the data
10:50
transform the data by cleaning the data and then we're going to perform some
10:52
and then we're going to perform some
10:52
and then we're going to perform some aggregation that some calculation in the
10:54
aggregation that some calculation in the
10:54
aggregation that some calculation in the gold layer and afterwards we're going to
10:56
gold layer and afterwards we're going to
10:56
gold layer and afterwards we're going to create a Power report using a direct
10:58
create a Power report using a direct
10:58
create a Power report using a direct query so I'm going to just move that a
11:01
query so I'm going to just move that a
11:01
query so I'm going to just move that a little bit and then I've got some notes
11:02
little bit and then I've got some notes
11:02
little bit and then I've got some notes at the bottom here so you can take your
11:03
at the bottom here so you can take your
11:03
at the bottom here so you can take your time and look at them now it's really
11:05
time and look at them now it's really
11:05
time and look at them now it's really important we attach the cluster to the
11:08
important we attach the cluster to the
11:08
important we attach the cluster to the specific book which I've already done
11:10
specific book which I've already done
11:10
specific book which I've already done and then we're going to start by
11:12
and then we're going to start by
11:12
and then we're going to start by importing some of the required libraries
11:14
importing some of the required libraries
11:14
importing some of the required libraries so we've got all of this so from ppar
11:16
so we've got all of this so from ppar
11:16
so we've got all of this so from ppar seo. functions so we're going to import
11:18
seo. functions so we're going to import
11:19
seo. functions so we're going to import Star and then we're going to also import
11:21
Star and then we're going to also import
11:21
Star and then we're going to also import the call from Json struct type struct
11:24
the call from Json struct type struct
11:24
the call from Json struct type struct field string type double type and so on
11:26
field string type double type and so on
11:26
field string type double type and so on and so forth so I'm just going to press
11:28
and so forth so I'm just going to press
11:28
and so forth so I'm just going to press control enter to run this code and this
11:31
control enter to run this code and this
11:31
control enter to run this code and this executed nicely now I'm going to screw
11:33
executed nicely now I'm going to screw
11:33
executed nicely now I'm going to screw down so we're going to set up the Azure
11:35
down so we're going to set up the Azure
11:35
down so we're going to set up the Azure event of connection so we can see we
11:37
event of connection so we can see we
11:37
event of connection so we can see we have the connection string and I'm going
11:38
have the connection string and I'm going
11:38
have the connection string and I'm going to pass in the primary connection string
11:40
to pass in the primary connection string
11:40
to pass in the primary connection string and then the event of name and I'm going
11:43
and then the event of name and I'm going
11:43
and then the event of name and I'm going to quickly fetch that so I'm going to
11:44
to quickly fetch that so I'm going to
11:44
to quickly fetch that so I'm going to come back here and then I want to go to
11:46
come back here and then I want to go to
11:46
come back here and then I want to go to the Shar access policies and I want to
11:48
the Shar access policies and I want to
11:48
the Shar access policies and I want to go to the weather SAS key and I'm going
11:51
go to the weather SAS key and I'm going
11:51
go to the weather SAS key and I'm going to copy the primary connection string
11:53
to copy the primary connection string
11:53
to copy the primary connection string similar to what we use in the vs code
11:55
similar to what we use in the vs code
11:55
similar to what we use in the vs code editor so come back in here so I'm going
11:57
editor so come back in here so I'm going
11:57
editor so come back in here so I'm going to select this placeholder value
11:59
to select this placeholder value
11:59
to select this placeholder value carefully and I'm going to paste the
12:01
carefully and I'm going to paste the
12:01
carefully and I'm going to paste the string and then for the event up name
12:04
string and then for the event up name
12:04
string and then for the event up name I'm going to just come back here and I'm
12:05
I'm going to just come back here and I'm
12:05
I'm going to just come back here and I'm going to copy this weather streaming
12:07
going to copy this weather streaming
12:07
going to copy this weather streaming data come back here I'm going to get rid
12:10
data come back here I'm going to get rid
12:10
data come back here I'm going to get rid of this placeholder value and inside the
12:13
of this placeholder value and inside the
12:13
of this placeholder value and inside the double quote I'm going to paste the
12:15
double quote I'm going to paste the
12:15
double quote I'm going to paste the event up name and then we have the
12:17
event up name and then we have the
12:17
event up name and then we have the connection string dictionary now in the
12:19
connection string dictionary now in the
12:19
connection string dictionary now in the dictionary we have this event ops.
12:20
dictionary we have this event ops.
12:21
dictionary we have this event ops. connection string and I'm going to be
12:22
connection string and I'm going to be
12:22
connection string and I'm going to be using the event UPS util to encrypt the
12:25
using the event UPS util to encrypt the
12:25
using the event UPS util to encrypt the connection string and then we're going
12:27
connection string and then we're going
12:27
connection string and then we're going to also pass in the event up name which
12:29
to also pass in the event up name which
12:29
to also pass in the event up name which is the same thing as what we have here
12:31
is the same thing as what we have here
12:31
is the same thing as what we have here and similar to the connection string
12:32
and similar to the connection string
12:32
and similar to the connection string here we just pass it inside this encrypt
12:35
here we just pass it inside this encrypt
12:35
here we just pass it inside this encrypt and I'm going to go ahead and run this
12:37
and I'm going to go ahead and run this
12:37
and I'm going to go ahead and run this code so press control enter and this
12:39
code so press control enter and this
12:39
code so press control enter and this executed nicely so let's move to how we
12:42
executed nicely so let's move to how we
12:42
executed nicely so let's move to how we can create the medon architecture want
12:44
can create the medon architecture want
12:44
can create the medon architecture want to create catalog schemas and volumes so
12:47
to create catalog schemas and volumes so
12:47
to create catalog schemas and volumes so first we're going to define the storage
12:49
first we're going to define the storage
12:49
first we're going to define the storage location for our catalog so we have this
12:51
location for our catalog so we have this
12:51
location for our catalog so we have this storage location and we're going to
12:53
storage location and we're going to
12:53
storage location and we're going to points to the storage location AB FSS
12:56
points to the storage location AB FSS
12:56
points to the storage location AB FSS path so how do we get this I'm going to
12:58
path so how do we get this I'm going to
12:58
path so how do we get this I'm going to come to this tab and then I want to go
13:00
come to this tab and then I want to go
13:00
come to this tab and then I want to go to the catalog and in the catalog click
13:02
to the catalog and in the catalog click
13:02
to the catalog and in the catalog click on this settings and I want to go to you
13:05
on this settings and I want to go to you
13:05
on this settings and I want to go to you know create external locations so we
13:06
know create external locations so we
13:06
know create external locations so we have this URL so I'm going to move this
13:08
have this URL so I'm going to move this
13:08
have this URL so I'm going to move this expand a little bit and this is our
13:10
expand a little bit and this is our
13:10
expand a little bit and this is our storage location so I'm going to copy
13:12
storage location so I'm going to copy
13:12
storage location so I'm going to copy that come back in here get rid of this
13:15
that come back in here get rid of this
13:15
that come back in here get rid of this place hold value inside this single
13:17
place hold value inside this single
13:17
place hold value inside this single quote control V and oops okay let me
13:20
quote control V and oops okay let me
13:20
quote control V and oops okay let me just move this line properly delete okay
13:23
just move this line properly delete okay
13:23
just move this line properly delete okay so this is the historic um location and
13:25
so this is the historic um location and
13:25
so this is the historic um location and we're going to create a unit catalog
13:27
we're going to create a unit catalog
13:27
we're going to create a unit catalog named weather and this going to be a
13:29
named weather and this going to be a
13:29
named weather and this going to be a managed location so we can see we have
13:31
managed location so we can see we have
13:31
managed location so we can see we have this store location pass inside this
13:33
this store location pass inside this
13:33
this store location pass inside this call bracket in it and this is going to
13:35
call bracket in it and this is going to
13:35
call bracket in it and this is going to be where the unit catalog will be
13:37
be where the unit catalog will be
13:37
be where the unit catalog will be managed and then we're going to create
13:39
managed and then we're going to create
13:39
managed and then we're going to create bronze layer schema so this is named
13:42
bronze layer schema so this is named
13:42
bronze layer schema so this is named bronze so this is going to be within the
13:44
bronze so this is going to be within the
13:44
bronze so this is going to be within the weather unit catalog so we're going to
13:46
weather unit catalog so we're going to
13:46
weather unit catalog so we're going to create the silver and the gold layer
13:48
create the silver and the gold layer
13:48
create the silver and the gold layer within the same weather unit catalog and
13:50
within the same weather unit catalog and
13:50
within the same weather unit catalog and then we're going to create the bronze
13:52
then we're going to create the bronze
13:52
then we're going to create the bronze layer for checkpoints now the checkpoint
13:55
layer for checkpoints now the checkpoint
13:55
layer for checkpoints now the checkpoint is going to be used for processing our
13:56
is going to be used for processing our
13:56
is going to be used for processing our data coming from the a event up to
13:58
data coming from the a event up to
13:58
data coming from the a event up to ensure for tolerance and data recovery
14:01
ensure for tolerance and data recovery
14:01
ensure for tolerance and data recovery in case of failures so we're going to
14:03
in case of failures so we're going to
14:03
in case of failures so we're going to create the volume for each of the layers
14:06
create the volume for each of the layers
14:06
create the volume for each of the layers so we have this create volume which is
14:09
so we have this create volume which is
14:09
so we have this create volume which is within the weather catalog bronze.
14:12
within the weather catalog bronze.
14:12
within the weather catalog bronze. bronze volume so we can see we have a
14:14
bronze volume so we can see we have a
14:14
bronze volume so we can see we have a spark SQ so the same thing for the
14:16
spark SQ so the same thing for the
14:16
spark SQ so the same thing for the silver so create volume weather. silver.
14:19
silver so create volume weather. silver.
14:19
silver so create volume weather. silver. silver volume and then going to also
14:21
silver volume and then going to also
14:21
silver volume and then going to also create volume weather go good volume so
14:23
create volume weather go good volume so
14:24
create volume weather go good volume so I'm going to just go ahead and run this
14:25
I'm going to just go ahead and run this
14:25
I'm going to just go ahead and run this code so this should work nicely
14:30
code so this should work nicely
14:30
code so this should work nicely okay so it's executed nicely now we can
14:33
okay so it's executed nicely now we can
14:33
okay so it's executed nicely now we can see that catalog in it now going to
14:35
see that catalog in it now going to
14:35
see that catalog in it now going to click on this refresh and then we should
14:37
click on this refresh and then we should
14:37
click on this refresh and then we should see the weather catalog so I'm going to
14:40
see the weather catalog so I'm going to
14:40
see the weather catalog so I'm going to expand and we can see we have the bronze
14:42
expand and we can see we have the bronze
14:42
expand and we can see we have the bronze can expand and I'm going to see the
14:44
can expand and I'm going to see the
14:44
can expand and I'm going to see the volume we just created for the BRS and
14:46
volume we just created for the BRS and
14:46
volume we just created for the BRS and then when I expand the go I'm going to
14:47
then when I expand the go I'm going to
14:47
then when I expand the go I'm going to see the volume and so on and so forth
14:49
see the volume and so on and so forth
14:49
see the volume and so on and so forth that is absolutely cool now let's go to
14:52
that is absolutely cool now let's go to
14:52
that is absolutely cool now let's go to the data processing that is the bronze
14:54
the data processing that is the bronze
14:54
the data processing that is the bronze layer so for the bronze layer we're
14:55
layer so for the bronze layer we're
14:55
layer so for the bronze layer we're going to use the spark. rstream
14:59
going to use the spark. rstream
14:59
going to use the spark. rstream format now the format is going to be the
15:01
format now the format is going to be the
15:01
format now the format is going to be the event up now because we are reading the
15:03
event up now because we are reading the
15:03
event up now because we are reading the data from the event of that you know we
15:05
data from the event of that you know we
15:05
data from the event of that you know we sending data from the vs code in so we
15:07
sending data from the vs code in so we
15:07
sending data from the vs code in so we have the event up as the format and the
15:09
have the event up as the format and the
15:09
have the event up as the format and the option is going to be the connection
15:11
option is going to be the connection
15:11
option is going to be the connection configuration dictionary that we
15:13
configuration dictionary that we
15:13
configuration dictionary that we specified in the Azure event connection
15:16
specified in the Azure event connection
15:16
specified in the Azure event connection here this value connection config and
15:18
here this value connection config and
15:18
here this value connection config and then we're going to move to the load so
15:20
then we're going to move to the load so
15:20
then we're going to move to the load so we're going to load and after loading
15:23
we're going to load and after loading
15:23
we're going to load and after loading we're going to just display the DF so
15:25
we're going to just display the DF so
15:25
we're going to just display the DF so after reading the stream we're going to
15:27
after reading the stream we're going to
15:27
after reading the stream we're going to write the stream into a checkpoint so
15:29
write the stream into a checkpoint so
15:30
write the stream into a checkpoint so the checkpoint is going to be the bronze
15:32
the checkpoint is going to be the bronze
15:32
the checkpoint is going to be the bronze volume so I'm going to provide the
15:35
volume so I'm going to provide the
15:35
volume so I'm going to provide the volume path for the bronze as the
15:37
volume path for the bronze as the
15:37
volume path for the bronze as the checkpoint so I'm going to click on this
15:38
checkpoint so I'm going to click on this
15:38
checkpoint so I'm going to click on this to expand and I can see the bronze
15:40
to expand and I can see the bronze
15:40
to expand and I can see the bronze volume now this is the icon for volume
15:42
volume now this is the icon for volume
15:42
volume now this is the icon for volume in dat breaks environment if you don't
15:43
in dat breaks environment if you don't
15:43
in dat breaks environment if you don't know and I'm going to click on the skab
15:46
know and I'm going to click on the skab
15:46
know and I'm going to click on the skab and then I want to copy the volume path
15:48
and then I want to copy the volume path
15:48
and then I want to copy the volume path so I'm going to come inside this place
15:50
so I'm going to come inside this place
15:50
so I'm going to come inside this place holder value carefully control V so
15:52
holder value carefully control V so
15:52
holder value carefully control V so that's going to be the checkpoint in
15:54
that's going to be the checkpoint in
15:54
that's going to be the checkpoint in order to ensure we have F tolerance and
15:57
order to ensure we have F tolerance and
15:57
order to ensure we have F tolerance and data recovery in case of failure and
15:59
data recovery in case of failure and
15:59
data recovery in case of failure and then the output is going to be a pain
16:01
then the output is going to be a pain
16:01
then the output is going to be a pain and then the format going to be in form
16:03
and then the format going to be in form
16:03
and then the format going to be in form of Delta and then to table so this going
16:05
of Delta and then to table so this going
16:05
of Delta and then to table so this going to be written into the weather. bronze.
16:10
to be written into the weather. bronze.
16:10
to be written into the weather. bronze. bronze data now this is going to be the
16:12
bronze data now this is going to be the
16:12
bronze data now this is going to be the name of the table the name of the schema
16:14
name of the table the name of the schema
16:14
name of the table the name of the schema and the name of the catalog so we have
16:16
and the name of the catalog so we have
16:16
and the name of the catalog so we have the three name space so I can go on and
16:18
the three name space so I can go on and
16:18
the three name space so I can go on and control enter to run the code and let's
16:21
control enter to run the code and let's
16:21
control enter to run the code and let's see what happens okay so we can see
16:24
see what happens okay so we can see
16:24
see what happens okay so we can see stream initializing so now I'm just
16:27
stream initializing so now I'm just
16:27
stream initializing so now I'm just going to wait for some couple of moments
16:29
going to wait for some couple of moments
16:29
going to wait for some couple of moments and I'm going to expand this display
16:30
and I'm going to expand this display
16:30
and I'm going to expand this display query one or let me just expand this one
16:32
query one or let me just expand this one
16:32
query one or let me just expand this one first now we're going to see that we
16:34
first now we're going to see that we
16:34
first now we're going to see that we have the input F processing rate and
16:37
have the input F processing rate and
16:37
have the input F processing rate and then we have the records per seconds and
16:39
then we have the records per seconds and
16:39
then we have the records per seconds and then we have the batch duration in
16:40
then we have the batch duration in
16:40
then we have the batch duration in seconds so we can see data is now coming
16:43
seconds so we can see data is now coming
16:43
seconds so we can see data is now coming into our bronze um volume of the bronze
16:46
into our bronze um volume of the bronze
16:46
into our bronze um volume of the bronze um layer and then we can see we have
16:48
um layer and then we can see we have
16:48
um layer and then we can see we have this data in form of binary format now
16:50
this data in form of binary format now
16:50
this data in form of binary format now this actually difficult to say so for
16:52
this actually difficult to say so for
16:52
this actually difficult to say so for example we can see we have the body
16:54
example we can see we have the body
16:54
example we can see we have the body column we have the paration and then we
16:56
column we have the paration and then we
16:56
column we have the paration and then we have the offset we have the sequence
16:58
have the offset we have the sequence
16:58
have the offset we have the sequence number the ened time the publisher
17:01
number the ened time the publisher
17:01
number the ened time the publisher Partition by properties and so on and so
17:03
Partition by properties and so on and so
17:03
Partition by properties and so on and so forth now we're going to conver the
17:05
forth now we're going to conver the
17:05
forth now we're going to conver the event scent which is in binary format to
17:07
event scent which is in binary format to
17:07
event scent which is in binary format to a tabular format and that's what we're
17:09
a tabular format and that's what we're
17:09
a tabular format and that's what we're going to do in the silver layer so I'm
17:12
going to do in the silver layer so I'm
17:12
going to do in the silver layer so I'm going to scroll down and as you can see
17:14
going to scroll down and as you can see
17:14
going to scroll down and as you can see even before we go on we can see this
17:16
even before we go on we can see this
17:16
even before we go on we can see this actually growing so this is actually a
17:18
actually growing so this is actually a
17:18
actually growing so this is actually a real time data processing so this going
17:19
real time data processing so this going
17:19
real time data processing so this going to be increasing from 13 rows to 14 and
17:22
to be increasing from 13 rows to 14 and
17:22
to be increasing from 13 rows to 14 and so on and so forth that is cool so let's
17:24
so on and so forth that is cool so let's
17:24
so on and so forth that is cool so let's go to the silver layer now in the silver
17:26
go to the silver layer now in the silver
17:26
go to the silver layer now in the silver layer we want to define the Json object
17:29
layer we want to define the Json object
17:29
layer we want to define the Json object using the struct type and struct field
17:31
using the struct type and struct field
17:31
using the struct type and struct field so we're going to call each of the um
17:34
so we're going to call each of the um
17:34
so we're going to call each of the um the keys so we have the um City latitude
17:37
the keys so we have the um City latitude
17:37
the keys so we have the um City latitude longitude and so on and so forth and
17:38
longitude and so on and so forth and
17:38
longitude and so on and so forth and then we have the appropriate data types
17:40
then we have the appropriate data types
17:40
then we have the appropriate data types so I'm going to run this code to define
17:42
so I'm going to run this code to define
17:42
so I'm going to run this code to define the Jon object and this executed nicely
17:45
the Jon object and this executed nicely
17:45
the Jon object and this executed nicely and then we want to move to reading the
17:47
and then we want to move to reading the
17:47
and then we want to move to reading the data from the bronze layer so you can
17:50
data from the bronze layer so you can
17:50
data from the bronze layer so you can see we are reading from this weather.
17:52
see we are reading from this weather.
17:53
see we are reading from this weather. bronze. bronze data table so this is the
17:55
bronze. bronze data table so this is the
17:55
bronze. bronze data table so this is the r stream and then we're going to use the
17:58
r stream and then we're going to use the
17:58
r stream and then we're going to use the width column so want to use this body
18:00
width column so want to use this body
18:00
width column so want to use this body and then we're going to use the call
18:01
and then we're going to use the call
18:02
and then we're going to use the call method to extract the body do cast we're
18:04
method to extract the body do cast we're
18:04
method to extract the body do cast we're going to basically cast the Json object
18:07
going to basically cast the Json object
18:07
going to basically cast the Json object as a string values so we can see we have
18:10
as a string values so we can see we have
18:10
as a string values so we can see we have this this um section here we're using
18:11
this this um section here we're using
18:11
this this um section here we're using the DF do select so we are selecting um
18:14
the DF do select so we are selecting um
18:14
the DF do select so we are selecting um this body do City keys and then we're
18:17
this body do City keys and then we're
18:17
this body do City keys and then we're casting as a string and then we also
18:19
casting as a string and then we also
18:19
casting as a string and then we also aliasing or alas as a city so this is
18:22
aliasing or alas as a city so this is
18:22
aliasing or alas as a city so this is going to be the name of the columns in
18:23
going to be the name of the columns in
18:24
going to be the name of the columns in the table of format so this is how we
18:25
the table of format so this is how we
18:25
the table of format so this is how we can transform the binary format of the
18:28
can transform the binary format of the
18:28
can transform the binary format of the Json data into a table format and I'm
18:31
Json data into a table format and I'm
18:31
Json data into a table format and I'm going to be displaying the DF string
18:33
going to be displaying the DF string
18:34
going to be displaying the DF string which is exactly this variable we
18:35
which is exactly this variable we
18:35
which is exactly this variable we declared here so after doing that we're
18:38
declared here so after doing that we're
18:38
declared here so after doing that we're going to also create a right Stream So
18:41
going to also create a right Stream So
18:41
going to also create a right Stream So for the right stream we have the option
18:43
for the right stream we have the option
18:43
for the right stream we have the option so we want want to create another
18:44
so we want want to create another
18:44
so we want want to create another checkpoint and the checkpoint is going
18:46
checkpoint and the checkpoint is going
18:46
checkpoint and the checkpoint is going to be to the silver layer so I'm going
18:49
to be to the silver layer so I'm going
18:49
to be to the silver layer so I'm going to come to the in the catalog here I'm
18:51
to come to the in the catalog here I'm
18:51
to come to the in the catalog here I'm going to expand the silver layer and I
18:53
going to expand the silver layer and I
18:53
going to expand the silver layer and I can see the silver volume I'm going to
18:55
can see the silver volume I'm going to
18:55
can see the silver volume I'm going to click on this and I want to copy the
18:57
click on this and I want to copy the
18:57
click on this and I want to copy the path so I'm going to come in here inside
18:59
path so I'm going to come in here inside
18:59
path so I'm going to come in here inside the double quote and delete and paste
19:02
the double quote and delete and paste
19:02
the double quote and delete and paste the path so we can see similar to the
19:04
the path so we can see similar to the
19:04
the path so we can see similar to the Bron we have the for SL volumes for SL
19:07
Bron we have the for SL volumes for SL
19:07
Bron we have the for SL volumes for SL weather for/ Silver for/ silver volume
19:10
weather for/ Silver for/ silver volume
19:10
weather for/ Silver for/ silver volume so this is going to be the checkpoints
19:12
so this is going to be the checkpoints
19:12
so this is going to be the checkpoints and then the output is going to be
19:13
and then the output is going to be
19:13
and then the output is going to be appended and the form is going to be a
19:15
appended and the form is going to be a
19:15
appended and the form is going to be a Dela and then to table is going to be
19:17
Dela and then to table is going to be
19:17
Dela and then to table is going to be written to the silver do weather.
19:21
written to the silver do weather.
19:21
written to the silver do weather. silver. silver data so this is going to
19:22
silver. silver data so this is going to
19:22
silver. silver data so this is going to be the three name space again so I'm
19:24
be the three name space again so I'm
19:24
be the three name space again so I'm going to press control enter to run the
19:27
going to press control enter to run the
19:27
going to press control enter to run the code and then let's see what happens and
19:29
code and then let's see what happens and
19:29
code and then let's see what happens and let's see the transform or the cleanse
19:31
let's see the transform or the cleanse
19:31
let's see the transform or the cleanse data in a tabular format so I'm going to
19:34
data in a tabular format so I'm going to
19:34
data in a tabular format so I'm going to screw a little bit and there we go so we
19:37
screw a little bit and there we go so we
19:37
screw a little bit and there we go so we can see our data is now returned in a
19:39
can see our data is now returned in a
19:39
can see our data is now returned in a tabular columnal format so we can see we
19:42
tabular columnal format so we can see we
19:42
tabular columnal format so we can see we have the city column the latitude the
19:44
have the city column the latitude the
19:44
have the city column the latitude the longitude temperature humidity wind
19:47
longitude temperature humidity wind
19:47
longitude temperature humidity wind speed um we have the pressure
19:50
speed um we have the pressure
19:50
speed um we have the pressure precipitation and so on and so forth to
19:52
precipitation and so on and so forth to
19:52
precipitation and so on and so forth to the time stamp now I'm going to expand
19:54
the time stamp now I'm going to expand
19:54
the time stamp now I'm going to expand this and I'm going to see also the input
19:56
this and I'm going to see also the input
19:56
this and I'm going to see also the input and processing rate and so on and so so
19:58
and processing rate and so on and so so
19:58
and processing rate and so on and so so for which is absolutely fantastic now
20:02
for which is absolutely fantastic now
20:02
for which is absolutely fantastic now I'm going to go to the last one which is
20:03
I'm going to go to the last one which is
20:03
I'm going to go to the last one which is the gold layer so for the gold layer
20:05
the gold layer so for the gold layer
20:05
the gold layer so for the gold layer again we're going to be using the spark.
20:08
again we're going to be using the spark.
20:08
again we're going to be using the spark. rstream and the format going to be a
20:10
rstream and the format going to be a
20:10
rstream and the format going to be a Delta and then we're going to call the
20:12
Delta and then we're going to call the
20:12
Delta and then we're going to call the table which is weather. silver. silver
20:14
table which is weather. silver. silver
20:14
table which is weather. silver. silver data table the three name space applied
20:17
data table the three name space applied
20:17
data table the three name space applied and I'm going to just perform this
20:18
and I'm going to just perform this
20:18
and I'm going to just perform this aggregation so we just want to see the
20:20
aggregation so we just want to see the
20:20
aggregation so we just want to see the average temperature average humidity
20:22
average temperature average humidity
20:22
average temperature average humidity average wind speed Group by the city
20:25
average wind speed Group by the city
20:25
average wind speed Group by the city column and this going to be a
20:27
column and this going to be a
20:27
column and this going to be a aggregation a calculation for us and
20:29
aggregation a calculation for us and
20:29
aggregation a calculation for us and then we're going to also display the
20:30
then we're going to also display the
20:30
then we're going to also display the aggregation and we're going to also
20:32
aggregation and we're going to also
20:32
aggregation and we're going to also write the data into this checkpoint so
20:35
write the data into this checkpoint so
20:35
write the data into this checkpoint so I'm going to carefully come back to this
20:37
I'm going to carefully come back to this
20:37
I'm going to carefully come back to this gold expand I can copy the part of the
20:41
gold expand I can copy the part of the
20:42
gold expand I can copy the part of the um the volume I'm going to come here
20:43
um the volume I'm going to come here
20:43
um the volume I'm going to come here carefully select delete and I'm going to
20:46
carefully select delete and I'm going to
20:46
carefully select delete and I'm going to paste that volume path so we're going to
20:48
paste that volume path so we're going to
20:48
paste that volume path so we're going to see we have this part and then for the
20:50
see we have this part and then for the
20:50
see we have this part and then for the output is going to be complete okay but
20:52
output is going to be complete okay but
20:52
output is going to be complete okay but that's the last layer and then this
20:53
that's the last layer and then this
20:53
that's the last layer and then this going to be a Delta format and then this
20:55
going to be a Delta format and then this
20:55
going to be a Delta format and then this going to be written into the weather
20:58
going to be written into the weather
20:58
going to be written into the weather catalog and then the gold schema and the
21:00
catalog and then the gold schema and the
21:00
catalog and then the gold schema and the gold data table so I can go on and
21:03
gold data table so I can go on and
21:03
gold data table so I can go on and execute control enter to run the gold
21:06
execute control enter to run the gold
21:06
execute control enter to run the gold layer and let's see what happens so we
21:10
layer and let's see what happens so we
21:10
layer and let's see what happens so we can see we have the read stream
21:12
can see we have the read stream
21:12
can see we have the read stream initializing and we're going to see our
21:15
initializing and we're going to see our
21:15
initializing and we're going to see our results this um average humidity average
21:18
results this um average humidity average
21:18
results this um average humidity average wind temperature and so on and so for by
21:20
wind temperature and so on and so for by
21:20
wind temperature and so on and so for by city so we going to see that in a moment
21:22
city so we going to see that in a moment
21:22
city so we going to see that in a moment let me just expand so we can see the
21:23
let me just expand so we can see the
21:23
let me just expand so we can see the silver layout so this are the number of
21:26
silver layout so this are the number of
21:26
silver layout so this are the number of the um columns that is the p to. datf
21:29
the um columns that is the p to. datf
21:29
the um columns that is the p to. datf frame. datf frame so let's just wait for
21:31
frame. datf frame so let's just wait for
21:31
frame. datf frame so let's just wait for a moment and we can see this is going on
21:33
a moment and we can see this is going on
21:34
a moment and we can see this is going on and we should see the final results
21:37
and we should see the final results
21:37
and we should see the final results amazing so we can see we have the city
21:40
amazing so we can see we have the city
21:40
amazing so we can see we have the city group by the average temperature average
21:43
group by the average temperature average
21:43
group by the average temperature average humidity average wind speed and the
21:46
humidity average wind speed and the
21:46
humidity average wind speed and the record count which is absolutely
21:48
record count which is absolutely
21:48
record count which is absolutely fantastic I can expand this and we can
21:50
fantastic I can expand this and we can
21:50
fantastic I can expand this and we can also say all of this and dashboard we
21:52
also say all of this and dashboard we
21:52
also say all of this and dashboard we can even come here and see the raw data
21:55
can even come here and see the raw data
21:55
can even come here and see the raw data in form of the Json object that is cool
21:58
in form of the Json object that is cool
21:58
in form of the Json object that is cool now everything is working fine now let's
22:00
now everything is working fine now let's
22:00
now everything is working fine now let's quickly come to this tab and by the way
22:02
quickly come to this tab and by the way
22:02
quickly come to this tab and by the way you can see we have this icon this tells
22:04
you can see we have this icon this tells
22:04
you can see we have this icon this tells you that we have a streaming data going
22:06
you that we have a streaming data going
22:06
you that we have a streaming data going on so I'm not going to interrupt or
22:08
on so I'm not going to interrupt or
22:08
on so I'm not going to interrupt or terminate um this job so I'm going to
22:11
terminate um this job so I'm going to
22:11
terminate um this job so I'm going to come to this tab and then I want to go
22:12
come to this tab and then I want to go
22:12
come to this tab and then I want to go to the compute and I'm sorry I'm going
22:14
to the compute and I'm sorry I'm going
22:14
to the compute and I'm sorry I'm going to go to the catalog and in the catalog
22:16
to go to the catalog and in the catalog
22:16
to go to the catalog and in the catalog I'm going to expand the data now let's
22:18
I'm going to expand the data now let's
22:18
I'm going to expand the data now let's just view the data in the Bron so I'm
22:20
just view the data in the Bron so I'm
22:21
just view the data in the Bron so I'm going to expand now we can see we have
22:22
going to expand now we can see we have
22:22
going to expand now we can see we have tables one and then we have volumes one
22:25
tables one and then we have volumes one
22:25
tables one and then we have volumes one so these are different things so I'm
22:26
so these are different things so I'm
22:26
so these are different things so I'm going to expand the table and when I
22:28
going to expand the table and when I
22:28
going to expand the table and when I click on the I can see the overview so
22:30
click on the I can see the overview so
22:30
click on the I can see the overview so we can see the columns now this is the
22:33
we can see the columns now this is the
22:33
we can see the columns now this is the binary format and let me go to the
22:36
binary format and let me go to the
22:36
binary format and let me go to the silver layer now in the silver layer
22:38
silver layer now in the silver layer
22:38
silver layer now in the silver layer table we're going to see that we have
22:41
table we're going to see that we have
22:41
table we're going to see that we have the names of the columns in a columnal
22:43
the names of the columns in a columnal
22:44
the names of the columns in a columnal method so you have the city latitude and
22:45
method so you have the city latitude and
22:45
method so you have the city latitude and so on and so forth now I'm going to
22:47
so on and so forth now I'm going to
22:47
so on and so forth now I'm going to click on Sample data when I CLI on the
22:49
click on Sample data when I CLI on the
22:49
click on Sample data when I CLI on the sample data I should be able to see at
22:51
sample data I should be able to see at
22:51
sample data I should be able to see at least some of the data and then we can
22:54
least some of the data and then we can
22:54
least some of the data and then we can move on so there we go so this is
22:56
move on so there we go so this is
22:56
move on so there we go so this is everything is looking absolutely
22:57
everything is looking absolutely
22:57
everything is looking absolutely fantastic now we can also go to the
22:59
fantastic now we can also go to the
22:59
fantastic now we can also go to the volume now again so when I go to the
23:01
volume now again so when I go to the
23:01
volume now again so when I go to the volume we can see the details so we have
23:04
volume we can see the details so we have
23:04
volume we can see the details so we have the temp path directory and then we have
23:06
the temp path directory and then we have
23:06
the temp path directory and then we have the commit and then we have all this
23:08
the commit and then we have all this
23:08
the commit and then we have all this metadata and the offset so when I go to
23:10
metadata and the offset so when I go to
23:11
metadata and the offset so when I go to even the gold layer we have similar
23:12
even the gold layer we have similar
23:12
even the gold layer we have similar things let's go to the gold layer let's
23:14
things let's go to the gold layer let's
23:14
things let's go to the gold layer let's see the table for us before we go to the
23:15
see the table for us before we go to the
23:15
see the table for us before we go to the volume so for the table so this is our
23:18
volume so for the table so this is our
23:18
volume so for the table so this is our calculation that we generated and then
23:19
calculation that we generated and then
23:19
calculation that we generated and then we can click on the sample data and then
23:22
we can click on the sample data and then
23:22
we can click on the sample data and then we going to see that um city by average
23:25
we going to see that um city by average
23:25
we going to see that um city by average temperature humidity and so on and so
23:26
temperature humidity and so on and so
23:26
temperature humidity and so on and so forth and I'm want to go to the volumes
23:28
forth and I'm want to go to the volumes
23:29
forth and I'm want to go to the volumes we can come here and we can see all the
23:30
we can come here and we can see all the
23:30
we can come here and we can see all the metadata which is absolutely fantastic
23:34
metadata which is absolutely fantastic
23:34
metadata which is absolutely fantastic now let's go ahead and create a power
23:36
now let's go ahead and create a power
23:36
now let's go ahead and create a power bear report on top of this now to do
23:38
bear report on top of this now to do
23:38
bear report on top of this now to do that is so easy I'm going to come to the
23:41
that is so easy I'm going to come to the
23:41
that is so easy I'm going to come to the marketplace and we have the partner
23:42
marketplace and we have the partner
23:42
marketplace and we have the partner connet which is now in this environment
23:44
connet which is now in this environment
23:44
connet which is now in this environment and you can see we have the DBT Cloud F
23:47
and you can see we have the DBT Cloud F
23:47
and you can see we have the DBT Cloud F Tran tablet desktop and so on and so
23:49
Tran tablet desktop and so on and so
23:49
Tran tablet desktop and so on and so forth and then I'm going to choose the
23:51
forth and then I'm going to choose the
23:51
forth and then I'm going to choose the powerbi click on that now this going to
23:53
powerbi click on that now this going to
23:53
powerbi click on that now this going to ask me to choose the compute that I want
23:55
ask me to choose the compute that I want
23:55
ask me to choose the compute that I want to connect to so I can see the connect
23:58
to connect to so I can see the connect
23:58
to connect to so I can see the connect function fun ality and then I'm going to
24:00
function fun ality and then I'm going to
24:00
function fun ality and then I'm going to browse and choose the weather streaming
24:02
browse and choose the weather streaming
24:02
browse and choose the weather streaming computer that we going to be use for
24:03
computer that we going to be use for
24:04
computer that we going to be use for this job so I'm going to click on
24:05
this job so I'm going to click on
24:05
this job so I'm going to click on download collection file and then it
24:07
download collection file and then it
24:07
download collection file and then it copies down downloaded to my download so
24:09
copies down downloaded to my download so
24:09
copies down downloaded to my download so I can click on that to open so when I
24:11
I can click on that to open so when I
24:11
I can click on that to open so when I click on that this going to launch the
24:13
click on that this going to launch the
24:13
click on that this going to launch the powerbi desktop okay so the first thing
24:16
powerbi desktop okay so the first thing
24:16
powerbi desktop okay so the first thing we're going to see is this environment
24:17
we're going to see is this environment
24:17
we're going to see is this environment which allows us to authenticate in order
24:20
which allows us to authenticate in order
24:20
which allows us to authenticate in order to access the arual data braks now I'm
24:22
to access the arual data braks now I'm
24:22
to access the arual data braks now I'm going to use the active aure active
24:24
going to use the active aure active
24:24
going to use the active aure active directory which is now known as the
24:25
directory which is now known as the
24:26
directory which is now known as the enter ID so I want to make sure I click
24:27
enter ID so I want to make sure I click
24:27
enter ID so I want to make sure I click on the sign I'm going to pick my
24:30
on the sign I'm going to pick my
24:30
on the sign I'm going to pick my account and then I'm going to click on
24:32
account and then I'm going to click on
24:32
account and then I'm going to click on connect so this is going to
24:33
connect so this is going to
24:34
connect so this is going to automatically connect to my data break
24:36
automatically connect to my data break
24:36
automatically connect to my data break cluster and then I can bring the data
24:38
cluster and then I can bring the data
24:38
cluster and then I can bring the data into the powerbi for further um
24:42
into the powerbi for further um
24:42
into the powerbi for further um visualization so we can see we we have
24:44
visualization so we can see we we have
24:44
visualization so we can see we we have in this Navigator we have all of this U
24:47
in this Navigator we have all of this U
24:47
in this Navigator we have all of this U analytics I met store sales I'm going to
24:50
analytics I met store sales I'm going to
24:50
analytics I met store sales I'm going to click on the weather and when I click on
24:52
click on the weather and when I click on
24:52
click on the weather and when I click on the we I'm going to see all the three
24:54
the we I'm going to see all the three
24:54
the we I'm going to see all the three layers including the default so we can
24:56
layers including the default so we can
24:56
layers including the default so we can see the bronse gold silver and the the
24:59
see the bronse gold silver and the the
24:59
see the bronse gold silver and the the default now I'm going to use the sver
25:02
default now I'm going to use the sver
25:02
default now I'm going to use the sver layer in order to create power be inside
25:05
layer in order to create power be inside
25:05
layer in order to create power be inside so I'm going to expand I can see the
25:07
so I'm going to expand I can see the
25:07
so I'm going to expand I can see the table icon so when I click on this I'm
25:08
table icon so when I click on this I'm
25:08
table icon so when I click on this I'm going to see the preview now this might
25:10
going to see the preview now this might
25:10
going to see the preview now this might take up to some maybe one minute
25:13
take up to some maybe one minute
25:13
take up to some maybe one minute depending on the strength of your
25:15
depending on the strength of your
25:15
depending on the strength of your computer so I'm just going to wait and
25:17
computer so I'm just going to wait and
25:17
computer so I'm just going to wait and after the data is up and running I'm
25:19
after the data is up and running I'm
25:19
after the data is up and running I'm going to click on transform data now the
25:21
going to click on transform data now the
25:21
going to click on transform data now the reason why I'm going to trans data is
25:23
reason why I'm going to trans data is
25:23
reason why I'm going to trans data is because all the columns in our table are
25:27
because all the columns in our table are
25:27
because all the columns in our table are being treated as as a string it can say
25:29
being treated as as a string it can say
25:29
being treated as as a string it can say everything is now a string so I'm going
25:31
everything is now a string so I'm going
25:31
everything is now a string so I'm going to actually just perform a little
25:33
to actually just perform a little
25:33
to actually just perform a little transformation in powerbi to transform
25:35
transformation in powerbi to transform
25:35
transformation in powerbi to transform the numerical columns as a whole number
25:37
the numerical columns as a whole number
25:37
the numerical columns as a whole number data types so I'm going to click on this
25:39
data types so I'm going to click on this
25:39
data types so I'm going to click on this and I can go to transform data and this
25:41
and I can go to transform data and this
25:41
and I can go to transform data and this going to automatically launch the data
25:43
going to automatically launch the data
25:43
going to automatically launch the data in the powerbi power query which allows
25:46
in the powerbi power query which allows
25:46
in the powerbi power query which allows us to apply further transformation okay
25:49
us to apply further transformation okay
25:49
us to apply further transformation okay so the data is now in the power query so
25:53
so the data is now in the power query so
25:53
so the data is now in the power query so we can say we have the string Text data
25:55
we can say we have the string Text data
25:55
we can say we have the string Text data types all through now I'm going to S the
25:56
types all through now I'm going to S the
25:56
types all through now I'm going to S the latitude and then I'm going to scroll to
25:58
latitude and then I'm going to scroll to
25:58
latitude and then I'm going to scroll to the right and I'm going to order on the
26:00
the right and I'm going to order on the
26:00
the right and I'm going to order on the shift key and then I want to select the
26:02
shift key and then I want to select the
26:02
shift key and then I want to select the cloud cover so I'm going to right click
26:05
cloud cover so I'm going to right click
26:05
cloud cover so I'm going to right click and I can go to the transform data and
26:07
and I can go to the transform data and
26:07
and I can go to the transform data and I'm going to choose a whole number data
26:09
I'm going to choose a whole number data
26:09
I'm going to choose a whole number data type so this going to automatically
26:11
type so this going to automatically
26:11
type so this going to automatically convert them to a whole number for me
26:12
convert them to a whole number for me
26:12
convert them to a whole number for me which is absolutely cool for um for the
26:15
which is absolutely cool for um for the
26:15
which is absolutely cool for um for the analysis so I'm going to just click on
26:16
analysis so I'm going to just click on
26:16
analysis so I'm going to just click on close and apply so this going to close
26:18
close and apply so this going to close
26:18
close and apply so this going to close the power query and apply the
26:20
the power query and apply the
26:20
the power query and apply the transformation into the powerbi data
26:22
transformation into the powerbi data
26:22
transformation into the powerbi data model and we can start building Insight
26:25
model and we can start building Insight
26:25
model and we can start building Insight around the data that is coming now this
26:28
around the data that is coming now this
26:28
around the data that is coming now this a near real time you know dashboard in
26:31
a near real time you know dashboard in
26:31
a near real time you know dashboard in power we're going to create so there we
26:33
power we're going to create so there we
26:33
power we're going to create so there we go we have the silver data table and can
26:36
go we have the silver data table and can
26:36
go we have the silver data table and can see at the bottom you have the storage
26:37
see at the bottom you have the storage
26:37
see at the bottom you have the storage mode which is the direct query now this
26:39
mode which is the direct query now this
26:39
mode which is the direct query now this simply means for every visual that we
26:41
simply means for every visual that we
26:41
simply means for every visual that we build in powerbi the powerbi was going
26:43
build in powerbi the powerbi was going
26:43
build in powerbi the powerbi was going to send the queries to the data brakes
26:45
to send the queries to the data brakes
26:45
to send the queries to the data brakes and then the data break will release the
26:47
and then the data break will release the
26:47
and then the data break will release the query back to the power and then we have
26:48
query back to the power and then we have
26:48
query back to the power and then we have the visualization so I'm going to expand
26:50
the visualization so I'm going to expand
26:50
the visualization so I'm going to expand this and then we can see we have this um
26:53
this and then we can see we have this um
26:54
this and then we can see we have this um a categorical field like the city and
26:56
a categorical field like the city and
26:56
a categorical field like the city and the weather condition and let's want to
26:58
the weather condition and let's want to
26:58
the weather condition and let's want to see the City by temperature I'm going to
27:01
see the City by temperature I'm going to
27:01
see the City by temperature I'm going to just drag the temperature in there and
27:02
just drag the temperature in there and
27:02
just drag the temperature in there and this going to create an implicit measure
27:04
this going to create an implicit measure
27:04
this going to create an implicit measure no problems now I'm going to just as
27:06
no problems now I'm going to just as
27:07
no problems now I'm going to just as that's going on I'm going to copy and
27:09
that's going on I'm going to copy and
27:09
that's going on I'm going to copy and paste just duplicate and let's I want to
27:11
paste just duplicate and let's I want to
27:11
paste just duplicate and let's I want to see the um weather condition by the wind
27:17
see the um weather condition by the wind
27:17
see the um weather condition by the wind speed and I'm just going to wait now
27:19
speed and I'm just going to wait now
27:19
speed and I'm just going to wait now this might take a while because the
27:21
this might take a while because the
27:21
this might take a while because the power there will have to send the query
27:23
power there will have to send the query
27:23
power there will have to send the query to data Brakes in order to generate the
27:25
to data Brakes in order to generate the
27:25
to data Brakes in order to generate the inside I'm just going to wait and see
27:27
inside I'm just going to wait and see
27:27
inside I'm just going to wait and see okay so we have the result back so we
27:30
okay so we have the result back so we
27:30
okay so we have the result back so we can see we have the sum of temperature
27:32
can see we have the sum of temperature
27:32
can see we have the sum of temperature by city so we can say that MOS is at the
27:34
by city so we can say that MOS is at the
27:34
by city so we can say that MOS is at the top of the list and follow by eou
27:37
top of the list and follow by eou
27:37
top of the list and follow by eou Seattle Chicago and so on and so forth
27:39
Seattle Chicago and so on and so forth
27:39
Seattle Chicago and so on and so forth now I can turn this Su of wind speed by
27:42
now I can turn this Su of wind speed by
27:42
now I can turn this Su of wind speed by weather condition to let's say um a
27:45
weather condition to let's say um a
27:45
weather condition to let's say um a donut chart so you can you know use
27:47
donut chart so you can you know use
27:47
donut chart so you can you know use whatever visualization you want to use
27:48
whatever visualization you want to use
27:48
whatever visualization you want to use in power and then we can see we have the
27:51
in power and then we can see we have the
27:51
in power and then we can see we have the weather condition such as storming foggy
27:54
weather condition such as storming foggy
27:54
weather condition such as storming foggy cloudy and so on and so forth now this
27:56
cloudy and so on and so forth now this
27:56
cloudy and so on and so forth now this is a m real now when I click or I can
28:00
is a m real now when I click or I can
28:00
is a m real now when I click or I can see now we have this value 162 for mosco
28:02
see now we have this value 162 for mosco
28:02
see now we have this value 162 for mosco now I can click on refresh now when I
28:04
now I can click on refresh now when I
28:04
now I can click on refresh now when I click on refresh I'm just going to wait
28:06
click on refresh I'm just going to wait
28:06
click on refresh I'm just going to wait for some couple of moments and the data
28:08
for some couple of moments and the data
28:08
for some couple of moments and the data will automatically you know update based
28:10
will automatically you know update based
28:11
will automatically you know update based on the data coming into the um events
28:14
on the data coming into the um events
28:15
on the data coming into the um events ofs and then into our data brakes
28:18
ofs and then into our data brakes
28:18
ofs and then into our data brakes notebook so I'm just going to wait and
28:20
notebook so I'm just going to wait and
28:20
notebook so I'm just going to wait and let's see what happens this should
28:22
let's see what happens this should
28:22
let's see what happens this should change from 162 to something else okay
28:25
change from 162 to something else okay
28:25
change from 162 to something else okay you can see it has changed so in is now
28:27
you can see it has changed so in is now
28:27
you can see it has changed so in is now at the top of the list so this is 182
28:30
at the top of the list so this is 182
28:30
at the top of the list so this is 182 187 now this is how we can build a Power
28:34
187 now this is how we can build a Power
28:34
187 now this is how we can build a Power report on top of the streaming data from
28:38
report on top of the streaming data from
28:38
report on top of the streaming data from data breakes so we can say we've gone
28:40
data breakes so we can say we've gone
28:40
data breakes so we can say we've gone through all of these steps we started
28:41
through all of these steps we started
28:41
through all of these steps we started off by going to the vs code start by
28:43
off by going to the vs code start by
28:43
off by going to the vs code start by generating data and then we send the
28:45
generating data and then we send the
28:45
generating data and then we send the data into the event H that we created
28:48
data into the event H that we created
28:48
data into the event H that we created and then we establish connection to the
28:50
and then we establish connection to the
28:50
and then we establish connection to the event up and then we implement or
28:52
event up and then we implement or
28:52
event up and then we implement or architect the medon architecture so
28:54
architect the medon architecture so
28:54
architect the medon architecture so these are all the code you can go
28:56
these are all the code you can go
28:56
these are all the code you can go through it now if you need um this all
28:59
through it now if you need um this all
28:59
through it now if you need um this all this theour data that I use I'm going to
29:01
this theour data that I use I'm going to
29:01
this theour data that I use I'm going to put the link of my repo this um um
29:04
put the link of my repo this um um
29:04
put the link of my repo this um um repository into this video so you can
29:06
repository into this video so you can
29:06
repository into this video so you can watch and by the way please subscribe to
29:09
watch and by the way please subscribe to
29:09
watch and by the way please subscribe to this channel if enjoy this video and
29:11
this channel if enjoy this video and
29:11
this channel if enjoy this video and then you've learned from it so thank you
29:12
then you've learned from it so thank you
29:12
then you've learned from it so thank you for watching bye for now