Up next in 10
Join us on April 20-21, 2021 for the very first Azure Cosmos DB Conf. Azure Cosmos DB Conf is a free online virtual developer event organized in collaboration with the Azure Cosmos DB community, and sessions will be delivered by community members and Microsoft The event will be streamed in three, 3-hour live segments with unique content in each
This live session starts at
4/20 9:00AM (Seattle)
20-4 17:00-20:00 (London)
20-4 21:30-00:30 (Mumbai)
21-4 02:00-04:00 (Sydney)
4/20/2021 at 9:30 PM (IST)
Session: Azure Cosmos DB Conf - Keynote
Join Director of Product Management for Azure Cosmos DB, Kirill Gavrylyuk to kick off Azure Cosmos DB Conf.
Speaker: Kirill Gavrylyuk
4/20/2021 at 10:30 PM (IST)
Session:Online and Real-Time Database Migration to Azure Cosmos DB with Striim
There is significant demand for moving and continuous data integration as workloads shift to the cloud. Modernizing databases by offloading workloads to cloud requires building real-time data pipelines from legacy systems. The Striim® platform is an enterprise-grade cloud data integration solution that continuously ingests, processes, and delivers high volumes of streaming data from diverse sources, on-premises or in the cloud. Cloud architects, data architects, and data engineers can use Striim to move data into Azure Cosmos DB in a consumable form, quickly and with sub-second latency to easily run critical transactional and analytical workloads in Microsoft Azure. We will provide a live demo showing how Striim is used to migrate enterprise databases to Azure continuously and in real time with no downtime. In this session you will learn how to: Prepare Azure Cosmos DB for data integration with automatic target schema and tables creation that reflects the source database Set up in-flight transformations right in the GUI to minimize end-to-end latency enable real-time analytics & operational reporting Deploy zero-downtime migrations to Azure Cosmos DB from existing enterprise databases anywhere Increase IT productivity and reduce cost of ownership by integrating and enriching data from multiple sources
Show More Show Less View Video Transcript
0:01
[Music]
0:24
hello
0:25
hello
0:25
hello and welcome to the very first cosmos db
0:28
and welcome to the very first cosmos db
0:28
and welcome to the very first cosmos db conference we have some awesome content
0:31
conference we have some awesome content
0:31
conference we have some awesome content for you today and we're so excited to
0:33
for you today and we're so excited to
0:33
for you today and we're so excited to have you here with us
0:35
have you here with us
0:35
have you here with us my name is tim sander and i'm here with
0:37
my name is tim sander and i'm here with
0:37
my name is tim sander and i'm here with my colleagues
0:38
my colleagues
0:38
my colleagues thomas weiss and deborah chen
0:41
thomas weiss and deborah chen
0:41
thomas weiss and deborah chen from the cosmos db engineering team i'm
0:44
from the cosmos db engineering team i'm
0:44
from the cosmos db engineering team i'm going to now turn it over
0:45
going to now turn it over
0:45
going to now turn it over to thomas to give a quick introduction
0:47
to thomas to give a quick introduction
0:47
to thomas to give a quick introduction to the format of the conference
0:50
to the format of the conference
0:50
to the format of the conference thanks tim so here is how it's going to
0:52
thanks tim so here is how it's going to
0:52
thanks tim so here is how it's going to work this
0:53
work this
0:53
work this very first remote conference will run as
0:56
very first remote conference will run as
0:56
very first remote conference will run as the three different live streams we have
0:58
the three different live streams we have
0:58
the three different live streams we have one stream for the americas one for emea
1:01
one stream for the americas one for emea
1:01
one stream for the americas one for emea and one for asia pacific
1:02
and one for asia pacific
1:02
and one for asia pacific and each of these live streams will last
1:04
and each of these live streams will last
1:04
and each of these live streams will last for three hours
1:06
for three hours
1:06
for three hours we also have an entire track of
1:07
we also have an entire track of
1:07
we also have an entire track of on-demand sessions that are available
1:09
on-demand sessions that are available
1:09
on-demand sessions that are available right now
1:10
right now
1:10
right now but don't go anywhere because we are
1:11
but don't go anywhere because we are
1:11
but don't go anywhere because we are just about to launch our
1:13
just about to launch our
1:13
just about to launch our live stream for the americans now each
1:15
live stream for the americans now each
1:15
live stream for the americans now each of these live streams will have its own
1:16
of these live streams will have its own
1:16
of these live streams will have its own unique content
1:17
unique content
1:18
unique content delivered by local members of our
1:19
delivered by local members of our
1:20
delivered by local members of our beloved cosmos db community
1:21
beloved cosmos db community
1:22
beloved cosmos db community now if you happen to miss any of the
1:23
now if you happen to miss any of the
1:23
now if you happen to miss any of the live sessions no stress
1:25
live sessions no stress
1:25
live sessions no stress don't need to worry because we will make
1:26
don't need to worry because we will make
1:26
don't need to worry because we will make these available on demand
1:29
these available on demand
1:29
these available on demand shortly after each stream concludes now
1:32
shortly after each stream concludes now
1:32
shortly after each stream concludes now also good to remember that each of the
1:33
also good to remember that each of the
1:34
also good to remember that each of the live sessions we are going to present
1:35
live sessions we are going to present
1:35
live sessions we are going to present right now
1:36
right now
1:36
right now we have a live q a with the speakers so
1:38
we have a live q a with the speakers so
1:38
we have a live q a with the speakers so if you would like to ask any question
1:40
if you would like to ask any question
1:40
if you would like to ask any question to our speakers uh please make sure to
1:42
to our speakers uh please make sure to
1:42
to our speakers uh please make sure to join the stream
1:43
join the stream
1:43
join the stream on learn tv deborah thanks thomas
1:46
on learn tv deborah thanks thomas
1:46
on learn tv deborah thanks thomas as thomas mentioned we have a lot of
1:48
as thomas mentioned we have a lot of
1:48
as thomas mentioned we have a lot of great exciting speakers and sessions
1:50
great exciting speakers and sessions
1:50
great exciting speakers and sessions in this three hour live stream we'll
1:51
in this three hour live stream we'll
1:51
in this three hour live stream we'll hear from customers and members of the
1:53
hear from customers and members of the
1:53
hear from customers and members of the community
1:54
community
1:54
community on how they've built their applications
1:55
on how they've built their applications
1:55
on how they've built their applications on cosmo cb
1:57
on cosmo cb
1:57
on cosmo cb how to do real-time migrations into the
1:59
how to do real-time migrations into the
1:59
how to do real-time migrations into the service and how to do fundamentals like
2:01
service and how to do fundamentals like
2:01
service and how to do fundamentals like data model
2:02
data model
2:02
data model data modeling and partitioning your data
2:04
data modeling and partitioning your data
2:04
data modeling and partitioning your data i'm really excited to introduce our
2:05
i'm really excited to introduce our
2:05
i'm really excited to introduce our first session
2:06
first session
2:06
first session and keynote speaker kirill gavrilook
2:08
and keynote speaker kirill gavrilook
2:08
and keynote speaker kirill gavrilook director
2:15
well hello everyone and welcome to our
2:17
well hello everyone and welcome to our
2:17
well hello everyone and welcome to our first cosmos db conference
2:22
first cosmos db conference
2:22
first cosmos db conference we're going to talk about how the world
2:24
we're going to talk about how the world
2:24
we're going to talk about how the world is turning upside down
2:25
is turning upside down
2:26
is turning upside down with apps and our patterns changing
2:29
with apps and our patterns changing
2:29
with apps and our patterns changing and how cosmos db helps you sail through
2:31
and how cosmos db helps you sail through
2:31
and how cosmos db helps you sail through these turbulent and exciting times
2:34
these turbulent and exciting times
2:34
these turbulent and exciting times let's gonna go back a little bit in time
2:37
let's gonna go back a little bit in time
2:37
let's gonna go back a little bit in time back in decade maybe
2:38
back in decade maybe
2:38
back in decade maybe and remember how most of the apps looked
2:42
and remember how most of the apps looked
2:42
and remember how most of the apps looked largely the same you had an application
2:46
largely the same you had an application
2:46
largely the same you had an application you had an app here some api layer and
2:49
you had an app here some api layer and
2:49
you had an app here some api layer and then the data slowly was dripping
2:51
then the data slowly was dripping
2:51
then the data slowly was dripping to a database applications were very
2:55
to a database applications were very
2:55
to a database applications were very reactive
2:56
reactive
2:56
reactive someone bought purchased a car
2:59
someone bought purchased a car
3:00
someone bought purchased a car then at some point that this record made
3:01
then at some point that this record made
3:01
then at some point that this record made it into a database
3:03
it into a database
3:03
it into a database and probably nothing nothing accesses
3:05
and probably nothing nothing accesses
3:05
and probably nothing nothing accesses this record for a while
3:07
this record for a while
3:07
this record for a while uh until maybe the next time the person
3:09
uh until maybe the next time the person
3:09
uh until maybe the next time the person comes to an appointment
3:11
comes to an appointment
3:11
comes to an appointment and we need to fetch some history
3:14
and we need to fetch some history
3:14
and we need to fetch some history compare this to what's happening today
3:17
compare this to what's happening today
3:17
compare this to what's happening today applications the dash the data is
3:19
applications the dash the data is
3:19
applications the dash the data is gushing in
3:20
gushing in
3:20
gushing in as it's a river of data that is coming
3:23
as it's a river of data that is coming
3:23
as it's a river of data that is coming straight to the database
3:26
straight to the database
3:26
straight to the database and then applications are
3:30
and then applications are
3:30
and then applications are moving in and trying to use this data to
3:33
moving in and trying to use this data to
3:33
moving in and trying to use this data to predict
3:33
predict
3:34
predict what the customers need what do the
3:36
what the customers need what do the
3:36
what the customers need what do the users need
3:38
users need
3:38
users need very different times that require very
3:41
very different times that require very
3:41
very different times that require very different databases
3:43
different databases
3:43
different databases very different scale let's look through
3:47
very different scale let's look through
3:47
very different scale let's look through what do we need from a database in this
3:49
what do we need from a database in this
3:49
what do we need from a database in this new environment
3:51
new environment
3:51
new environment first of all it needs to scale instantly
3:54
first of all it needs to scale instantly
3:54
first of all it needs to scale instantly transparently and guarantee speed at any
3:58
transparently and guarantee speed at any
3:58
transparently and guarantee speed at any point in time
4:00
point in time
4:00
point in time tco is very important because the volume
4:02
tco is very important because the volume
4:02
tco is very important because the volume of data can change rapidly
4:04
of data can change rapidly
4:04
of data can change rapidly as we walk as we see we'll have couple
4:07
as we walk as we see we'll have couple
4:07
as we walk as we see we'll have couple conversations
4:08
conversations
4:08
conversations later in this talk and we'll hear from
4:11
later in this talk and we'll hear from
4:11
later in this talk and we'll hear from hawkes
4:12
hawkes
4:12
hawkes building really exciting and large scale
4:14
building really exciting and large scale
4:14
building really exciting and large scale applications
4:15
applications
4:15
applications uh that where and the scale changes
4:18
uh that where and the scale changes
4:18
uh that where and the scale changes rapidly the database needs to react
4:21
rapidly the database needs to react
4:21
rapidly the database needs to react instantly
4:21
instantly
4:22
instantly the data is database needs to be
4:23
the data is database needs to be
4:23
the data is database needs to be serverless we are not
4:25
serverless we are not
4:25
serverless we are not we don't live anymore in the world where
4:26
we don't live anymore in the world where
4:26
we don't live anymore in the world where you can think about instances and vms
4:30
you can think about instances and vms
4:30
you can think about instances and vms we are past that we need to react and
4:32
we are past that we need to react and
4:32
we are past that we need to react and provide performance database needs to
4:34
provide performance database needs to
4:34
provide performance database needs to provide performance and scale when the
4:36
provide performance and scale when the
4:36
provide performance and scale when the app needs it as a granularity
4:38
app needs it as a granularity
4:38
app needs it as a granularity that the app needs the data
4:42
that the app needs the data
4:42
that the app needs the data changes all the time it's a river of
4:45
changes all the time it's a river of
4:45
changes all the time it's a river of data
4:45
data
4:46
data you can't you don't control the river
4:47
you can't you don't control the river
4:47
you can't you don't control the river you don't control the schema
4:49
you don't control the schema
4:49
you don't control the schema the database needs to be flexible and
4:51
the database needs to be flexible and
4:52
the database needs to be flexible and accommodate any changes since
4:53
accommodate any changes since
4:53
accommodate any changes since the in in the schema in the data
4:55
the in in the schema in the data
4:55
the in in the schema in the data structure
4:56
structure
4:56
structure it needs to make it easy to build these
4:58
it needs to make it easy to build these
4:58
it needs to make it easy to build these apps that are
5:00
apps that are
5:00
apps that are robust to changes in the data shape
5:04
of course there are many established
5:08
of course there are many established
5:08
of course there are many established very popular programming models and the
5:11
very popular programming models and the
5:11
very popular programming models and the database
5:12
database
5:12
database needs to cater towards developers
5:16
needs to cater towards developers
5:16
needs to cater towards developers um developers as those who are building
5:18
um developers as those who are building
5:18
um developers as those who are building these apps and need to build these apps
5:20
these apps and need to build these apps
5:20
these apps and need to build these apps fast
5:20
fast
5:20
fast and familiarity with tools and
5:22
and familiarity with tools and
5:22
and familiarity with tools and technologies is important
5:24
technologies is important
5:24
technologies is important there are great programming models um
5:27
there are great programming models um
5:27
there are great programming models um mongodb
5:28
mongodb
5:28
mongodb drive and api cassandra api and
5:32
drive and api cassandra api and
5:32
drive and api cassandra api and there are more more and more graph
5:34
there are more more and more graph
5:34
there are more more and more graph databases today
5:36
databases today
5:36
databases today it's important for us uh to have to
5:39
it's important for us uh to have to
5:39
it's important for us uh to have to offer developers what they're
5:41
offer developers what they're
5:41
offer developers what they're familiar with
5:44
now the applications are the lifeline of
5:47
now the applications are the lifeline of
5:47
now the applications are the lifeline of the
5:48
the
5:48
the of the companies right the company does
5:51
of the companies right the company does
5:51
of the companies right the company does not exist anymore without with
5:52
not exist anymore without with
5:52
not exist anymore without with application application even
5:55
application application even
5:56
application application even stops working even for a second the
5:58
stops working even for a second the
5:58
stops working even for a second the database needs to be
6:00
database needs to be
6:00
database needs to be reliable provide five nine slas needs to
6:04
reliable provide five nine slas needs to
6:04
reliable provide five nine slas needs to be
6:05
be
6:05
be resilient to failure even if the entire
6:07
resilient to failure even if the entire
6:08
resilient to failure even if the entire region goes down in the cloud
6:09
region goes down in the cloud
6:09
region goes down in the cloud the application must continue to work
6:13
the application must continue to work
6:13
the application must continue to work at the same time applications are also
6:16
at the same time applications are also
6:16
at the same time applications are also inherently
6:17
inherently
6:17
inherently omnipresent they're everywhere and so
6:19
omnipresent they're everywhere and so
6:19
omnipresent they're everywhere and so the database needs to be present whereas
6:21
the database needs to be present whereas
6:21
the database needs to be present whereas the
6:21
the
6:21
the where the customers are and so it needs
6:24
where the customers are and so it needs
6:24
where the customers are and so it needs to be globally distributed so that you
6:26
to be globally distributed so that you
6:26
to be globally distributed so that you can bring data
6:27
can bring data
6:27
can bring data where the users are
6:30
where the users are
6:30
where the users are finally applications are uh
6:34
finally applications are uh
6:34
finally applications are uh in need to ever as ever to
6:37
in need to ever as ever to
6:37
in need to ever as ever to close the digital loop to provide uh
6:40
close the digital loop to provide uh
6:40
close the digital loop to provide uh customers
6:42
customers
6:42
customers timely and productive uh services
6:45
timely and productive uh services
6:45
timely and productive uh services before they know they even need them and
6:48
before they know they even need them and
6:48
before they know they even need them and for that we need to have rich
6:50
for that we need to have rich
6:50
for that we need to have rich insights in real time without any delay
6:52
insights in real time without any delay
6:52
insights in real time without any delay without
6:53
without
6:53
without any of this complex etl complexity must
6:56
any of this complex etl complexity must
6:56
any of this complex etl complexity must be very fast
6:57
be very fast
6:57
be very fast in real time the provide provide
7:00
in real time the provide provide
7:00
in real time the provide provide analytics to drive the app
7:02
analytics to drive the app
7:02
analytics to drive the app and provide bi so that the um so the
7:05
and provide bi so that the um so the
7:05
and provide bi so that the um so the company can
7:06
company can
7:06
company can react quickly to the change to the
7:08
react quickly to the change to the
7:08
react quickly to the change to the changes
7:10
changes
7:10
changes now we have this database um many of you
7:13
now we have this database um many of you
7:13
now we have this database um many of you are using it today
7:14
are using it today
7:14
are using it today it's called azure cosmos db it's fast no
7:17
it's called azure cosmos db it's fast no
7:17
it's called azure cosmos db it's fast no sql database
7:18
sql database
7:18
sql database with open api for any scale now let's
7:21
with open api for any scale now let's
7:21
with open api for any scale now let's talk so we'll walk through a few of the
7:23
talk so we'll walk through a few of the
7:23
talk so we'll walk through a few of the updates where we're taking this database
7:25
updates where we're taking this database
7:25
updates where we're taking this database forward
7:26
forward
7:26
forward we'll talk to a few folks uh that use
7:28
we'll talk to a few folks uh that use
7:28
we'll talk to a few folks uh that use database today and share the experiences
7:30
database today and share the experiences
7:30
database today and share the experiences share their patterns
7:32
share their patterns
7:32
share their patterns we are continuing to double down on
7:34
we are continuing to double down on
7:34
we are continuing to double down on serverless we continue to
7:35
serverless we continue to
7:35
serverless we continue to to offer uh more and more flexibility
7:38
to offer uh more and more flexibility
7:38
to offer uh more and more flexibility uh we continue to improve the cost
7:40
uh we continue to improve the cost
7:40
uh we continue to improve the cost profile of the database
7:42
profile of the database
7:42
profile of the database making it more more friendly to
7:44
making it more more friendly to
7:44
making it more more friendly to developers we're adding
7:46
developers we're adding
7:46
developers we're adding notebook supports where uh
7:50
notebook supports where uh
7:50
notebook supports where uh we're we're offering serverless
7:53
we're we're offering serverless
7:53
we're we're offering serverless as we recently announced uh we keep in
7:55
as we recently announced uh we keep in
7:56
as we recently announced uh we keep in time in
7:57
time in
7:57
time in step with the development of the open
7:59
step with the development of the open
7:59
step with the development of the open source apis for those who are familiar
8:01
source apis for those who are familiar
8:01
source apis for those who are familiar and like the open source apis
8:03
and like the open source apis
8:03
and like the open source apis um and we continue working to make sure
8:06
um and we continue working to make sure
8:06
um and we continue working to make sure that
8:07
that
8:07
that the database uh is the backbone of your
8:10
the database uh is the backbone of your
8:10
the database uh is the backbone of your application the database never goes down
8:14
application the database never goes down
8:14
application the database never goes down um at the same time there are some
8:15
um at the same time there are some
8:15
um at the same time there are some exciting things happening around the
8:17
exciting things happening around the
8:17
exciting things happening around the real time and closing the digital loop
8:19
real time and closing the digital loop
8:19
real time and closing the digital loop let's think about the common pattern
8:21
let's think about the common pattern
8:21
let's think about the common pattern today it's very different from these
8:23
today it's very different from these
8:23
today it's very different from these data drives that we
8:24
data drives that we
8:24
data drives that we are used to in the past the data is
8:27
are used to in the past the data is
8:27
are used to in the past the data is coming into cosmos db straight
8:31
coming into cosmos db straight
8:31
coming into cosmos db straight uh through ingestion through through
8:33
uh through ingestion through through
8:33
uh through ingestion through through kafka service bus
8:35
kafka service bus
8:35
kafka service bus through any of the technologies or
8:37
through any of the technologies or
8:37
through any of the technologies or directly
8:39
directly
8:39
directly the database instantly reacts and out of
8:42
the database instantly reacts and out of
8:42
the database instantly reacts and out of scales
8:43
scales
8:43
scales thanks to the instant out of the scales
8:44
thanks to the instant out of the scales
8:44
thanks to the instant out of the scales that cosmos db provides the acce
8:46
that cosmos db provides the acce
8:46
that cosmos db provides the acce database it can instantly react to the
8:48
database it can instantly react to the
8:48
database it can instantly react to the volume of data
8:50
volume of data
8:50
volume of data and then the application is built in as
8:53
and then the application is built in as
8:53
and then the application is built in as a set of micro
8:53
a set of micro
8:54
a set of micro services the typical cosmos db
8:55
services the typical cosmos db
8:55
services the typical cosmos db application is built as a set of micro
8:57
application is built as a set of micro
8:57
application is built as a set of micro services that react to the change feed
8:59
services that react to the change feed
9:00
services that react to the change feed this
9:00
this
9:00
this real-time interactive transaction logs
9:03
real-time interactive transaction logs
9:03
real-time interactive transaction logs that cosmos db offers to you
9:05
that cosmos db offers to you
9:05
that cosmos db offers to you um so that you can augment and build and
9:07
um so that you can augment and build and
9:07
um so that you can augment and build and develop
9:09
develop
9:09
develop and enrich your application and close
9:11
and enrich your application and close
9:11
and enrich your application and close the loop
9:12
the loop
9:12
the loop with the user close the loop with your
9:14
with the user close the loop with your
9:14
with the user close the loop with your customers
9:16
customers
9:16
customers that's kind of that's the predominant
9:17
that's kind of that's the predominant
9:18
that's kind of that's the predominant app pattern today and let's talk a few
9:20
app pattern today and let's talk a few
9:20
app pattern today and let's talk a few examples for example toyota
9:22
examples for example toyota
9:22
examples for example toyota um toyota offers predictive analytics if
9:25
um toyota offers predictive analytics if
9:25
um toyota offers predictive analytics if you're using uh toyota
9:27
you're using uh toyota
9:27
you're using uh toyota uh the lots of sensors are sending data
9:29
uh the lots of sensors are sending data
9:29
uh the lots of sensors are sending data the data goes to cosmos db
9:31
the data goes to cosmos db
9:31
the data goes to cosmos db uh based on the data um you may get
9:34
uh based on the data um you may get
9:34
uh based on the data um you may get you'll get
9:37
you get predictive uh outreach um
9:41
you get predictive uh outreach um
9:41
you get predictive uh outreach um and predictive maintenance
9:42
and predictive maintenance
9:42
and predictive maintenance recommendations in case of something
9:45
recommendations in case of something
9:46
recommendations in case of something we think is going wrong before you even
9:49
we think is going wrong before you even
9:49
we think is going wrong before you even face with an issue but it is not
9:53
face with an issue but it is not
9:53
face with an issue but it is not i iot represents more than 20 percent of
9:55
i iot represents more than 20 percent of
9:56
i iot represents more than 20 percent of cosmos db usage
9:57
cosmos db usage
9:57
cosmos db usage worldwide but it is that pattern does
9:59
worldwide but it is that pattern does
9:59
worldwide but it is that pattern does not is not limited to iot only
10:01
not is not limited to iot only
10:02
not is not limited to iot only if you look at e-commerce think about an
10:04
if you look at e-commerce think about an
10:04
if you look at e-commerce think about an example
10:05
example
10:05
example and we'll see we'll hear from our
10:07
and we'll see we'll hear from our
10:07
and we'll see we'll hear from our friends and aces
10:08
friends and aces
10:08
friends and aces later in the day but i think of asus
10:10
later in the day but i think of asus
10:10
later in the day but i think of asus example like in a very simple flow
10:13
example like in a very simple flow
10:13
example like in a very simple flow returns returns are going the signals
10:16
returns returns are going the signals
10:16
returns returns are going the signals from the app
10:17
from the app
10:17
from the app are going to the uh into the database
10:19
are going to the uh into the database
10:19
are going to the uh into the database state
10:20
state
10:20
state and then the application itself the
10:22
and then the application itself the
10:22
and then the application itself the logic is
10:26
a set of microservices each of them
10:29
a set of microservices each of them
10:29
a set of microservices each of them knows what to do with particular type of
10:31
knows what to do with particular type of
10:31
knows what to do with particular type of change
10:31
change
10:31
change and so they're all listening on change
10:33
and so they're all listening on change
10:33
and so they're all listening on change feed and uh
10:34
feed and uh
10:34
feed and uh and processing the data creating
10:36
and processing the data creating
10:36
and processing the data creating materialized views
10:38
materialized views
10:38
materialized views and uh sending back the signals and
10:40
and uh sending back the signals and
10:40
and uh sending back the signals and driving the application behaviors this
10:42
driving the application behaviors this
10:42
driving the application behaviors this way we can
10:43
way we can
10:43
way we can asus can provide predictive application
10:45
asus can provide predictive application
10:45
asus can provide predictive application behaviors to their customers
10:48
behaviors to their customers
10:48
behaviors to their customers now let me invite on stage uh
10:51
now let me invite on stage uh
10:51
now let me invite on stage uh anand krishnamurti a group engineering
10:54
anand krishnamurti a group engineering
10:54
anand krishnamurti a group engineering manager manager from microsoft teams
10:56
manager manager from microsoft teams
10:56
manager manager from microsoft teams microsoft teams does not really need
10:58
microsoft teams does not really need
10:58
microsoft teams does not really need introduction many of you use this today
11:01
introduction many of you use this today
11:01
introduction many of you use this today but let's talk about how microsoft teams
11:03
but let's talk about how microsoft teams
11:04
but let's talk about how microsoft teams uses costumes db today
11:06
uses costumes db today
11:06
uses costumes db today welcome hello folks uh um
11:10
welcome hello folks uh um
11:10
welcome hello folks uh um thanks a lot for inviting me to it's a
11:12
thanks a lot for inviting me to it's a
11:12
thanks a lot for inviting me to it's a pleasure
11:13
pleasure
11:13
pleasure to to join all of you um my name is
11:15
to to join all of you um my name is
11:15
to to join all of you um my name is anand and i
11:16
anand and i
11:16
anand and i am a group engineering manager um for
11:18
am a group engineering manager um for
11:18
am a group engineering manager um for microsoft teams uh primarily on the
11:20
microsoft teams uh primarily on the
11:20
microsoft teams uh primarily on the messaging side of things
11:22
messaging side of things
11:22
messaging side of things all of you use microsoft teams um
11:25
all of you use microsoft teams um
11:25
all of you use microsoft teams um it's been a phenomenal experience and i
11:27
it's been a phenomenal experience and i
11:27
it's been a phenomenal experience and i want to share a little bit
11:29
want to share a little bit
11:29
want to share a little bit around the same uh you know a few years
11:31
around the same uh you know a few years
11:32
around the same uh you know a few years back
11:32
back
11:32
back we had you know prioritized our moving
11:34
we had you know prioritized our moving
11:34
we had you know prioritized our moving off our legacy legacy store stacks uh on
11:36
off our legacy legacy store stacks uh on
11:36
off our legacy legacy store stacks uh on cosmos tv and that uh you know that was
11:38
cosmos tv and that uh you know that was
11:38
cosmos tv and that uh you know that was one of the best decisions we made
11:40
one of the best decisions we made
11:40
one of the best decisions we made um we kind of use cosmos db
11:43
um we kind of use cosmos db
11:43
um we kind of use cosmos db uh you know all over our uh all through
11:46
uh you know all over our uh all through
11:46
uh you know all over our uh all through our our messaging services side uh you
11:48
our our messaging services side uh you
11:48
our our messaging services side uh you know our aggregation side
11:50
know our aggregation side
11:50
know our aggregation side even the transport services actually use
11:51
even the transport services actually use
11:51
even the transport services actually use cosmos tv uh we're roughly
11:54
cosmos tv uh we're roughly
11:54
cosmos tv uh we're roughly doing about six seven petabytes of data
11:56
doing about six seven petabytes of data
11:56
doing about six seven petabytes of data we're doing about
11:57
we're doing about
11:58
we're doing about you know close to two trillion
11:59
you know close to two trillion
11:59
you know close to two trillion transactions a second and the skill that
12:01
transactions a second and the skill that
12:01
transactions a second and the skill that we've been able to achieve with cosmos
12:03
we've been able to achieve with cosmos
12:03
we've been able to achieve with cosmos db as our partner
12:04
db as our partner
12:04
db as our partner was phenomenal especially given the kobe
12:06
was phenomenal especially given the kobe
12:06
was phenomenal especially given the kobe time frame and
12:07
time frame and
12:08
time frame and a lot of people um you know latched on
12:10
a lot of people um you know latched on
12:10
a lot of people um you know latched on to microsoft teams in order to get the
12:12
to microsoft teams in order to get the
12:12
to microsoft teams in order to get the remote work going
12:13
remote work going
12:13
remote work going when you look at uh somewhere about um
12:16
when you look at uh somewhere about um
12:16
when you look at uh somewhere about um you know a 2x to 3x growth
12:20
you know a 2x to 3x growth
12:20
you know a 2x to 3x growth the first day when we got here i think
12:22
the first day when we got here i think
12:22
the first day when we got here i think it was march 12th
12:23
it was march 12th
12:23
it was march 12th of last year um you know we just scaled
12:27
of last year um you know we just scaled
12:27
of last year um you know we just scaled like crazy and and it was amazing to see
12:29
like crazy and and it was amazing to see
12:29
like crazy and and it was amazing to see the decision that we made on the
12:30
the decision that we made on the
12:30
the decision that we made on the technology stack
12:32
technology stack
12:32
technology stack now we use cosmos db uh pretty much from
12:35
now we use cosmos db uh pretty much from
12:35
now we use cosmos db uh pretty much from uh you know bottoms up all kinds of
12:37
uh you know bottoms up all kinds of
12:37
uh you know bottoms up all kinds of features all the way from multimasters
12:39
features all the way from multimasters
12:39
features all the way from multimasters to
12:40
to
12:40
to you know different sets of consistency
12:41
you know different sets of consistency
12:42
you know different sets of consistency levels and and um
12:44
levels and and um
12:44
levels and and um you know all on the replication or
12:45
you know all on the replication or
12:45
you know all on the replication or compliance engines run
12:47
compliance engines run
12:47
compliance engines run on cosmos db and so you know we were
12:49
on cosmos db and so you know we were
12:49
on cosmos db and so you know we were pretty
12:50
pretty
12:50
pretty um you know we were watching it pretty
12:52
um you know we were watching it pretty
12:52
um you know we were watching it pretty close but uh it was
12:53
close but uh it was
12:53
close but uh it was it was phenomenal to see that over the
12:55
it was phenomenal to see that over the
12:55
it was phenomenal to see that over the last year alone our usage has increased
12:58
last year alone our usage has increased
12:58
last year alone our usage has increased by over 10x
12:59
by over 10x
12:59
by over 10x um you know this is i'm talking about
13:01
um you know this is i'm talking about
13:01
um you know this is i'm talking about core messaging services we're talking
13:02
core messaging services we're talking
13:02
core messaging services we're talking about billions
13:03
about billions
13:03
about billions of messages in a day you know billions
13:05
of messages in a day you know billions
13:05
of messages in a day you know billions of uh meetings
13:07
of uh meetings
13:07
of uh meetings that that that use this uh
13:08
that that that use this uh
13:08
that that that use this uh infrastructure as well
13:10
infrastructure as well
13:10
infrastructure as well and so um it's uh it's been a great
13:12
and so um it's uh it's been a great
13:12
and so um it's uh it's been a great experience
13:13
experience
13:13
experience uh you know moving to cosmos db being
13:16
uh you know moving to cosmos db being
13:16
uh you know moving to cosmos db being able to achieve that level of scale
13:18
able to achieve that level of scale
13:18
able to achieve that level of scale and the kind of reliability uh you know
13:20
and the kind of reliability uh you know
13:20
and the kind of reliability uh you know guaranteed four nines reliability and
13:22
guaranteed four nines reliability and
13:22
guaranteed four nines reliability and there's a lot of these
13:23
there's a lot of these
13:23
there's a lot of these beautiful level features in terms of
13:24
beautiful level features in terms of
13:24
beautiful level features in terms of automatic failover and
13:26
automatic failover and
13:26
automatic failover and and uh things that have been helping us
13:28
and uh things that have been helping us
13:28
and uh things that have been helping us you know kind of achieve that kind of a
13:30
you know kind of achieve that kind of a
13:30
you know kind of achieve that kind of a transaction model that that that soon as
13:32
transaction model that that that soon as
13:32
transaction model that that that soon as the real-time
13:33
the real-time
13:33
the real-time communication platform thank you
13:37
communication platform thank you
13:37
communication platform thank you it's definitely kind of clear from this
13:38
it's definitely kind of clear from this
13:38
it's definitely kind of clear from this chart uh this is where we're having
13:40
chart uh this is where we're having
13:40
chart uh this is where we're having daily transactions that are going from
13:42
daily transactions that are going from
13:42
daily transactions that are going from teams to cosmos db and we're talking
13:44
teams to cosmos db and we're talking
13:44
teams to cosmos db and we're talking about close to two and a half trillion
13:46
about close to two and a half trillion
13:46
about close to two and a half trillion transactions per day
13:48
transactions per day
13:48
transactions per day and you can see that it changes very
13:49
and you can see that it changes very
13:49
and you can see that it changes very rapidly from day to day obviously
13:51
rapidly from day to day obviously
13:51
rapidly from day to day obviously during the weekends you have uh more
13:54
during the weekends you have uh more
13:54
during the weekends you have uh more than
13:54
than
13:54
than 2x uh dip as folks thankfully don't work
13:58
2x uh dip as folks thankfully don't work
13:58
2x uh dip as folks thankfully don't work on weekends
13:59
on weekends
13:59
on weekends and then they come back to work and the
14:01
and then they come back to work and the
14:01
and then they come back to work and the usage comes back
14:02
usage comes back
14:02
usage comes back through the weekdays and it changes
14:04
through the weekdays and it changes
14:04
through the weekdays and it changes through the year there's seasonalities
14:06
through the year there's seasonalities
14:06
through the year there's seasonalities there's obviously
14:07
there's obviously
14:07
there's obviously a little bit of idle periods oh it's not
14:10
a little bit of idle periods oh it's not
14:10
a little bit of idle periods oh it's not not really idle
14:11
not really idle
14:11
not really idle but less active during the holidays how
14:13
but less active during the holidays how
14:13
but less active during the holidays how do you guys
14:14
do you guys
14:14
do you guys react to these uh rapidly changing
14:16
react to these uh rapidly changing
14:16
react to these uh rapidly changing volumes of data
14:19
volumes of data
14:19
volumes of data um so you know you know this
14:22
um so you know you know this
14:22
um so you know you know this multi-faceted
14:23
multi-faceted
14:23
multi-faceted approach that we have uh in terms of in
14:25
approach that we have uh in terms of in
14:25
approach that we have uh in terms of in terms of uh reacting to this
14:27
terms of uh reacting to this
14:27
terms of uh reacting to this you know this there's different models
14:28
you know this there's different models
14:28
you know this there's different models that we follow in terms of
14:30
that we follow in terms of
14:30
that we follow in terms of you know what are the total amount of uh
14:32
you know what are the total amount of uh
14:32
you know what are the total amount of uh provisioning that's been done
14:33
provisioning that's been done
14:33
provisioning that's been done uh for example like edu has become one
14:35
uh for example like edu has become one
14:35
uh for example like edu has become one of our major consumers today
14:36
of our major consumers today
14:36
of our major consumers today a lot of uh a lot of you know colleges
14:39
a lot of uh a lot of you know colleges
14:39
a lot of uh a lot of you know colleges in school are
14:40
in school are
14:40
in school are our teams and we've you know
14:43
our teams and we've you know
14:43
our teams and we've you know from a cosmos db standpoint we work very
14:45
from a cosmos db standpoint we work very
14:45
from a cosmos db standpoint we work very closely with some of the teams in cosmos
14:47
closely with some of the teams in cosmos
14:47
closely with some of the teams in cosmos db in order to understand the usage
14:48
db in order to understand the usage
14:48
db in order to understand the usage patterns where to optimize and how to
14:50
patterns where to optimize and how to
14:50
patterns where to optimize and how to optimize it
14:51
optimize it
14:51
optimize it uh the one thing that we've been pretty
14:54
uh the one thing that we've been pretty
14:54
uh the one thing that we've been pretty happy about
14:55
happy about
14:55
happy about is the scale itself so from uh from a
14:57
is the scale itself so from uh from a
14:57
is the scale itself so from uh from a compute side yes we gotta go put in some
14:59
compute side yes we gotta go put in some
14:59
compute side yes we gotta go put in some work to build data centers and
15:01
work to build data centers and
15:01
work to build data centers and and and achieve that scale but from uh
15:03
and and achieve that scale but from uh
15:03
and and achieve that scale but from uh from a storage perspective we've been
15:04
from a storage perspective we've been
15:04
from a storage perspective we've been able to scale
15:05
able to scale
15:05
able to scale pretty pretty high even even though our
15:07
pretty pretty high even even though our
15:07
pretty pretty high even even though our throughput uh our read and write
15:09
throughput uh our read and write
15:09
throughput uh our read and write throughput has been increasing like
15:10
throughput has been increasing like
15:10
throughput has been increasing like crazy uh you know over the last year
15:12
crazy uh you know over the last year
15:12
crazy uh you know over the last year um so you know it's it's basically a
15:15
um so you know it's it's basically a
15:15
um so you know it's it's basically a combination of multiple things that we
15:16
combination of multiple things that we
15:16
combination of multiple things that we do try to try to keep up with the
15:18
do try to try to keep up with the
15:18
do try to try to keep up with the the current usages this is really
15:21
the current usages this is really
15:21
the current usages this is really amazing
15:22
amazing
15:22
amazing here we are looking at the database of
15:24
here we are looking at the database of
15:24
here we are looking at the database of the size of more than five petabytes
15:27
the size of more than five petabytes
15:27
the size of more than five petabytes that can take kind of two and a half
15:28
that can take kind of two and a half
15:28
that can take kind of two and a half trillion transactions
15:30
trillion transactions
15:30
trillion transactions per day amazing amazing like i'm really
15:33
per day amazing amazing like i'm really
15:33
per day amazing amazing like i'm really glad what you were what we were
15:37
glad what you were what we were
15:37
glad what you were what we were be able to do together and certainly
15:39
be able to do together and certainly
15:39
be able to do together and certainly wish
15:40
wish
15:40
wish microsoft teams nothing more but
15:42
microsoft teams nothing more but
15:42
microsoft teams nothing more but continuing the 10x growth
15:44
continuing the 10x growth
15:44
continuing the 10x growth uh this is really exciting
15:47
uh this is really exciting
15:48
uh this is really exciting absolutely thank you so much and
15:51
absolutely thank you so much and
15:52
absolutely thank you so much and so as you can see um
15:55
so as you can see um
15:55
so as you can see um uh elasticity and scale
15:58
uh elasticity and scale
15:58
uh elasticity and scale are one of the key reasons kind of uh
16:02
are one of the key reasons kind of uh
16:02
are one of the key reasons kind of uh applications use cosmos db and if you
16:04
applications use cosmos db and if you
16:04
applications use cosmos db and if you just think about kind of
16:06
just think about kind of
16:06
just think about kind of five petabytes of data and you continue
16:08
five petabytes of data and you continue
16:08
five petabytes of data and you continue to use the same database
16:10
to use the same database
16:10
to use the same database as if nothing changes like whether
16:12
as if nothing changes like whether
16:12
as if nothing changes like whether you're using it with just one terabyte
16:13
you're using it with just one terabyte
16:14
you're using it with just one terabyte or whether you're using with five
16:15
or whether you're using with five
16:15
or whether you're using with five petabytes
16:16
petabytes
16:16
petabytes it's the same programming model it
16:18
it's the same programming model it
16:18
it's the same programming model it scales out the same uh
16:19
scales out the same uh
16:19
scales out the same uh same sdk same app um
16:23
same sdk same app um
16:23
same sdk same app um and it can react to the change to the
16:26
and it can react to the change to the
16:26
and it can react to the change to the changes in transaction volume
16:28
changes in transaction volume
16:28
changes in transaction volume at ease obviously a lot of work goes
16:31
at ease obviously a lot of work goes
16:31
at ease obviously a lot of work goes behind the scenes into this
16:33
behind the scenes into this
16:33
behind the scenes into this let's let's look through some of the
16:34
let's let's look through some of the
16:34
let's let's look through some of the changes some of the improvements that we
16:36
changes some of the improvements that we
16:36
changes some of the improvements that we are making to make it
16:37
are making to make it
16:37
are making to make it uh even easier for folks to deal with uh
16:41
uh even easier for folks to deal with uh
16:41
uh even easier for folks to deal with uh rapid change our first uh this has been
16:44
rapid change our first uh this has been
16:44
rapid change our first uh this has been a lot
16:45
a lot
16:45
a lot of reward but we keep working and
16:47
of reward but we keep working and
16:47
of reward but we keep working and tweaking and improving is
16:49
tweaking and improving is
16:49
tweaking and improving is instant auto scale this is a picture of
16:52
instant auto scale this is a picture of
16:52
instant auto scale this is a picture of cosmos db where you can configure
16:54
cosmos db where you can configure
16:54
cosmos db where you can configure uh cosmos to be another scale and it
16:56
uh cosmos to be another scale and it
16:56
uh cosmos to be another scale and it will drive the volume of
16:58
will drive the volume of
16:58
will drive the volume of the transaction volume and scale
17:01
the transaction volume and scale
17:01
the transaction volume and scale instantly within 10x
17:04
instantly within 10x
17:04
instantly within 10x um range and it will scale scale
17:07
um range and it will scale scale
17:07
um range and it will scale scale asynchronously
17:08
asynchronously
17:08
asynchronously beyond the mx range uh responding to
17:11
beyond the mx range uh responding to
17:11
beyond the mx range uh responding to your traffic
17:13
your traffic
17:13
your traffic again we maintain the same guarantees uh
17:16
again we maintain the same guarantees uh
17:16
again we maintain the same guarantees uh as when you just perform this
17:19
as when you just perform this
17:19
as when you just perform this manually we maintain the same guarantees
17:21
manually we maintain the same guarantees
17:21
manually we maintain the same guarantees around the throughput
17:23
around the throughput
17:23
around the throughput around the latency so you never see if
17:25
around the latency so you never see if
17:25
around the latency so you never see if you're using auto scale
17:27
you're using auto scale
17:27
you're using auto scale the application does not face with the
17:29
the application does not face with the
17:29
the application does not face with the throttling errors
17:30
throttling errors
17:30
throttling errors or not enough resources wait until i
17:32
or not enough resources wait until i
17:32
or not enough resources wait until i scale uh we can risk this good we offer
17:36
scale uh we can risk this good we offer
17:36
scale uh we can risk this good we offer this
17:36
this
17:36
this capacity to instantly into the
17:38
capacity to instantly into the
17:38
capacity to instantly into the application instantly as it's required
17:41
application instantly as it's required
17:41
application instantly as it's required uh with throughput and availability
17:43
uh with throughput and availability
17:43
uh with throughput and availability guarantees
17:45
guarantees
17:45
guarantees now some of the improvements we're
17:46
now some of the improvements we're
17:46
now some of the improvements we're looking at in this area
17:48
looking at in this area
17:48
looking at in this area one of them is ability for uh folks to
17:51
one of them is ability for uh folks to
17:51
one of them is ability for uh folks to uh do adaptive change to the
17:55
uh do adaptive change to the
17:55
uh do adaptive change to the um to the range of the other scale
17:58
um to the range of the other scale
17:58
um to the range of the other scale you don't always need 10x sometimes you
18:01
you don't always need 10x sometimes you
18:01
you don't always need 10x sometimes you need less you actually prefer that
18:03
need less you actually prefer that
18:03
need less you actually prefer that that we kind of
18:06
slow down the app a little bit so that
18:08
slow down the app a little bit so that
18:08
slow down the app a little bit so that you don't for example for cost reasons
18:10
you don't for example for cost reasons
18:10
you don't for example for cost reasons you don't want to
18:11
you don't want to
18:11
you don't want to scale beyond certain threshold so this
18:14
scale beyond certain threshold so this
18:14
scale beyond certain threshold so this this this is coming and can please reach
18:16
this this is coming and can please reach
18:16
this this is coming and can please reach out uh we can
18:17
out uh we can
18:17
out uh we can we can enroll in a preview of this uh
18:20
we can enroll in a preview of this uh
18:20
we can enroll in a preview of this uh there is
18:21
there is
18:21
there is work going on at asynchronously and
18:24
work going on at asynchronously and
18:24
work going on at asynchronously and automatically adjusting the range itself
18:26
automatically adjusting the range itself
18:26
automatically adjusting the range itself of the auto scale this is
18:27
of the auto scale this is
18:27
of the auto scale this is this is coming um so today you have to
18:31
this is coming um so today you have to
18:31
this is coming um so today you have to when you
18:32
when you
18:32
when you increase the throughput when when when
18:34
increase the throughput when when when
18:34
increase the throughput when when when your app needs
18:35
your app needs
18:35
your app needs more than 10x to grow more than 10x as
18:37
more than 10x to grow more than 10x as
18:37
more than 10x to grow more than 10x as you've seen with teams like this
18:39
you've seen with teams like this
18:39
you've seen with teams like this that went through the roof over uh
18:42
that went through the roof over uh
18:42
that went through the roof over uh really overnight
18:43
really overnight
18:43
really overnight once the uh we all went into quarantine
18:46
once the uh we all went into quarantine
18:46
once the uh we all went into quarantine and
18:46
and
18:46
and uh the kids went to school online my
18:49
uh the kids went to school online my
18:49
uh the kids went to school online my team's usage went through the roof
18:51
team's usage went through the roof
18:51
team's usage went through the roof um they needed to scale out rapidly and
18:54
um they needed to scale out rapidly and
18:54
um they needed to scale out rapidly and so this is today it requires
18:56
so this is today it requires
18:56
so this is today it requires kind of going and manually adjusting
18:57
kind of going and manually adjusting
18:57
kind of going and manually adjusting your auto scale um
18:59
your auto scale um
18:59
your auto scale um range and increasing uh for youtube
19:03
range and increasing uh for youtube
19:03
range and increasing uh for youtube through the portal
19:04
through the portal
19:04
through the portal but in the future uh you won't have you
19:06
but in the future uh you won't have you
19:06
but in the future uh you won't have you wanna have to do that you can just set
19:08
wanna have to do that you can just set
19:08
wanna have to do that you can just set it on autopilot if you need to
19:10
it on autopilot if you need to
19:10
it on autopilot if you need to um many other improvements kind of we're
19:14
um many other improvements kind of we're
19:14
um many other improvements kind of we're looking at
19:15
looking at
19:15
looking at in the other scale area how to kind of
19:17
in the other scale area how to kind of
19:17
in the other scale area how to kind of deal with uh
19:18
deal with uh
19:18
deal with uh hot partition keys when in the other
19:20
hot partition keys when in the other
19:20
hot partition keys when in the other scale and how to make it
19:22
scale and how to make it
19:22
scale and how to make it more easier for your application we also
19:25
more easier for your application we also
19:25
more easier for your application we also recently introduced
19:26
recently introduced
19:26
recently introduced serverless um not everyone may know this
19:30
serverless um not everyone may know this
19:30
serverless um not everyone may know this we're using cosmos db uh but cause today
19:33
we're using cosmos db uh but cause today
19:33
we're using cosmos db uh but cause today you have several servers capability
19:35
you have several servers capability
19:35
you have several servers capability available with cosmos db
19:37
available with cosmos db
19:37
available with cosmos db now of course this db always was
19:38
now of course this db always was
19:38
now of course this db always was serverless i mean there are no servers
19:39
serverless i mean there are no servers
19:39
serverless i mean there are no servers behind the scenes
19:40
behind the scenes
19:40
behind the scenes you just provisions throughput or you
19:42
you just provisions throughput or you
19:42
you just provisions throughput or you set it to another scale and it goes
19:44
set it to another scale and it goes
19:44
set it to another scale and it goes uh serverless capacity mode for cosmos
19:46
uh serverless capacity mode for cosmos
19:46
uh serverless capacity mode for cosmos db means that you
19:48
db means that you
19:48
db means that you pay per request so
19:51
pay per request so
19:51
pay per request so one million requests cost 25 cents
19:54
one million requests cost 25 cents
19:54
one million requests cost 25 cents um you send it if you don't send any
19:56
um you send it if you don't send any
19:56
um you send it if you don't send any requests you do not pay anything
19:58
requests you do not pay anything
19:58
requests you do not pay anything uh similarly you pay for storage the
20:01
uh similarly you pay for storage the
20:01
uh similarly you pay for storage the same way 25 cents per gigabyte
20:03
same way 25 cents per gigabyte
20:03
same way 25 cents per gigabyte you don't store anything you don't pay
20:04
you don't store anything you don't pay
20:04
you don't store anything you don't pay for anything uh so it's pure
20:06
for anything uh so it's pure
20:06
for anything uh so it's pure consumption model uh you don't you don't
20:09
consumption model uh you don't you don't
20:09
consumption model uh you don't you don't pre-provision anything it's great when
20:11
pre-provision anything it's great when
20:11
pre-provision anything it's great when you don't know what to expect from the
20:12
you don't know what to expect from the
20:12
you don't know what to expect from the application when you're just
20:13
application when you're just
20:13
application when you're just getting started it's great when you have
20:15
getting started it's great when you have
20:15
getting started it's great when you have bursty intermittent traffic that is hard
20:17
bursty intermittent traffic that is hard
20:17
bursty intermittent traffic that is hard to predict
20:19
to predict
20:19
to predict it's uh awesome for when you have
20:22
it's uh awesome for when you have
20:22
it's uh awesome for when you have scenarios that have really low
20:24
scenarios that have really low
20:24
scenarios that have really low average to peak uh traffic ratio
20:27
average to peak uh traffic ratio
20:27
average to peak uh traffic ratio uh it's very economical it's fantastic
20:30
uh it's very economical it's fantastic
20:30
uh it's very economical it's fantastic for that test
20:31
for that test
20:31
for that test you don't have to drop turn off your
20:33
you don't have to drop turn off your
20:33
you don't have to drop turn off your database anymore when you go on on
20:34
database anymore when you go on on
20:34
database anymore when you go on on weekends
20:35
weekends
20:35
weekends uh or worries that you forgot to turn it
20:37
uh or worries that you forgot to turn it
20:37
uh or worries that you forgot to turn it off um
20:39
off um
20:39
off um it's if you don't send traffic you don't
20:42
it's if you don't send traffic you don't
20:42
it's if you don't send traffic you don't pay
20:43
pay
20:43
pay this serverless is available today for
20:44
this serverless is available today for
20:44
this serverless is available today for all apis
20:47
all apis
20:47
all apis and we keep we keep tweaking on it like
20:49
and we keep we keep tweaking on it like
20:49
and we keep we keep tweaking on it like we will we keep it we'll keep increasing
20:51
we will we keep it we'll keep increasing
20:51
we will we keep it we'll keep increasing the limits
20:52
the limits
20:52
the limits today not all the features are available
20:54
today not all the features are available
20:54
today not all the features are available for serverless
20:55
for serverless
20:55
for serverless this will change over time the intent is
20:57
this will change over time the intent is
20:57
this will change over time the intent is to offer all of the database
20:59
to offer all of the database
20:59
to offer all of the database for example uh global distribution today
21:01
for example uh global distribution today
21:01
for example uh global distribution today is not available for serverless
21:02
is not available for serverless
21:02
is not available for serverless applications
21:03
applications
21:03
applications this will change um will increase as
21:06
this will change um will increase as
21:06
this will change um will increase as soon as the storage
21:07
soon as the storage
21:07
soon as the storage uh limits for serverless
21:11
uh limits for serverless
21:11
uh limits for serverless databases uh many many other
21:13
databases uh many many other
21:13
databases uh many many other improvements are coming in
21:15
improvements are coming in
21:15
improvements are coming in for serverless stay tuned
21:18
for serverless stay tuned
21:18
for serverless stay tuned uh he was exciting um
21:21
uh he was exciting um
21:21
uh he was exciting um since we're afraid among friends here
21:24
since we're afraid among friends here
21:24
since we're afraid among friends here are
21:24
are
21:24
are other exciting things coming down the
21:26
other exciting things coming down the
21:26
other exciting things coming down the road um we are removing our
21:28
road um we are removing our
21:28
road um we are removing our gp limits some of you may know it but
21:30
gp limits some of you may know it but
21:30
gp limits some of you may know it but everyone
21:31
everyone
21:31
everyone is something when you can when you uh
21:34
is something when you can when you uh
21:34
is something when you can when you uh the application really uses cosmos dp to
21:36
the application really uses cosmos dp to
21:36
the application really uses cosmos dp to store a lot of data but the traffic is
21:37
store a lot of data but the traffic is
21:37
store a lot of data but the traffic is not high
21:39
not high
21:39
not high in the past you had to always provision
21:41
in the past you had to always provision
21:41
in the past you had to always provision some amount of throughput
21:43
some amount of throughput
21:43
some amount of throughput that uh will be gone uh this
21:46
that uh will be gone uh this
21:46
that uh will be gone uh this this spring and it's actually already
21:49
this spring and it's actually already
21:49
this spring and it's actually already you just need to
21:50
you just need to
21:50
you just need to reach out to us in documentation there
21:52
reach out to us in documentation there
21:52
reach out to us in documentation there is a link
21:53
is a link
21:53
is a link uh if you request will remove that limit
21:55
uh if you request will remove that limit
21:55
uh if you request will remove that limit for you
21:56
for you
21:56
for you um pretty exciting about this so you can
21:59
um pretty exciting about this so you can
21:59
um pretty exciting about this so you can we're expanding the
22:00
we're expanding the
22:00
we're expanding the um scenarios for cosmos db into kind of
22:04
um scenarios for cosmos db into kind of
22:04
um scenarios for cosmos db into kind of high storage scenarios um
22:07
high storage scenarios um
22:07
high storage scenarios um not everyone runs teams with trillion
22:09
not everyone runs teams with trillion
22:10
not everyone runs teams with trillion transactions per day
22:12
transactions per day
22:12
transactions per day but lots of very exciting scenarios
22:14
but lots of very exciting scenarios
22:14
but lots of very exciting scenarios where maybe you need high throughput but
22:16
where maybe you need high throughput but
22:16
where maybe you need high throughput but only for a limited amount of time
22:18
only for a limited amount of time
22:18
only for a limited amount of time or maybe your throughput is not as high
22:19
or maybe your throughput is not as high
22:19
or maybe your throughput is not as high but your data is really high
22:21
but your data is really high
22:21
but your data is really high uh really large now we're adding
22:23
uh really large now we're adding
22:23
uh really large now we're adding petition key advisor
22:25
petition key advisor
22:25
petition key advisor partition the choice of partition key is
22:27
partition the choice of partition key is
22:27
partition the choice of partition key is very critical when you use cosmos db
22:29
very critical when you use cosmos db
22:29
very critical when you use cosmos db and it's uh
22:33
and it's uh
22:33
and it's uh it's important um to choose it wisely uh
22:36
it's important um to choose it wisely uh
22:36
it's important um to choose it wisely uh and sometimes you don't always can
22:38
and sometimes you don't always can
22:38
and sometimes you don't always can predict the future
22:39
predict the future
22:40
predict the future um so you can it help partition key
22:42
um so you can it help partition key
22:42
um so you can it help partition key advisor helps you guide
22:43
advisor helps you guide
22:43
advisor helps you guide how to provide a kind of um
22:47
good future proof choice of partition
22:49
good future proof choice of partition
22:49
good future proof choice of partition key for your application
22:51
key for your application
22:52
key for your application we're also adding a capability called
22:54
we're also adding a capability called
22:54
we're also adding a capability called sub-partitioning
22:55
sub-partitioning
22:55
sub-partitioning uh this is where you can select uh
22:59
uh this is where you can select uh
22:59
uh this is where you can select uh add additional partition keys over time
23:02
add additional partition keys over time
23:02
add additional partition keys over time um kind of nested so that you don't have
23:04
um kind of nested so that you don't have
23:04
um kind of nested so that you don't have to
23:05
to
23:06
to see uh again if you over time if you
23:08
see uh again if you over time if you
23:08
see uh again if you over time if you discover
23:09
discover
23:09
discover uh you know uh there's always a joke
23:11
uh you know uh there's always a joke
23:11
uh you know uh there's always a joke among the scale-out systems that
23:13
among the scale-out systems that
23:13
among the scale-out systems that when you discover london and new york
23:15
when you discover london and new york
23:15
when you discover london and new york among your user base
23:17
among your user base
23:17
among your user base uh among your partition keys you have to
23:19
uh among your partition keys you have to
23:19
uh among your partition keys you have to do something about it
23:20
do something about it
23:20
do something about it uh so if you discover a really large
23:22
uh so if you discover a really large
23:22
uh so if you discover a really large partition key
23:23
partition key
23:23
partition key subpartition will help you subdivide it
23:26
subpartition will help you subdivide it
23:26
subpartition will help you subdivide it and so that you can take full advantage
23:27
and so that you can take full advantage
23:27
and so that you can take full advantage of cosmos db
23:28
of cosmos db
23:28
of cosmos db uh seamless transparent partitioning
23:31
uh seamless transparent partitioning
23:31
uh seamless transparent partitioning behind the scenes
23:32
behind the scenes
23:32
behind the scenes um and we talked about how to adjustable
23:35
um and we talked about how to adjustable
23:36
um and we talked about how to adjustable auto scale range lots of other
23:37
auto scale range lots of other
23:38
auto scale range lots of other improvements coming
23:41
now let's talk about apis uh cosmos db
23:44
now let's talk about apis uh cosmos db
23:44
now let's talk about apis uh cosmos db uh is a special attempt pay special
23:48
uh is a special attempt pay special
23:48
uh is a special attempt pay special attention to offering easy to use
23:50
attention to offering easy to use
23:50
attention to offering easy to use api and programming model whether you
23:51
api and programming model whether you
23:52
api and programming model whether you choose sql api
23:53
choose sql api
23:53
choose sql api and you can use our drivers so whether
23:55
and you can use our drivers so whether
23:55
and you can use our drivers so whether you want to use mongodb drivers
23:57
you want to use mongodb drivers
23:57
you want to use mongodb drivers and programming model or cassandra
23:59
and programming model or cassandra
23:59
and programming model or cassandra programming model uh we are
24:01
programming model uh we are
24:01
programming model uh we are there for you and cosmos dp api for
24:03
there for you and cosmos dp api for
24:03
there for you and cosmos dp api for mongodb is a very popular api
24:05
mongodb is a very popular api
24:05
mongodb is a very popular api among the bfs is a very popular
24:07
among the bfs is a very popular
24:07
among the bfs is a very popular programming language
24:09
programming language
24:09
programming language it's a great community uh we love
24:11
it's a great community uh we love
24:11
it's a great community uh we love mongodb developers
24:12
mongodb developers
24:12
mongodb developers um and we offer we bring
24:16
um and we offer we bring
24:16
um and we offer we bring all the this uh definition
24:19
all the this uh definition
24:19
all the this uh definition all the cosmos db goodness this
24:21
all the cosmos db goodness this
24:22
all the cosmos db goodness this transparent partitioning
24:23
transparent partitioning
24:23
transparent partitioning a limitless scale elasticity to
24:27
a limitless scale elasticity to
24:27
a limitless scale elasticity to the table offering you through mongodb
24:30
the table offering you through mongodb
24:30
the table offering you through mongodb api
24:32
api
24:32
api now we recently announced support from
24:34
now we recently announced support from
24:34
now we recently announced support from onbdb for all and we'll follow up with
24:37
onbdb for all and we'll follow up with
24:37
onbdb for all and we'll follow up with mongodb 4.2
24:38
mongodb 4.2
24:38
mongodb 4.2 and we'll continue adding and securing
24:40
and we'll continue adding and securing
24:40
and we'll continue adding and securing keeping steps with the development of
24:42
keeping steps with the development of
24:42
keeping steps with the development of mongodb
24:44
mongodb
24:44
mongodb programming models uh uh cosmos db
24:47
programming models uh uh cosmos db
24:47
programming models uh uh cosmos db offers wire compatible api for mobidb
24:49
offers wire compatible api for mobidb
24:50
offers wire compatible api for mobidb uh the nice thing about it is we can
24:52
uh the nice thing about it is we can
24:52
uh the nice thing about it is we can transparently offer you both
24:54
transparently offer you both
24:54
transparently offer you both right so they might for example upgrade
24:56
right so they might for example upgrade
24:56
right so they might for example upgrade from three point six to four
24:58
from three point six to four
24:58
from three point six to four should you need it is transparent you
25:00
should you need it is transparent you
25:00
should you need it is transparent you don't need to bring anything down
25:02
don't need to bring anything down
25:02
don't need to bring anything down there is no kind of rolling upgrade or
25:04
there is no kind of rolling upgrade or
25:04
there is no kind of rolling upgrade or any of the constants
25:05
any of the constants
25:05
any of the constants uh you just turn it button you turn a
25:08
uh you just turn it button you turn a
25:08
uh you just turn it button you turn a button
25:08
button
25:08
button and photo version is available to you
25:13
some interesting and exciting features
25:15
some interesting and exciting features
25:15
some interesting and exciting features we brought uh kind of client-driven
25:17
we brought uh kind of client-driven
25:17
we brought uh kind of client-driven transaction support
25:18
transaction support
25:18
transaction support to cosmos db we had transaction support
25:21
to cosmos db we had transaction support
25:22
to cosmos db we had transaction support in the past but it required user
25:23
in the past but it required user
25:23
in the past but it required user using stored procedures uh not
25:26
using stored procedures uh not
25:26
using stored procedures uh not everyone's cup of tea to use kind of
25:28
everyone's cup of tea to use kind of
25:28
everyone's cup of tea to use kind of stock procedures on the server side
25:30
stock procedures on the server side
25:30
stock procedures on the server side we brought uh built-in uh transpar
25:32
we brought uh built-in uh transpar
25:32
we brought uh built-in uh transpar transaction support
25:34
transaction support
25:34
transaction support as part of mongodb for uh for all and it
25:37
as part of mongodb for uh for all and it
25:37
as part of mongodb for uh for all and it will be offered across
25:38
will be offered across
25:38
will be offered across all apis um we
25:41
all apis um we
25:42
all apis um we offer the um zero downtime upgrade uh
25:46
offer the um zero downtime upgrade uh
25:46
offer the um zero downtime upgrade uh the binary encoding got really really
25:48
the binary encoding got really really
25:48
the binary encoding got really really efficient okay
25:49
efficient okay
25:49
efficient okay in fact if you upgrade if you run using
25:51
in fact if you upgrade if you run using
25:51
in fact if you upgrade if you run using mongodb 3.6
25:53
mongodb 3.6
25:53
mongodb 3.6 today um with cosmos db
25:56
today um with cosmos db
25:56
today um with cosmos db uh i really recommend you to upgrade if
25:58
uh i really recommend you to upgrade if
25:58
uh i really recommend you to upgrade if you can because you'll get significant
26:00
you can because you'll get significant
26:00
you can because you'll get significant improvements
26:01
improvements
26:01
improvements from the efficient binary coding um you
26:04
from the efficient binary coding um you
26:04
from the efficient binary coding um you you may save up to 40
26:06
you may save up to 40
26:06
you may save up to 40 in cost um both on the encodings as well
26:09
in cost um both on the encodings as well
26:09
in cost um both on the encodings as well as
26:10
as
26:10
as the uh are you cost of your query
26:12
the uh are you cost of your query
26:12
the uh are you cost of your query operations and the
26:13
operations and the
26:13
operations and the read operations um and i
26:17
read operations um and i
26:17
read operations um and i of course if you're using today today
26:19
of course if you're using today today
26:19
of course if you're using today today mongodb on virtual machines or on
26:21
mongodb on virtual machines or on
26:21
mongodb on virtual machines or on premises
26:22
premises
26:22
premises it's very easy to migrate take that pain
26:25
it's very easy to migrate take that pain
26:25
it's very easy to migrate take that pain away
26:25
away
26:25
away uh from your from you focus on on the
26:28
uh from your from you focus on on the
26:28
uh from your from you focus on on the application
26:30
application
26:30
application um we can manage customers db
26:33
um we can manage customers db
26:33
um we can manage customers db is a fully managed database you don't
26:34
is a fully managed database you don't
26:34
is a fully managed database you don't need to worry about vms you don't need
26:36
need to worry about vms you don't need
26:36
need to worry about vms you don't need to put it
26:37
to put it
26:37
to put it but managing the database there's no
26:41
but managing the database there's no
26:41
but managing the database there's no concept of dba with cosmos db it's a
26:44
concept of dba with cosmos db it's a
26:44
concept of dba with cosmos db it's a database for developers developers can
26:45
database for developers developers can
26:46
database for developers developers can focus on the application
26:47
focus on the application
26:47
focus on the application we have your back we take care of it no
26:49
we have your back we take care of it no
26:49
we have your back we take care of it no matter where the database
26:51
matter where the database
26:51
matter where the database growth 10x overnight it will be it'll be
26:54
growth 10x overnight it will be it'll be
26:54
growth 10x overnight it will be it'll be there for you
26:55
there for you
26:55
there for you just choose the right partition key
26:58
just choose the right partition key
26:58
just choose the right partition key pretty exciting stuff let me show you
27:02
pretty exciting stuff let me show you
27:02
pretty exciting stuff let me show you a few demos
27:15
so first um i want to show you how easy
27:18
so first um i want to show you how easy
27:18
so first um i want to show you how easy it is
27:19
it is
27:19
it is for mongodb developers to start with
27:21
for mongodb developers to start with
27:21
for mongodb developers to start with cosmos db
27:22
cosmos db
27:22
cosmos db right so uh you can actually let's start
27:24
right so uh you can actually let's start
27:24
right so uh you can actually let's start with
27:31
a particular search engine of choice
27:35
that's fine try cosmos db this is a very
27:39
that's fine try cosmos db this is a very
27:39
that's fine try cosmos db this is a very handy and
27:40
handy and
27:40
handy and useful sandbox that you can select
27:46
it does not require you can start
27:47
it does not require you can start
27:47
it does not require you can start working with customers db without
27:50
working with customers db without
27:50
working with customers db without uh signing up for azure without
27:52
uh signing up for azure without
27:52
uh signing up for azure without providing your credit card
27:54
providing your credit card
27:54
providing your credit card i want to create mongodb account
27:57
i want to create mongodb account
27:57
i want to create mongodb account i'm already saved signed in with
27:59
i'm already saved signed in with
27:59
i'm already saved signed in with microsoft account uh
28:00
microsoft account uh
28:00
microsoft account uh but if i weren't it would it was it
28:03
but if i weren't it would it was it
28:03
but if i weren't it would it was it would ask me to
28:05
would ask me to
28:05
would ask me to now it just took seconds and i now have
28:07
now it just took seconds and i now have
28:07
now it just took seconds and i now have my cost
28:08
my cost
28:08
my cost free cosmos db account again as far as
28:11
free cosmos db account again as far as
28:11
free cosmos db account again as far as this is concerned i don't have either
28:13
this is concerned i don't have either
28:13
this is concerned i don't have either subscription i didn't provide any
28:14
subscription i didn't provide any
28:14
subscription i didn't provide any creative card
28:15
creative card
28:15
creative card i just click a button and it gave me a
28:17
i just click a button and it gave me a
28:17
i just click a button and it gave me a free cosmos db
28:18
free cosmos db
28:18
free cosmos db and it's a very beefy cosmos db account
28:21
and it's a very beefy cosmos db account
28:21
and it's a very beefy cosmos db account it can give you up to 20 000 reviews
28:23
it can give you up to 20 000 reviews
28:23
it can give you up to 20 000 reviews for free for a month with limitless
28:26
for free for a month with limitless
28:26
for free for a month with limitless renewals
28:27
renewals
28:28
renewals to kick the tires um great environment
28:30
to kick the tires um great environment
28:30
to kick the tires um great environment uh
28:31
uh
28:31
uh twenty thousand are uses a lot it's like
28:33
twenty thousand are uses a lot it's like
28:33
twenty thousand are uses a lot it's like two uh up to
28:34
two uh up to
28:34
two uh up to two thousand or more writes per second
28:37
two thousand or more writes per second
28:37
two thousand or more writes per second uh twenty thousand reads per second
28:40
uh twenty thousand reads per second
28:40
uh twenty thousand reads per second and we give you portal experience with
28:42
and we give you portal experience with
28:42
and we give you portal experience with cosmos db now you can't
28:44
cosmos db now you can't
28:44
cosmos db now you can't use all of azure in this portal window
28:47
use all of azure in this portal window
28:47
use all of azure in this portal window just because
28:48
just because
28:48
just because well you don't have azure subscription
28:50
well you don't have azure subscription
28:50
well you don't have azure subscription but you can use cosmos db and all the
28:51
but you can use cosmos db and all the
28:52
but you can use cosmos db and all the features
28:53
features
28:53
features uh through through the portal and then
28:55
uh through through the portal and then
28:55
uh through through the portal and then switch to a
28:56
switch to a
28:56
switch to a proper azure subscriptions once you feel
28:59
proper azure subscriptions once you feel
28:59
proper azure subscriptions once you feel like this is your cup of tea
29:00
like this is your cup of tea
29:00
like this is your cup of tea so let's create some collection um i can
29:03
so let's create some collection um i can
29:03
so let's create some collection um i can play with it right here
29:04
play with it right here
29:04
play with it right here in my data explorer i can
29:08
in my data explorer i can
29:08
in my data explorer i can i can go and look i can create
29:13
here i can enable notebooks a bunch of i
29:16
here i can enable notebooks a bunch of i
29:16
here i can enable notebooks a bunch of i can use shell
29:18
can use shell
29:18
can use shell i can create
29:21
new documents right here
29:27
new documents right here
29:27
new documents right here save etc now but this is a mongodb
29:30
save etc now but this is a mongodb
29:30
save etc now but this is a mongodb database
29:31
database
29:31
database it's just better right it's like uh i
29:33
it's just better right it's like uh i
29:33
it's just better right it's like uh i can use work with it as a mongodb
29:35
can use work with it as a mongodb
29:35
can use work with it as a mongodb database that's the whole point of it so
29:36
database that's the whole point of it so
29:36
database that's the whole point of it so let me go to connection string
29:38
let me go to connection string
29:38
let me go to connection string let me take this connection string from
29:40
let me take this connection string from
29:40
let me take this connection string from here
29:41
here
29:41
here and let me walk over to uh one of the
29:44
and let me walk over to uh one of the
29:44
and let me walk over to uh one of the popular
29:45
popular
29:45
popular mongodb tools and connect
29:49
mongodb tools and connect
29:49
mongodb tools and connect now studio 3t is a popular tool out
29:51
now studio 3t is a popular tool out
29:51
now studio 3t is a popular tool out there there are many others and you can
29:52
there there are many others and you can
29:52
there there are many others and you can use any of them with
29:54
use any of them with
29:54
use any of them with this tool does not know that i'm working
29:55
this tool does not know that i'm working
29:55
this tool does not know that i'm working with cosmos db okay
29:57
with cosmos db okay
29:57
with cosmos db okay and knows nothing about questions db um
30:01
and knows nothing about questions db um
30:01
and knows nothing about questions db um but it detected that i have a mongodb
30:03
but it detected that i have a mongodb
30:03
but it detected that i have a mongodb connection string in the clipboard
30:05
connection string in the clipboard
30:05
connection string in the clipboard um said okay let's import it
30:09
um said okay let's import it
30:09
um said okay let's import it now we can see the url is actually what
30:12
now we can see the url is actually what
30:12
now we can see the url is actually what we just copied
30:13
we just copied
30:13
we just copied right um so that's great let's test the
30:16
right um so that's great let's test the
30:16
right um so that's great let's test the connection
30:21
again it just thinks that it's mortgage
30:23
again it just thinks that it's mortgage
30:23
again it just thinks that it's mortgage b out there
30:24
b out there
30:24
b out there and it's excited to talk to it awesome
30:27
and it's excited to talk to it awesome
30:27
and it's excited to talk to it awesome let's connect to this
30:32
and here i have my to-do list
30:35
and here i have my to-do list
30:35
and here i have my to-do list collections that i just created i can go
30:36
collections that i just created i can go
30:36
collections that i just created i can go to items i can
30:42
i can see the documents that i added i
30:44
i can see the documents that i added i
30:44
i can see the documents that i added i can add more documents if needed
31:03
this is all great i can go back to my
31:04
this is all great i can go back to my
31:04
this is all great i can go back to my data explorer i can see that
31:07
data explorer i can see that
31:07
data explorer i can see that the tools are working as expected i can
31:08
the tools are working as expected i can
31:08
the tools are working as expected i can look at the
31:10
look at the
31:10
look at the um now i have two documents
31:13
um now i have two documents
31:13
um now i have two documents one of them created created by creator
31:15
one of them created created by creator
31:15
one of them created created by creator studio
31:16
studio
31:16
studio excellent now this is of course hello
31:18
excellent now this is of course hello
31:18
excellent now this is of course hello world this has been out there for a
31:19
world this has been out there for a
31:19
world this has been out there for a while
31:20
while
31:20
while uh some of you have uh if you use smokey
31:23
uh some of you have uh if you use smokey
31:23
uh some of you have uh if you use smokey db api or cosmos db you're familiar with
31:24
db api or cosmos db you're familiar with
31:24
db api or cosmos db you're familiar with all that
31:25
all that
31:25
all that now what's exciting is that let's take a
31:28
now what's exciting is that let's take a
31:28
now what's exciting is that let's take a look at another database that i have
31:29
look at another database that i have
31:29
look at another database that i have here
31:31
here
31:31
here and it's a little bit bigger
31:35
okay so this one is let's see well it's
31:38
okay so this one is let's see well it's
31:38
okay so this one is let's see well it's about 17 terabytes right
31:40
about 17 terabytes right
31:40
about 17 terabytes right so
31:43
it takes some effort to set up a 17
31:46
it takes some effort to set up a 17
31:46
it takes some effort to set up a 17 terabyte small db database on virtual
31:48
terabyte small db database on virtual
31:48
terabyte small db database on virtual machines
31:49
machines
31:49
machines and good luck doing it on any of the
31:51
and good luck doing it on any of the
31:51
and good luck doing it on any of the mongodb
31:53
mongodb
31:53
mongodb services out there
31:56
services out there
31:56
services out there with cosmos db it's a piece of cake you
31:59
with cosmos db it's a piece of cake you
31:59
with cosmos db it's a piece of cake you create we
32:00
create we
32:00
create we keep pumping data whether it's 17
32:01
keep pumping data whether it's 17
32:01
keep pumping data whether it's 17 terabytes whether it's five terabytes
32:04
terabytes whether it's five terabytes
32:04
terabytes whether it's five terabytes but we've seen with steams you just keep
32:06
but we've seen with steams you just keep
32:06
but we've seen with steams you just keep ingesting data into
32:07
ingesting data into
32:07
ingesting data into into cosmos db it'll scale out
32:09
into cosmos db it'll scale out
32:09
into cosmos db it'll scale out transparently to you
32:11
transparently to you
32:11
transparently to you as long as you provide for us a shard
32:13
as long as you provide for us a shard
32:13
as long as you provide for us a shard key now let's
32:14
key now let's
32:14
key now let's uh go to our connection string actually
32:17
uh go to our connection string actually
32:17
uh go to our connection string actually i think i already have it opened here um
32:21
i think i already have it opened here um
32:21
i think i already have it opened here um yeah so this is how this this is the
32:24
yeah so this is how this this is the
32:24
yeah so this is how this this is the database i have i can work with it
32:26
database i have i can work with it
32:26
database i have i can work with it let me go to data explorer i can work
32:29
let me go to data explorer i can work
32:29
let me go to data explorer i can work with it here
32:30
with it here
32:30
with it here uh or i can uh
32:33
uh or i can uh
32:33
uh or i can uh i can use my there's an exciting kind of
32:38
i can i can use work with it here uh or
32:41
i can i can use work with it here uh or
32:41
i can i can use work with it here uh or i can
32:42
i can
32:42
i can uh and the data
32:45
uh and the data
32:45
uh and the data distort here is kind of derivative of
32:47
distort here is kind of derivative of
32:47
distort here is kind of derivative of the popular flight information
32:51
uh or i can go and you can see this this
32:54
uh or i can go and you can see this this
32:54
uh or i can go and you can see this this has all the flights from the
32:56
has all the flights from the
32:56
has all the flights from the transportation bureau
32:57
transportation bureau
32:57
transportation bureau and we kind of multiply this a few times
32:59
and we kind of multiply this a few times
32:59
and we kind of multiply this a few times to get
33:00
to get
33:00
to get to get to watch enough size or i can go
33:03
to get to watch enough size or i can go
33:03
to get to watch enough size or i can go right here and i can
33:08
and i can run a proper uh
33:13
and i can run a proper uh
33:13
and i can run a proper uh aggregation uh pipeline query against
33:16
aggregation uh pipeline query against
33:16
aggregation uh pipeline query against this database
33:17
this database
33:17
this database and again this is kind of 17 terabytes
33:20
and again this is kind of 17 terabytes
33:20
and again this is kind of 17 terabytes so in american if you're running only to
33:22
so in american if you're running only to
33:22
so in american if you're running only to be on vms let me take a while
33:24
be on vms let me take a while
33:24
be on vms let me take a while but with cosmos db that's pretty
33:36
straightforward and it will
33:38
straightforward and it will
33:38
straightforward and it will it will run a little bit and then it
33:40
it will run a little bit and then it
33:40
it will run a little bit and then it will return back the results
33:51
and there you go well this is actually
33:54
and there you go well this is actually
33:54
and there you go well this is actually did explain um but my point is that
33:57
did explain um but my point is that
33:57
did explain um but my point is that you can you can aggregate you cannot
33:59
you can you can aggregate you cannot
33:59
you can you can aggregate you cannot aggregation queries very fast
34:01
aggregation queries very fast
34:01
aggregation queries very fast against any size of data and that's
34:03
against any size of data and that's
34:03
against any size of data and that's where kind of the
34:04
where kind of the
34:04
where kind of the benefits of cosmos db transpire right
34:09
so there are a few reasons why uh folks
34:12
so there are a few reasons why uh folks
34:12
so there are a few reasons why uh folks choose customs db api from only db
34:14
choose customs db api from only db
34:14
choose customs db api from only db one is that you don't need to worry
34:15
one is that you don't need to worry
34:16
one is that you don't need to worry about manually sharding kind of creating
34:18
about manually sharding kind of creating
34:18
about manually sharding kind of creating shards managing shards it's automatic
34:20
shards managing shards it's automatic
34:20
shards managing shards it's automatic and transparent for you
34:22
and transparent for you
34:22
and transparent for you limitless scale you can start with a
34:24
limitless scale you can start with a
34:24
limitless scale you can start with a megabyte and you can
34:25
megabyte and you can
34:25
megabyte and you can end with it and you can get to petabytes
34:28
end with it and you can get to petabytes
34:28
end with it and you can get to petabytes of data
34:29
of data
34:29
of data in fact the single largest mongodb
34:30
in fact the single largest mongodb
34:30
in fact the single largest mongodb database you have on cosmos db it's got
34:32
database you have on cosmos db it's got
34:32
database you have on cosmos db it's got more than 600 terabytes
34:34
more than 600 terabytes
34:34
more than 600 terabytes good luck doing it on the native one
34:35
good luck doing it on the native one
34:35
good luck doing it on the native one will be engine
34:37
will be engine
34:37
will be engine uh it's uh it's transparency
34:40
uh it's uh it's transparency
34:40
uh it's uh it's transparency um automatic instant and granular
34:43
um automatic instant and granular
34:43
um automatic instant and granular scaling
34:44
scaling
34:44
scaling this instant out of scale that i talked
34:45
this instant out of scale that i talked
34:45
this instant out of scale that i talked about um
34:47
about um
34:47
about um it's a signifi it's mission critical
34:50
it's a signifi it's mission critical
34:50
it's a signifi it's mission critical already
34:51
already
34:51
already you've seen kind of measure the critical
34:54
you've seen kind of measure the critical
34:54
you've seen kind of measure the critical applications that we all rely on
34:56
applications that we all rely on
34:56
applications that we all rely on um whether you're having your kids
34:59
um whether you're having your kids
34:59
um whether you're having your kids attend school
35:00
attend school
35:00
attend school whether you're going and ordering cup of
35:01
whether you're going and ordering cup of
35:02
whether you're going and ordering cup of coffee you buy something in a grocery
35:03
coffee you buy something in a grocery
35:04
coffee you buy something in a grocery store
35:04
store
35:04
store you use cosmos db in your daily life
35:07
you use cosmos db in your daily life
35:07
you use cosmos db in your daily life every one of us
35:08
every one of us
35:08
every one of us and we rely on it working and this is
35:11
and we rely on it working and this is
35:11
and we rely on it working and this is this is guys the values that cosmos db
35:13
this is guys the values that cosmos db
35:13
this is guys the values that cosmos db brings
35:13
brings
35:13
brings is that they even given azure region
35:16
is that they even given azure region
35:16
is that they even given azure region goes down
35:17
goes down
35:17
goes down rto is zero if you're using cosmos db
35:19
rto is zero if you're using cosmos db
35:19
rto is zero if you're using cosmos db active active
35:21
active active
35:21
active active uh it's serverless you don't need to
35:23
uh it's serverless you don't need to
35:23
uh it's serverless you don't need to worry about instances you pay only when
35:25
worry about instances you pay only when
35:25
worry about instances you pay only when you use it
35:26
you use it
35:26
you use it and we'll talk about real-time analytics
35:28
and we'll talk about real-time analytics
35:28
and we'll talk about real-time analytics uh in
35:29
uh in
35:29
uh in later in the talk now just few
35:33
later in the talk now just few
35:33
later in the talk now just few things that we recently announced uh
35:35
things that we recently announced uh
35:35
things that we recently announced uh advancements to our cassandra api this
35:37
advancements to our cassandra api this
35:37
advancements to our cassandra api this is another api we love
35:39
is another api we love
35:39
is another api we love uh great great developers out there
35:41
uh great great developers out there
35:41
uh great great developers out there using the cassandra
35:43
using the cassandra
35:43
using the cassandra we have now and managing since for
35:45
we have now and managing since for
35:45
we have now and managing since for apache cassandra that helps you build
35:47
apache cassandra that helps you build
35:47
apache cassandra that helps you build hybrid cassandra deployments and you can
35:50
hybrid cassandra deployments and you can
35:50
hybrid cassandra deployments and you can you can take cosmos db you can take this
35:52
you can take cosmos db you can take this
35:52
you can take cosmos db you can take this manage instances for pasha cassandra
35:54
manage instances for pasha cassandra
35:54
manage instances for pasha cassandra and you can join an existing cassandra
35:56
and you can join an existing cassandra
35:56
and you can join an existing cassandra ring to facilitate migrations
35:59
ring to facilitate migrations
35:59
ring to facilitate migrations you get flexibility and control uh
36:02
you get flexibility and control uh
36:02
you get flexibility and control uh you can join you can take advantage of
36:06
you can join you can take advantage of
36:06
you can join you can take advantage of cosmos db serverless
36:07
cosmos db serverless
36:07
cosmos db serverless always in your existing cassandra
36:09
always in your existing cassandra
36:09
always in your existing cassandra application without making any changes
36:11
application without making any changes
36:11
application without making any changes to it and you can turn on things like
36:13
to it and you can turn on things like
36:13
to it and you can turn on things like real-time already analytics twist and
36:14
real-time already analytics twist and
36:14
real-time already analytics twist and apps link
36:17
apps link
36:17
apps link now with that let me bring
36:20
now with that let me bring
36:20
now with that let me bring uh to on stage and welcome on stage the
36:23
uh to on stage and welcome on stage the
36:23
uh to on stage and welcome on stage the person that doesn't really need an
36:24
person that doesn't really need an
36:24
person that doesn't really need an introduction guillermo raj
36:26
introduction guillermo raj
36:26
introduction guillermo raj uh ceo of versailles very well known in
36:29
uh ceo of versailles very well known in
36:29
uh ceo of versailles very well known in developer community
36:30
developer community
36:30
developer community uh the many contributions many modules
36:34
uh the many contributions many modules
36:34
uh the many contributions many modules that we all used
36:35
that we all used
36:35
that we all used uh in the past yeah welcome thanks for
36:39
uh in the past yeah welcome thanks for
36:39
uh in the past yeah welcome thanks for having me
36:40
having me
36:40
having me excited to have you tell us a little
36:42
excited to have you tell us a little
36:42
excited to have you tell us a little more about verso
36:44
more about verso
36:44
more about verso let me bring you guys yeah um thanks for
36:47
let me bring you guys yeah um thanks for
36:47
let me bring you guys yeah um thanks for bringing up our website
36:48
bringing up our website
36:48
bringing up our website um so uh versailles is a serverless
36:50
um so uh versailles is a serverless
36:50
um so uh versailles is a serverless platform uh for modern
36:52
platform uh for modern
36:52
platform uh for modern front end focused applications and
36:55
front end focused applications and
36:55
front end focused applications and teams so customers like tripadvisor
36:58
teams so customers like tripadvisor
36:58
teams so customers like tripadvisor patreon washington post they host and
37:02
patreon washington post they host and
37:02
patreon washington post they host and deploy
37:02
deploy
37:02
deploy large parts of their online presence on
37:05
large parts of their online presence on
37:05
large parts of their online presence on versailles
37:06
versailles
37:06
versailles they're using modern frameworks like
37:08
they're using modern frameworks like
37:08
they're using modern frameworks like nexjs
37:09
nexjs
37:09
nexjs which we create and maintain or others
37:13
which we create and maintain or others
37:13
which we create and maintain or others to create really rich interactive and
37:15
to create really rich interactive and
37:16
to create really rich interactive and performant
37:17
performant
37:17
performant front-end experiences for their
37:18
front-end experiences for their
37:18
front-end experiences for their customers so
37:20
customers so
37:20
customers so when you come to versailles we give you
37:21
when you come to versailles we give you
37:21
when you come to versailles we give you this journey of
37:23
this journey of
37:23
this journey of develop preview ship you start with
37:26
develop preview ship you start with
37:26
develop preview ship you start with developing right so you start with
37:27
developing right so you start with
37:27
developing right so you start with a what we call a serverless programming
37:30
a what we call a serverless programming
37:30
a what we call a serverless programming model
37:31
model
37:31
model when you use a front framework like
37:32
when you use a front framework like
37:32
when you use a front framework like nexjs we take care of
37:34
nexjs we take care of
37:34
nexjs we take care of everything that relates to caching to
37:37
everything that relates to caching to
37:37
everything that relates to caching to seo
37:38
seo
37:38
seo to high performance and we give you a
37:40
to high performance and we give you a
37:40
to high performance and we give you a really amazing developer experience
37:43
really amazing developer experience
37:43
really amazing developer experience so typically front-end developers had to
37:45
so typically front-end developers had to
37:45
so typically front-end developers had to choose sort of between like
37:46
choose sort of between like
37:46
choose sort of between like i'll have a great dx but i'll give up on
37:49
i'll have a great dx but i'll give up on
37:49
i'll have a great dx but i'll give up on seo
37:49
seo
37:50
seo or i'll have a great dx and i'm going to
37:52
or i'll have a great dx and i'm going to
37:52
or i'll have a great dx and i'm going to forego server rendering and other
37:54
forego server rendering and other
37:54
forego server rendering and other capabilities that allow you to really
37:55
capabilities that allow you to really
37:55
capabilities that allow you to really really scale
37:57
really scale
37:57
really scale your front applications so our customers
37:59
your front applications so our customers
37:59
your front applications so our customers kind of get this
38:00
kind of get this
38:00
kind of get this mix of best of both worlds they get the
38:03
mix of best of both worlds they get the
38:03
mix of best of both worlds they get the ability to deploy static applications
38:05
ability to deploy static applications
38:05
ability to deploy static applications and they get the ability to deploy
38:07
and they get the ability to deploy
38:07
and they get the ability to deploy highly dynamic or what we call hybrid
38:10
highly dynamic or what we call hybrid
38:10
highly dynamic or what we call hybrid applications so we also power
38:14
applications so we also power
38:14
applications so we also power on the preview side of things we every
38:15
on the preview side of things we every
38:16
on the preview side of things we every time you push code
38:17
time you push code
38:17
time you push code you get a live url on top of which you
38:20
you get a live url on top of which you
38:20
you get a live url on top of which you can collaborate around the evolution of
38:22
can collaborate around the evolution of
38:22
can collaborate around the evolution of your frontend
38:23
your frontend
38:24
your frontend so instead of collaborating sort of on
38:25
so instead of collaborating sort of on
38:25
so instead of collaborating sort of on top of mockups
38:27
top of mockups
38:27
top of mockups or a single staging server that everyone
38:29
or a single staging server that everyone
38:29
or a single staging server that everyone fights over
38:31
fights over
38:31
fights over every single deploy you make on
38:32
every single deploy you make on
38:32
every single deploy you make on versailles gets this what we call this
38:34
versailles gets this what we call this
38:34
versailles gets this what we call this preview url
38:36
preview url
38:36
preview url which we can share around and
38:38
which we can share around and
38:38
which we can share around and collaborate with the rest of your team
38:40
collaborate with the rest of your team
38:40
collaborate with the rest of your team so to give you a little bit of context
38:41
so to give you a little bit of context
38:41
so to give you a little bit of context of our scale um
38:43
of our scale um
38:43
of our scale um since november we've doubled the our
38:45
since november we've doubled the our
38:45
since november we've doubled the our global edge network traffic
38:47
global edge network traffic
38:47
global edge network traffic to more than 11 billion requests a week
38:50
to more than 11 billion requests a week
38:50
to more than 11 billion requests a week from just five billion requests a week
38:52
from just five billion requests a week
38:52
from just five billion requests a week last november
38:54
last november
38:54
last november uh with 3x the number of this deploy
38:57
uh with 3x the number of this deploy
38:57
uh with 3x the number of this deploy previews that we generate so we see
38:59
previews that we generate so we see
38:59
previews that we generate so we see four million unique deployments
39:02
four million unique deployments
39:02
four million unique deployments passing through our network every week
39:05
passing through our network every week
39:05
passing through our network every week of teams collaborating on their future
39:07
of teams collaborating on their future
39:07
of teams collaborating on their future front end projects um and cosmo civ has
39:10
front end projects um and cosmo civ has
39:10
front end projects um and cosmo civ has been a really big part of our story
39:12
been a really big part of our story
39:12
been a really big part of our story since the very beginning so when we set
39:14
since the very beginning so when we set
39:14
since the very beginning so when we set out to create this platform that would
39:16
out to create this platform that would
39:16
out to create this platform that would uh you know uh sort of be batteries
39:19
uh you know uh sort of be batteries
39:19
uh you know uh sort of be batteries included
39:20
included
39:20
included for every front end for every nexjs
39:22
for every front end for every nexjs
39:22
for every front end for every nexjs developer in the world
39:24
developer in the world
39:24
developer in the world we kind of decided like look for a
39:26
we kind of decided like look for a
39:26
we kind of decided like look for a database that
39:28
database that
39:28
database that shared our values and shared our design
39:30
shared our values and shared our design
39:30
shared our values and shared our design and our approach to the market as well
39:32
and our approach to the market as well
39:32
and our approach to the market as well so the first the very first thing that
39:34
so the first the very first thing that
39:34
so the first the very first thing that drew us to uh cosmos db was
39:37
drew us to uh cosmos db was
39:37
drew us to uh cosmos db was we kind of saw this distinction in the
39:38
we kind of saw this distinction in the
39:38
we kind of saw this distinction in the marketplace early in
39:40
marketplace early in
39:40
marketplace early in even as early as 2017 when we're making
39:42
even as early as 2017 when we're making
39:42
even as early as 2017 when we're making this decision
39:43
this decision
39:44
this decision of there were databases that were sort
39:46
of there were databases that were sort
39:46
of there were databases that were sort of
39:47
of
39:47
of databases as a service and we also saw
39:50
databases as a service and we also saw
39:50
databases as a service and we also saw that there were
39:51
that there were
39:51
that there were services that provided managed databases
39:53
services that provided managed databases
39:54
services that provided managed databases on top of vms
39:55
on top of vms
39:55
on top of vms and if you sort of zoom out they look
39:57
and if you sort of zoom out they look
39:57
and if you sort of zoom out they look very similar
39:58
very similar
39:58
very similar but there's a very big distinction and
40:00
but there's a very big distinction and
40:00
but there's a very big distinction and we saw in cosmos db that this was a
40:03
we saw in cosmos db that this was a
40:03
we saw in cosmos db that this was a database
40:03
database
40:03
database as a service it was a serverless
40:06
as a service it was a serverless
40:06
as a service it was a serverless approach
40:07
approach
40:07
approach it was cloud native and they would meet
40:09
it was cloud native and they would meet
40:09
it was cloud native and they would meet our scale
40:10
our scale
40:10
our scale and the number two thing i mentioned we
40:12
and the number two thing i mentioned we
40:12
and the number two thing i mentioned we operate a global network a global edge
40:14
operate a global network a global edge
40:14
operate a global network a global edge network
40:15
network
40:15
network on top of which we publish your frontend
40:17
on top of which we publish your frontend
40:17
on top of which we publish your frontend projects
40:18
projects
40:18
projects we have our ingress layer starting in 70
40:22
we have our ingress layer starting in 70
40:22
we have our ingress layer starting in 70 cities worldwide
40:23
cities worldwide
40:23
cities worldwide we operate we generate and cache your
40:26
we operate we generate and cache your
40:26
we operate we generate and cache your pages in 17 regions
40:28
pages in 17 regions
40:28
pages in 17 regions worldwide so the global aspects of
40:31
worldwide so the global aspects of
40:31
worldwide so the global aspects of cosmos db were absolutely critical
40:33
cosmos db were absolutely critical
40:33
cosmos db were absolutely critical in this mission that's amazing i guess
40:38
in this mission that's amazing i guess
40:38
in this mission that's amazing i guess i like we all run global teams these
40:40
i like we all run global teams these
40:40
i like we all run global teams these days right and we have
40:41
days right and we have
40:42
days right and we have uh and we all constantly need to
40:43
uh and we all constantly need to
40:43
uh and we all constantly need to collaborate this is
40:45
collaborate this is
40:45
collaborate this is exactly developer experience that people
40:46
exactly developer experience that people
40:46
exactly developer experience that people aspire to where
40:48
aspire to where
40:48
aspire to where every every check-in is immediately
40:50
every every check-in is immediately
40:50
every every check-in is immediately available in preview to the entire team
40:52
available in preview to the entire team
40:52
available in preview to the entire team everyone can comment uh how do you do
40:55
everyone can comment uh how do you do
40:55
everyone can comment uh how do you do this with
40:56
this with
40:56
this with can you mention the global uh
40:58
can you mention the global uh
40:58
can you mention the global uh development
40:59
development
40:59
development is important part of the vision of
41:02
is important part of the vision of
41:02
is important part of the vision of versailles
41:03
versailles
41:03
versailles enabling global development how does
41:05
enabling global development how does
41:05
enabling global development how does cosmos db help you there
41:07
cosmos db help you there
41:07
cosmos db help you there absolutely so uh one of the key tenets
41:10
absolutely so uh one of the key tenets
41:10
absolutely so uh one of the key tenets of nexjs
41:11
of nexjs
41:11
of nexjs and versailles is pages have to be very
41:15
and versailles is pages have to be very
41:15
and versailles is pages have to be very very very fast for the end user right so
41:18
very very fast for the end user right so
41:18
very very fast for the end user right so i mentioned earlier
41:19
i mentioned earlier
41:19
i mentioned earlier sometimes it's really sort of a
41:20
sometimes it's really sort of a
41:20
sometimes it's really sort of a dichotomy between i have a great
41:22
dichotomy between i have a great
41:22
dichotomy between i have a great developer experience
41:23
developer experience
41:23
developer experience but i also need my business to succeed
41:25
but i also need my business to succeed
41:26
but i also need my business to succeed at the end of the day
41:26
at the end of the day
41:26
at the end of the day i need my pages to be really really
41:29
i need my pages to be really really
41:29
i need my pages to be really really really fast
41:30
really fast
41:30
really fast so when you go to a website hosted on
41:31
so when you go to a website hosted on
41:31
so when you go to a website hosted on the versailles platform like
41:33
the versailles platform like
41:33
the versailles platform like hashicorp.com what happens is we have we
41:36
hashicorp.com what happens is we have we
41:36
hashicorp.com what happens is we have we operate in any cast
41:37
operate in any cast
41:37
operate in any cast network that automatically routes you to
41:40
network that automatically routes you to
41:40
network that automatically routes you to the nearest location
41:42
the nearest location
41:42
the nearest location where that website could live at that
41:45
where that website could live at that
41:45
where that website could live at that point
41:46
point
41:46
point we try to give developers tools
41:49
we try to give developers tools
41:49
we try to give developers tools to automatically cache and optimize
41:51
to automatically cache and optimize
41:51
to automatically cache and optimize their websites
41:52
their websites
41:52
their websites so as an example when you create an xjs
41:54
so as an example when you create an xjs
41:54
so as an example when you create an xjs application
41:55
application
41:56
application we notice all the parts of your app that
41:58
we notice all the parts of your app that
41:58
we notice all the parts of your app that are static
41:59
are static
41:59
are static in fact we're just looking at a static
42:01
in fact we're just looking at a static
42:01
in fact we're just looking at a static page within a very large next.js
42:03
page within a very large next.js
42:03
page within a very large next.js application which is
42:04
application which is
42:04
application which is versal.com itself this what you're
42:07
versal.com itself this what you're
42:07
versal.com itself this what you're looking at
42:07
looking at
42:07
looking at is a static so what happened was you
42:09
is a static so what happened was you
42:10
is a static so what happened was you went to versailles.com in seattle
42:12
went to versailles.com in seattle
42:12
went to versailles.com in seattle we terminated your connection at the
42:14
we terminated your connection at the
42:14
we terminated your connection at the nearest edge and we serve this page in a
42:16
nearest edge and we serve this page in a
42:16
nearest edge and we serve this page in a really really performant manner
42:18
really really performant manner
42:18
really really performant manner so behind all these pages behind all our
42:22
so behind all these pages behind all our
42:22
so behind all these pages behind all our capabilities for
42:23
capabilities for
42:24
capabilities for incremental static generation for deploy
42:26
incremental static generation for deploy
42:26
incremental static generation for deploy previews
42:27
previews
42:27
previews is customers db providing the metadata
42:29
is customers db providing the metadata
42:29
is customers db providing the metadata layer
42:30
layer
42:30
layer so every deploy every build every page
42:34
so every deploy every build every page
42:34
so every deploy every build every page we pre-render
42:35
we pre-render
42:35
we pre-render is touches customs to be a sort of the
42:37
is touches customs to be a sort of the
42:37
is touches customs to be a sort of the central
42:38
central
42:38
central intelligence and control plane of our
42:41
intelligence and control plane of our
42:41
intelligence and control plane of our platform
42:42
platform
42:42
platform so instead of depending on manually
42:45
so instead of depending on manually
42:45
so instead of depending on manually scaling database instances for something
42:47
scaling database instances for something
42:47
scaling database instances for something so mission critical for us or having to
42:50
so mission critical for us or having to
42:50
so mission critical for us or having to choose
42:51
choose
42:51
choose an individual region to host our
42:52
an individual region to host our
42:52
an individual region to host our database we let cosmos db do all the
42:55
database we let cosmos db do all the
42:56
database we let cosmos db do all the scaling and global replication magic
42:59
scaling and global replication magic
42:59
scaling and global replication magic that's amazing yeah like i love the
43:00
that's amazing yeah like i love the
43:00
that's amazing yeah like i love the scenario and we have
43:02
scenario and we have
43:02
scenario and we have many uh these kind of globally
43:04
many uh these kind of globally
43:04
many uh these kind of globally distributed control planes
43:06
distributed control planes
43:06
distributed control planes kind of built using this kind of
43:08
kind of built using this kind of
43:08
kind of built using this kind of actually azure resource manager
43:10
actually azure resource manager
43:10
actually azure resource manager which is a globally distributed control
43:11
which is a globally distributed control
43:11
which is a globally distributed control plane uses cosmos db underneath the
43:13
plane uses cosmos db underneath the
43:13
plane uses cosmos db underneath the covers
43:13
covers
43:13
covers all the azure resources they're all from
43:15
all the azure resources they're all from
43:15
all the azure resources they're all from cosmos db uh
43:17
cosmos db uh
43:17
cosmos db uh but they uh give it i think you guys
43:21
but they uh give it i think you guys
43:21
but they uh give it i think you guys because we can evolve together from the
43:23
because we can evolve together from the
43:23
because we can evolve together from the very early age and i
43:24
very early age and i
43:24
very early age and i admit there were bumpy places on the
43:26
admit there were bumpy places on the
43:26
admit there were bumpy places on the road ah thank you
43:28
road ah thank you
43:28
road ah thank you it's been an amazing journey and as a
43:31
it's been an amazing journey and as a
43:31
it's been an amazing journey and as a result and
43:32
result and
43:32
result and uh one of our senior engineers will be
43:34
uh one of our senior engineers will be
43:34
uh one of our senior engineers will be giving a presentation
43:36
giving a presentation
43:36
giving a presentation we've collected a lot of valuable
43:37
we've collected a lot of valuable
43:38
we've collected a lot of valuable lessons that are developer focused
43:39
lessons that are developer focused
43:40
lessons that are developer focused especially for developers that like us
43:42
especially for developers that like us
43:42
especially for developers that like us are part of the javascript world
43:44
are part of the javascript world
43:44
are part of the javascript world in the node.js ecosystem if you're
43:46
in the node.js ecosystem if you're
43:46
in the node.js ecosystem if you're curious about how we've scaled
43:48
curious about how we've scaled
43:48
curious about how we've scaled with cosmos db the tooling that we've
43:51
with cosmos db the tooling that we've
43:51
with cosmos db the tooling that we've used the sdks and layers on top of the
43:53
used the sdks and layers on top of the
43:53
used the sdks and layers on top of the sdks that we've developed
43:55
sdks that we've developed
43:55
sdks that we've developed including a custom sdb simulator for
43:57
including a custom sdb simulator for
43:57
including a custom sdb simulator for node a cosmos db cli for node
44:00
node a cosmos db cli for node
44:00
node a cosmos db cli for node and a lot of our layers of
44:02
and a lot of our layers of
44:02
and a lot of our layers of instrumentation that we've added
44:04
instrumentation that we've added
44:04
instrumentation that we've added to connect our observability
44:07
to connect our observability
44:07
to connect our observability uh centers with cosmos db you'll learn
44:10
uh centers with cosmos db you'll learn
44:10
uh centers with cosmos db you'll learn all that uh
44:11
all that uh
44:11
all that uh check out javi velasco's presentation
44:13
check out javi velasco's presentation
44:13
check out javi velasco's presentation from the versailles team
44:15
from the versailles team
44:15
from the versailles team thank you so much guillermo folks please
44:17
thank you so much guillermo folks please
44:17
thank you so much guillermo folks please stay uh tune in
44:18
stay uh tune in
44:18
stay uh tune in uh for versailles presentation uh from
44:21
uh for versailles presentation uh from
44:21
uh for versailles presentation uh from javier uh
44:22
javier uh
44:22
javier uh really awesome uh great uh
44:25
really awesome uh great uh
44:25
really awesome uh great uh dev test tools were built along the way
44:27
dev test tools were built along the way
44:27
dev test tools were built along the way i've used them myself
44:29
i've used them myself
44:29
i've used them myself and versailles has been
44:32
and versailles has been
44:32
and versailles has been really awesome partner thank you so much
44:38
okay so can i give you a few words about
44:40
okay so can i give you a few words about
44:40
okay so can i give you a few words about the one capabilities that
44:41
the one capabilities that
44:41
the one capabilities that uh glamorous mentioned uh which is this
44:44
uh glamorous mentioned uh which is this
44:44
uh glamorous mentioned uh which is this global distribution that
44:46
global distribution that
44:46
global distribution that which is the features that cosmos db was
44:47
which is the features that cosmos db was
44:48
which is the features that cosmos db was born with it's a cloud native database
44:50
born with it's a cloud native database
44:50
born with it's a cloud native database built around global distribution it's
44:52
built around global distribution it's
44:52
built around global distribution it's not something that was
44:53
not something that was
44:53
not something that was bundled on top it is part of the core
44:56
bundled on top it is part of the core
44:56
bundled on top it is part of the core engine and therefore we are able to
44:59
engine and therefore we are able to
44:59
engine and therefore we are able to achieve very fast
45:00
achieve very fast
45:00
achieve very fast uh replication between regions
45:04
uh replication between regions
45:04
uh replication between regions i can replication happens in line uh
45:07
i can replication happens in line uh
45:07
i can replication happens in line uh quickly
45:07
quickly
45:07
quickly in real time obviously bound by the
45:10
in real time obviously bound by the
45:10
in real time obviously bound by the speed of light
45:11
speed of light
45:11
speed of light but nevertheless uh one of the key
45:13
but nevertheless uh one of the key
45:13
but nevertheless uh one of the key attributes of this global distribution
45:15
attributes of this global distribution
45:15
attributes of this global distribution is ability to do
45:16
is ability to do
45:16
is ability to do true active active uh
45:20
deployment multiple region rights so
45:22
deployment multiple region rights so
45:22
deployment multiple region rights so application can write into any regions
45:25
application can write into any regions
45:25
application can write into any regions no failover options needed and achieve
45:27
no failover options needed and achieve
45:27
no failover options needed and achieve zero rto
45:29
zero rto
45:29
zero rto as uh obviously to build these
45:32
as uh obviously to build these
45:32
as uh obviously to build these applications you have to
45:33
applications you have to
45:33
applications you have to account for conflicts conflicts may
45:35
account for conflicts conflicts may
45:35
account for conflicts conflicts may arise
45:36
arise
45:36
arise when you achieve this zero uh recovery
45:39
when you achieve this zero uh recovery
45:40
when you achieve this zero uh recovery time objective
45:40
time objective
45:40
time objective when an entire region can go down and
45:43
when an entire region can go down and
45:43
when an entire region can go down and your application does not notice
45:45
your application does not notice
45:45
your application does not notice obviously laws of physics your rpo
45:48
obviously laws of physics your rpo
45:48
obviously laws of physics your rpo cannot be zero
45:49
cannot be zero
45:50
cannot be zero but we are working on this on on this
45:52
but we are working on this on on this
45:52
but we are working on this on on this t-shirt
45:53
t-shirt
45:53
t-shirt that brings both of them very close to
45:55
that brings both of them very close to
45:55
that brings both of them very close to zero
45:56
zero
45:56
zero um so you can use a strong consistency
45:59
um so you can use a strong consistency
45:59
um so you can use a strong consistency and closely and achieve close to zero
46:01
and closely and achieve close to zero
46:01
and closely and achieve close to zero rto as well
46:03
rto as well
46:03
rto as well um lots of exciting things happening in
46:05
um lots of exciting things happening in
46:05
um lots of exciting things happening in the global distribution
46:06
the global distribution
46:06
the global distribution uh i can uh do you can appoint you to
46:10
uh i can uh do you can appoint you to
46:10
uh i can uh do you can appoint you to one example
46:11
one example
46:11
one example uh there is a very nice blog by walmart
46:14
uh there is a very nice blog by walmart
46:14
uh there is a very nice blog by walmart engineering team
46:15
engineering team
46:16
engineering team uh that describes how they moved
46:19
uh that describes how they moved
46:19
uh that describes how they moved their ecommerce platform walmart.com
46:22
their ecommerce platform walmart.com
46:22
their ecommerce platform walmart.com um one of the largest uh over to cosmos
46:26
um one of the largest uh over to cosmos
46:26
um one of the largest uh over to cosmos db
46:27
db
46:27
db uh with aks and how they
46:30
uh with aks and how they
46:30
uh with aks and how they uh kind of use that uh microservices
46:33
uh kind of use that uh microservices
46:33
uh kind of use that uh microservices architecture
46:34
architecture
46:34
architecture backed by cosmos db and achieve and
46:37
backed by cosmos db and achieve and
46:37
backed by cosmos db and achieve and achieve the
46:38
achieve the
46:38
achieve the multi-region rights uh five nights
46:42
multi-region rights uh five nights
46:42
multi-region rights uh five nights availability
46:43
availability
46:43
availability uh even even if there is a regional
46:46
uh even even if there is a regional
46:46
uh even even if there is a regional failure
46:46
failure
46:46
failure with the cloud and clouds do have
46:48
with the cloud and clouds do have
46:48
with the cloud and clouds do have sometimes outages
46:50
sometimes outages
46:50
sometimes outages um as much as we don't like it and try
46:53
um as much as we don't like it and try
46:53
um as much as we don't like it and try to avoid them
46:54
to avoid them
46:54
to avoid them uh application does not notice them
46:58
uh application does not notice them
46:58
uh application does not notice them thanks to that capability of cosmos db
47:00
thanks to that capability of cosmos db
47:00
thanks to that capability of cosmos db the multi-region rights
47:02
the multi-region rights
47:02
the multi-region rights check out this blog uh really awesome
47:04
check out this blog uh really awesome
47:04
check out this blog uh really awesome write-up uh
47:05
write-up uh
47:05
write-up uh lots of lots of details around how they
47:08
lots of lots of details around how they
47:08
lots of lots of details around how they did it
47:08
did it
47:08
did it um kind of very exciting to see a
47:12
um kind of very exciting to see a
47:12
um kind of very exciting to see a vlog from walmart.com now
47:15
vlog from walmart.com now
47:15
vlog from walmart.com now i would like to invite on stage uh
47:18
i would like to invite on stage uh
47:18
i would like to invite on stage uh another really exciting application many
47:20
another really exciting application many
47:20
another really exciting application many of you
47:20
of you
47:20
of you have used it in the past the
47:25
have used it in the past the
47:25
have used it in the past the a developed key developer behind yammer
47:28
a developed key developer behind yammer
47:28
a developed key developer behind yammer feeds
47:29
feeds
47:30
feeds uh will join me meg welcome
47:33
uh will join me meg welcome
47:33
uh will join me meg welcome hi karel thank you so much for having me
47:36
hi karel thank you so much for having me
47:36
hi karel thank you so much for having me it's great to have you could you tell us
47:38
it's great to have you could you tell us
47:38
it's great to have you could you tell us a little bit about younger
47:40
a little bit about younger
47:40
a little bit about younger let me bring you an example of the
47:41
let me bring you an example of the
47:41
let me bring you an example of the yammer feed
47:43
yammer feed
47:43
yammer feed just because yeah so yeah for those of
47:46
just because yeah so yeah for those of
47:46
just because yeah so yeah for those of you who don't know
47:47
you who don't know
47:47
you who don't know yammer is basically an enterprise social
47:49
yammer is basically an enterprise social
47:50
yammer is basically an enterprise social network
47:51
network
47:51
network you can see here on this this is a
47:52
you can see here on this this is a
47:52
you can see here on this this is a typical landing page this screenshot
47:54
typical landing page this screenshot
47:54
typical landing page this screenshot here yeah
47:55
here yeah
47:55
here yeah so um so here you can see people are
47:58
so um so here you can see people are
47:58
so um so here you can see people are posting you can reply
47:59
posting you can reply
47:59
posting you can reply comment like and we call this a feed in
48:02
comment like and we call this a feed in
48:02
comment like and we call this a feed in yammer and feeds our core to our product
48:05
yammer and feeds our core to our product
48:05
yammer and feeds our core to our product so keep in mind that this is a really
48:06
so keep in mind that this is a really
48:06
so keep in mind that this is a really high fan out model
48:08
high fan out model
48:08
high fan out model with a lot of data and by that i mean if
48:11
with a lot of data and by that i mean if
48:11
with a lot of data and by that i mean if we deliver a message
48:12
we deliver a message
48:12
we deliver a message it may need to be delivered to hundreds
48:14
it may need to be delivered to hundreds
48:14
it may need to be delivered to hundreds of thousands of users
48:16
of thousands of users
48:16
of thousands of users and we used to store these feeds in
48:18
and we used to store these feeds in
48:18
and we used to store these feeds in hbase a self-managed hbase cluster
48:20
hbase a self-managed hbase cluster
48:20
hbase a self-managed hbase cluster but we now serve feeds entirely from
48:22
but we now serve feeds entirely from
48:22
but we now serve feeds entirely from cosmos db
48:25
that's really awesome what was kind of
48:28
that's really awesome what was kind of
48:28
that's really awesome what was kind of the most
48:29
the most
48:29
the most challenging part of kind of building
48:32
challenging part of kind of building
48:32
challenging part of kind of building this really high pan out
48:34
this really high pan out
48:34
this really high pan out architecture where you need to push
48:36
architecture where you need to push
48:36
architecture where you need to push these updates to so many users and how
48:38
these updates to so many users and how
48:38
these updates to so many users and how do you do this on cosmos db
48:41
do you do this on cosmos db
48:41
do you do this on cosmos db right so i mean i guess i can give a
48:42
right so i mean i guess i can give a
48:42
right so i mean i guess i can give a little background of where we came from
48:44
little background of where we came from
48:44
little background of where we came from um you know we started we have our own
48:46
um you know we started we have our own
48:46
um you know we started we have our own self-managed hbase we had a really small
48:49
self-managed hbase we had a really small
48:49
self-managed hbase we had a really small team
48:49
team
48:49
team managing that and at the scale that
48:51
managing that and at the scale that
48:51
managing that and at the scale that we're operating it just wasn't
48:53
we're operating it just wasn't
48:53
we're operating it just wasn't manageable anymore it just wasn't
48:54
manageable anymore it just wasn't
48:54
manageable anymore it just wasn't sustainable we wanted to get out of the
48:56
sustainable we wanted to get out of the
48:56
sustainable we wanted to get out of the business of
48:57
business of
48:57
business of storing you know managing our own
48:58
storing you know managing our own
48:58
storing you know managing our own storage and build features instead
49:01
storage and build features instead
49:01
storage and build features instead so that's why we needed a new database
49:03
so that's why we needed a new database
49:03
so that's why we needed a new database solution um and it's how we chose cosmos
49:05
solution um and it's how we chose cosmos
49:05
solution um and it's how we chose cosmos to be
49:06
to be
49:06
to be i think we had a few key requirements
49:08
i think we had a few key requirements
49:08
i think we had a few key requirements you know we have that that fan out
49:10
you know we have that that fan out
49:10
you know we have that that fan out um we have a ton of data we have seven
49:12
um we have a ton of data we have seven
49:12
um we have a ton of data we have seven terabytes in cosmos db
49:14
terabytes in cosmos db
49:14
terabytes in cosmos db but what we really needed was something
49:16
but what we really needed was something
49:16
but what we really needed was something fully managed um
49:17
fully managed um
49:18
fully managed um we needed something with super low
49:19
we needed something with super low
49:19
we needed something with super low latency and we wanted something that
49:21
latency and we wanted something that
49:21
latency and we wanted something that worked nicely with our
49:22
worked nicely with our
49:22
worked nicely with our java environment um so obviously cosmos
49:25
java environment um so obviously cosmos
49:25
java environment um so obviously cosmos hits all those
49:26
hits all those
49:26
hits all those hits all those buttons um it's fully
49:27
hits all those buttons um it's fully
49:28
hits all those buttons um it's fully managed we trust the cosmos team with
49:29
managed we trust the cosmos team with
49:29
managed we trust the cosmos team with our data
49:29
our data
49:30
our data we run in four regions and probably more
49:32
we run in four regions and probably more
49:32
we run in four regions and probably more in the future in our production
49:34
in the future in our production
49:34
in the future in our production regions we can add regions with the
49:36
regions we can add regions with the
49:36
regions we can add regions with the click of a button
49:37
click of a button
49:37
click of a button we can fail over with the click of a
49:38
we can fail over with the click of a
49:38
we can fail over with the click of a button those are things that we could
49:40
button those are things that we could
49:40
button those are things that we could never do before on hbase that was
49:42
never do before on hbase that was
49:42
never do before on hbase that was a date couple hours or a day multiple
49:44
a date couple hours or a day multiple
49:44
a date couple hours or a day multiple engineers
49:45
engineers
49:45
engineers and now we just click the button and we
49:48
and now we just click the button and we
49:48
and now we just click the button and we also love
49:48
also love
49:48
also love auto scale as the previous speaker
49:51
auto scale as the previous speaker
49:51
auto scale as the previous speaker mentioned
49:53
mentioned
49:53
mentioned because yammer has sort of ebbs and
49:55
because yammer has sort of ebbs and
49:55
because yammer has sort of ebbs and flows in traffic throughout the day as
49:56
flows in traffic throughout the day as
49:56
flows in traffic throughout the day as people come online
49:57
people come online
49:58
people come online during business hours and with auto
50:00
during business hours and with auto
50:00
during business hours and with auto scale we just pay for what we use
50:02
scale we just pay for what we use
50:02
scale we just pay for what we use it instantly scales for you know traffic
50:04
it instantly scales for you know traffic
50:04
it instantly scales for you know traffic peaks that come through
50:06
peaks that come through
50:06
peaks that come through and uh you know we don't have to worry
50:08
and uh you know we don't have to worry
50:08
and uh you know we don't have to worry about any of that cosmos does it all for
50:09
about any of that cosmos does it all for
50:09
about any of that cosmos does it all for us
50:11
us
50:11
us yes the second thing was latency you
50:12
yes the second thing was latency you
50:12
yes the second thing was latency you know coming from our own managed
50:14
know coming from our own managed
50:14
know coming from our own managed storage system in our in-house you know
50:16
storage system in our in-house you know
50:16
storage system in our in-house you know in our same data center
50:17
in our same data center
50:18
in our same data center same network we relied the feeds
50:20
same network we relied the feeds
50:20
same network we relied the feeds infrastructure relied on super low
50:22
infrastructure relied on super low
50:22
infrastructure relied on super low latency and cosmos ticks that as well
50:25
latency and cosmos ticks that as well
50:25
latency and cosmos ticks that as well with the you know sub 10 millisecond sla
50:28
with the you know sub 10 millisecond sla
50:28
with the you know sub 10 millisecond sla and um the other thing you know i
50:30
and um the other thing you know i
50:30
and um the other thing you know i mentioned is that
50:31
mentioned is that
50:31
mentioned is that we're a java shop you think of microsoft
50:33
we're a java shop you think of microsoft
50:33
we're a java shop you think of microsoft sometimes it's like.net and c-sharp but
50:35
sometimes it's like.net and c-sharp but
50:35
sometimes it's like.net and c-sharp but yammer is actually almost entirely java
50:38
yammer is actually almost entirely java
50:38
yammer is actually almost entirely java micro services
50:39
micro services
50:39
micro services and we needed something that would work
50:41
and we needed something that would work
50:41
and we needed something that would work with our java structure
50:43
with our java structure
50:43
with our java structure and i can attest that the java v4 sdk is
50:47
and i can attest that the java v4 sdk is
50:47
and i can attest that the java v4 sdk is fully mature
50:48
fully mature
50:48
fully mature and definitely production ready we had
50:50
and definitely production ready we had
50:50
and definitely production ready we had some you know a little rough in the
50:52
some you know a little rough in the
50:52
some you know a little rough in the earlier stages but um
50:54
earlier stages but um
50:54
earlier stages but um the current sdk is definitely you know
50:57
the current sdk is definitely you know
50:57
the current sdk is definitely you know go-to
50:57
go-to
50:57
go-to and wonderful to work with um and you
51:00
and wonderful to work with um and you
51:00
and wonderful to work with um and you know as an added bonus being on cosmos
51:02
know as an added bonus being on cosmos
51:02
know as an added bonus being on cosmos db unlocked a ton of feature development
51:04
db unlocked a ton of feature development
51:04
db unlocked a ton of feature development that just
51:05
that just
51:05
that just was not possible before on hbase
51:08
was not possible before on hbase
51:08
was not possible before on hbase i'm talking things like secondary
51:09
i'm talking things like secondary
51:09
i'm talking things like secondary indexes allows for
51:11
indexes allows for
51:11
indexes allows for filtering and sorting which we couldn't
51:14
filtering and sorting which we couldn't
51:14
filtering and sorting which we couldn't do before
51:15
do before
51:15
do before and it's just as soon as we got to
51:17
and it's just as soon as we got to
51:17
and it's just as soon as we got to cosmos db our feature development just
51:18
cosmos db our feature development just
51:18
cosmos db our feature development just exploded
51:19
exploded
51:19
exploded because we had so much flexibility with
51:21
because we had so much flexibility with
51:21
because we had so much flexibility with the model there are more benefits i can
51:23
the model there are more benefits i can
51:23
the model there are more benefits i can go into if we have time to
51:26
go into if we have time to
51:26
go into if we have time to that's awesome i think what we really do
51:28
that's awesome i think what we really do
51:28
that's awesome i think what we really do what we want to do is kind of we want to
51:29
what we want to do is kind of we want to
51:29
what we want to do is kind of we want to set up
51:29
set up
51:29
set up a uh we'll give a uh an on-demand
51:32
a uh we'll give a uh an on-demand
51:32
a uh we'll give a uh an on-demand session
51:33
session
51:33
session and sit down with you and go into
51:35
and sit down with you and go into
51:35
and sit down with you and go into details it's a really exciting use case
51:37
details it's a really exciting use case
51:37
details it's a really exciting use case many customers have faced with kind of
51:39
many customers have faced with kind of
51:39
many customers have faced with kind of similar challenges of kind of having
51:41
similar challenges of kind of having
51:41
similar challenges of kind of having these
51:42
these
51:42
these these feeds also they can defeat they
51:44
these feeds also they can defeat they
51:44
these feeds also they can defeat they can be sequences
51:45
can be sequences
51:45
can be sequences uh and kind of solving this large
51:48
uh and kind of solving this large
51:48
uh and kind of solving this large phenomenon
51:49
phenomenon
51:49
phenomenon large fan out problem where you have to
51:51
large fan out problem where you have to
51:52
large fan out problem where you have to deliver this information to hundreds of
51:53
deliver this information to hundreds of
51:53
deliver this information to hundreds of thousands
51:54
thousands
51:54
thousands and millions millions of users we use
51:57
and millions millions of users we use
51:57
and millions millions of users we use yammer every day at microsoft
51:59
yammer every day at microsoft
51:59
yammer every day at microsoft it's hundreds of thousands of employees
52:01
it's hundreds of thousands of employees
52:01
it's hundreds of thousands of employees and can we all get them uh
52:04
and can we all get them uh
52:04
and can we all get them uh and uh it's great that cosmos db enables
52:07
and uh it's great that cosmos db enables
52:07
and uh it's great that cosmos db enables you to
52:07
you to
52:07
you to uh ride this wave of growth with ease
52:11
uh ride this wave of growth with ease
52:11
uh ride this wave of growth with ease thank you so much uh and we'll do a more
52:14
thank you so much uh and we'll do a more
52:14
thank you so much uh and we'll do a more detailed
52:15
detailed
52:15
detailed uh deep dive on this thank you
52:21
all right um well i wanted to show uh
52:25
all right um well i wanted to show uh
52:25
all right um well i wanted to show uh about one more really exciting uh
52:27
about one more really exciting uh
52:27
about one more really exciting uh feature
52:28
feature
52:28
feature that we have and somewhat related to the
52:29
that we have and somewhat related to the
52:30
that we have and somewhat related to the mongodb demonstrations that they gave
52:32
mongodb demonstrations that they gave
52:32
mongodb demonstrations that they gave uh earlier and which is a real-time
52:35
uh earlier and which is a real-time
52:35
uh earlier and which is a real-time analytics
52:36
analytics
52:36
analytics uh strengths with our system team
52:39
uh strengths with our system team
52:39
uh strengths with our system team are just synapse uh that we've built
52:41
are just synapse uh that we've built
52:42
are just synapse uh that we've built together
52:43
together
52:43
together um now with cosmos dp in this new world
52:47
um now with cosmos dp in this new world
52:47
um now with cosmos dp in this new world right when the
52:47
right when the
52:47
right when the data is crashing into the database and
52:49
data is crashing into the database and
52:49
data is crashing into the database and you have to make real-time decisions
52:51
you have to make real-time decisions
52:51
you have to make real-time decisions fast
52:52
fast
52:52
fast you have to enable your application to
52:54
you have to enable your application to
52:54
you have to enable your application to make those real-time decisions to close
52:56
make those real-time decisions to close
52:56
make those real-time decisions to close the loop with the customer
52:57
the loop with the customer
52:57
the loop with the customer uh you also have to make some of these
52:59
uh you also have to make some of these
52:59
uh you also have to make some of these decisions you have to make yourself
53:01
decisions you have to make yourself
53:01
decisions you have to make yourself um whether it's to bring more and from
53:04
um whether it's to bring more and from
53:04
um whether it's to bring more and from supply
53:05
supply
53:06
supply whether it's to ultra whether it's to
53:07
whether it's to ultra whether it's to
53:07
whether it's to ultra whether it's to kind of react whether it's to call then
53:09
kind of react whether it's to call then
53:09
kind of react whether it's to call then reach out
53:10
reach out
53:10
reach out to someone maybe there is an air airbag
53:12
to someone maybe there is an air airbag
53:12
to someone maybe there is an air airbag deployed you want to be there
53:15
deployed you want to be there
53:15
deployed you want to be there for your customer in real time how do
53:17
for your customer in real time how do
53:17
for your customer in real time how do you do this
53:18
you do this
53:18
you do this so cosmos db uh offers the real time
53:21
so cosmos db uh offers the real time
53:21
so cosmos db uh offers the real time reaching sites
53:23
reaching sites
53:23
reaching sites uh thanks to synapse link feature that
53:25
uh thanks to synapse link feature that
53:25
uh thanks to synapse link feature that enables this real-time dashboards
53:29
enables this real-time dashboards
53:29
enables this real-time dashboards bi predictive analytics and it's
53:31
bi predictive analytics and it's
53:32
bi predictive analytics and it's accessible
53:32
accessible
53:32
accessible through the application
53:36
the way it works is that we have two
53:38
the way it works is that we have two
53:38
the way it works is that we have two stores effective we have a transactional
53:40
stores effective we have a transactional
53:40
stores effective we have a transactional store
53:41
store
53:41
store and we have an analytical store and so
53:43
and we have an analytical store and so
53:44
and we have an analytical store and so as data is coming in
53:45
as data is coming in
53:45
as data is coming in in the transactional store it's made
53:46
in the transactional store it's made
53:46
in the transactional store it's made automatically available in a columnar
53:49
automatically available in a columnar
53:49
automatically available in a columnar parquet format in the analytical store
53:52
parquet format in the analytical store
53:52
parquet format in the analytical store and offered to synapse link to synapse
53:55
and offered to synapse link to synapse
53:55
and offered to synapse link to synapse to any of the runtimes whether it's uh
53:58
to any of the runtimes whether it's uh
53:58
to any of the runtimes whether it's uh snap sql servers
53:59
snap sql servers
53:59
snap sql servers it's a full t sequel that you can use to
54:01
it's a full t sequel that you can use to
54:02
it's a full t sequel that you can use to power bi or you can
54:03
power bi or you can
54:03
power bi or you can directly talk to it as an ogbc uh
54:06
directly talk to it as an ogbc uh
54:06
directly talk to it as an ogbc uh using your dbc driver or through the
54:09
using your dbc driver or through the
54:09
using your dbc driver or through the synapse spark and you can
54:10
synapse spark and you can
54:10
synapse spark and you can build your uh spark pipelines uh
54:14
build your uh spark pipelines uh
54:14
build your uh spark pipelines uh spark applications um querying this
54:17
spark applications um querying this
54:17
spark applications um querying this columnar store and this columnar score
54:19
columnar store and this columnar score
54:19
columnar store and this columnar score is optimized for analytical case it's
54:21
is optimized for analytical case it's
54:21
is optimized for analytical case it's columnar right so analytical queries
54:23
columnar right so analytical queries
54:23
columnar right so analytical queries becomes cheap they become fast
54:26
becomes cheap they become fast
54:26
becomes cheap they become fast you don't have to build your own etl
54:27
you don't have to build your own etl
54:27
you don't have to build your own etl pipeline right it's all managed by the
54:30
pipeline right it's all managed by the
54:30
pipeline right it's all managed by the service for you
54:31
service for you
54:31
service for you um if you need to transform the data of
54:33
um if you need to transform the data of
54:33
um if you need to transform the data of course you can do it in synapse but
54:35
course you can do it in synapse but
54:35
course you can do it in synapse but but the data just to do analytical query
54:37
but the data just to do analytical query
54:37
but the data just to do analytical query you don't need to do anything
54:38
you don't need to do anything
54:38
you don't need to do anything uh the data is available to you in real
54:41
uh the data is available to you in real
54:41
uh the data is available to you in real time the delays
54:42
time the delays
54:42
time the delays sega is kind of really low seconds to
54:45
sega is kind of really low seconds to
54:45
sega is kind of really low seconds to low minutes
54:46
low minutes
54:46
low minutes uh between the data hitting
54:47
uh between the data hitting
54:47
uh between the data hitting transactional store and being available
54:49
transactional store and being available
54:49
transactional store and being available to you
54:50
to you
54:50
to you in the analytical store and
54:53
in the analytical store and
54:53
in the analytical store and as you make these analytical queries
54:55
as you make these analytical queries
54:55
as you make these analytical queries they do not impact
54:56
they do not impact
54:56
they do not impact at all the performance of your
54:57
at all the performance of your
54:57
at all the performance of your transactional store you don't need to
54:59
transactional store you don't need to
54:59
transactional store you don't need to burn or use anymore at all
55:02
burn or use anymore at all
55:02
burn or use anymore at all obviously analytical store is much
55:03
obviously analytical store is much
55:03
obviously analytical store is much cheaper than transactional as you would
55:05
cheaper than transactional as you would
55:05
cheaper than transactional as you would expect
55:05
expect
55:06
expect this is three cents per gigabyte it has
55:08
this is three cents per gigabyte it has
55:08
this is three cents per gigabyte it has uh the queries are cheaper
55:10
uh the queries are cheaper
55:10
uh the queries are cheaper faster more efficient and you don't burn
55:12
faster more efficient and you don't burn
55:12
faster more efficient and you don't burn our use on this
55:14
our use on this
55:14
our use on this uh which is kind of really valuable uh
55:17
uh which is kind of really valuable uh
55:17
uh which is kind of really valuable uh so
55:18
so
55:18
so backing out in my uh longer db demo i
55:21
backing out in my uh longer db demo i
55:21
backing out in my uh longer db demo i ran this query right this is a mongodb
55:23
ran this query right this is a mongodb
55:23
ran this query right this is a mongodb aggregation query
55:24
aggregation query
55:24
aggregation query and and can if i remove this explain uh
55:28
and and can if i remove this explain uh
55:28
and and can if i remove this explain uh we'll get more sensible results it will
55:30
we'll get more sensible results it will
55:30
we'll get more sensible results it will give it will give me the
55:31
give it will give me the
55:31
give it will give me the average flights number of flights
55:35
average flights number of flights
55:35
average flights number of flights per month and few other aggregations on
55:38
per month and few other aggregations on
55:38
per month and few other aggregations on our flights database
55:40
our flights database
55:40
our flights database uh and there's a lot of synthetic data
55:42
uh and there's a lot of synthetic data
55:42
uh and there's a lot of synthetic data in the database so that it's 17
55:44
in the database so that it's 17
55:44
in the database so that it's 17 terabytes and it ran very fast right
55:46
terabytes and it ran very fast right
55:46
terabytes and it ran very fast right um so i now have have my data um
55:50
um so i now have have my data um
55:50
um so i now have have my data um i can see how many how many uh average
55:52
i can see how many how many uh average
55:52
i can see how many how many uh average per month so obviously in
55:54
per month so obviously in
55:54
per month so obviously in 2020 uh there will be there are much
55:56
2020 uh there will be there are much
55:56
2020 uh there will be there are much fewer
55:57
fewer
55:57
fewer flights than in 2019 now
56:00
flights than in 2019 now
56:00
flights than in 2019 now what i can do is if i look at the the
56:02
what i can do is if i look at the the
56:02
what i can do is if i look at the the geek and me wants to know how is the
56:04
geek and me wants to know how is the
56:04
geek and me wants to know how is the screen is so fast even at 17
56:07
screen is so fast even at 17
56:07
screen is so fast even at 17 data let me run it again with explain
56:10
data let me run it again with explain
56:10
data let me run it again with explain and get back
56:16
and get back
56:16
and get back okay so if i look at this here
56:21
in stages this is kind of a query plan
56:23
in stages this is kind of a query plan
56:23
in stages this is kind of a query plan if you will right
56:28
you can see that what it did behind the
56:30
you can see that what it did behind the
56:30
you can see that what it did behind the scenes that he translated into a key sql
56:32
scenes that he translated into a key sql
56:32
scenes that he translated into a key sql query
56:33
query
56:33
query and it uses synapse laying this is how
56:35
and it uses synapse laying this is how
56:35
and it uses synapse laying this is how we achieve the scale
56:36
we achieve the scale
56:36
we achieve the scale right you can run this on analytical
56:38
right you can run this on analytical
56:38
right you can run this on analytical queries your application cannot use
56:39
queries your application cannot use
56:39
queries your application cannot use analytical queries over terabytes and
56:41
analytical queries over terabytes and
56:41
analytical queries over terabytes and petabytes of data
56:43
petabytes of data
56:43
petabytes of data and instead of relying on it pegging
56:45
and instead of relying on it pegging
56:45
and instead of relying on it pegging your operational data store
56:47
your operational data store
56:47
your operational data store uh with this with analytical queries
56:49
uh with this with analytical queries
56:49
uh with this with analytical queries that i tend to be heavy
56:51
that i tend to be heavy
56:51
that i tend to be heavy right and can i and killing it we
56:53
right and can i and killing it we
56:53
right and can i and killing it we instead we use synapse link
56:56
instead we use synapse link
56:56
instead we use synapse link and brings power to the application so
56:58
and brings power to the application so
56:58
and brings power to the application so you can take advantage of the real-time
57:00
you can take advantage of the real-time
57:00
you can take advantage of the real-time analytics
57:01
analytics
57:01
analytics and uh close the digital loop with the
57:03
and uh close the digital loop with the
57:03
and uh close the digital loop with the customer uh
57:04
customer uh
57:04
customer uh that's an example of the power this
57:06
that's an example of the power this
57:06
that's an example of the power this particular capability is coming soon
57:08
particular capability is coming soon
57:08
particular capability is coming soon but this is example of the power the
57:10
but this is example of the power the
57:10
but this is example of the power the synapse link provides
57:13
synapse link provides
57:13
synapse link provides um which we are very really excited
57:15
um which we are very really excited
57:15
um which we are very really excited about
57:16
about
57:16
about now to conclude uh lots of exciting
57:18
now to conclude uh lots of exciting
57:18
now to conclude uh lots of exciting things are coming online i didn't
57:20
things are coming online i didn't
57:20
things are coming online i didn't mention
57:21
mention
57:21
mention uh lots of security improvements ad
57:23
uh lots of security improvements ad
57:23
uh lots of security improvements ad integration our bag
57:24
integration our bag
57:24
integration our bag we're bringing uh cosmos dml emulator to
57:27
we're bringing uh cosmos dml emulator to
57:27
we're bringing uh cosmos dml emulator to linux
57:28
linux
57:28
linux uh cosmos db is used across the board in
57:31
uh cosmos db is used across the board in
57:31
uh cosmos db is used across the board in many
57:32
many
57:32
many uh verticals we just touched gave you a
57:34
uh verticals we just touched gave you a
57:34
uh verticals we just touched gave you a few examples and our wonderful guests
57:36
few examples and our wonderful guests
57:36
few examples and our wonderful guests shared with you their stories of using
57:38
shared with you their stories of using
57:38
shared with you their stories of using cosmos db and riding this wave
57:41
cosmos db and riding this wave
57:41
cosmos db and riding this wave of uh sailing through this uh world
57:44
of uh sailing through this uh world
57:44
of uh sailing through this uh world going upside down with the
57:46
going upside down with the
57:46
going upside down with the terabytes and petabytes of data rushing
57:48
terabytes and petabytes of data rushing
57:48
terabytes and petabytes of data rushing to your database
57:50
to your database
57:50
to your database with ease uh thanks to cosmos db scale
57:53
with ease uh thanks to cosmos db scale
57:53
with ease uh thanks to cosmos db scale and performance
57:54
and performance
57:54
and performance whether you're using uh whether you're
57:56
whether you're using uh whether you're
57:56
whether you're using uh whether you're building apps for retail manufacturing
57:58
building apps for retail manufacturing
57:58
building apps for retail manufacturing financial services
58:00
financial services
58:00
financial services cosmosdb is there for you you can read
58:02
cosmosdb is there for you you can read
58:02
cosmosdb is there for you you can read our
58:03
our
58:03
our stories of our customers you can tune in
58:06
stories of our customers you can tune in
58:06
stories of our customers you can tune in to our uh
58:09
to our uh
58:09
to our uh to our uh live blogs and
58:13
to our uh live blogs and
58:13
to our uh live blogs and i hope you enjoy our uh conference many
58:15
i hope you enjoy our uh conference many
58:15
i hope you enjoy our uh conference many more stories are coming
58:16
more stories are coming
58:16
more stories are coming uh folks are gonna go into a great level
58:18
uh folks are gonna go into a great level
58:18
uh folks are gonna go into a great level of detail
58:19
of detail
58:19
of detail about the advancements to cosmos db as
58:22
about the advancements to cosmos db as
58:22
about the advancements to cosmos db as well as
58:22
well as
58:22
well as how they build applications on top of
58:24
how they build applications on top of
58:24
how they build applications on top of cosmos db thank you so much
58:27
cosmos db thank you so much
58:27
cosmos db thank you so much and enjoy the show
58:31
thank you gary that was a lot of great
58:33
thank you gary that was a lot of great
58:33
thank you gary that was a lot of great content for our
58:34
content for our
58:34
content for our viewers whether they are you know new
58:36
viewers whether they are you know new
58:36
viewers whether they are you know new cosmos db users or
58:38
cosmos db users or
58:38
cosmos db users or seasoned cosmos db developers i think
58:40
seasoned cosmos db developers i think
58:40
seasoned cosmos db developers i think that was a lot of
58:41
that was a lot of
58:41
that was a lot of good content to catch up with everything
58:43
good content to catch up with everything
58:43
good content to catch up with everything that's happening in the cosmos did you
58:44
that's happening in the cosmos did you
58:44
that's happening in the cosmos did you landscape
58:45
landscape
58:45
landscape we got a lot of great questions in the
58:47
we got a lot of great questions in the
58:47
we got a lot of great questions in the chat unfortunately we are a bit tight on
58:49
chat unfortunately we are a bit tight on
58:49
chat unfortunately we are a bit tight on time so we won't be able
58:50
time so we won't be able
58:50
time so we won't be able to answer all of them uh a quick note
58:52
to answer all of them uh a quick note
58:52
to answer all of them uh a quick note for our for our
58:54
for our for our
58:54
for our for our viewers some questions haven't been
58:55
viewers some questions haven't been
58:55
viewers some questions haven't been answered because we have kept them uh
58:57
answered because we have kept them uh
58:57
answered because we have kept them uh for
58:57
for
58:57
for our speaker but no worries we will
58:59
our speaker but no worries we will
58:59
our speaker but no worries we will eventually get to answer all of them uh
59:01
eventually get to answer all of them uh
59:01
eventually get to answer all of them uh maybe just one if we have time material
59:02
maybe just one if we have time material
59:02
maybe just one if we have time material we have
59:03
we have
59:03
we have a good question um from one of our
59:06
a good question um from one of our
59:06
a good question um from one of our viewers
59:07
viewers
59:07
viewers asking what is the current studies and
59:09
asking what is the current studies and
59:09
asking what is the current studies and level of mongodb compatibility
59:11
level of mongodb compatibility
59:12
level of mongodb compatibility with mongodb's 3.6 and for all we
59:14
with mongodb's 3.6 and for all we
59:14
with mongodb's 3.6 and for all we achieved are
59:15
achieved are
59:15
achieved are very significant there may be an
59:18
very significant there may be an
59:18
very significant there may be an operator
59:19
operator
59:19
operator or two uh that you may find that we
59:21
or two uh that you may find that we
59:21
or two uh that you may find that we don't uh
59:22
don't uh
59:22
don't uh that we don't support and again uh we
59:25
that we don't support and again uh we
59:25
that we don't support and again uh we have all the telemetry so spokes bring
59:27
have all the telemetry so spokes bring
59:27
have all the telemetry so spokes bring the applications
59:28
the applications
59:28
the applications we're always on the lookout and if we
59:30
we're always on the lookout and if we
59:30
we're always on the lookout and if we missed one
59:31
missed one
59:31
missed one we added right away we know about it uh
59:35
we added right away we know about it uh
59:35
we added right away we know about it uh at the same time kind of numerous
59:37
at the same time kind of numerous
59:37
at the same time kind of numerous applications are now running
59:38
applications are now running
59:38
applications are now running um a large large portion of cosmos db
59:41
um a large large portion of cosmos db
59:42
um a large large portion of cosmos db usage are coming from mongodb api
59:44
usage are coming from mongodb api
59:44
usage are coming from mongodb api can have pretty significant uh customers
59:47
can have pretty significant uh customers
59:47
can have pretty significant uh customers running their lifeline
59:48
running their lifeline
59:48
running their lifeline on mongodb api um because there's
59:51
on mongodb api um because there's
59:52
on mongodb api um because there's quite quite a few case studies on this
59:53
quite quite a few case studies on this
59:54
quite quite a few case studies on this effort and as i mentioned kind of we
59:55
effort and as i mentioned kind of we
59:55
effort and as i mentioned kind of we have kind of
59:56
have kind of
59:56
have kind of hundreds of terabytes multiple databases
59:59
hundreds of terabytes multiple databases
59:59
hundreds of terabytes multiple databases running on cosmos db today
1:00:01
running on cosmos db today
1:00:01
running on cosmos db today uh compatibility is high but the
1:00:03
uh compatibility is high but the
1:00:03
uh compatibility is high but the important part is that it's not just
1:00:05
important part is that it's not just
1:00:05
important part is that it's not just it's just it's a compatible mobidb
1:00:06
it's just it's a compatible mobidb
1:00:06
it's just it's a compatible mobidb database
1:00:07
database
1:00:07
database it's a database that brings you this
1:00:09
it's a database that brings you this
1:00:09
it's a database that brings you this transparent sharding
1:00:11
transparent sharding
1:00:11
transparent sharding painless experience no pain no worrying
1:00:13
painless experience no pain no worrying
1:00:14
painless experience no pain no worrying about scale
1:00:15
about scale
1:00:15
about scale a limitless scale instant out of scale
1:00:18
a limitless scale instant out of scale
1:00:18
a limitless scale instant out of scale serverless
1:00:19
serverless
1:00:19
serverless it's a service mongodb database
1:00:21
it's a service mongodb database
1:00:21
it's a service mongodb database something that you cannot find
1:00:26
elsewhere
1:00:29
elsewhere
1:00:29
elsewhere all right uh we are just on time tim do
1:00:31
all right uh we are just on time tim do
1:00:31
all right uh we are just on time tim do you want to introduce our
1:00:32
you want to introduce our
1:00:32
you want to introduce our next speaker yeah that sounds great and
1:00:35
next speaker yeah that sounds great and
1:00:35
next speaker yeah that sounds great and thank you so much karil for that awesome
1:00:36
thank you so much karil for that awesome
1:00:36
thank you so much karil for that awesome session
1:00:37
session
1:00:37
session uh and it was great to hear i mean just
1:00:38
uh and it was great to hear i mean just
1:00:38
uh and it was great to hear i mean just how easy it is to migrate your mongodb
1:00:41
how easy it is to migrate your mongodb
1:00:41
how easy it is to migrate your mongodb applications to
1:00:42
applications to
1:00:42
applications to azure cosmos dbs api for mongodb uh
1:00:45
azure cosmos dbs api for mongodb uh
1:00:45
azure cosmos dbs api for mongodb uh and speaking of migration uh it's
1:00:47
and speaking of migration uh it's
1:00:47
and speaking of migration uh it's actually what our next session is going
1:00:49
actually what our next session is going
1:00:49
actually what our next session is going to be about so it's uh that question was
1:00:50
to be about so it's uh that question was
1:00:50
to be about so it's uh that question was a great transition
1:00:51
a great transition
1:00:52
a great transition uh to what olok and andrew are going to
1:00:54
uh to what olok and andrew are going to
1:00:54
uh to what olok and andrew are going to talk about today
1:00:55
talk about today
1:00:55
talk about today and they're going to talk about uh how
1:00:58
and they're going to talk about uh how
1:00:58
and they're going to talk about uh how to
1:00:58
to
1:00:58
to basically uh migrate in real time with
1:01:01
basically uh migrate in real time with
1:01:01
basically uh migrate in real time with db
1:01:02
db
1:01:02
db so now i'm going to turn it over to golo
1:01:04
so now i'm going to turn it over to golo
1:01:04
so now i'm going to turn it over to golo online in real time database migrations
1:01:06
online in real time database migrations
1:01:06
online in real time database migrations to azure cosmos cp with screen
1:01:11
to azure cosmos cp with screen
1:01:11
to azure cosmos cp with screen hey everyone um this is a look and
1:01:15
hey everyone um this is a look and
1:01:15
hey everyone um this is a look and andrew here
1:01:15
andrew here
1:01:15
andrew here uh so my name's andrew i've been working
1:01:18
uh so my name's andrew i've been working
1:01:18
uh so my name's andrew i've been working on the azure cosmos db
1:01:19
on the azure cosmos db
1:01:19
on the azure cosmos db uh product for about the past seven
1:01:21
uh product for about the past seven
1:01:21
uh product for about the past seven years and over the years
1:01:23
years and over the years
1:01:23
years and over the years i felt very very blessed to work with
1:01:25
i felt very very blessed to work with
1:01:25
i felt very very blessed to work with many developers like you
1:01:27
many developers like you
1:01:27
many developers like you in this session i'd like to cover one of
1:01:28
in this session i'd like to cover one of
1:01:28
in this session i'd like to cover one of the most common challenges i frequently
1:01:31
the most common challenges i frequently
1:01:31
the most common challenges i frequently see
1:01:31
see
1:01:31
see in modernizing applications and to set
1:01:34
in modernizing applications and to set
1:01:34
in modernizing applications and to set some context on this
1:01:35
some context on this
1:01:35
some context on this the common use case i frequently see on
1:01:37
the common use case i frequently see on
1:01:37
the common use case i frequently see on why developers choose cosmos db
1:01:39
why developers choose cosmos db
1:01:39
why developers choose cosmos db over traditional databases is to take
1:01:41
over traditional databases is to take
1:01:41
over traditional databases is to take advantage of its distributed nature
1:01:43
advantage of its distributed nature
1:01:43
advantage of its distributed nature um if you've worked with data at scale
1:01:45
um if you've worked with data at scale
1:01:45
um if you've worked with data at scale before this is personally known as the
1:01:46
before this is personally known as the
1:01:46
before this is personally known as the partitioning and replication problem
1:01:48
partitioning and replication problem
1:01:48
partitioning and replication problem and azure cosmos db provides the ability
1:01:51
and azure cosmos db provides the ability
1:01:51
and azure cosmos db provides the ability to automatically scale up via
1:01:52
to automatically scale up via
1:01:52
to automatically scale up via partitioning
1:01:53
partitioning
1:01:53
partitioning or known as sharding which enables our
1:01:55
or known as sharding which enables our
1:01:55
or known as sharding which enables our applications to remain highly responsive
1:01:57
applications to remain highly responsive
1:01:57
applications to remain highly responsive even when they grow to massive scale in
1:02:00
even when they grow to massive scale in
1:02:00
even when they grow to massive scale in the previous session karel showed us how
1:02:02
the previous session karel showed us how
1:02:02
the previous session karel showed us how microsoft
1:02:02
microsoft
1:02:02
microsoft builds applications like microsoft teams
1:02:04
builds applications like microsoft teams
1:02:04
builds applications like microsoft teams on cosmos db
1:02:06
on cosmos db
1:02:06
on cosmos db which allows the underlying data tier to
1:02:08
which allows the underlying data tier to
1:02:08
which allows the underlying data tier to scale to trillions of transactions per
1:02:10
scale to trillions of transactions per
1:02:10
scale to trillions of transactions per day
1:02:10
day
1:02:10
day while maintaining latencies measured in
1:02:12
while maintaining latencies measured in
1:02:12
while maintaining latencies measured in single digit milliseconds
1:02:14
single digit milliseconds
1:02:14
single digit milliseconds i also frequently see developers
1:02:16
i also frequently see developers
1:02:16
i also frequently see developers leverage the other distributed aspect of
1:02:18
leverage the other distributed aspect of
1:02:18
leverage the other distributed aspect of cosmo cd which is replication
1:02:20
cosmo cd which is replication
1:02:20
cosmo cd which is replication in which we can easily build
1:02:21
in which we can easily build
1:02:21
in which we can easily build applications that enjoy a high degree of
1:02:23
applications that enjoy a high degree of
1:02:23
applications that enjoy a high degree of availability
1:02:24
availability
1:02:24
availability and resiliency and this is achieved
1:02:26
and resiliency and this is achieved
1:02:26
and resiliency and this is achieved through redundancy by replicating that
1:02:28
through redundancy by replicating that
1:02:28
through redundancy by replicating that data live across
1:02:29
data live across
1:02:30
data live across numerous data centers now a question
1:02:33
numerous data centers now a question
1:02:33
numerous data centers now a question that i'm frequently asked is
1:02:35
that i'm frequently asked is
1:02:35
that i'm frequently asked is how do i take advantage of cosmos db for
1:02:37
how do i take advantage of cosmos db for
1:02:37
how do i take advantage of cosmos db for my existing mission critical
1:02:38
my existing mission critical
1:02:38
my existing mission critical applications
1:02:39
applications
1:02:40
applications more specifically how do i facilitate
1:02:42
more specifically how do i facilitate
1:02:42
more specifically how do i facilitate live data movement
1:02:43
live data movement
1:02:43
live data movement from my existing legacy database can we
1:02:46
from my existing legacy database can we
1:02:46
from my existing legacy database can we do this while keeping our applications
1:02:48
do this while keeping our applications
1:02:48
do this while keeping our applications online
1:02:48
online
1:02:48
online and avoid linking maintenance periods
1:02:51
and avoid linking maintenance periods
1:02:51
and avoid linking maintenance periods what if that data is coming from a
1:02:52
what if that data is coming from a
1:02:52
what if that data is coming from a traditional relational database
1:02:54
traditional relational database
1:02:54
traditional relational database the data model on my relational database
1:02:56
the data model on my relational database
1:02:56
the data model on my relational database tends to be highly normalized
1:02:57
tends to be highly normalized
1:02:57
tends to be highly normalized in which we optimize for storage over
1:02:59
in which we optimize for storage over
1:02:59
in which we optimize for storage over compute by
1:03:00
compute by
1:03:00
compute by shredding and joining that data across
1:03:03
shredding and joining that data across
1:03:03
shredding and joining that data across many tables on each request
1:03:04
many tables on each request
1:03:04
many tables on each request but in cosmos db we're taught to revisit
1:03:07
but in cosmos db we're taught to revisit
1:03:07
but in cosmos db we're taught to revisit how we think about data modeling
1:03:08
how we think about data modeling
1:03:08
how we think about data modeling which we need to optimize for compute in
1:03:10
which we need to optimize for compute in
1:03:10
which we need to optimize for compute in the form of our use
1:03:12
the form of our use
1:03:12
the form of our use instead of storage this is achieved by
1:03:14
instead of storage this is achieved by
1:03:14
instead of storage this is achieved by denormalizing that data and which
1:03:15
denormalizing that data and which
1:03:15
denormalizing that data and which relationships are represented through
1:03:17
relationships are represented through
1:03:17
relationships are represented through nested objects and arrays
1:03:19
nested objects and arrays
1:03:19
nested objects and arrays instead of joining across tables moving
1:03:21
instead of joining across tables moving
1:03:21
instead of joining across tables moving from normalized to denormalized data
1:03:23
from normalized to denormalized data
1:03:23
from normalized to denormalized data requires
1:03:23
requires
1:03:23
requires quite a bit of data transformation can
1:03:26
quite a bit of data transformation can
1:03:26
quite a bit of data transformation can we do this as part of our live data
1:03:28
we do this as part of our live data
1:03:28
we do this as part of our live data movement process
1:03:29
movement process
1:03:29
movement process so to solve this problem i've had the
1:03:31
so to solve this problem i've had the
1:03:31
so to solve this problem i've had the great pleasure of working with a very
1:03:33
great pleasure of working with a very
1:03:33
great pleasure of working with a very very good friend of mine a lock
1:03:35
very good friend of mine a lock
1:03:35
very good friend of mine a lock who is the founder and chief product
1:03:37
who is the founder and chief product
1:03:37
who is the founder and chief product officer at stream
1:03:38
officer at stream
1:03:38
officer at stream shrime builds some amazing tooling that
1:03:40
shrime builds some amazing tooling that
1:03:40
shrime builds some amazing tooling that targets solving this very specific pain
1:03:42
targets solving this very specific pain
1:03:42
targets solving this very specific pain point uh so a lot it's
1:03:45
point uh so a lot it's
1:03:45
point uh so a lot it's very very nice to have you with us uh
1:03:47
very very nice to have you with us uh
1:03:47
very very nice to have you with us uh would you like to tell us more on this
1:03:50
would you like to tell us more on this
1:03:50
would you like to tell us more on this sure absolutely thank you so much uh
1:03:52
sure absolutely thank you so much uh
1:03:52
sure absolutely thank you so much uh andrew and uh thanks a lot
1:03:53
andrew and uh thanks a lot
1:03:53
andrew and uh thanks a lot um i hope everyone's enjoying this
1:03:55
um i hope everyone's enjoying this
1:03:55
um i hope everyone's enjoying this virtual cosmos db conference
1:03:57
virtual cosmos db conference
1:03:58
virtual cosmos db conference i'm really delighted to be part of this
1:04:00
i'm really delighted to be part of this
1:04:00
i'm really delighted to be part of this and let me pick it off from
1:04:01
and let me pick it off from
1:04:01
and let me pick it off from where curiel and andrew left off um
1:04:04
where curiel and andrew left off um
1:04:04
where curiel and andrew left off um yeah so in this session in the next 30
1:04:06
yeah so in this session in the next 30
1:04:06
yeah so in this session in the next 30 minutes
1:04:07
minutes
1:04:08
minutes i'm going to be talking about primarily
1:04:10
i'm going to be talking about primarily
1:04:10
i'm going to be talking about primarily the data movement and the data migration
1:04:12
the data movement and the data migration
1:04:12
the data movement and the data migration problem that was just touched on i'm
1:04:15
problem that was just touched on i'm
1:04:15
problem that was just touched on i'm going to start off with a
1:04:16
going to start off with a
1:04:16
going to start off with a with a platform overview so that's going
1:04:18
with a platform overview so that's going
1:04:18
with a platform overview so that's going to give you some
1:04:19
to give you some
1:04:19
to give you some idea about the feature functional
1:04:21
idea about the feature functional
1:04:21
idea about the feature functional capabilities of what can be achieved and
1:04:23
capabilities of what can be achieved and
1:04:23
capabilities of what can be achieved and how to
1:04:24
how to
1:04:24
how to you know set up the actual migration and
1:04:26
you know set up the actual migration and
1:04:26
you know set up the actual migration and the integration data pipelines
1:04:28
the integration data pipelines
1:04:28
the integration data pipelines um and uh i'm also going to point out
1:04:31
um and uh i'm also going to point out
1:04:31
um and uh i'm also going to point out some of the challenges along the way
1:04:32
some of the challenges along the way
1:04:32
some of the challenges along the way um you know as it comes to both the
1:04:35
um you know as it comes to both the
1:04:35
um you know as it comes to both the denomination
1:04:36
denomination
1:04:36
denomination uh moving data from your legacy systems
1:04:38
uh moving data from your legacy systems
1:04:38
uh moving data from your legacy systems onto cosmos db
1:04:40
onto cosmos db
1:04:40
onto cosmos db uh how do you do uh mapping between
1:04:42
uh how do you do uh mapping between
1:04:42
uh how do you do uh mapping between different uh
1:04:44
different uh
1:04:44
different uh for example from one primary key to a
1:04:46
for example from one primary key to a
1:04:46
for example from one primary key to a partition key
1:04:47
partition key
1:04:47
partition key how do you scale the workloads and so
1:04:49
how do you scale the workloads and so
1:04:49
how do you scale the workloads and so forth um and i'm going to wrap up and
1:04:51
forth um and i'm going to wrap up and
1:04:51
forth um and i'm going to wrap up and i'm going to try
1:04:51
i'm going to try
1:04:51
i'm going to try to show you a couple of demos hopefully
1:04:55
to show you a couple of demos hopefully
1:04:55
to show you a couple of demos hopefully i'll have time to show you both so with
1:04:57
i'll have time to show you both so with
1:04:57
i'll have time to show you both so with that let's get started
1:04:59
that let's get started
1:04:59
that let's get started so first about stream um so a stream as
1:05:02
so first about stream um so a stream as
1:05:02
so first about stream um so a stream as a
1:05:03
a
1:05:03
a partner and we've been working with
1:05:05
partner and we've been working with
1:05:05
partner and we've been working with cosmos db for several years so we enable
1:05:08
cosmos db for several years so we enable
1:05:08
cosmos db for several years so we enable continuous
1:05:09
continuous
1:05:09
continuous enterprise data movement to the cloud
1:05:12
enterprise data movement to the cloud
1:05:12
enterprise data movement to the cloud and
1:05:12
and
1:05:12
and um the core idea is that you might have
1:05:15
um the core idea is that you might have
1:05:15
um the core idea is that you might have a number of different
1:05:16
a number of different
1:05:16
a number of different applications running against you know
1:05:19
applications running against you know
1:05:19
applications running against you know different types of
1:05:20
different types of
1:05:20
different types of database systems nosql systems many of
1:05:23
database systems nosql systems many of
1:05:23
database systems nosql systems many of them could be running on premise
1:05:25
them could be running on premise
1:05:25
them could be running on premise they could be running in other clouds
1:05:27
they could be running in other clouds
1:05:27
they could be running in other clouds and oftentimes to sort of take advantage
1:05:29
and oftentimes to sort of take advantage
1:05:29
and oftentimes to sort of take advantage of the
1:05:30
of the
1:05:30
of the you know very low latency access in a
1:05:33
you know very low latency access in a
1:05:33
you know very low latency access in a distributed fashion
1:05:34
distributed fashion
1:05:34
distributed fashion for the cosmos db workloads you want to
1:05:37
for the cosmos db workloads you want to
1:05:37
for the cosmos db workloads you want to move the data
1:05:37
move the data
1:05:37
move the data into cosmos db so what we do is we
1:05:40
into cosmos db so what we do is we
1:05:40
into cosmos db so what we do is we facilitate that
1:05:41
facilitate that
1:05:41
facilitate that and the core focus that we have in line
1:05:44
and the core focus that we have in line
1:05:44
and the core focus that we have in line with
1:05:44
with
1:05:44
with why you want to deploy a lot of these
1:05:46
why you want to deploy a lot of these
1:05:46
why you want to deploy a lot of these applications on cosmos db
1:05:48
applications on cosmos db
1:05:48
applications on cosmos db is the real-time aspect of it now if you
1:05:51
is the real-time aspect of it now if you
1:05:51
is the real-time aspect of it now if you sort of take a look at the picture to
1:05:53
sort of take a look at the picture to
1:05:53
sort of take a look at the picture to the right
1:05:53
the right
1:05:53
the right um and take a look at um you know data
1:05:56
um and take a look at um you know data
1:05:56
um and take a look at um you know data as an iceberg
1:05:57
as an iceberg
1:05:57
as an iceberg there's ways to come at the data um
1:06:00
there's ways to come at the data um
1:06:00
there's ways to come at the data um directly through the apis of the
1:06:02
directly through the apis of the
1:06:02
directly through the apis of the application
1:06:03
application
1:06:03
application and then move that on to let's say a
1:06:05
and then move that on to let's say a
1:06:05
and then move that on to let's say a system like cosmos db
1:06:07
system like cosmos db
1:06:07
system like cosmos db but oftentimes when the data
1:06:10
but oftentimes when the data
1:06:10
but oftentimes when the data volumes are super high and we're really
1:06:13
volumes are super high and we're really
1:06:13
volumes are super high and we're really talking about
1:06:14
talking about
1:06:14
talking about tens to hundreds of thousands of events
1:06:16
tens to hundreds of thousands of events
1:06:16
tens to hundreds of thousands of events per second in a traditional legacy
1:06:18
per second in a traditional legacy
1:06:18
per second in a traditional legacy database
1:06:19
database
1:06:19
database then how do you actually come at it from
1:06:21
then how do you actually come at it from
1:06:21
then how do you actually come at it from an api that's at the application level
1:06:23
an api that's at the application level
1:06:23
an api that's at the application level that becomes a critical challenge so
1:06:25
that becomes a critical challenge so
1:06:25
that becomes a critical challenge so oftentimes what people do is they tend
1:06:27
oftentimes what people do is they tend
1:06:27
oftentimes what people do is they tend to come at it
1:06:29
to come at it
1:06:29
to come at it more in a batch basis and what that does
1:06:31
more in a batch basis and what that does
1:06:31
more in a batch basis and what that does is
1:06:32
is
1:06:32
is it introduces latency between
1:06:35
it introduces latency between
1:06:35
it introduces latency between the traditional system and cosmos db and
1:06:38
the traditional system and cosmos db and
1:06:38
the traditional system and cosmos db and that's one of the things you're trying
1:06:39
that's one of the things you're trying
1:06:39
that's one of the things you're trying to avoid
1:06:40
to avoid
1:06:40
to avoid because of the sensitive nature in terms
1:06:42
because of the sensitive nature in terms
1:06:42
because of the sensitive nature in terms of low latency for many of these
1:06:43
of low latency for many of these
1:06:43
of low latency for many of these applications
1:06:44
applications
1:06:44
applications so one of the one of the sort of the
1:06:47
so one of the one of the sort of the
1:06:47
so one of the one of the sort of the revolutionary techniques here
1:06:49
revolutionary techniques here
1:06:49
revolutionary techniques here is something called you know change data
1:06:51
is something called you know change data
1:06:51
is something called you know change data capture and the ability to stream
1:06:53
capture and the ability to stream
1:06:53
capture and the ability to stream in a very non-intrusive fashion data uh
1:06:56
in a very non-intrusive fashion data uh
1:06:56
in a very non-intrusive fashion data uh from many of these systems into cosmos
1:06:59
from many of these systems into cosmos
1:06:59
from many of these systems into cosmos db so i'm going to spend
1:07:00
db so i'm going to spend
1:07:00
db so i'm going to spend a significant amount of time getting
1:07:02
a significant amount of time getting
1:07:02
a significant amount of time getting into that area today
1:07:04
into that area today
1:07:04
into that area today okay so um just overall to set the stage
1:07:08
okay so um just overall to set the stage
1:07:08
okay so um just overall to set the stage um as to how the pipelines would work
1:07:11
um as to how the pipelines would work
1:07:11
um as to how the pipelines would work and you know as developers you know how
1:07:13
and you know as developers you know how
1:07:13
and you know as developers you know how are you guys able to work with stream as
1:07:15
are you guys able to work with stream as
1:07:15
are you guys able to work with stream as a platform to
1:07:16
a platform to
1:07:16
a platform to ultimately work with any one of the apis
1:07:19
ultimately work with any one of the apis
1:07:19
ultimately work with any one of the apis for cosmos db any one of the different
1:07:21
for cosmos db any one of the different
1:07:21
for cosmos db any one of the different models
1:07:22
models
1:07:22
models and then stream the data there let me
1:07:24
and then stream the data there let me
1:07:24
and then stream the data there let me just set up the high level picture
1:07:26
just set up the high level picture
1:07:26
just set up the high level picture so um you have a lot of different
1:07:28
so um you have a lot of different
1:07:28
so um you have a lot of different applications that you're running today
1:07:29
applications that you're running today
1:07:29
applications that you're running today they could be running on
1:07:30
they could be running on
1:07:30
they could be running on databases uh there could be some
1:07:32
databases uh there could be some
1:07:32
databases uh there could be some applications that are generating
1:07:34
applications that are generating
1:07:34
applications that are generating uh log files or writing data to
1:07:36
uh log files or writing data to
1:07:36
uh log files or writing data to messaging systems
1:07:38
messaging systems
1:07:38
messaging systems uh which could be for example kafka and
1:07:40
uh which could be for example kafka and
1:07:40
uh which could be for example kafka and data could also be coming in over um you
1:07:42
data could also be coming in over um you
1:07:42
data could also be coming in over um you know different uh protocols let's say
1:07:43
know different uh protocols let's say
1:07:44
know different uh protocols let's say tcp udp http etc
1:07:46
tcp udp http etc
1:07:46
tcp udp http etc so uh within the platform the entire
1:07:49
so uh within the platform the entire
1:07:49
so uh within the platform the entire um orientation of building a pipeline is
1:07:52
um orientation of building a pipeline is
1:07:52
um orientation of building a pipeline is having a drag and drop
1:07:53
having a drag and drop
1:07:53
having a drag and drop type of an interface and then at the
1:07:55
type of an interface and then at the
1:07:55
type of an interface and then at the back end
1:07:56
back end
1:07:56
back end there's java code that actually gets
1:07:58
there's java code that actually gets
1:07:58
there's java code that actually gets executed for continuous data collection
1:08:00
executed for continuous data collection
1:08:00
executed for continuous data collection and at a very uh straightforward type of
1:08:03
and at a very uh straightforward type of
1:08:03
and at a very uh straightforward type of a pipeline
1:08:04
a pipeline
1:08:04
a pipeline you can continuously collect this and
1:08:06
you can continuously collect this and
1:08:06
you can continuously collect this and you can actually deliver this into
1:08:07
you can actually deliver this into
1:08:07
you can actually deliver this into different uh collections onto
1:08:09
different uh collections onto
1:08:10
different uh collections onto cosmos db into different containers um
1:08:13
cosmos db into different containers um
1:08:13
cosmos db into different containers um more interestingly you know as you start
1:08:15
more interestingly you know as you start
1:08:15
more interestingly you know as you start moving this data
1:08:16
moving this data
1:08:16
moving this data andrew talked about some of the
1:08:18
andrew talked about some of the
1:08:18
andrew talked about some of the challenges that come along you might
1:08:19
challenges that come along you might
1:08:20
challenges that come along you might have you know
1:08:21
have you know
1:08:21
have you know very normalized data and then you want
1:08:23
very normalized data and then you want
1:08:23
very normalized data and then you want to sort of
1:08:24
to sort of
1:08:24
to sort of split that up and write different
1:08:26
split that up and write different
1:08:26
split that up and write different portions of that data to different
1:08:27
portions of that data to different
1:08:27
portions of that data to different containers
1:08:28
containers
1:08:28
containers or you want to coalesce it uh sometimes
1:08:30
or you want to coalesce it uh sometimes
1:08:30
or you want to coalesce it uh sometimes so that you can bring it together into
1:08:32
so that you can bring it together into
1:08:32
so that you can bring it together into one container
1:08:33
one container
1:08:33
one container so that you can have denormalized access
1:08:35
so that you can have denormalized access
1:08:35
so that you can have denormalized access to that data
1:08:36
to that data
1:08:36
to that data for your applications that are running
1:08:37
for your applications that are running
1:08:38
for your applications that are running on cosmos tv
1:08:39
on cosmos tv
1:08:39
on cosmos tv so one of the things that you can do is
1:08:41
so one of the things that you can do is
1:08:41
so one of the things that you can do is as the data comes in
1:08:43
as the data comes in
1:08:43
as the data comes in you can either event by event or
1:08:46
you can either event by event or
1:08:46
you can either event by event or you know on a state like a like a little
1:08:49
you know on a state like a like a little
1:08:49
you know on a state like a like a little window
1:08:50
window
1:08:50
window which is temporal or it could be batch
1:08:52
which is temporal or it could be batch
1:08:52
which is temporal or it could be batch based you can hold that data
1:08:54
based you can hold that data
1:08:54
based you can hold that data and this is all happening in memory and
1:08:57
and this is all happening in memory and
1:08:57
and this is all happening in memory and you're able to apply
1:08:58
you're able to apply
1:08:58
you're able to apply a number of different uh you know
1:09:02
a number of different uh you know
1:09:02
a number of different uh you know transformations or aggregations or
1:09:05
transformations or aggregations or
1:09:05
transformations or aggregations or enrichment or even join this with
1:09:08
enrichment or even join this with
1:09:08
enrichment or even join this with data that is reference data and this is
1:09:11
data that is reference data and this is
1:09:11
data that is reference data and this is one of the
1:09:11
one of the
1:09:11
one of the critical requirements that often come
1:09:13
critical requirements that often come
1:09:13
critical requirements that often come you know we face as developers where we
1:09:15
you know we face as developers where we
1:09:15
you know we face as developers where we want to move the data
1:09:16
want to move the data
1:09:16
want to move the data to sort of you know create the overall
1:09:19
to sort of you know create the overall
1:09:19
to sort of you know create the overall uh document
1:09:20
uh document
1:09:20
uh document as we move the move the data so in this
1:09:22
as we move the move the data so in this
1:09:22
as we move the move the data so in this example you can see that logically you
1:09:24
example you can see that logically you
1:09:24
example you can see that logically you are able to
1:09:25
are able to
1:09:26
are able to not only transform uh the data that's
1:09:28
not only transform uh the data that's
1:09:28
not only transform uh the data that's coming in
1:09:29
coming in
1:09:29
coming in you can also you know
1:09:32
you can also you know
1:09:32
you can also you know massage it using an external context so
1:09:34
massage it using an external context so
1:09:34
massage it using an external context so you could preload data
1:09:36
you could preload data
1:09:36
you could preload data from additional sources into the stream
1:09:38
from additional sources into the stream
1:09:38
from additional sources into the stream platform
1:09:39
platform
1:09:39
platform and now you can literally join that data
1:09:42
and now you can literally join that data
1:09:42
and now you can literally join that data um
1:09:42
um
1:09:42
um you know as you take that data this is
1:09:44
you know as you take that data this is
1:09:44
you know as you take that data this is this might be for example for
1:09:46
this might be for example for
1:09:46
this might be for example for for enrichment of that data some more
1:09:49
for enrichment of that data some more
1:09:49
for enrichment of that data some more advanced capabilities might be
1:09:51
advanced capabilities might be
1:09:51
advanced capabilities might be that our operators um one thing i should
1:09:54
that our operators um one thing i should
1:09:54
that our operators um one thing i should have mentioned is that
1:09:55
have mentioned is that
1:09:55
have mentioned is that a lot of the capability is uh through
1:09:57
a lot of the capability is uh through
1:09:57
a lot of the capability is uh through the ui
1:09:58
the ui
1:09:58
the ui uh so you are able to actually and i'm
1:09:59
uh so you are able to actually and i'm
1:09:59
uh so you are able to actually and i'm gonna show you that later
1:10:01
gonna show you that later
1:10:01
gonna show you that later you're able to take a lot of the
1:10:03
you're able to take a lot of the
1:10:03
you're able to take a lot of the operators that we have
1:10:04
operators that we have
1:10:04
operators that we have and apply that on the record itself or
1:10:07
and apply that on the record itself or
1:10:07
and apply that on the record itself or on the window
1:10:08
on the window
1:10:08
on the window and if you wanted to extend it you're
1:10:11
and if you wanted to extend it you're
1:10:11
and if you wanted to extend it you're also able to write your own queries
1:10:14
also able to write your own queries
1:10:14
also able to write your own queries in sql um or if you wanted to
1:10:17
in sql um or if you wanted to
1:10:17
in sql um or if you wanted to do some advanced uh integration with
1:10:19
do some advanced uh integration with
1:10:19
do some advanced uh integration with your own machine learning models for
1:10:21
your own machine learning models for
1:10:21
your own machine learning models for example
1:10:22
example
1:10:22
example uh because you know the model might want
1:10:23
uh because you know the model might want
1:10:23
uh because you know the model might want to give you a a value that you want to
1:10:25
to give you a a value that you want to
1:10:26
to give you a a value that you want to store as part of that data
1:10:27
store as part of that data
1:10:27
store as part of that data then you're able to also write your own
1:10:28
then you're able to also write your own
1:10:28
then you're able to also write your own java code and you can actually link in
1:10:30
java code and you can actually link in
1:10:30
java code and you can actually link in with that
1:10:31
with that
1:10:31
with that um along the pipeline so again you know
1:10:34
um along the pipeline so again you know
1:10:34
um along the pipeline so again you know just stressing on the collection piece
1:10:35
just stressing on the collection piece
1:10:35
just stressing on the collection piece of it
1:10:36
of it
1:10:36
of it the transformation piece of it and the
1:10:38
the transformation piece of it and the
1:10:38
the transformation piece of it and the transformation could involve multiple
1:10:39
transformation could involve multiple
1:10:39
transformation could involve multiple different
1:10:40
different
1:10:40
different uh tables for example coming in from an
1:10:42
uh tables for example coming in from an
1:10:42
uh tables for example coming in from an oracle database or a sql server database
1:10:44
oracle database or a sql server database
1:10:44
oracle database or a sql server database mysql database
1:10:46
mysql database
1:10:46
mysql database and you can actually massage it or you
1:10:47
and you can actually massage it or you
1:10:47
and you can actually massage it or you could split it apart and then give that
1:10:49
could split it apart and then give that
1:10:49
could split it apart and then give that as different
1:10:50
as different
1:10:50
as different um you know event streams to the
1:10:52
um you know event streams to the
1:10:52
um you know event streams to the platform and each event stream could
1:10:54
platform and each event stream could
1:10:54
platform and each event stream could then be talking to
1:10:55
then be talking to
1:10:55
then be talking to a different container and we'll get into
1:10:57
a different container and we'll get into
1:10:57
a different container and we'll get into the guts of that i'm just trying to set
1:10:58
the guts of that i'm just trying to set
1:10:58
the guts of that i'm just trying to set up the high level picture here
1:11:00
up the high level picture here
1:11:00
up the high level picture here along the way there's a lot of
1:11:01
along the way there's a lot of
1:11:01
along the way there's a lot of operational monitoring so there's a lot
1:11:03
operational monitoring so there's a lot
1:11:03
operational monitoring so there's a lot of debugging capabilities
1:11:05
of debugging capabilities
1:11:05
of debugging capabilities um that you can you can you can take a
1:11:07
um that you can you can you can take a
1:11:07
um that you can you can you can take a look at the
1:11:08
look at the
1:11:08
look at the kind of the image of the data that's
1:11:09
kind of the image of the data that's
1:11:09
kind of the image of the data that's coming in and once you actually apply
1:11:11
coming in and once you actually apply
1:11:11
coming in and once you actually apply the transformation operator
1:11:13
the transformation operator
1:11:13
the transformation operator there's also a post view of that of that
1:11:16
there's also a post view of that of that
1:11:16
there's also a post view of that of that record
1:11:16
record
1:11:16
record so let's say if you have a record that's
1:11:19
so let's say if you have a record that's
1:11:19
so let's say if you have a record that's coming in which is a straightforward
1:11:21
coming in which is a straightforward
1:11:22
coming in which is a straightforward you know tuple from a relational
1:11:23
you know tuple from a relational
1:11:23
you know tuple from a relational database you can actually convert that
1:11:25
database you can actually convert that
1:11:25
database you can actually convert that to a json object and you're able to
1:11:27
to a json object and you're able to
1:11:27
to a json object and you're able to actually visualize the json object along
1:11:29
actually visualize the json object along
1:11:29
actually visualize the json object along the way
1:11:29
the way
1:11:29
the way prior to it delivering that over the api
1:11:33
prior to it delivering that over the api
1:11:33
prior to it delivering that over the api on the cosmos tv site okay
1:11:36
on the cosmos tv site okay
1:11:36
on the cosmos tv site okay um so let me just quickly get into why
1:11:39
um so let me just quickly get into why
1:11:39
um so let me just quickly get into why is this interesting and where
1:11:40
is this interesting and where
1:11:40
is this interesting and where we see a lot of our joint customers um
1:11:43
we see a lot of our joint customers um
1:11:43
we see a lot of our joint customers um who have these development teams
1:11:45
who have these development teams
1:11:45
who have these development teams you know uh struggle um and you know
1:11:47
you know uh struggle um and you know
1:11:47
you know uh struggle um and you know what are the challenges they're trying
1:11:48
what are the challenges they're trying
1:11:48
what are the challenges they're trying to solve
1:11:49
to solve
1:11:49
to solve so i think some of them are mentioned um
1:11:51
so i think some of them are mentioned um
1:11:51
so i think some of them are mentioned um obviously you might actually be trying
1:11:53
obviously you might actually be trying
1:11:53
obviously you might actually be trying to sunset an application that may be
1:11:54
to sunset an application that may be
1:11:54
to sunset an application that may be running
1:11:55
running
1:11:55
running let's say in a legacy database like
1:11:58
let's say in a legacy database like
1:11:58
let's say in a legacy database like oracle
1:11:59
oracle
1:11:59
oracle uh you might actually have uh data
1:12:00
uh you might actually have uh data
1:12:00
uh you might actually have uh data coming in from a
1:12:02
coming in from a
1:12:02
coming in from a a nosql system like mongodb and you
1:12:04
a nosql system like mongodb and you
1:12:04
a nosql system like mongodb and you wanted to do a migration and eventually
1:12:06
wanted to do a migration and eventually
1:12:06
wanted to do a migration and eventually phase that application to cosmos db
1:12:09
phase that application to cosmos db
1:12:09
phase that application to cosmos db um you may want to do data denomination
1:12:11
um you may want to do data denomination
1:12:11
um you may want to do data denomination uh like i mentioned
1:12:13
uh like i mentioned
1:12:13
uh like i mentioned um where you know it's not your goal may
1:12:16
um where you know it's not your goal may
1:12:16
um where you know it's not your goal may not be to sunset an application
1:12:18
not be to sunset an application
1:12:18
not be to sunset an application but there's a new application that's
1:12:19
but there's a new application that's
1:12:19
but there's a new application that's being deployed uh to kind of you know do
1:12:22
being deployed uh to kind of you know do
1:12:22
being deployed uh to kind of you know do digital transformation or data
1:12:23
digital transformation or data
1:12:23
digital transformation or data modernization and as part of that you
1:12:25
modernization and as part of that you
1:12:25
modernization and as part of that you want to sort of you know
1:12:26
want to sort of you know
1:12:26
want to sort of you know have a brand new application run on on
1:12:29
have a brand new application run on on
1:12:29
have a brand new application run on on cosmos db
1:12:30
cosmos db
1:12:30
cosmos db um real-time telemetry and event feeds
1:12:32
um real-time telemetry and event feeds
1:12:32
um real-time telemetry and event feeds into cosmos db that's another critical
1:12:34
into cosmos db that's another critical
1:12:34
into cosmos db that's another critical uh use case
1:12:35
uh use case
1:12:35
uh use case and this has to do with the ability to
1:12:37
and this has to do with the ability to
1:12:38
and this has to do with the ability to ingest at very very high volumes so
1:12:40
ingest at very very high volumes so
1:12:40
ingest at very very high volumes so typically millions of events per second
1:12:41
typically millions of events per second
1:12:41
typically millions of events per second if that's the kind of use case that you
1:12:43
if that's the kind of use case that you
1:12:43
if that's the kind of use case that you have um
1:12:45
have um
1:12:45
have um doing parallel reads if you have a kafka
1:12:48
doing parallel reads if you have a kafka
1:12:48
doing parallel reads if you have a kafka cluster
1:12:49
cluster
1:12:49
cluster and there's a large number of partitions
1:12:51
and there's a large number of partitions
1:12:51
and there's a large number of partitions then how do you actually go in
1:12:53
then how do you actually go in
1:12:53
then how do you actually go in and do continuous data collection across
1:12:56
and do continuous data collection across
1:12:56
and do continuous data collection across the
1:12:56
the
1:12:56
the across the kafka topic which may have
1:12:59
across the kafka topic which may have
1:12:59
across the kafka topic which may have hundreds of partitions
1:13:00
hundreds of partitions
1:13:00
hundreds of partitions using a very simple uh you know pipeline
1:13:04
using a very simple uh you know pipeline
1:13:04
using a very simple uh you know pipeline that allows you to capture the data
1:13:06
that allows you to capture the data
1:13:06
that allows you to capture the data continuously
1:13:07
continuously
1:13:07
continuously and then move it to the cosmos db uh
1:13:09
and then move it to the cosmos db uh
1:13:09
and then move it to the cosmos db uh target system
1:13:11
target system
1:13:11
target system and lastly even if you have just normal
1:13:13
and lastly even if you have just normal
1:13:13
and lastly even if you have just normal uh json data that's coming in from some
1:13:15
uh json data that's coming in from some
1:13:15
uh json data that's coming in from some application in a file
1:13:17
application in a file
1:13:17
application in a file or maybe over over some uh you know
1:13:19
or maybe over over some uh you know
1:13:20
or maybe over over some uh you know network protocol
1:13:21
network protocol
1:13:21
network protocol then you can actually take those and
1:13:22
then you can actually take those and
1:13:22
then you can actually take those and then also move that onto the cosmos db
1:13:24
then also move that onto the cosmos db
1:13:24
then also move that onto the cosmos db side so these are sort of what i call
1:13:25
side so these are sort of what i call
1:13:25
side so these are sort of what i call the use cases uh if you will
1:13:27
the use cases uh if you will
1:13:27
the use cases uh if you will which are um which are meant to be
1:13:29
which are um which are meant to be
1:13:29
which are um which are meant to be addressed of course we
1:13:31
addressed of course we
1:13:31
addressed of course we are focusing today on the migration the
1:13:33
are focusing today on the migration the
1:13:33
are focusing today on the migration the data movement which is you know very
1:13:34
data movement which is you know very
1:13:34
data movement which is you know very real time so that you can ultimately
1:13:36
real time so that you can ultimately
1:13:36
real time so that you can ultimately you know you have maybe uh let's say an
1:13:39
you know you have maybe uh let's say an
1:13:39
you know you have maybe uh let's say an oracle system running
1:13:40
oracle system running
1:13:40
oracle system running you set up your cosmos db system and
1:13:42
you set up your cosmos db system and
1:13:42
you set up your cosmos db system and then you sort of you know sync them up
1:13:44
then you sort of you know sync them up
1:13:44
then you sort of you know sync them up and ultimately you're
1:13:45
and ultimately you're
1:13:45
and ultimately you're to pull the plug and then just have a
1:13:47
to pull the plug and then just have a
1:13:47
to pull the plug and then just have a new application running on the cosmos db
1:13:48
new application running on the cosmos db
1:13:48
new application running on the cosmos db site
1:13:49
site
1:13:50
site okay um so let me just uh quickly
1:13:53
okay um so let me just uh quickly
1:13:53
okay um so let me just uh quickly get to um you know the how is this
1:13:56
get to um you know the how is this
1:13:56
get to um you know the how is this actually done
1:13:57
actually done
1:13:57
actually done so typically the objective is hey i want
1:14:00
so typically the objective is hey i want
1:14:00
so typically the objective is hey i want to migrate some existing tables or
1:14:01
to migrate some existing tables or
1:14:01
to migrate some existing tables or collections
1:14:02
collections
1:14:02
collections let's say from um you know with an
1:14:05
let's say from um you know with an
1:14:05
let's say from um you know with an intent to cut over to the cosmos dd
1:14:07
intent to cut over to the cosmos dd
1:14:07
intent to cut over to the cosmos dd database and um the idea really is that
1:14:10
database and um the idea really is that
1:14:10
database and um the idea really is that hey
1:14:10
hey
1:14:10
hey i you know i may want to just completely
1:14:12
i you know i may want to just completely
1:14:12
i you know i may want to just completely migrate or keep the two sides in sync
1:14:14
migrate or keep the two sides in sync
1:14:14
migrate or keep the two sides in sync and along the way there's these
1:14:15
and along the way there's these
1:14:15
and along the way there's these transformation um
1:14:17
transformation um
1:14:17
transformation um and let's say denomination capabilities
1:14:19
and let's say denomination capabilities
1:14:19
and let's say denomination capabilities that i want to actually introduce
1:14:20
that i want to actually introduce
1:14:20
that i want to actually introduce so typically there are two things going
1:14:22
so typically there are two things going
1:14:22
so typically there are two things going on here one of them is
1:14:24
on here one of them is
1:14:24
on here one of them is how do i do sort of like a scale
1:14:26
how do i do sort of like a scale
1:14:26
how do i do sort of like a scale instantiation
1:14:27
instantiation
1:14:27
instantiation or often known as an initial load where
1:14:29
or often known as an initial load where
1:14:29
or often known as an initial load where you might have
1:14:30
you might have
1:14:30
you might have existing data for a decade or two
1:14:33
existing data for a decade or two
1:14:33
existing data for a decade or two decades and you want to now move
1:14:34
decades and you want to now move
1:14:34
decades and you want to now move all of that data over so you kind of
1:14:36
all of that data over so you kind of
1:14:36
all of that data over so you kind of have an initial snapshot
1:14:38
have an initial snapshot
1:14:38
have an initial snapshot and and the other part aspect of this is
1:14:40
and and the other part aspect of this is
1:14:40
and and the other part aspect of this is well what happens because
1:14:42
well what happens because
1:14:42
well what happens because you can't really take down time so the
1:14:44
you can't really take down time so the
1:14:44
you can't really take down time so the data is actually changing on the on your
1:14:46
data is actually changing on the on your
1:14:46
data is actually changing on the on your source system
1:14:47
source system
1:14:47
source system so how do you how do you um accommodate
1:14:49
so how do you how do you um accommodate
1:14:49
so how do you how do you um accommodate for that data and that's
1:14:50
for that data and that's
1:14:50
for that data and that's uh is the synchronization phase right
1:14:52
uh is the synchronization phase right
1:14:52
uh is the synchronization phase right that accommodates for your ongoing
1:14:54
that accommodates for your ongoing
1:14:54
that accommodates for your ongoing changes
1:14:55
changes
1:14:56
changes so within the synchronization you know
1:14:58
so within the synchronization you know
1:14:58
so within the synchronization you know there's a couple of things that you face
1:14:59
there's a couple of things that you face
1:15:00
there's a couple of things that you face um you know as when you're developing
1:15:01
um you know as when you're developing
1:15:01
um you know as when you're developing these pipelines uh number one is how do
1:15:03
these pipelines uh number one is how do
1:15:03
these pipelines uh number one is how do i keep it real time uh because like i
1:15:05
i keep it real time uh because like i
1:15:05
i keep it real time uh because like i mentioned
1:15:06
mentioned
1:15:06
mentioned if you're gonna go take a batch based
1:15:07
if you're gonna go take a batch based
1:15:08
if you're gonna go take a batch based approach then oftentimes
1:15:10
approach then oftentimes
1:15:10
approach then oftentimes you don't want to impact the source
1:15:12
you don't want to impact the source
1:15:12
you don't want to impact the source database which is mission running on
1:15:14
database which is mission running on
1:15:14
database which is mission running on mission critical workload
1:15:15
mission critical workload
1:15:15
mission critical workload so you're going to add a lot of impact
1:15:17
so you're going to add a lot of impact
1:15:17
so you're going to add a lot of impact and overhead if you're constantly
1:15:19
and overhead if you're constantly
1:15:19
and overhead if you're constantly pulling that
1:15:20
pulling that
1:15:20
pulling that uh the the system to detect for any
1:15:22
uh the the system to detect for any
1:15:22
uh the the system to detect for any delta changes so that turns out to be a
1:15:24
delta changes so that turns out to be a
1:15:24
delta changes so that turns out to be a non-starter typically
1:15:26
non-starter typically
1:15:26
non-starter typically um number two is you do want to have
1:15:28
um number two is you do want to have
1:15:28
um number two is you do want to have some sort of a high speed interface
1:15:31
some sort of a high speed interface
1:15:31
some sort of a high speed interface and that's really what you know uh often
1:15:33
and that's really what you know uh often
1:15:33
and that's really what you know uh often times people want to use a
1:15:35
times people want to use a
1:15:35
times people want to use a a an interface like ctc which is a
1:15:37
a an interface like ctc which is a
1:15:37
a an interface like ctc which is a change data capture interface
1:15:38
change data capture interface
1:15:38
change data capture interface and many of the these systems allow you
1:15:41
and many of the these systems allow you
1:15:41
and many of the these systems allow you to have an api that where you can mine
1:15:44
to have an api that where you can mine
1:15:44
to have an api that where you can mine the transaction log or the redo log
1:15:46
the transaction log or the redo log
1:15:46
the transaction log or the redo log or maybe the other or the ops log and
1:15:49
or maybe the other or the ops log and
1:15:49
or maybe the other or the ops log and then you can actually
1:15:50
then you can actually
1:15:50
then you can actually take the changes from there that turns
1:15:51
take the changes from there that turns
1:15:51
take the changes from there that turns out to be one of the most efficient ways
1:15:53
out to be one of the most efficient ways
1:15:53
out to be one of the most efficient ways in which
1:15:54
in which
1:15:54
in which um you can you could grab the event
1:15:56
um you can you could grab the event
1:15:56
um you can you could grab the event stream for
1:15:57
stream for
1:15:57
stream for you know uh applying on the on the
1:15:59
you know uh applying on the on the
1:15:59
you know uh applying on the on the target system
1:16:01
target system
1:16:01
target system um the other part is that you want to
1:16:03
um the other part is that you want to
1:16:03
um the other part is that you want to make sure that you're able to keep the
1:16:05
make sure that you're able to keep the
1:16:05
make sure that you're able to keep the two sides
1:16:06
two sides
1:16:06
two sides in sync which means that not only are
1:16:07
in sync which means that not only are
1:16:07
in sync which means that not only are you collecting the data you're also
1:16:09
you collecting the data you're also
1:16:09
you collecting the data you're also applying it
1:16:10
applying it
1:16:10
applying it using the apis on the cosmos db site and
1:16:12
using the apis on the cosmos db site and
1:16:12
using the apis on the cosmos db site and you have to be able to paralyze that
1:16:14
you have to be able to paralyze that
1:16:14
you have to be able to paralyze that you know the the the rights uh to
1:16:17
you know the the the rights uh to
1:16:17
you know the the the rights uh to different containers
1:16:18
different containers
1:16:18
different containers um so you want to make sure that you
1:16:20
um so you want to make sure that you
1:16:20
um so you want to make sure that you know you're able to keep up um
1:16:22
know you're able to keep up um
1:16:22
know you're able to keep up um you know in near real time between the
1:16:23
you know in near real time between the
1:16:23
you know in near real time between the two systems so that ultimately you can
1:16:25
two systems so that ultimately you can
1:16:25
two systems so that ultimately you can actually you know fail over
1:16:26
actually you know fail over
1:16:26
actually you know fail over or cut over to the to the to the cosmos
1:16:29
or cut over to the to the to the cosmos
1:16:29
or cut over to the to the to the cosmos db side
1:16:30
db side
1:16:30
db side okay so these are kind of some of the
1:16:31
okay so these are kind of some of the
1:16:31
okay so these are kind of some of the synchronization challenges so let me
1:16:32
synchronization challenges so let me
1:16:32
synchronization challenges so let me just
1:16:33
just
1:16:33
just talk about what is the big deal here why
1:16:35
talk about what is the big deal here why
1:16:35
talk about what is the big deal here why you know what is the complexity here
1:16:37
you know what is the complexity here
1:16:37
you know what is the complexity here so oftentimes right people are not going
1:16:39
so oftentimes right people are not going
1:16:39
so oftentimes right people are not going to allow you to put triggers on the
1:16:40
to allow you to put triggers on the
1:16:40
to allow you to put triggers on the source system because that turns out to
1:16:42
source system because that turns out to
1:16:42
source system because that turns out to be just super expensive
1:16:43
be just super expensive
1:16:43
be just super expensive uh on a critical application for that's
1:16:45
uh on a critical application for that's
1:16:45
uh on a critical application for that's running on that on that on that
1:16:47
running on that on that on that
1:16:47
running on that on that on that database system so the problem with the
1:16:49
database system so the problem with the
1:16:50
database system so the problem with the log records is you're not writing
1:16:52
log records is you're not writing
1:16:52
log records is you're not writing them as you're applying records to a
1:16:53
them as you're applying records to a
1:16:54
them as you're applying records to a table typically these are very compact
1:16:56
table typically these are very compact
1:16:56
table typically these are very compact binary representations of the change
1:16:59
binary representations of the change
1:16:59
binary representations of the change vectors that you produce against data
1:17:01
vectors that you produce against data
1:17:01
vectors that you produce against data blocks
1:17:02
blocks
1:17:02
blocks on the actual tables which are
1:17:04
on the actual tables which are
1:17:04
on the actual tables which are represented and coded as redo records in
1:17:06
represented and coded as redo records in
1:17:06
represented and coded as redo records in the transaction log
1:17:07
the transaction log
1:17:07
the transaction log so you know that structure you need to
1:17:10
so you know that structure you need to
1:17:10
so you know that structure you need to interpret that and then you need to be
1:17:11
interpret that and then you need to be
1:17:11
interpret that and then you need to be able to map that
1:17:13
able to map that
1:17:13
able to map that onto you know the model on the cosmos db
1:17:15
onto you know the model on the cosmos db
1:17:15
onto you know the model on the cosmos db side and that turns out to be
1:17:17
side and that turns out to be
1:17:17
side and that turns out to be challenging
1:17:17
challenging
1:17:17
challenging you have to write a lot of code you have
1:17:18
you have to write a lot of code you have
1:17:18
you have to write a lot of code you have to understand the format and that's not
1:17:20
to understand the format and that's not
1:17:20
to understand the format and that's not that simple
1:17:21
that simple
1:17:21
that simple um oftentimes you might do things like
1:17:24
um oftentimes you might do things like
1:17:24
um oftentimes you might do things like partial updates um
1:17:25
partial updates um
1:17:25
partial updates um so if you do a partial update then what
1:17:27
so if you do a partial update then what
1:17:27
so if you do a partial update then what happens if you already have
1:17:29
happens if you already have
1:17:29
happens if you already have a complete image of that of that
1:17:32
a complete image of that of that
1:17:32
a complete image of that of that document in cosmos db and now someone
1:17:34
document in cosmos db and now someone
1:17:34
document in cosmos db and now someone just updated let's say
1:17:35
just updated let's say
1:17:35
just updated let's say on the source system just two of the
1:17:37
on the source system just two of the
1:17:37
on the source system just two of the columns out of the 100 columns
1:17:39
columns out of the 100 columns
1:17:39
columns out of the 100 columns well then then the document that i
1:17:41
well then then the document that i
1:17:41
well then then the document that i construct might be a partial document so
1:17:43
construct might be a partial document so
1:17:43
construct might be a partial document so how do i deal with things like that
1:17:45
how do i deal with things like that
1:17:45
how do i deal with things like that right so these are the things that that
1:17:47
right so these are the things that that
1:17:47
right so these are the things that that um you know you need to pay attention to
1:17:49
um you know you need to pay attention to
1:17:49
um you know you need to pay attention to uh if there are failures along the way
1:17:51
uh if there are failures along the way
1:17:51
uh if there are failures along the way then what are the event guarantees in
1:17:53
then what are the event guarantees in
1:17:53
then what are the event guarantees in the change data stream
1:17:54
the change data stream
1:17:54
the change data stream to am i you know able to pick up things
1:17:57
to am i you know able to pick up things
1:17:57
to am i you know able to pick up things from where i left off or do i have to
1:17:58
from where i left off or do i have to
1:17:58
from where i left off or do i have to worry about writing the logic
1:18:00
worry about writing the logic
1:18:00
worry about writing the logic to figure out where things are so these
1:18:02
to figure out where things are so these
1:18:02
to figure out where things are so these are again some of the complexities that
1:18:04
are again some of the complexities that
1:18:04
are again some of the complexities that come in
1:18:04
come in
1:18:04
come in as you're trying to handle the the the
1:18:07
as you're trying to handle the the the
1:18:07
as you're trying to handle the the the the
1:18:07
the
1:18:08
the change data records one also very
1:18:11
change data records one also very
1:18:11
change data records one also very interesting thing is
1:18:11
interesting thing is
1:18:12
interesting thing is that the many of the underlying
1:18:14
that the many of the underlying
1:18:14
that the many of the underlying databases may not give you the changes
1:18:16
databases may not give you the changes
1:18:16
databases may not give you the changes in a transaction commit order so they
1:18:18
in a transaction commit order so they
1:18:18
in a transaction commit order so they might actually be writing the records in
1:18:20
might actually be writing the records in
1:18:20
might actually be writing the records in an interleaved fashion
1:18:21
an interleaved fashion
1:18:21
an interleaved fashion in the redo log so um and then
1:18:24
in the redo log so um and then
1:18:24
in the redo log so um and then ultimately the transaction possibly
1:18:26
ultimately the transaction possibly
1:18:26
ultimately the transaction possibly could even roll back
1:18:27
could even roll back
1:18:27
could even roll back so if you're mining the changes in real
1:18:30
so if you're mining the changes in real
1:18:30
so if you're mining the changes in real time
1:18:30
time
1:18:30
time then you also need to assemble them so
1:18:32
then you also need to assemble them so
1:18:32
then you also need to assemble them so that you may have like a complete
1:18:34
that you may have like a complete
1:18:34
that you may have like a complete transaction
1:18:34
transaction
1:18:34
transaction and then you need to map this
1:18:36
and then you need to map this
1:18:36
and then you need to map this transaction for example if it goes
1:18:37
transaction for example if it goes
1:18:38
transaction for example if it goes across an
1:18:39
across an
1:18:39
across an orders table you know and then also like
1:18:41
orders table you know and then also like
1:18:41
orders table you know and then also like a line item table
1:18:42
a line item table
1:18:42
a line item table then you want to have that be
1:18:44
then you want to have that be
1:18:44
then you want to have that be represented in maybe one document so
1:18:46
represented in maybe one document so
1:18:46
represented in maybe one document so then you need to strip out
1:18:47
then you need to strip out
1:18:47
then you need to strip out portions of the values from those
1:18:50
portions of the values from those
1:18:50
portions of the values from those records
1:18:51
records
1:18:51
records you know and then create a document and
1:18:52
you know and then create a document and
1:18:52
you know and then create a document and then move them forward okay so these are
1:18:54
then move them forward okay so these are
1:18:54
then move them forward okay so these are sort of you know some of the interesting
1:18:55
sort of you know some of the interesting
1:18:55
sort of you know some of the interesting things that come up
1:18:56
things that come up
1:18:56
things that come up as you're dealing with uh with the cdc
1:18:59
as you're dealing with uh with the cdc
1:18:59
as you're dealing with uh with the cdc uh in your pipelines
1:19:01
uh in your pipelines
1:19:01
uh in your pipelines so in stream how do you do this thing uh
1:19:03
so in stream how do you do this thing uh
1:19:03
so in stream how do you do this thing uh so my next few slides are gonna talk
1:19:05
so my next few slides are gonna talk
1:19:05
so my next few slides are gonna talk about that
1:19:06
about that
1:19:06
about that so um you know what we've done is we've
1:19:09
so um you know what we've done is we've
1:19:09
so um you know what we've done is we've sort of you know
1:19:10
sort of you know
1:19:10
sort of you know given a visual environment where there
1:19:12
given a visual environment where there
1:19:12
given a visual environment where there are pre-built wizards uh so you can see
1:19:14
are pre-built wizards uh so you can see
1:19:14
are pre-built wizards uh so you can see that you know there's wizards from
1:19:15
that you know there's wizards from
1:19:15
that you know there's wizards from mongodb to
1:19:16
mongodb to
1:19:16
mongodb to cosmos tv and oracle cdc to cosmos db
1:19:19
cosmos tv and oracle cdc to cosmos db
1:19:20
cosmos tv and oracle cdc to cosmos db microsoft sql server and there's really
1:19:21
microsoft sql server and there's really
1:19:21
microsoft sql server and there's really like you know
1:19:22
like you know
1:19:22
like you know a lot of the popular sources that you
1:19:24
a lot of the popular sources that you
1:19:24
a lot of the popular sources that you want to develop uh
1:19:25
want to develop uh
1:19:25
want to develop uh for so the idea is that you're starting
1:19:27
for so the idea is that you're starting
1:19:27
for so the idea is that you're starting off uh
1:19:28
off uh
1:19:28
off uh with saying here's my source and then
1:19:30
with saying here's my source and then
1:19:30
with saying here's my source and then you're able to write
1:19:31
you're able to write
1:19:31
you're able to write you know as developers uh some either
1:19:34
you know as developers uh some either
1:19:34
you know as developers uh some either you can you know take our pre-built
1:19:35
you can you know take our pre-built
1:19:35
you can you know take our pre-built functions and apply those to the
1:19:37
functions and apply those to the
1:19:37
functions and apply those to the pipeline or you're welcome to write your
1:19:38
pipeline or you're welcome to write your
1:19:38
pipeline or you're welcome to write your own
1:19:39
own
1:19:39
own in sql or in java um if let's say i
1:19:42
in sql or in java um if let's say i
1:19:42
in sql or in java um if let's say i mentioned like i mentioned
1:19:43
mentioned like i mentioned
1:19:43
mentioned like i mentioned if the if the out of the box you know
1:19:45
if the if the out of the box you know
1:19:46
if the if the out of the box you know function is not satisfactory and you
1:19:47
function is not satisfactory and you
1:19:47
function is not satisfactory and you really want to invent your own
1:19:49
really want to invent your own
1:19:49
really want to invent your own you know interesting uh algorithm to
1:19:52
you know interesting uh algorithm to
1:19:52
you know interesting uh algorithm to maybe massage the data often time that
1:19:55
maybe massage the data often time that
1:19:55
maybe massage the data often time that maybe that may be required
1:19:56
maybe that may be required
1:19:56
maybe that may be required particularly if you are denormalizing
1:19:58
particularly if you are denormalizing
1:19:58
particularly if you are denormalizing the data across multiple different
1:19:59
the data across multiple different
1:19:59
the data across multiple different sources
1:20:00
sources
1:20:00
sources because it might require custom logic um
1:20:03
because it might require custom logic um
1:20:03
because it might require custom logic um so
1:20:04
so
1:20:04
so you pick one of these wizards um and the
1:20:06
you pick one of these wizards um and the
1:20:06
you pick one of these wizards um and the way you know this thing
1:20:08
way you know this thing
1:20:08
way you know this thing happens is in is in three high-level
1:20:10
happens is in is in three high-level
1:20:10
happens is in is in three high-level steps um
1:20:11
steps um
1:20:11
steps um and i'm gonna show you the demo as well
1:20:12
and i'm gonna show you the demo as well
1:20:12
and i'm gonna show you the demo as well so you select the source as out of the
1:20:14
so you select the source as out of the
1:20:14
so you select the source as out of the box you can actually select a number of
1:20:16
box you can actually select a number of
1:20:16
box you can actually select a number of different
1:20:17
different
1:20:17
different sources like you know oracle or mongodb
1:20:20
sources like you know oracle or mongodb
1:20:20
sources like you know oracle or mongodb or
1:20:20
or
1:20:20
or mysql or postgres or kafka etc so in
1:20:23
mysql or postgres or kafka etc so in
1:20:23
mysql or postgres or kafka etc so in this case i've selected
1:20:24
this case i've selected
1:20:24
this case i've selected maybe the the the oracle reader which is
1:20:27
maybe the the the oracle reader which is
1:20:27
maybe the the the oracle reader which is getting my inventory data
1:20:29
getting my inventory data
1:20:29
getting my inventory data and then it does a there's a designer
1:20:31
and then it does a there's a designer
1:20:31
and then it does a there's a designer palette which allows you to
1:20:33
palette which allows you to
1:20:33
palette which allows you to say hey am i trying to transform an
1:20:35
say hey am i trying to transform an
1:20:35
say hey am i trying to transform an event that's a normal
1:20:36
event that's a normal
1:20:36
event that's a normal event coming from a file or is it coming
1:20:38
event coming from a file or is it coming
1:20:38
event coming from a file or is it coming from a database so the database event
1:20:41
from a database so the database event
1:20:41
from a database so the database event allows you to deal with the cdc record
1:20:43
allows you to deal with the cdc record
1:20:43
allows you to deal with the cdc record that i mentioned earlier
1:20:45
that i mentioned earlier
1:20:45
that i mentioned earlier because remember that is not specific to
1:20:47
because remember that is not specific to
1:20:47
because remember that is not specific to a single table
1:20:48
a single table
1:20:48
a single table so its schema doesn't conform to you
1:20:50
so its schema doesn't conform to you
1:20:50
so its schema doesn't conform to you know just
1:20:51
know just
1:20:51
know just one one type you might have changes to
1:20:54
one one type you might have changes to
1:20:54
one one type you might have changes to you know
1:20:55
you know
1:20:55
you know hundreds of tables but they're all being
1:20:56
hundreds of tables but they're all being
1:20:56
hundreds of tables but they're all being represented as one logical change record
1:20:59
represented as one logical change record
1:20:59
represented as one logical change record so we have operators that allow you to
1:21:01
so we have operators that allow you to
1:21:02
so we have operators that allow you to you know
1:21:02
you know
1:21:02
you know massage that data in that logical change
1:21:04
massage that data in that logical change
1:21:04
massage that data in that logical change record based on these database event
1:21:06
record based on these database event
1:21:06
record based on these database event transformers
1:21:07
transformers
1:21:07
transformers so you can actually add your own custom
1:21:08
so you can actually add your own custom
1:21:08
so you can actually add your own custom fields you can modify the data
1:21:11
fields you can modify the data
1:21:11
fields you can modify the data you can add your own metadata if you
1:21:12
you can add your own metadata if you
1:21:12
you can add your own metadata if you wanted to often times because
1:21:14
wanted to often times because
1:21:14
wanted to often times because uh sometimes we've seen cases where
1:21:16
uh sometimes we've seen cases where
1:21:16
uh sometimes we've seen cases where there may not be a good partition key
1:21:18
there may not be a good partition key
1:21:18
there may not be a good partition key so you want us to add you know your own
1:21:22
so you want us to add you know your own
1:21:22
so you want us to add you know your own uh you know key along with the data
1:21:24
uh you know key along with the data
1:21:24
uh you know key along with the data record so that you know on the cosmos db
1:21:26
record so that you know on the cosmos db
1:21:26
record so that you know on the cosmos db side we're able to
1:21:27
side we're able to
1:21:27
side we're able to you know be be super efficient and and
1:21:29
you know be be super efficient and and
1:21:29
you know be be super efficient and and uh get to the right partition
1:21:31
uh get to the right partition
1:21:31
uh get to the right partition and identify the the subsequent record
1:21:34
and identify the the subsequent record
1:21:34
and identify the the subsequent record okay and step
1:21:34
okay and step
1:21:34
okay and step three is there's a writer piece of it so
1:21:37
three is there's a writer piece of it so
1:21:37
three is there's a writer piece of it so far there's a there's a reader
1:21:38
far there's a there's a reader
1:21:38
far there's a there's a reader there's like this transformation
1:21:39
there's like this transformation
1:21:39
there's like this transformation capability and this is a very linear
1:21:41
capability and this is a very linear
1:21:41
capability and this is a very linear simple
1:21:42
simple
1:21:42
simple graph you could actually the middle tier
1:21:43
graph you could actually the middle tier
1:21:43
graph you could actually the middle tier here could have like 10 different
1:21:45
here could have like 10 different
1:21:45
here could have like 10 different components and you could have a
1:21:46
components and you could have a
1:21:46
components and you could have a you know kind of a fan in model back
1:21:47
you know kind of a fan in model back
1:21:48
you know kind of a fan in model back into a data stream which is then talking
1:21:50
into a data stream which is then talking
1:21:50
into a data stream which is then talking to
1:21:50
to
1:21:50
to you know the cosmos db writer which is
1:21:52
you know the cosmos db writer which is
1:21:52
you know the cosmos db writer which is another component within the stream
1:21:54
another component within the stream
1:21:54
another component within the stream platform and that's the one that
1:21:56
platform and that's the one that
1:21:56
platform and that's the one that actually leverages all of the apis
1:21:58
actually leverages all of the apis
1:21:58
actually leverages all of the apis to then finally deliver the data onto
1:22:00
to then finally deliver the data onto
1:22:00
to then finally deliver the data onto the cosmos db side
1:22:02
the cosmos db side
1:22:02
the cosmos db side okay so um so with this so this is sort
1:22:05
okay so um so with this so this is sort
1:22:05
okay so um so with this so this is sort of like how you set up the
1:22:06
of like how you set up the
1:22:06
of like how you set up the the the configuration of the pipeline
1:22:08
the the configuration of the pipeline
1:22:08
the the configuration of the pipeline itself
1:22:10
itself
1:22:10
itself and there's like a lot of you know
1:22:11
and there's like a lot of you know
1:22:11
and there's like a lot of you know useful functions there that you might
1:22:13
useful functions there that you might
1:22:13
useful functions there that you might want for example if you want to do some
1:22:14
want for example if you want to do some
1:22:14
want for example if you want to do some aggregations or if you want to do
1:22:16
aggregations or if you want to do
1:22:16
aggregations or if you want to do split records or mask records you can do
1:22:18
split records or mask records you can do
1:22:18
split records or mask records you can do stuff like that
1:22:20
stuff like that
1:22:20
stuff like that we do support all of the popular apis
1:22:23
we do support all of the popular apis
1:22:23
we do support all of the popular apis that
1:22:24
that
1:22:24
that kirill talked about earlier um so if
1:22:26
kirill talked about earlier um so if
1:22:26
kirill talked about earlier um so if you're familiar with the sql
1:22:27
you're familiar with the sql
1:22:28
you're familiar with the sql core sql api or the api cassandra
1:22:30
core sql api or the api cassandra
1:22:30
core sql api or the api cassandra api we support all those apis
1:22:32
api we support all those apis
1:22:32
api we support all those apis these are within the cosmos db
1:22:35
these are within the cosmos db
1:22:36
these are within the cosmos db writer okay uh for for stream so the so
1:22:39
writer okay uh for for stream so the so
1:22:39
writer okay uh for for stream so the so the incoming data format this is an
1:22:40
the incoming data format this is an
1:22:40
the incoming data format this is an interesting point i think i
1:22:42
interesting point i think i
1:22:42
interesting point i think i uh talked about this briefly so the
1:22:44
uh talked about this briefly so the
1:22:44
uh talked about this briefly so the incoming type could just be a
1:22:45
incoming type could just be a
1:22:45
incoming type could just be a straightforward record that's coming
1:22:46
straightforward record that's coming
1:22:46
straightforward record that's coming from a table
1:22:47
from a table
1:22:47
from a table it could be coming in from the actual
1:22:50
it could be coming in from the actual
1:22:50
it could be coming in from the actual transaction log
1:22:51
transaction log
1:22:51
transaction log so in this case you can see that you
1:22:53
so in this case you can see that you
1:22:53
so in this case you can see that you know not only do you have the data but
1:22:54
know not only do you have the data but
1:22:54
know not only do you have the data but you also have the metadata things like
1:22:56
you also have the metadata things like
1:22:56
you also have the metadata things like the table names and
1:22:57
the table names and
1:22:57
the table names and the the the redo byte address where the
1:23:00
the the the redo byte address where the
1:23:00
the the the redo byte address where the record was generated and the system
1:23:01
record was generated and the system
1:23:01
record was generated and the system commit number
1:23:02
commit number
1:23:02
commit number all those uh things that the the
1:23:04
all those uh things that the the
1:23:04
all those uh things that the the transaction lock keeps track of
1:23:06
transaction lock keeps track of
1:23:06
transaction lock keeps track of if the data is coming in from mongodb
1:23:08
if the data is coming in from mongodb
1:23:08
if the data is coming in from mongodb then again you could have
1:23:09
then again you could have
1:23:09
then again you could have um just the the the document itself
1:23:12
um just the the the document itself
1:23:12
um just the the the document itself um you know which is coming from the
1:23:14
um you know which is coming from the
1:23:14
um you know which is coming from the from the collection or you could have
1:23:16
from the collection or you could have
1:23:16
from the collection or you could have the ops log record which again has the
1:23:18
the ops log record which again has the
1:23:18
the ops log record which again has the data and the metadata and these are just
1:23:20
data and the metadata and these are just
1:23:20
data and the metadata and these are just for your convenience because if
1:23:21
for your convenience because if
1:23:21
for your convenience because if sometimes you may want to add some
1:23:23
sometimes you may want to add some
1:23:23
sometimes you may want to add some metadata
1:23:23
metadata
1:23:24
metadata as part of your collection you know
1:23:25
as part of your collection you know
1:23:25
as part of your collection you know before you apply to the
1:23:27
before you apply to the
1:23:27
before you apply to the to the container on cosmos and finally
1:23:29
to the container on cosmos and finally
1:23:29
to the container on cosmos and finally it could be just coming in as
1:23:31
it could be just coming in as
1:23:31
it could be just coming in as straight through json documents uh from
1:23:33
straight through json documents uh from
1:23:33
straight through json documents uh from from a file or from kafka
1:23:35
from a file or from kafka
1:23:35
from a file or from kafka okay so um i think we have about uh
1:23:38
okay so um i think we have about uh
1:23:38
okay so um i think we have about uh seven more minutes so i'm going to speed
1:23:39
seven more minutes so i'm going to speed
1:23:39
seven more minutes so i'm going to speed up
1:23:40
up
1:23:40
up this in my last couple of slides before
1:23:41
this in my last couple of slides before
1:23:41
this in my last couple of slides before i get to the demo um so the so the idea
1:23:44
i get to the demo um so the so the idea
1:23:44
i get to the demo um so the so the idea in cosmos db writer which is the main
1:23:45
in cosmos db writer which is the main
1:23:45
in cosmos db writer which is the main component which is
1:23:47
component which is
1:23:47
component which is leveraging the apis are we received the
1:23:49
leveraging the apis are we received the
1:23:49
leveraging the apis are we received the data from the data sources
1:23:50
data from the data sources
1:23:50
data from the data sources that's coming in as an event stream you
1:23:53
that's coming in as an event stream you
1:23:53
that's coming in as an event stream you can join that with a
1:23:54
can join that with a
1:23:54
can join that with a data preloaded from a cache in stream
1:23:57
data preloaded from a cache in stream
1:23:57
data preloaded from a cache in stream there's a data structure like that
1:23:58
there's a data structure like that
1:23:58
there's a data structure like that or we can even do lookups to an external
1:24:00
or we can even do lookups to an external
1:24:00
or we can even do lookups to an external system let's say over
1:24:02
system let's say over
1:24:02
system let's say over an api like jdbc grab that and then you
1:24:05
an api like jdbc grab that and then you
1:24:05
an api like jdbc grab that and then you can actually go ahead and
1:24:06
can actually go ahead and
1:24:06
can actually go ahead and and apply that supporting
1:24:10
and apply that supporting
1:24:10
and apply that supporting and writing different tables to
1:24:12
and writing different tables to
1:24:12
and writing different tables to different uh
1:24:13
different uh
1:24:13
different uh containers is supported um you can
1:24:15
containers is supported um you can
1:24:16
containers is supported um you can replay inserts updates deletes
1:24:17
replay inserts updates deletes
1:24:18
replay inserts updates deletes um that are coming in from the source
1:24:19
um that are coming in from the source
1:24:19
um that are coming in from the source side um
1:24:21
side um
1:24:21
side um and mapping is just supported pretty
1:24:23
and mapping is just supported pretty
1:24:23
and mapping is just supported pretty much out of the box
1:24:25
much out of the box
1:24:25
much out of the box uh there is advanced capability so if
1:24:26
uh there is advanced capability so if
1:24:26
uh there is advanced capability so if you wanted to map specific columns
1:24:29
you wanted to map specific columns
1:24:29
you wanted to map specific columns to specific fields then you're able to
1:24:31
to specific fields then you're able to
1:24:31
to specific fields then you're able to do that as well
1:24:32
do that as well
1:24:32
do that as well okay um finally just from a scale point
1:24:35
okay um finally just from a scale point
1:24:35
okay um finally just from a scale point of view stream can run as a cluster
1:24:37
of view stream can run as a cluster
1:24:37
of view stream can run as a cluster so it can scale up with similar to
1:24:39
so it can scale up with similar to
1:24:39
so it can scale up with similar to cosmos db
1:24:40
cosmos db
1:24:40
cosmos db um so literally we're moving
1:24:44
um so literally we're moving
1:24:44
um so literally we're moving billions of records uh over a day um
1:24:47
billions of records uh over a day um
1:24:47
billions of records uh over a day um at many of the retail uh sites uh
1:24:50
at many of the retail uh sites uh
1:24:50
at many of the retail uh sites uh worldwide
1:24:51
worldwide
1:24:51
worldwide um and we do take advantage of batch
1:24:54
um and we do take advantage of batch
1:24:54
um and we do take advantage of batch optimizations
1:24:55
optimizations
1:24:55
optimizations uh on in in all of the apis uh for the
1:24:58
uh on in in all of the apis uh for the
1:24:58
uh on in in all of the apis uh for the various models that i mentioned earlier
1:25:00
various models that i mentioned earlier
1:25:00
various models that i mentioned earlier okay so with that i'm going to give you
1:25:02
okay so with that i'm going to give you
1:25:02
okay so with that i'm going to give you a uh
1:25:03
a uh
1:25:03
a uh demo i have about six minutes so let's
1:25:06
demo i have about six minutes so let's
1:25:06
demo i have about six minutes so let's actually step into the
1:25:08
actually step into the
1:25:08
actually step into the um the oracle to cosmos db demo first
1:25:13
um the oracle to cosmos db demo first
1:25:13
um the oracle to cosmos db demo first okay
1:25:17
perfect so um this is a this is a
1:25:19
perfect so um this is a this is a
1:25:19
perfect so um this is a this is a recorded demo so i have the luxury of
1:25:21
recorded demo so i have the luxury of
1:25:21
recorded demo so i have the luxury of actually pausing and highlighting some
1:25:22
actually pausing and highlighting some
1:25:22
actually pausing and highlighting some interesting things that you might be
1:25:23
interesting things that you might be
1:25:23
interesting things that you might be interested in
1:25:24
interested in
1:25:24
interested in and of course you know um you know
1:25:26
and of course you know um you know
1:25:26
and of course you know um you know please feel free to send questions um
1:25:28
please feel free to send questions um
1:25:28
please feel free to send questions um as as you dig deeper into this thing so
1:25:31
as as you dig deeper into this thing so
1:25:31
as as you dig deeper into this thing so uh this is the so there's a web server
1:25:33
uh this is the so there's a web server
1:25:33
uh this is the so there's a web server as part of stream and we're
1:25:35
as part of stream and we're
1:25:35
as part of stream and we're just connecting through the ui um the
1:25:37
just connecting through the ui um the
1:25:37
just connecting through the ui um the apps represent the pipelines
1:25:39
apps represent the pipelines
1:25:39
apps represent the pipelines so i'm going to step into the apps and
1:25:42
so i'm going to step into the apps and
1:25:42
so i'm going to step into the apps and here you can see that there are
1:25:43
here you can see that there are
1:25:43
here you can see that there are two pre-built pipelines one of them is
1:25:45
two pre-built pipelines one of them is
1:25:46
two pre-built pipelines one of them is called aura to cosmos the other is cdc
1:25:48
called aura to cosmos the other is cdc
1:25:48
called aura to cosmos the other is cdc azure monitor app
1:25:49
azure monitor app
1:25:49
azure monitor app okay so let's step into the aura to
1:25:52
okay so let's step into the aura to
1:25:52
okay so let's step into the aura to cosmos
1:25:53
cosmos
1:25:53
cosmos app okay so once i step into the
1:25:57
app okay so once i step into the
1:25:57
app okay so once i step into the into the flow uh let me just pause this
1:25:59
into the flow uh let me just pause this
1:25:59
into the flow uh let me just pause this you're seeing
1:26:00
you're seeing
1:26:00
you're seeing um top to bottom uh a graph and this is
1:26:03
um top to bottom uh a graph and this is
1:26:03
um top to bottom uh a graph and this is how
1:26:04
how
1:26:04
how the data flows from the data source into
1:26:07
the data flows from the data source into
1:26:07
the data flows from the data source into the small component with the squiggly
1:26:09
the small component with the squiggly
1:26:09
the small component with the squiggly lines that's a stream
1:26:10
lines that's a stream
1:26:10
lines that's a stream uh so the stream is sort of where each
1:26:12
uh so the stream is sort of where each
1:26:12
uh so the stream is sort of where each one of the records comes in
1:26:13
one of the records comes in
1:26:13
one of the records comes in and then there's another component which
1:26:15
and then there's another component which
1:26:15
and then there's another component which is enrich cq
1:26:16
is enrich cq
1:26:16
is enrich cq that's a continuous query that's
1:26:18
that's a continuous query that's
1:26:18
that's a continuous query that's constantly and continuously enriching
1:26:20
constantly and continuously enriching
1:26:20
constantly and continuously enriching the
1:26:21
the
1:26:21
the the event that's coming in and then it's
1:26:23
the event that's coming in and then it's
1:26:23
the event that's coming in and then it's going to go right it
1:26:24
going to go right it
1:26:24
going to go right it to uh cause uh two ways one to azure
1:26:27
to uh cause uh two ways one to azure
1:26:27
to uh cause uh two ways one to azure blob storage and the other two
1:26:28
blob storage and the other two
1:26:28
blob storage and the other two cosmos db okay so and the way this is
1:26:31
cosmos db okay so and the way this is
1:26:31
cosmos db okay so and the way this is built is by drag
1:26:32
built is by drag
1:26:32
built is by drag you know doing a drag and drop from the
1:26:34
you know doing a drag and drop from the
1:26:34
you know doing a drag and drop from the components on the left side where we
1:26:35
components on the left side where we
1:26:36
components on the left side where we have support for lots of different
1:26:37
have support for lots of different
1:26:37
have support for lots of different sources
1:26:38
sources
1:26:38
sources uh to talk to your cosmos db target okay
1:26:40
uh to talk to your cosmos db target okay
1:26:40
uh to talk to your cosmos db target okay that's how this application this graph
1:26:42
that's how this application this graph
1:26:42
that's how this application this graph itself was generated
1:26:43
itself was generated
1:26:43
itself was generated okay here's the the operators that i was
1:26:46
okay here's the the operators that i was
1:26:46
okay here's the the operators that i was talking about earlier
1:26:47
talking about earlier
1:26:47
talking about earlier so let's step into this is the
1:26:48
so let's step into this is the
1:26:48
so let's step into this is the configuration panel on the right side
1:26:51
configuration panel on the right side
1:26:51
configuration panel on the right side here's where you can actually say hey
1:26:52
here's where you can actually say hey
1:26:52
here's where you can actually say hey what are the tables that i'm interested
1:26:54
what are the tables that i'm interested
1:26:54
what are the tables that i'm interested in
1:26:54
in
1:26:54
in moving live to cosmos db
1:26:58
moving live to cosmos db
1:26:58
moving live to cosmos db in this case you know i mentioned
1:26:59
in this case you know i mentioned
1:26:59
in this case you know i mentioned earlier that you can enrich your data
1:27:01
earlier that you can enrich your data
1:27:01
earlier that you can enrich your data so at the top you know if you take a
1:27:03
so at the top you know if you take a
1:27:04
so at the top you know if you take a look at
1:27:04
look at
1:27:04
look at the oracle reader that was where the
1:27:06
the oracle reader that was where the
1:27:06
the oracle reader that was where the that's the connection to the database
1:27:08
that's the connection to the database
1:27:08
that's the connection to the database the other one is this sales rep cache so
1:27:11
the other one is this sales rep cache so
1:27:11
the other one is this sales rep cache so we have some pre-loaded data
1:27:13
we have some pre-loaded data
1:27:13
we have some pre-loaded data from in this case you know just let's
1:27:15
from in this case you know just let's
1:27:15
from in this case you know just let's say a file um and you can actually join
1:27:17
say a file um and you can actually join
1:27:17
say a file um and you can actually join across these two based on the rep id the
1:27:19
across these two based on the rep id the
1:27:19
across these two based on the rep id the sales rep id because the name may not be
1:27:21
sales rep id because the name may not be
1:27:21
sales rep id because the name may not be available and we want to push the name
1:27:23
available and we want to push the name
1:27:23
available and we want to push the name into the record so this is an example of
1:27:25
into the record so this is an example of
1:27:25
into the record so this is an example of how you can actually you know
1:27:26
how you can actually you know
1:27:26
how you can actually you know denormalize some of the data
1:27:28
denormalize some of the data
1:27:28
denormalize some of the data and then um and i'll show you the query
1:27:30
and then um and i'll show you the query
1:27:30
and then um and i'll show you the query as well in just one second
1:27:32
as well in just one second
1:27:32
as well in just one second okay so this is uh currently we're
1:27:34
okay so this is uh currently we're
1:27:34
okay so this is uh currently we're looking at the the cache definition here
1:27:37
looking at the the cache definition here
1:27:38
looking at the the cache definition here okay let's take a look at the query that
1:27:40
okay let's take a look at the query that
1:27:40
okay let's take a look at the query that joined does the join here
1:27:42
joined does the join here
1:27:42
joined does the join here so you can see that you know i'm just
1:27:43
so you can see that you know i'm just
1:27:43
so you can see that you know i'm just writing a simple select statement across
1:27:45
writing a simple select statement across
1:27:45
writing a simple select statement across my
1:27:46
my
1:27:46
my oracle source as well as the sales rep
1:27:48
oracle source as well as the sales rep
1:27:48
oracle source as well as the sales rep cache and then i'm joining that on the
1:27:50
cache and then i'm joining that on the
1:27:50
cache and then i'm joining that on the under on the rep id so that we can pull
1:27:53
under on the rep id so that we can pull
1:27:53
under on the rep id so that we can pull up the rep name
1:27:58
okay
1:28:00
okay
1:28:00
okay and then let's just go forward a little
1:28:02
and then let's just go forward a little
1:28:02
and then let's just go forward a little bit on the targets i'm going to jump
1:28:04
bit on the targets i'm going to jump
1:28:04
bit on the targets i'm going to jump into the cosmos db side
1:28:05
into the cosmos db side
1:28:05
into the cosmos db side that's where you provide your access
1:28:07
that's where you provide your access
1:28:07
that's where you provide your access keys etc
1:28:09
keys etc
1:28:09
keys etc and the mapping capabilities on the
1:28:11
and the mapping capabilities on the
1:28:11
and the mapping capabilities on the cosmos db
1:28:12
cosmos db
1:28:12
cosmos db let's go ahead and deploy this
1:28:13
let's go ahead and deploy this
1:28:13
let's go ahead and deploy this application um and once i deploy it i'm
1:28:15
application um and once i deploy it i'm
1:28:15
application um and once i deploy it i'm going to start it
1:28:16
going to start it
1:28:16
going to start it the start of the application is what
1:28:17
the start of the application is what
1:28:18
the start of the application is what gets the data flowing i had mentioned
1:28:19
gets the data flowing i had mentioned
1:28:19
gets the data flowing i had mentioned earlier that you can actually preview
1:28:21
earlier that you can actually preview
1:28:21
earlier that you can actually preview the data
1:28:21
the data
1:28:21
the data so in this case um once the source
1:28:24
so in this case um once the source
1:28:24
so in this case um once the source starts speeding the data
1:28:26
starts speeding the data
1:28:26
starts speeding the data you can actually start previewing and
1:28:27
you can actually start previewing and
1:28:27
you can actually start previewing and you're seeing the actual
1:28:29
you're seeing the actual
1:28:29
you're seeing the actual um you know the the records that are
1:28:31
um you know the the records that are
1:28:31
um you know the the records that are coming in um and you can see that i
1:28:33
coming in um and you can see that i
1:28:33
coming in um and you can see that i pulled in the sales rep name as well
1:28:35
pulled in the sales rep name as well
1:28:35
pulled in the sales rep name as well from the cache
1:28:36
from the cache
1:28:36
from the cache so so far we're getting this data um and
1:28:38
so so far we're getting this data um and
1:28:38
so so far we're getting this data um and we're previewing it
1:28:39
we're previewing it
1:28:39
we're previewing it we're gonna then take this data you know
1:28:42
we're gonna then take this data you know
1:28:42
we're gonna then take this data you know and then
1:28:43
and then
1:28:43
and then depending on which api i'm using it's
1:28:45
depending on which api i'm using it's
1:28:45
depending on which api i'm using it's going to actually go ahead and
1:28:46
going to actually go ahead and
1:28:46
going to actually go ahead and and and format that and push that on the
1:28:48
and and format that and push that on the
1:28:48
and and format that and push that on the cosmos db side so let's take a look at
1:28:50
cosmos db side so let's take a look at
1:28:50
cosmos db side so let's take a look at that
1:28:51
that
1:28:51
that okay so here's the the um i'm logged
1:28:54
okay so here's the the um i'm logged
1:28:54
okay so here's the the um i'm logged into the into the cosmos db
1:28:56
into the into the cosmos db
1:28:56
into the into the cosmos db database and you can see that the
1:28:58
database and you can see that the
1:28:58
database and you can see that the specific
1:29:00
specific
1:29:00
specific orders and the items are available now
1:29:03
orders and the items are available now
1:29:03
orders and the items are available now on the cosmos db site
1:29:11
on the cosmos db site
1:29:11
on the cosmos db site okay and the the
1:29:14
okay and the the
1:29:14
okay and the the the the similar to this you know i
1:29:16
the the similar to this you know i
1:29:16
the the similar to this you know i showed you the oracle demo we have about
1:29:17
showed you the oracle demo we have about
1:29:18
showed you the oracle demo we have about a minute
1:29:18
a minute
1:29:18
a minute there's a lot of monitoring capability
1:29:20
there's a lot of monitoring capability
1:29:20
there's a lot of monitoring capability here um the key
1:29:21
here um the key
1:29:21
here um the key i'm going to skip past some of this
1:29:22
i'm going to skip past some of this
1:29:22
i'm going to skip past some of this stuff to get to the interesting pieces
1:29:24
stuff to get to the interesting pieces
1:29:24
stuff to get to the interesting pieces that you
1:29:25
that you
1:29:25
that you as developers might be concerned with so
1:29:26
as developers might be concerned with so
1:29:26
as developers might be concerned with so you want to often see
1:29:28
you want to often see
1:29:28
you want to often see what is the resource usage is my data um
1:29:31
what is the resource usage is my data um
1:29:31
what is the resource usage is my data um on an aggregate basis let's say i'm
1:29:34
on an aggregate basis let's say i'm
1:29:34
on an aggregate basis let's say i'm moving you know so many events per
1:29:35
moving you know so many events per
1:29:35
moving you know so many events per second
1:29:36
second
1:29:36
second you know what is the rate that at which
1:29:37
you know what is the rate that at which
1:29:38
you know what is the rate that at which i'm applying it to cosmos db
1:29:39
i'm applying it to cosmos db
1:29:39
i'm applying it to cosmos db so that you can scale into parallel
1:29:41
so that you can scale into parallel
1:29:41
so that you can scale into parallel flows a lot of the capability can
1:29:43
flows a lot of the capability can
1:29:43
flows a lot of the capability can um can so you can configure uh and scale
1:29:46
um can so you can configure uh and scale
1:29:46
um can so you can configure uh and scale the the stream platform
1:29:48
the the stream platform
1:29:48
the the stream platform along with how many containers you're
1:29:50
along with how many containers you're
1:29:50
along with how many containers you're writing to and also on the
1:29:51
writing to and also on the
1:29:52
writing to and also on the depending on the data that's available
1:29:53
depending on the data that's available
1:29:53
depending on the data that's available from your source depending on the
1:29:55
from your source depending on the
1:29:55
from your source depending on the throughput that you're
1:29:56
throughput that you're
1:29:56
throughput that you're receiving there so i'm going to stop
1:29:58
receiving there so i'm going to stop
1:29:58
receiving there so i'm going to stop there i don't think i have time to get
1:29:59
there i don't think i have time to get
1:29:59
there i don't think i have time to get into the second part of the demo
1:30:01
into the second part of the demo
1:30:01
into the second part of the demo which is the the mongodb reader it's
1:30:03
which is the the mongodb reader it's
1:30:03
which is the the mongodb reader it's very similar
1:30:04
very similar
1:30:04
very similar uh so um let me just uh
1:30:07
uh so um let me just uh
1:30:07
uh so um let me just uh give room for just maybe a quick
1:30:09
give room for just maybe a quick
1:30:09
give room for just maybe a quick question or two if we have the time
1:30:11
question or two if we have the time
1:30:11
question or two if we have the time because about 10 29 if you do have any
1:30:14
because about 10 29 if you do have any
1:30:14
because about 10 29 if you do have any other
1:30:15
other
1:30:15
other any other questions and comments you can
1:30:17
any other questions and comments you can
1:30:17
any other questions and comments you can go to our website at stream.com or you
1:30:19
go to our website at stream.com or you
1:30:20
go to our website at stream.com or you can also go to the azure marketplace and
1:30:21
can also go to the azure marketplace and
1:30:21
can also go to the azure marketplace and just
1:30:22
just
1:30:22
just take it for a spin um so both options
1:30:24
take it for a spin um so both options
1:30:24
take it for a spin um so both options are available
1:30:25
are available
1:30:25
are available um and with that i'm going to take any
1:30:27
um and with that i'm going to take any
1:30:27
um and with that i'm going to take any questions uh
1:30:28
questions uh
1:30:28
questions uh that might uh that might be there thank
1:30:31
that might uh that might be there thank
1:30:31
that might uh that might be there thank you
1:30:33
you
1:30:33
you thanks a lot and andrew actually we are
1:30:35
thanks a lot and andrew actually we are
1:30:35
thanks a lot and andrew actually we are a bit uh
1:30:36
a bit uh
1:30:36
a bit uh short on time again but we will we will
1:30:38
short on time again but we will we will
1:30:38
short on time again but we will we will take any questions we had offline and
1:30:40
take any questions we had offline and
1:30:40
take any questions we had offline and we'll get your help to answer the
1:30:42
we'll get your help to answer the
1:30:42
we'll get your help to answer the questions we had um
1:30:43
questions we had um
1:30:43
questions we had um thanks that was again great content and
1:30:45
thanks that was again great content and
1:30:45
thanks that was again great content and uh hopefully it makes it a little bit
1:30:47
uh hopefully it makes it a little bit
1:30:48
uh hopefully it makes it a little bit more obvious for our users
1:30:49
more obvious for our users
1:30:49
more obvious for our users uh to understand how to migrate data to
1:30:52
uh to understand how to migrate data to
1:30:52
uh to understand how to migrate data to cosmos db with
1:30:53
cosmos db with
1:30:53
cosmos db with we stream um so again we are right on
1:30:55
we stream um so again we are right on
1:30:55
we stream um so again we are right on time so let's
1:30:56
time so let's
1:30:56
time so let's move to our next speaker and next up we
1:30:59
move to our next speaker and next up we
1:30:59
move to our next speaker and next up we have
1:30:59
have
1:30:59
have sandeep nawate from adobe who is going
1:31:02
sandeep nawate from adobe who is going
1:31:02
sandeep nawate from adobe who is going to share his experience
1:31:03
to share his experience
1:31:03
to share his experience in creating a unified view from multiple
1:31:06
in creating a unified view from multiple
1:31:06
in creating a unified view from multiple data sources which
1:31:07
data sources which
1:31:07
data sources which i think is something that should
1:31:08
i think is something that should
1:31:08
i think is something that should resonate with many of our viewers today
1:31:10
resonate with many of our viewers today
1:31:10
resonate with many of our viewers today because more and more
1:31:12
because more and more
1:31:12
because more and more we need to combine multiple data
1:31:13
we need to combine multiple data
1:31:13
we need to combine multiple data services together and this can prove to
1:31:15
services together and this can prove to
1:31:15
services together and this can prove to be a challenge
1:31:16
be a challenge
1:31:16
be a challenge to operate on to you sandeep take it
1:31:18
to operate on to you sandeep take it
1:31:18
to operate on to you sandeep take it away
1:31:22
thank you thank you very much i want to
1:31:25
thank you thank you very much i want to
1:31:25
thank you thank you very much i want to start with
1:31:26
start with
1:31:26
start with uh my thanks to microsoft team here for
1:31:29
uh my thanks to microsoft team here for
1:31:29
uh my thanks to microsoft team here for giving me this opportunity to talk to
1:31:31
giving me this opportunity to talk to
1:31:31
giving me this opportunity to talk to all of you guys as well all the viewers
1:31:33
all of you guys as well all the viewers
1:31:33
all of you guys as well all the viewers who are tuning in to this particular
1:31:35
who are tuning in to this particular
1:31:35
who are tuning in to this particular presentation uh i'm sandeep navate i run
1:31:38
presentation uh i'm sandeep navate i run
1:31:38
presentation uh i'm sandeep navate i run engineering for adobe's experience
1:31:41
engineering for adobe's experience
1:31:41
engineering for adobe's experience platform
1:31:41
platform
1:31:42
platform i will be talking about what we're
1:31:44
i will be talking about what we're
1:31:44
i will be talking about what we're calling unified profile and identity
1:31:46
calling unified profile and identity
1:31:46
calling unified profile and identity architecture for
1:31:47
architecture for
1:31:47
architecture for adobe's experience platform there's a
1:31:49
adobe's experience platform there's a
1:31:49
adobe's experience platform there's a lot of work we have done over the last
1:31:51
lot of work we have done over the last
1:31:51
lot of work we have done over the last three four years
1:31:52
three four years
1:31:52
three four years with a fairly large scale system that
1:31:54
with a fairly large scale system that
1:31:54
with a fairly large scale system that has been built
1:31:55
has been built
1:31:55
has been built and it's heavily using right now cosmos
1:31:58
and it's heavily using right now cosmos
1:31:58
and it's heavily using right now cosmos db
1:31:59
db
1:31:59
db as well as many other azure technologies
1:32:01
as well as many other azure technologies
1:32:01
as well as many other azure technologies so i'll be actually walking you through
1:32:03
so i'll be actually walking you through
1:32:03
so i'll be actually walking you through that
1:32:03
that
1:32:03
that the agenda today i will actually firstly
1:32:05
the agenda today i will actually firstly
1:32:05
the agenda today i will actually firstly introduce you to adobe's experience
1:32:07
introduce you to adobe's experience
1:32:07
introduce you to adobe's experience cloud
1:32:08
cloud
1:32:08
cloud position of experience platform within
1:32:11
position of experience platform within
1:32:11
position of experience platform within that and the next layer down below that
1:32:13
that and the next layer down below that
1:32:13
that and the next layer down below that is where the profile and identity
1:32:15
is where the profile and identity
1:32:15
is where the profile and identity structures actually emerge
1:32:17
structures actually emerge
1:32:17
structures actually emerge and the importance of these structures
1:32:18
and the importance of these structures
1:32:18
and the importance of these structures in the customer era
1:32:20
in the customer era
1:32:20
in the customer era as well actually getting into the
1:32:23
as well actually getting into the
1:32:23
as well actually getting into the architecture some of the issues that we
1:32:25
architecture some of the issues that we
1:32:25
architecture some of the issues that we actually saw that we're trying to solve
1:32:26
actually saw that we're trying to solve
1:32:26
actually saw that we're trying to solve talk about the scale that we are
1:32:28
talk about the scale that we are
1:32:28
talk about the scale that we are achieving with the underlying
1:32:29
achieving with the underlying
1:32:29
achieving with the underlying technologies that we're using
1:32:31
technologies that we're using
1:32:31
technologies that we're using okay so uh let's let's look at the
1:32:34
okay so uh let's let's look at the
1:32:34
okay so uh let's let's look at the adobe's
1:32:35
adobe's
1:32:35
adobe's experience cloud okay as as a whole
1:32:38
experience cloud okay as as a whole
1:32:38
experience cloud okay as as a whole experience cloud applications are known
1:32:40
experience cloud applications are known
1:32:40
experience cloud applications are known in terms of let's say analytics club
1:32:42
in terms of let's say analytics club
1:32:42
in terms of let's say analytics club at the top most layer here which is
1:32:44
at the top most layer here which is
1:32:44
at the top most layer here which is analytics audience manager which is a
1:32:45
analytics audience manager which is a
1:32:45
analytics audience manager which is a dmp solution
1:32:47
dmp solution
1:32:47
dmp solution your marketing cloud which has solutions
1:32:48
your marketing cloud which has solutions
1:32:48
your marketing cloud which has solutions such as campaign which is email
1:32:50
such as campaign which is email
1:32:50
such as campaign which is email marketing solution
1:32:51
marketing solution
1:32:51
marketing solution market or b2b related marketing solution
1:32:55
market or b2b related marketing solution
1:32:55
market or b2b related marketing solution engage target experience manager
1:32:58
engage target experience manager
1:32:58
engage target experience manager you have commerce solutions which are
1:32:59
you have commerce solutions which are
1:32:59
you have commerce solutions which are magento all these solutions are
1:33:01
magento all these solutions are
1:33:01
magento all these solutions are essentially known as an application
1:33:03
essentially known as an application
1:33:03
essentially known as an application layer
1:33:04
layer
1:33:04
layer right below this you see the the entire
1:33:06
right below this you see the the entire
1:33:06
right below this you see the the entire layer which is the application services
1:33:08
layer which is the application services
1:33:08
layer which is the application services and the platform layer
1:33:10
and the platform layer
1:33:10
and the platform layer adobe's experience platform is something
1:33:12
adobe's experience platform is something
1:33:12
adobe's experience platform is something that was concealed over the last four
1:33:14
that was concealed over the last four
1:33:14
that was concealed over the last four years and we've been
1:33:15
years and we've been
1:33:15
years and we've been working on it for last four years today
1:33:17
working on it for last four years today
1:33:17
working on it for last four years today it's large-scale
1:33:19
it's large-scale
1:33:19
it's large-scale platform available as a product and
1:33:21
platform available as a product and
1:33:21
platform available as a product and selling in the marketplace
1:33:24
selling in the marketplace
1:33:24
selling in the marketplace it consists of some important components
1:33:26
it consists of some important components
1:33:26
it consists of some important components such as real-time customer profile
1:33:28
such as real-time customer profile
1:33:28
such as real-time customer profile this is going to be our focus area of
1:33:30
this is going to be our focus area of
1:33:30
this is going to be our focus area of discussion today uh there's a lot of ai
1:33:33
discussion today uh there's a lot of ai
1:33:33
discussion today uh there's a lot of ai machine learning techniques that are
1:33:34
machine learning techniques that are
1:33:34
machine learning techniques that are actually getting used this entire
1:33:36
actually getting used this entire
1:33:36
actually getting used this entire platform is built with the thinking of
1:33:37
platform is built with the thinking of
1:33:37
platform is built with the thinking of an open ecosystem
1:33:39
an open ecosystem
1:33:39
an open ecosystem extensible data model as well as
1:33:41
extensible data model as well as
1:33:41
extensible data model as well as complete micro services based
1:33:42
complete micro services based
1:33:42
complete micro services based architecture
1:33:44
architecture
1:33:44
architecture now the date the entire system has been
1:33:47
now the date the entire system has been
1:33:47
now the date the entire system has been built
1:33:47
built
1:33:47
built on top of cloud their application
1:33:49
on top of cloud their application
1:33:49
on top of cloud their application services build and this entire stack
1:33:51
services build and this entire stack
1:33:51
services build and this entire stack together
1:33:52
together
1:33:52
together is becoming the new generation of
1:33:54
is becoming the new generation of
1:33:54
is becoming the new generation of adobe's experience cloud
1:33:56
adobe's experience cloud
1:33:56
adobe's experience cloud okay now when we started looking at
1:33:58
okay now when we started looking at
1:33:58
okay now when we started looking at building experience cloud the question
1:34:00
building experience cloud the question
1:34:00
building experience cloud the question was okay
1:34:00
was okay
1:34:00
was okay how we're looking at different
1:34:02
how we're looking at different
1:34:02
how we're looking at different challenges that
1:34:04
challenges that
1:34:04
challenges that uh marketers are actually facing so
1:34:07
uh marketers are actually facing so
1:34:07
uh marketers are actually facing so let's look at the
1:34:08
let's look at the
1:34:08
let's look at the uh marketers challenges that we're
1:34:10
uh marketers challenges that we're
1:34:10
uh marketers challenges that we're trying to kind of solve
1:34:13
trying to kind of solve
1:34:13
trying to kind of solve if you start looking at the experience
1:34:15
if you start looking at the experience
1:34:15
if you start looking at the experience era okay
1:34:16
era okay
1:34:16
era okay the problem that we're looking at for
1:34:18
the problem that we're looking at for
1:34:18
the problem that we're looking at for solving the customer experience
1:34:20
solving the customer experience
1:34:20
solving the customer experience is that in the modern world today
1:34:23
is that in the modern world today
1:34:24
is that in the modern world today every single thing gets driven out of
1:34:26
every single thing gets driven out of
1:34:26
every single thing gets driven out of how
1:34:27
how
1:34:27
how a enterprise is interacting with
1:34:29
a enterprise is interacting with
1:34:29
a enterprise is interacting with consumers
1:34:30
consumers
1:34:30
consumers their experience there actually decides
1:34:33
their experience there actually decides
1:34:33
their experience there actually decides the long-term loyalty
1:34:34
the long-term loyalty
1:34:34
the long-term loyalty and this is the long-term value that
1:34:36
and this is the long-term value that
1:34:36
and this is the long-term value that we're trying to go after
1:34:37
we're trying to go after
1:34:37
we're trying to go after now to understand actually building this
1:34:39
now to understand actually building this
1:34:39
now to understand actually building this loyalty and providing the best
1:34:40
loyalty and providing the best
1:34:40
loyalty and providing the best experience
1:34:41
experience
1:34:41
experience most important thing for a marketeer is
1:34:43
most important thing for a marketeer is
1:34:44
most important thing for a marketeer is you must understand your consumer first
1:34:46
you must understand your consumer first
1:34:46
you must understand your consumer first you must understand your customers
1:34:47
you must understand your customers
1:34:47
you must understand your customers completely okay
1:34:49
completely okay
1:34:49
completely okay this is the 360 view of your customer
1:34:51
this is the 360 view of your customer
1:34:51
this is the 360 view of your customer that you need
1:34:52
that you need
1:34:52
that you need a comprehensive view of your your
1:34:54
a comprehensive view of your your
1:34:54
a comprehensive view of your your consumers or the customers is important
1:34:56
consumers or the customers is important
1:34:56
consumers or the customers is important and it's not sufficient to simply say
1:34:58
and it's not sufficient to simply say
1:34:58
and it's not sufficient to simply say that i'm going to just take all this
1:34:59
that i'm going to just take all this
1:34:59
that i'm going to just take all this data coming from
1:35:00
data coming from
1:35:00
data coming from enterprise systems as well as
1:35:02
enterprise systems as well as
1:35:02
enterprise systems as well as interaction systems laid
1:35:04
interaction systems laid
1:35:04
interaction systems laid down in the data lake that's not
1:35:05
down in the data lake that's not
1:35:05
down in the data lake that's not sufficient so that becomes one of the
1:35:07
sufficient so that becomes one of the
1:35:07
sufficient so that becomes one of the important challenges
1:35:08
important challenges
1:35:08
important challenges next part is that as you start figuring
1:35:10
next part is that as you start figuring
1:35:10
next part is that as you start figuring out how to bring this data
1:35:12
out how to bring this data
1:35:12
out how to bring this data the new world of activation which is the
1:35:15
the new world of activation which is the
1:35:15
the new world of activation which is the word activation actually refers to how
1:35:17
word activation actually refers to how
1:35:17
word activation actually refers to how do you take this data
1:35:18
do you take this data
1:35:18
do you take this data and take actions to provide the
1:35:20
and take actions to provide the
1:35:20
and take actions to provide the experience to the consumer
1:35:22
experience to the consumer
1:35:22
experience to the consumer this activation is more or less
1:35:24
this activation is more or less
1:35:24
this activation is more or less happening real time
1:35:25
happening real time
1:35:25
happening real time today's scenarios is 100 milliseconds to
1:35:28
today's scenarios is 100 milliseconds to
1:35:28
today's scenarios is 100 milliseconds to 250 milliseconds is becoming
1:35:30
250 milliseconds is becoming
1:35:30
250 milliseconds is becoming more or less than norm in the industry
1:35:32
more or less than norm in the industry
1:35:32
more or less than norm in the industry saying we must respond in this
1:35:34
saying we must respond in this
1:35:34
saying we must respond in this particular thing and we will look at
1:35:35
particular thing and we will look at
1:35:35
particular thing and we will look at some of the use cases there
1:35:38
some of the use cases there
1:35:38
some of the use cases there now this becomes a second challenge the
1:35:40
now this becomes a second challenge the
1:35:40
now this becomes a second challenge the next one actually gets down to saying
1:35:41
next one actually gets down to saying
1:35:41
next one actually gets down to saying that it's not just about being real on
1:35:43
that it's not just about being real on
1:35:43
that it's not just about being real on time it's real time across
1:35:45
time it's real time across
1:35:45
time it's real time across every single channel that the the
1:35:48
every single channel that the the
1:35:48
every single channel that the the customer is present
1:35:49
customer is present
1:35:49
customer is present the channel here means your phone
1:35:53
the channel here means your phone
1:35:53
the channel here means your phone your tablet your laptop
1:35:56
your tablet your laptop
1:35:56
your tablet your laptop all these are digital devices a consumer
1:35:59
all these are digital devices a consumer
1:35:59
all these are digital devices a consumer or a customer could be present
1:36:00
or a customer could be present
1:36:00
or a customer could be present anywhere and you must provide experience
1:36:03
anywhere and you must provide experience
1:36:03
anywhere and you must provide experience that is actually orchestrated well which
1:36:06
that is actually orchestrated well which
1:36:06
that is actually orchestrated well which is understanding
1:36:07
is understanding
1:36:07
is understanding the user's behavior across all these
1:36:09
the user's behavior across all these
1:36:09
the user's behavior across all these channels and providing experience
1:36:11
channels and providing experience
1:36:11
channels and providing experience becomes important
1:36:12
becomes important
1:36:12
becomes important now this scale of data if i try to bring
1:36:14
now this scale of data if i try to bring
1:36:14
now this scale of data if i try to bring together i run into the next challenge
1:36:16
together i run into the next challenge
1:36:16
together i run into the next challenge which is
1:36:16
which is
1:36:16
which is hey there is there is privacy
1:36:18
hey there is there is privacy
1:36:18
hey there is there is privacy regulations that are actually there
1:36:20
regulations that are actually there
1:36:20
regulations that are actually there everywhere
1:36:21
everywhere
1:36:21
everywhere user consent becomes important thing i
1:36:23
user consent becomes important thing i
1:36:23
user consent becomes important thing i need to start thinking about the gdpr
1:36:25
need to start thinking about the gdpr
1:36:25
need to start thinking about the gdpr and related
1:36:26
and related
1:36:26
and related ccpa these kind of regulations are
1:36:28
ccpa these kind of regulations are
1:36:28
ccpa these kind of regulations are actually coming into play so you need
1:36:29
actually coming into play so you need
1:36:30
actually coming into play so you need also a governance framework around this
1:36:32
also a governance framework around this
1:36:32
also a governance framework around this so these were some of the kind of
1:36:33
so these were some of the kind of
1:36:33
so these were some of the kind of important challenges for the experience
1:36:35
important challenges for the experience
1:36:35
important challenges for the experience maker that we were looking at
1:36:36
maker that we were looking at
1:36:36
maker that we were looking at and from this we started looking at
1:36:38
and from this we started looking at
1:36:38
and from this we started looking at saying we need to build the experience
1:36:40
saying we need to build the experience
1:36:40
saying we need to build the experience platform
1:36:40
platform
1:36:40
platform so let's look at the experience platform
1:36:42
so let's look at the experience platform
1:36:42
so let's look at the experience platform architecture at a high level
1:36:44
architecture at a high level
1:36:44
architecture at a high level this is more like an architectural view
1:36:47
this is more like an architectural view
1:36:47
this is more like an architectural view okay
1:36:48
okay
1:36:48
okay from the left hand side what i have is i
1:36:51
from the left hand side what i have is i
1:36:51
from the left hand side what i have is i have data sources that are trying to
1:36:52
have data sources that are trying to
1:36:52
have data sources that are trying to bring in the data
1:36:54
bring in the data
1:36:54
bring in the data the data sources essentially are data
1:36:56
the data sources essentially are data
1:36:56
the data sources essentially are data such as interactional data interactional
1:36:58
such as interactional data interactional
1:36:58
such as interactional data interactional data is what happens on the web what
1:37:00
data is what happens on the web what
1:37:00
data is what happens on the web what happens actually in your mobile devices
1:37:02
happens actually in your mobile devices
1:37:02
happens actually in your mobile devices what happens uh let's say transactional
1:37:05
what happens uh let's say transactional
1:37:05
what happens uh let's say transactional data this could be coming from
1:37:06
data this could be coming from
1:37:06
data this could be coming from e-commerce could be coming from bacon
1:37:08
e-commerce could be coming from bacon
1:37:08
e-commerce could be coming from bacon model right
1:37:09
model right
1:37:09
model right operational data the third-party data
1:37:11
operational data the third-party data
1:37:11
operational data the third-party data set all these data
1:37:12
set all these data
1:37:12
set all these data is arriving in the system today's world
1:37:14
is arriving in the system today's world
1:37:14
is arriving in the system today's world the data is coming more or less moving
1:37:16
the data is coming more or less moving
1:37:16
the data is coming more or less moving into everything becoming streaming
1:37:18
into everything becoming streaming
1:37:18
into everything becoming streaming just in time data arrives and you're
1:37:20
just in time data arrives and you're
1:37:20
just in time data arrives and you're expected to take actions on it
1:37:22
expected to take actions on it
1:37:22
expected to take actions on it but batch data still lives this data as
1:37:25
but batch data still lives this data as
1:37:25
but batch data still lives this data as it arrives in the system
1:37:27
it arrives in the system
1:37:27
it arrives in the system the next part is building essentially a
1:37:28
the next part is building essentially a
1:37:28
the next part is building essentially a data model
1:37:30
data model
1:37:30
data model the system is built with the thinking
1:37:31
the system is built with the thinking
1:37:32
the system is built with the thinking that is an extensible data model
1:37:34
that is an extensible data model
1:37:34
that is an extensible data model more object feels like a json structure
1:37:37
more object feels like a json structure
1:37:37
more object feels like a json structure with a schema
1:37:38
with a schema
1:37:38
with a schema okay and we call it xdm or the
1:37:40
okay and we call it xdm or the
1:37:40
okay and we call it xdm or the experience data model
1:37:41
experience data model
1:37:41
experience data model this data model is flexible where any
1:37:43
this data model is flexible where any
1:37:43
this data model is flexible where any customer can adapt to their business
1:37:45
customer can adapt to their business
1:37:45
customer can adapt to their business needs
1:37:46
needs
1:37:46
needs and we actually take all this data and
1:37:48
and we actually take all this data and
1:37:48
and we actually take all this data and convert there's etl processes as well as
1:37:50
convert there's etl processes as well as
1:37:50
convert there's etl processes as well as mapping that happens in real time and
1:37:52
mapping that happens in real time and
1:37:52
mapping that happens in real time and data gets collected
1:37:53
data gets collected
1:37:53
data gets collected the data gets laid down in data lake for
1:37:56
the data gets laid down in data lake for
1:37:56
the data gets laid down in data lake for certain purposes
1:37:57
certain purposes
1:37:57
certain purposes as well starts moving into the next leg
1:37:59
as well starts moving into the next leg
1:37:59
as well starts moving into the next leg which is
1:38:00
which is
1:38:00
which is the building the real-time customer
1:38:02
the building the real-time customer
1:38:02
the building the real-time customer profile the profile building exercise
1:38:04
profile the profile building exercise
1:38:04
profile the profile building exercise has two different aspects to it one is
1:38:06
has two different aspects to it one is
1:38:06
has two different aspects to it one is the profile
1:38:07
the profile
1:38:07
the profile and identity and we separate the two
1:38:10
and identity and we separate the two
1:38:10
and identity and we separate the two things out
1:38:10
things out
1:38:10
things out and i'll get into details of why the
1:38:12
and i'll get into details of why the
1:38:12
and i'll get into details of why the separation is needed and both of these
1:38:14
separation is needed and both of these
1:38:14
separation is needed and both of these components heavily
1:38:15
components heavily
1:38:15
components heavily depend upon using cosmos db and i'm
1:38:17
depend upon using cosmos db and i'm
1:38:18
depend upon using cosmos db and i'm actually talking about that
1:38:20
actually talking about that
1:38:20
actually talking about that then the part is that what kind of
1:38:21
then the part is that what kind of
1:38:21
then the part is that what kind of workloads this system actually starts
1:38:23
workloads this system actually starts
1:38:23
workloads this system actually starts supporting which is profile lookup
1:38:24
supporting which is profile lookup
1:38:24
supporting which is profile lookup segmentation batch segmentation
1:38:26
segmentation batch segmentation
1:38:26
segmentation batch segmentation real-time segmentation
1:38:27
real-time segmentation
1:38:28
real-time segmentation there are many workloads that actually
1:38:29
there are many workloads that actually
1:38:29
there are many workloads that actually system needs to start supporting
1:38:31
system needs to start supporting
1:38:31
system needs to start supporting this data goes to the next leg where we
1:38:33
this data goes to the next leg where we
1:38:33
this data goes to the next leg where we want to start building insights
1:38:35
want to start building insights
1:38:35
want to start building insights so using the data lake as well as using
1:38:38
so using the data lake as well as using
1:38:38
so using the data lake as well as using all the
1:38:38
all the
1:38:38
all the the assets that are being laid down
1:38:40
the assets that are being laid down
1:38:40
the assets that are being laid down within cosmos db we start looking at
1:38:42
within cosmos db we start looking at
1:38:42
within cosmos db we start looking at how i can start bringing insights
1:38:44
how i can start bringing insights
1:38:44
how i can start bringing insights insights come in terms of
1:38:46
insights come in terms of
1:38:46
insights come in terms of running simply queries analytical and vr
1:38:48
running simply queries analytical and vr
1:38:48
running simply queries analytical and vr workloads running data science
1:38:50
workloads running data science
1:38:50
workloads running data science model operationalization kind of aspects
1:38:53
model operationalization kind of aspects
1:38:53
model operationalization kind of aspects all these things bring together insights
1:38:56
all these things bring together insights
1:38:56
all these things bring together insights now what i have is
1:38:57
now what i have is
1:38:57
now what i have is i have profile which is unified i have
1:38:59
i have profile which is unified i have
1:39:00
i have profile which is unified i have insights
1:39:00
insights
1:39:00
insights and now i take it to my activation
1:39:02
and now i take it to my activation
1:39:02
and now i take it to my activation points
1:39:04
points
1:39:04
points activation points is where i deliver my
1:39:05
activation points is where i deliver my
1:39:05
activation points is where i deliver my experience this is where edge computing
1:39:08
experience this is where edge computing
1:39:08
experience this is where edge computing and many of these things start coming
1:39:10
and many of these things start coming
1:39:10
and many of these things start coming into play and in the modern era
1:39:12
into play and in the modern era
1:39:12
into play and in the modern era it's not just that i'm actually just
1:39:14
it's not just that i'm actually just
1:39:14
it's not just that i'm actually just taking the data and taking actions
1:39:16
taking the data and taking actions
1:39:16
taking the data and taking actions streaming services require actually
1:39:18
streaming services require actually
1:39:18
streaming services require actually doing processing of the queries
1:39:20
doing processing of the queries
1:39:20
doing processing of the queries as events actually arrive looking up the
1:39:24
as events actually arrive looking up the
1:39:24
as events actually arrive looking up the data from the profile and acting on it
1:39:26
data from the profile and acting on it
1:39:26
data from the profile and acting on it and then delivering actually the
1:39:28
and then delivering actually the
1:39:28
and then delivering actually the experience to the customer right on the
1:39:29
experience to the customer right on the
1:39:29
experience to the customer right on the time
1:39:30
time
1:39:30
time okay let's let's get to the next
1:39:33
okay let's let's get to the next
1:39:33
okay let's let's get to the next uh level of details with respect to
1:39:36
uh level of details with respect to
1:39:36
uh level of details with respect to profile and identity
1:39:38
profile and identity
1:39:38
profile and identity and why this is actually becoming
1:39:40
and why this is actually becoming
1:39:40
and why this is actually becoming important okay for the marketing use
1:39:42
important okay for the marketing use
1:39:42
important okay for the marketing use cases as well as reporting use cases
1:39:45
cases as well as reporting use cases
1:39:45
cases as well as reporting use cases so in the marketing i think many of you
1:39:46
so in the marketing i think many of you
1:39:46
so in the marketing i think many of you guys are already aware marketing use
1:39:48
guys are already aware marketing use
1:39:48
guys are already aware marketing use cases come out in many different
1:39:50
cases come out in many different
1:39:50
cases come out in many different ways there is definitely personalization
1:39:53
ways there is definitely personalization
1:39:53
ways there is definitely personalization that is actually happening in web and
1:39:55
that is actually happening in web and
1:39:55
that is actually happening in web and mobile and in-app and all these things
1:39:57
mobile and in-app and all these things
1:39:57
mobile and in-app and all these things which is
1:39:58
which is
1:39:58
which is what i look at a personalization systems
1:40:00
what i look at a personalization systems
1:40:00
what i look at a personalization systems is that
1:40:01
is that
1:40:01
is that an incoming request it's actually
1:40:03
an incoming request it's actually
1:40:03
an incoming request it's actually segmented
1:40:04
segmented
1:40:04
segmented all the audience is segmented into a
1:40:06
all the audience is segmented into a
1:40:06
all the audience is segmented into a large set of users
1:40:08
large set of users
1:40:08
large set of users and that set sees more or less the same
1:40:10
and that set sees more or less the same
1:40:10
and that set sees more or less the same exact experience
1:40:11
exact experience
1:40:11
exact experience that is generally called a
1:40:12
that is generally called a
1:40:12
that is generally called a personalization now set size can vary
1:40:14
personalization now set size can vary
1:40:14
personalization now set size can vary but we have seen that size is getting
1:40:16
but we have seen that size is getting
1:40:16
but we have seen that size is getting into 10 million to 100 million users
1:40:18
into 10 million to 100 million users
1:40:18
into 10 million to 100 million users falling into single set which is
1:40:20
falling into single set which is
1:40:20
falling into single set which is essentially you're not doing much
1:40:21
essentially you're not doing much
1:40:21
essentially you're not doing much personalization
1:40:22
personalization
1:40:22
personalization okay as you get this uh smaller and
1:40:24
okay as you get this uh smaller and
1:40:24
okay as you get this uh smaller and smaller personalization actually goes
1:40:26
smaller personalization actually goes
1:40:26
smaller personalization actually goes higher and higher now this is happening
1:40:28
higher and higher now this is happening
1:40:28
higher and higher now this is happening today whether it is happening in the
1:40:30
today whether it is happening in the
1:40:30
today whether it is happening in the email marketing is happening
1:40:32
email marketing is happening
1:40:32
email marketing is happening in push notifications or is happening on
1:40:34
in push notifications or is happening on
1:40:34
in push notifications or is happening on the inbound channels
1:40:36
the inbound channels
1:40:36
the inbound channels interaction is kind of a new kind of
1:40:38
interaction is kind of a new kind of
1:40:38
interaction is kind of a new kind of thing if i were to say that
1:40:39
thing if i were to say that
1:40:39
thing if i were to say that personalization is actually unimodal
1:40:41
personalization is actually unimodal
1:40:41
personalization is actually unimodal it's either inbound or it is outbound
1:40:43
it's either inbound or it is outbound
1:40:43
it's either inbound or it is outbound but it's not both at the same time
1:40:45
but it's not both at the same time
1:40:45
but it's not both at the same time interaction is bimodal and we are seeing
1:40:48
interaction is bimodal and we are seeing
1:40:48
interaction is bimodal and we are seeing a whole lot of interaction that is
1:40:49
a whole lot of interaction that is
1:40:49
a whole lot of interaction that is happening and i'll tell you a few
1:40:50
happening and i'll tell you a few
1:40:50
happening and i'll tell you a few examples of what interaction actually
1:40:52
examples of what interaction actually
1:40:52
examples of what interaction actually means
1:40:53
means
1:40:54
means interaction has one more characteristics
1:40:55
interaction has one more characteristics
1:40:56
interaction has one more characteristics it is about one individual
1:40:58
it is about one individual
1:40:58
it is about one individual so it becomes personalization to the
1:41:00
so it becomes personalization to the
1:41:00
so it becomes personalization to the next level which is
1:41:01
next level which is
1:41:01
next level which is a segment size of one are fully
1:41:03
a segment size of one are fully
1:41:03
a segment size of one are fully individualized
1:41:05
individualized
1:41:05
individualized where do you see interactions my most
1:41:07
where do you see interactions my most
1:41:07
where do you see interactions my most recent experience actually walking into
1:41:09
recent experience actually walking into
1:41:09
recent experience actually walking into dme was very
1:41:11
dme was very
1:41:11
dme was very very thrilling you know that in dmv you
1:41:13
very thrilling you know that in dmv you
1:41:13
very thrilling you know that in dmv you will just completely waste your time
1:41:15
will just completely waste your time
1:41:15
will just completely waste your time just actually
1:41:15
just actually
1:41:15
just actually standing in lines this was a situation
1:41:17
standing in lines this was a situation
1:41:17
standing in lines this was a situation where everything happened
1:41:19
where everything happened
1:41:19
where everything happened completely with devices where i had i
1:41:21
completely with devices where i had i
1:41:22
completely with devices where i had i was not standing
1:41:23
was not standing
1:41:23
was not standing anywhere physically in line my actual
1:41:25
anywhere physically in line my actual
1:41:25
anywhere physically in line my actual work within dme was five minutes walk to
1:41:27
work within dme was five minutes walk to
1:41:27
work within dme was five minutes walk to the counter get the job done get out
1:41:29
the counter get the job done get out
1:41:29
the counter get the job done get out everything happened through devices and
1:41:31
everything happened through devices and
1:41:31
everything happened through devices and there was a continuous interactions
1:41:32
there was a continuous interactions
1:41:32
there was a continuous interactions from my phone into the system starting
1:41:35
from my phone into the system starting
1:41:35
from my phone into the system starting initially from my browser then from the
1:41:37
initially from my browser then from the
1:41:37
initially from my browser then from the phone and everything actually worked out
1:41:39
phone and everything actually worked out
1:41:39
phone and everything actually worked out now this is happening in government
1:41:40
now this is happening in government
1:41:40
now this is happening in government offices i've seen this in libraries
1:41:42
offices i've seen this in libraries
1:41:42
offices i've seen this in libraries library checkouts are actually changed
1:41:44
library checkouts are actually changed
1:41:44
library checkouts are actually changed you're seeing this happening in
1:41:45
you're seeing this happening in
1:41:45
you're seeing this happening in enterprises you're seeing this happening
1:41:47
enterprises you're seeing this happening
1:41:47
enterprises you're seeing this happening in retail
1:41:48
in retail
1:41:48
in retail right buying online and picking up in
1:41:50
right buying online and picking up in
1:41:50
right buying online and picking up in store is one of those interactional
1:41:51
store is one of those interactional
1:41:51
store is one of those interactional examples we are actually seeing
1:41:53
examples we are actually seeing
1:41:53
examples we are actually seeing you seeing that happening in the food
1:41:54
you seeing that happening in the food
1:41:54
you seeing that happening in the food industry you're seeing in hospitals so
1:41:56
industry you're seeing in hospitals so
1:41:56
industry you're seeing in hospitals so you see this happening pretty much
1:41:58
you see this happening pretty much
1:41:58
you see this happening pretty much everywhere and covered environment has
1:42:00
everywhere and covered environment has
1:42:00
everywhere and covered environment has pushed it into high gear
1:42:02
pushed it into high gear
1:42:02
pushed it into high gear and much of this requires almost
1:42:04
and much of this requires almost
1:42:04
and much of this requires almost instantaneous response
1:42:05
instantaneous response
1:42:05
instantaneous response you cannot wait so these are becoming
1:42:07
you cannot wait so these are becoming
1:42:07
you cannot wait so these are becoming two dominant examples the next example
1:42:09
two dominant examples the next example
1:42:09
two dominant examples the next example is really reporting which is where
1:42:10
is really reporting which is where
1:42:10
is really reporting which is where you're trying to kind of
1:42:11
you're trying to kind of
1:42:12
you're trying to kind of reporting is kind of a broader title but
1:42:13
reporting is kind of a broader title but
1:42:13
reporting is kind of a broader title but really pulling analytics together bi
1:42:16
really pulling analytics together bi
1:42:16
really pulling analytics together bi attribution these are kind of use cases
1:42:18
attribution these are kind of use cases
1:42:18
attribution these are kind of use cases which require full
1:42:19
which require full
1:42:19
which require full understanding of single consumer or
1:42:21
understanding of single consumer or
1:42:21
understanding of single consumer or customer
1:42:22
customer
1:42:22
customer and their identities now let's look at
1:42:24
and their identities now let's look at
1:42:24
and their identities now let's look at the
1:42:26
the
1:42:26
the details of the problems with respect to
1:42:28
details of the problems with respect to
1:42:28
details of the problems with respect to identities and profile
1:42:31
identities and profile
1:42:31
identities and profile so here is kind of the view of the world
1:42:33
so here is kind of the view of the world
1:42:33
so here is kind of the view of the world i'm going to kind of talk about let's
1:42:34
i'm going to kind of talk about let's
1:42:34
i'm going to kind of talk about let's consider this is my picture poorly drawn
1:42:36
consider this is my picture poorly drawn
1:42:36
consider this is my picture poorly drawn picture of sandeep okay the consumer
1:42:39
picture of sandeep okay the consumer
1:42:39
picture of sandeep okay the consumer you are essentially looking at his
1:42:41
you are essentially looking at his
1:42:41
you are essentially looking at his interactions actually
1:42:42
interactions actually
1:42:42
interactions actually uh in all different devices and
1:42:45
uh in all different devices and
1:42:45
uh in all different devices and different places
1:42:46
different places
1:42:46
different places you have essentially a phone you have a
1:42:48
you have essentially a phone you have a
1:42:48
you have essentially a phone you have a tablet you have a computer that you're
1:42:50
tablet you have a computer that you're
1:42:50
tablet you have a computer that you're interacting with there's a brick and
1:42:51
interacting with there's a brick and
1:42:51
interacting with there's a brick and mortar interaction he's talking to a
1:42:53
mortar interaction he's talking to a
1:42:53
mortar interaction he's talking to a customer services representative
1:42:55
customer services representative
1:42:55
customer services representative every single place there is an identity
1:42:57
every single place there is an identity
1:42:57
every single place there is an identity there's a primary key if you were to
1:42:59
there's a primary key if you were to
1:42:59
there's a primary key if you were to think about database people like us
1:43:01
think about database people like us
1:43:01
think about database people like us right there is a primary key that we are
1:43:03
right there is a primary key that we are
1:43:04
right there is a primary key that we are actually talking about and this primary
1:43:05
actually talking about and this primary
1:43:05
actually talking about and this primary key essentially is like idfa the cookie
1:43:07
key essentially is like idfa the cookie
1:43:08
key essentially is like idfa the cookie actually are the crm id these identities
1:43:10
actually are the crm id these identities
1:43:10
actually are the crm id these identities exist
1:43:11
exist
1:43:11
exist and each one is different okay now let's
1:43:14
and each one is different okay now let's
1:43:14
and each one is different okay now let's look at the part of saying that okay
1:43:15
look at the part of saying that okay
1:43:15
look at the part of saying that okay well how does the data actually start
1:43:17
well how does the data actually start
1:43:17
well how does the data actually start coming together each channel in this
1:43:19
coming together each channel in this
1:43:19
coming together each channel in this case has its own data bucket i call the
1:43:22
case has its own data bucket i call the
1:43:22
case has its own data bucket i call the fragment of that profile for sandeep
1:43:24
fragment of that profile for sandeep
1:43:24
fragment of that profile for sandeep okay this fragment actually exists and
1:43:26
okay this fragment actually exists and
1:43:26
okay this fragment actually exists and it has one primary key associated with
1:43:28
it has one primary key associated with
1:43:28
it has one primary key associated with it
1:43:28
it
1:43:28
it but any one fragment is a partial
1:43:30
but any one fragment is a partial
1:43:30
but any one fragment is a partial understanding of what sunday's actual
1:43:32
understanding of what sunday's actual
1:43:32
understanding of what sunday's actual profile looks like
1:43:33
profile looks like
1:43:33
profile looks like so what is needed is that i need a way
1:43:35
so what is needed is that i need a way
1:43:36
so what is needed is that i need a way to bring all this thing together into a
1:43:38
to bring all this thing together into a
1:43:38
to bring all this thing together into a single view
1:43:39
single view
1:43:39
single view okay so first part is that we're
1:43:41
okay so first part is that we're
1:43:41
okay so first part is that we're separating the picture saying that there
1:43:42
separating the picture saying that there
1:43:42
separating the picture saying that there is fragments where the data
1:43:43
is fragments where the data
1:43:43
is fragments where the data resides for this particular consumer and
1:43:46
resides for this particular consumer and
1:43:46
resides for this particular consumer and there are identities okay
1:43:47
there are identities okay
1:43:47
there are identities okay these identities are actually just there
1:43:49
these identities are actually just there
1:43:49
these identities are actually just there across different channels
1:43:51
across different channels
1:43:51
across different channels so now let's look at the next picture
1:43:52
so now let's look at the next picture
1:43:52
so now let's look at the next picture which is how do we actually bring it
1:43:54
which is how do we actually bring it
1:43:54
which is how do we actually bring it together
1:43:54
together
1:43:54
together okay so
1:43:59
once we actually start look uh okay we
1:44:02
once we actually start look uh okay we
1:44:02
once we actually start look uh okay we are not okay
1:44:05
bringing it together as you start
1:44:06
bringing it together as you start
1:44:06
bringing it together as you start thinking about it all these identities
1:44:08
thinking about it all these identities
1:44:08
thinking about it all these identities are getting related to each other
1:44:10
are getting related to each other
1:44:10
are getting related to each other okay identity relationship with each
1:44:12
okay identity relationship with each
1:44:12
okay identity relationship with each other you can think about saying
1:44:13
other you can think about saying
1:44:13
other you can think about saying like i'm forming a graph of these
1:44:15
like i'm forming a graph of these
1:44:15
like i'm forming a graph of these identities this is the identity graph
1:44:18
identities this is the identity graph
1:44:18
identities this is the identity graph this needs to be built now once you
1:44:20
this needs to be built now once you
1:44:20
this needs to be built now once you build the identity graph that actually
1:44:22
build the identity graph that actually
1:44:22
build the identity graph that actually tells the profile
1:44:23
tells the profile
1:44:23
tells the profile how to bring together fragments these
1:44:25
how to bring together fragments these
1:44:25
how to bring together fragments these are logically brought together
1:44:27
are logically brought together
1:44:27
are logically brought together they're not physically brought together
1:44:28
they're not physically brought together
1:44:28
they're not physically brought together which means that as the graph evolves
1:44:31
which means that as the graph evolves
1:44:31
which means that as the graph evolves as things change you can actually start
1:44:33
as things change you can actually start
1:44:33
as things change you can actually start bringing together profiles
1:44:35
bringing together profiles
1:44:35
bringing together profiles and now i have a one single view of this
1:44:37
and now i have a one single view of this
1:44:37
and now i have a one single view of this particular consumer sandeep's profile is
1:44:39
particular consumer sandeep's profile is
1:44:39
particular consumer sandeep's profile is actually underserved from a given use
1:44:41
actually underserved from a given use
1:44:41
actually underserved from a given use case perspective
1:44:42
case perspective
1:44:42
case perspective now if you start thinking about what is
1:44:43
now if you start thinking about what is
1:44:43
now if you start thinking about what is happening behind this because you're
1:44:45
happening behind this because you're
1:44:46
happening behind this because you're bringing these identities together every
1:44:48
bringing these identities together every
1:44:48
bringing these identities together every enterprise needs to start thinking about
1:44:50
enterprise needs to start thinking about
1:44:50
enterprise needs to start thinking about how to bring this together
1:44:51
how to bring this together
1:44:51
how to bring this together and if you want to understand your
1:44:53
and if you want to understand your
1:44:53
and if you want to understand your consumer vow which is
1:44:55
consumer vow which is
1:44:55
consumer vow which is the the imperative for the experience
1:44:57
the the imperative for the experience
1:44:57
the the imperative for the experience era then you must understand the
1:44:59
era then you must understand the
1:44:59
era then you must understand the identities
1:44:59
identities
1:45:00
identities you must bring it together this is
1:45:02
you must bring it together this is
1:45:02
you must bring it together this is almost balance sheet worthy
1:45:04
almost balance sheet worthy
1:45:04
almost balance sheet worthy actually asset that any enterprise can
1:45:06
actually asset that any enterprise can
1:45:06
actually asset that any enterprise can build and this is becoming extremely
1:45:08
build and this is becoming extremely
1:45:08
build and this is becoming extremely important and we're noticing every
1:45:10
important and we're noticing every
1:45:10
important and we're noticing every customer
1:45:10
customer
1:45:10
customer of adobe actually is looking at saying
1:45:12
of adobe actually is looking at saying
1:45:12
of adobe actually is looking at saying how does identities
1:45:14
how does identities
1:45:14
how does identities actually work in this new world now
1:45:17
actually work in this new world now
1:45:17
actually work in this new world now let's take to now now we know how the
1:45:19
let's take to now now we know how the
1:45:19
let's take to now now we know how the profile comes together let's look at the
1:45:20
profile comes together let's look at the
1:45:20
profile comes together let's look at the next level of details on the profile
1:45:22
next level of details on the profile
1:45:22
next level of details on the profile architecture
1:45:25
architecture
1:45:25
architecture so this is essentially just a
1:45:28
so this is essentially just a
1:45:28
so this is essentially just a conceptual view of now how the profile
1:45:30
conceptual view of now how the profile
1:45:30
conceptual view of now how the profile is going to start coming together
1:45:31
is going to start coming together
1:45:31
is going to start coming together from the left hand side i'm showing the
1:45:33
from the left hand side i'm showing the
1:45:33
from the left hand side i'm showing the profile fragments
1:45:34
profile fragments
1:45:34
profile fragments okay the profile fragments is what i
1:45:36
okay the profile fragments is what i
1:45:36
okay the profile fragments is what i talked about a single
1:45:38
talked about a single
1:45:38
talked about a single minded view of saying that a single
1:45:39
minded view of saying that a single
1:45:39
minded view of saying that a single identity gives you a single part of data
1:45:41
identity gives you a single part of data
1:45:41
identity gives you a single part of data so web
1:45:42
so web
1:45:42
so web will have its own container of data
1:45:44
will have its own container of data
1:45:44
will have its own container of data every interaction element collected from
1:45:46
every interaction element collected from
1:45:46
every interaction element collected from the web
1:45:46
the web
1:45:46
the web will be sitting in that container mobile
1:45:48
will be sitting in that container mobile
1:45:48
will be sitting in that container mobile will have similar same thing
1:45:50
will have similar same thing
1:45:50
will have similar same thing all the crm data will be sitting inside
1:45:51
all the crm data will be sitting inside
1:45:52
all the crm data will be sitting inside one container
1:45:53
one container
1:45:53
one container these containers essentially bring
1:45:55
these containers essentially bring
1:45:55
these containers essentially bring together the data using the
1:45:56
together the data using the
1:45:56
together the data using the identity okay an identity graph that we
1:45:59
identity okay an identity graph that we
1:45:59
identity okay an identity graph that we stitch together
1:46:00
stitch together
1:46:00
stitch together we will bring together all the data on
1:46:02
we will bring together all the data on
1:46:02
we will bring together all the data on the profile that is becomes the story of
1:46:04
the profile that is becomes the story of
1:46:04
the profile that is becomes the story of saying
1:46:05
saying
1:46:05
saying i got a unified view of the profile this
1:46:07
i got a unified view of the profile this
1:46:07
i got a unified view of the profile this actually starts falling under the
1:46:09
actually starts falling under the
1:46:09
actually starts falling under the governance
1:46:10
governance
1:46:10
governance controls and now i need to start talking
1:46:12
controls and now i need to start talking
1:46:12
controls and now i need to start talking about what use cases i'm going to start
1:46:14
about what use cases i'm going to start
1:46:14
about what use cases i'm going to start supporting on top of this
1:46:15
supporting on top of this
1:46:15
supporting on top of this okay so as we start looking at the use
1:46:18
okay so as we start looking at the use
1:46:18
okay so as we start looking at the use cases part of it
1:46:19
cases part of it
1:46:19
cases part of it simple use cases is i need to provide
1:46:22
simple use cases is i need to provide
1:46:22
simple use cases is i need to provide point lookup point lookup is needed in
1:46:23
point lookup point lookup is needed in
1:46:24
point lookup point lookup is needed in many many use cases which is starting
1:46:26
many many use cases which is starting
1:46:26
many many use cases which is starting with simple like
1:46:27
with simple like
1:46:27
with simple like customer support applications or any
1:46:28
customer support applications or any
1:46:28
customer support applications or any high touch application needs to see
1:46:30
high touch application needs to see
1:46:30
high touch application needs to see the customer's profile that's the
1:46:32
the customer's profile that's the
1:46:32
the customer's profile that's the simplest part of it even data science
1:46:35
simplest part of it even data science
1:46:35
simplest part of it even data science workspace or any data science or
1:46:36
workspace or any data science or
1:46:36
workspace or any data science or modeling exercise time to time
1:46:38
modeling exercise time to time
1:46:38
modeling exercise time to time need to see it user interfaces need to
1:46:40
need to see it user interfaces need to
1:46:40
need to see it user interfaces need to kind of look at a point lookup
1:46:41
kind of look at a point lookup
1:46:42
kind of look at a point lookup point lookup means that on the fly
1:46:44
point lookup means that on the fly
1:46:44
point lookup means that on the fly figure out what the graphs look are
1:46:45
figure out what the graphs look are
1:46:45
figure out what the graphs look are going to come together like
1:46:46
going to come together like
1:46:46
going to come together like as well figure out what the different
1:46:48
as well figure out what the different
1:46:48
as well figure out what the different fragments are and put together a view of
1:46:50
fragments are and put together a view of
1:46:50
fragments are and put together a view of a person
1:46:51
a person
1:46:51
a person at that point in time okay so it's a
1:46:53
at that point in time okay so it's a
1:46:53
at that point in time okay so it's a very complicated query although it looks
1:46:55
very complicated query although it looks
1:46:55
very complicated query although it looks like a simple point lookup
1:46:56
like a simple point lookup
1:46:56
like a simple point lookup okay next part is that bad segmentation
1:46:59
okay next part is that bad segmentation
1:46:59
okay next part is that bad segmentation that segmentation means
1:47:00
that segmentation means
1:47:00
that segmentation means is actually means running a large number
1:47:02
is actually means running a large number
1:47:02
is actually means running a large number of queries in
1:47:04
of queries in
1:47:04
of queries in a periodic basis number of actually
1:47:07
a periodic basis number of actually
1:47:07
a periodic basis number of actually queries that actually run against this
1:47:08
queries that actually run against this
1:47:08
queries that actually run against this kind of a database
1:47:09
kind of a database
1:47:09
kind of a database actually gets into for a single
1:47:11
actually gets into for a single
1:47:11
actually gets into for a single enterprise into tens of thousands
1:47:12
enterprise into tens of thousands
1:47:12
enterprise into tens of thousands of queries actually run and that is run
1:47:15
of queries actually run and that is run
1:47:15
of queries actually run and that is run in a batch model which means that you're
1:47:16
in a batch model which means that you're
1:47:16
in a batch model which means that you're running on a periodic basis
1:47:18
running on a periodic basis
1:47:18
running on a periodic basis but in the modern world they also need
1:47:20
but in the modern world they also need
1:47:20
but in the modern world they also need to be done in a streaming world
1:47:22
to be done in a streaming world
1:47:22
to be done in a streaming world okay real-time world now here where the
1:47:24
okay real-time world now here where the
1:47:24
okay real-time world now here where the application starts getting very
1:47:25
application starts getting very
1:47:26
application starts getting very stringent you have less than 100
1:47:27
stringent you have less than 100
1:47:27
stringent you have less than 100 milliseconds to run
1:47:28
milliseconds to run
1:47:28
milliseconds to run queries so for the incoming single event
1:47:31
queries so for the incoming single event
1:47:32
queries so for the incoming single event okay you have to look up this profile
1:47:34
okay you have to look up this profile
1:47:34
okay you have to look up this profile which is a complex query
1:47:36
which is a complex query
1:47:36
which is a complex query and run 10 000 plus queries against all
1:47:38
and run 10 000 plus queries against all
1:47:38
and run 10 000 plus queries against all this data set
1:47:40
this data set
1:47:40
this data set okay and actually materialize the query
1:47:43
okay and actually materialize the query
1:47:43
okay and actually materialize the query results you have about 100 milliseconds
1:47:45
results you have about 100 milliseconds
1:47:45
results you have about 100 milliseconds and this is for one event typical
1:47:47
and this is for one event typical
1:47:47
and this is for one event typical enterprise actually will get hundreds of
1:47:49
enterprise actually will get hundreds of
1:47:49
enterprise actually will get hundreds of thousands of events actually arriving
1:47:51
thousands of events actually arriving
1:47:51
thousands of events actually arriving every single second
1:47:52
every single second
1:47:52
every single second so think about the database load
1:47:54
so think about the database load
1:47:54
so think about the database load characteristics that are trying to
1:47:55
characteristics that are trying to
1:47:55
characteristics that are trying to create in this one
1:47:57
create in this one
1:47:57
create in this one you have hundreds of thousands of events
1:47:58
you have hundreds of thousands of events
1:47:58
you have hundreds of thousands of events for each event ten thousand queries
1:48:00
for each event ten thousand queries
1:48:00
for each event ten thousand queries actually need to be run
1:48:01
actually need to be run
1:48:01
actually need to be run and the multiplication of two is the
1:48:03
and the multiplication of two is the
1:48:03
and the multiplication of two is the workload in terms of query execution
1:48:05
workload in terms of query execution
1:48:05
workload in terms of query execution needs to be run on top of this
1:48:06
needs to be run on top of this
1:48:06
needs to be run on top of this particular system now these kind of
1:48:08
particular system now these kind of
1:48:08
particular system now these kind of scales when you're looking at it of
1:48:09
scales when you're looking at it of
1:48:09
scales when you're looking at it of course one has to get to some other
1:48:11
course one has to get to some other
1:48:11
course one has to get to some other fundamentals in computer science saying
1:48:13
fundamentals in computer science saying
1:48:13
fundamentals in computer science saying query execution in a standard model such
1:48:15
query execution in a standard model such
1:48:15
query execution in a standard model such as like you're running simply spark
1:48:17
as like you're running simply spark
1:48:17
as like you're running simply spark okay running a sequel and window-based
1:48:19
okay running a sequel and window-based
1:48:19
okay running a sequel and window-based queries that's not actually workable
1:48:21
queries that's not actually workable
1:48:21
queries that's not actually workable model you can't run 10 000 queries per
1:48:23
model you can't run 10 000 queries per
1:48:23
model you can't run 10 000 queries per second
1:48:24
second
1:48:24
second across 100 000 events each one arriving
1:48:26
across 100 000 events each one arriving
1:48:26
across 100 000 events each one arriving and actually give 100 milliseconds
1:48:28
and actually give 100 milliseconds
1:48:28
and actually give 100 milliseconds so there's a lot of work that is done at
1:48:30
so there's a lot of work that is done at
1:48:30
so there's a lot of work that is done at some base level understanding of saying
1:48:32
some base level understanding of saying
1:48:32
some base level understanding of saying how query execution can be modulated and
1:48:35
how query execution can be modulated and
1:48:35
how query execution can be modulated and completely changing the tag forms and
1:48:37
completely changing the tag forms and
1:48:37
completely changing the tag forms and either sql oriented query execution or
1:48:39
either sql oriented query execution or
1:48:39
either sql oriented query execution or even
1:48:39
even
1:48:39
even text search oriented query execution to
1:48:41
text search oriented query execution to
1:48:41
text search oriented query execution to a different word we call the
1:48:43
a different word we call the
1:48:43
a different word we call the query indexing technologies okay now
1:48:47
query indexing technologies okay now
1:48:47
query indexing technologies okay now the all these things together along with
1:48:49
the all these things together along with
1:48:49
the all these things together along with the underlying database
1:48:50
the underlying database
1:48:50
the underlying database allows us now to start to get to saying
1:48:52
allows us now to start to get to saying
1:48:52
allows us now to start to get to saying that we can actually start supporting
1:48:53
that we can actually start supporting
1:48:53
that we can actually start supporting scale
1:48:54
scale
1:48:54
scale okay now given this this much of a
1:48:56
okay now given this this much of a
1:48:56
okay now given this this much of a background information on on what is
1:48:58
background information on on what is
1:48:58
background information on on what is happening let's look at the next level
1:49:00
happening let's look at the next level
1:49:00
happening let's look at the next level of details on the architecture so now we
1:49:02
of details on the architecture so now we
1:49:02
of details on the architecture so now we look we'll start looking at
1:49:04
look we'll start looking at
1:49:04
look we'll start looking at the architecture of profile as well as
1:49:07
the architecture of profile as well as
1:49:07
the architecture of profile as well as identity
1:49:10
identity
1:49:10
identity so this particular architecture as
1:49:12
so this particular architecture as
1:49:12
so this particular architecture as you're starting to look at
1:49:13
you're starting to look at
1:49:13
you're starting to look at has been divided into two different
1:49:15
has been divided into two different
1:49:15
has been divided into two different parts streaming world as well as batch
1:49:17
parts streaming world as well as batch
1:49:17
parts streaming world as well as batch world
1:49:19
world
1:49:19
world there's a lot of similarities between
1:49:20
there's a lot of similarities between
1:49:20
there's a lot of similarities between profile and identity as you will start
1:49:22
profile and identity as you will start
1:49:22
profile and identity as you will start noticing
1:49:24
noticing
1:49:24
noticing from the left hand side as i was talking
1:49:25
from the left hand side as i was talking
1:49:26
from the left hand side as i was talking about there is essentially data that is
1:49:28
about there is essentially data that is
1:49:28
about there is essentially data that is actually coming into the system
1:49:30
actually coming into the system
1:49:30
actually coming into the system the data that comes is from the
1:49:31
the data that comes is from the
1:49:31
the data that comes is from the interactional systems typically
1:49:33
interactional systems typically
1:49:33
interactional systems typically real-time data collection we call the
1:49:35
real-time data collection we call the
1:49:35
real-time data collection we call the data collection service dcs
1:49:37
data collection service dcs
1:49:37
data collection service dcs this is a box where all the data
1:49:39
this is a box where all the data
1:49:39
this is a box where all the data actually arrives this is streaming data
1:49:40
actually arrives this is streaming data
1:49:40
actually arrives this is streaming data collections
1:49:42
collections
1:49:42
collections now once the data actually starts
1:49:44
now once the data actually starts
1:49:44
now once the data actually starts arriving there is we're using apache
1:49:46
arriving there is we're using apache
1:49:46
arriving there is we're using apache flink
1:49:47
flink
1:49:47
flink and actually trying to do streaming
1:49:49
and actually trying to do streaming
1:49:49
and actually trying to do streaming graph compute on top of this
1:49:51
graph compute on top of this
1:49:51
graph compute on top of this added by reduce radius is being used as
1:49:53
added by reduce radius is being used as
1:49:53
added by reduce radius is being used as more like a
1:49:54
more like a
1:49:54
more like a a cache that is needed for some other
1:49:56
a cache that is needed for some other
1:49:56
a cache that is needed for some other computation
1:49:58
computation
1:49:58
computation but the results of all this information
1:50:00
but the results of all this information
1:50:00
but the results of all this information means we are continuously making
1:50:02
means we are continuously making
1:50:02
means we are continuously making updates into cosmos db where actually
1:50:04
updates into cosmos db where actually
1:50:04
updates into cosmos db where actually the graph will be stored
1:50:06
the graph will be stored
1:50:06
the graph will be stored similarly from the batch you have etl
1:50:09
similarly from the batch you have etl
1:50:09
similarly from the batch you have etl processes that are actually running
1:50:10
processes that are actually running
1:50:10
processes that are actually running incoming data is actually transformed
1:50:13
incoming data is actually transformed
1:50:13
incoming data is actually transformed laid down into azure data link
1:50:15
laid down into azure data link
1:50:15
laid down into azure data link these are essentially packet files where
1:50:17
these are essentially packet files where
1:50:17
these are essentially packet files where we are actually uh laying down the data
1:50:19
we are actually uh laying down the data
1:50:19
we are actually uh laying down the data on top of that we are running spark
1:50:21
on top of that we are running spark
1:50:21
on top of that we are running spark based graph compute jobs these graph
1:50:24
based graph compute jobs these graph
1:50:24
based graph compute jobs these graph compute jobs essentially are computing
1:50:25
compute jobs essentially are computing
1:50:26
compute jobs essentially are computing as the data arrives and the data gets
1:50:28
as the data arrives and the data gets
1:50:28
as the data arrives and the data gets again put into
1:50:29
again put into
1:50:29
again put into graph updates into cosmos db so this
1:50:31
graph updates into cosmos db so this
1:50:31
graph updates into cosmos db so this entire use case starts looking like
1:50:33
entire use case starts looking like
1:50:33
entire use case starts looking like cosmos db
1:50:34
cosmos db
1:50:34
cosmos db being used for graph computation and
1:50:37
being used for graph computation and
1:50:37
being used for graph computation and running graph api on top of it and we
1:50:39
running graph api on top of it and we
1:50:39
running graph api on top of it and we look at the scale at which these systems
1:50:40
look at the scale at which these systems
1:50:40
look at the scale at which these systems are actually running
1:50:42
are actually running
1:50:42
are actually running once you have a graph compute that is
1:50:44
once you have a graph compute that is
1:50:44
once you have a graph compute that is actually in place
1:50:46
actually in place
1:50:46
actually in place the next layer of this one actually
1:50:48
the next layer of this one actually
1:50:48
the next layer of this one actually starts looking at the apis that actually
1:50:50
starts looking at the apis that actually
1:50:50
starts looking at the apis that actually support it as i mentioned the entire
1:50:52
support it as i mentioned the entire
1:50:52
support it as i mentioned the entire system
1:50:53
system
1:50:53
system has been built as api first system
1:50:56
has been built as api first system
1:50:56
has been built as api first system so apis are continuously accessing and
1:50:58
so apis are continuously accessing and
1:50:58
so apis are continuously accessing and you have to understand that
1:50:59
you have to understand that
1:51:00
you have to understand that level of scale at which the apis have to
1:51:02
level of scale at which the apis have to
1:51:02
level of scale at which the apis have to run is that a graph lookup has to happen
1:51:04
run is that a graph lookup has to happen
1:51:04
run is that a graph lookup has to happen every single time that i'm either
1:51:06
every single time that i'm either
1:51:06
every single time that i'm either running a query
1:51:08
running a query
1:51:08
running a query okay or i'm actually doing a point
1:51:09
okay or i'm actually doing a point
1:51:10
okay or i'm actually doing a point lookup or i'm running
1:51:11
lookup or i'm running
1:51:11
lookup or i'm running bad segmentation jobs i need to do
1:51:13
bad segmentation jobs i need to do
1:51:13
bad segmentation jobs i need to do lookups on the cosmos db which is the
1:51:15
lookups on the cosmos db which is the
1:51:15
lookups on the cosmos db which is the graph needs to be looked up
1:51:16
graph needs to be looked up
1:51:16
graph needs to be looked up so the scale at which these runs are
1:51:18
so the scale at which these runs are
1:51:18
so the scale at which these runs are extremely heavy on cosmos db and we're
1:51:21
extremely heavy on cosmos db and we're
1:51:21
extremely heavy on cosmos db and we're running actually at fairly large scale
1:51:22
running actually at fairly large scale
1:51:22
running actually at fairly large scale i'll show you
1:51:23
i'll show you
1:51:23
i'll show you some of the numbers and the scales at
1:51:24
some of the numbers and the scales at
1:51:24
some of the numbers and the scales at which systems are running this data
1:51:26
which systems are running this data
1:51:26
which systems are running this data eventually then gets used by unified
1:51:28
eventually then gets used by unified
1:51:28
eventually then gets used by unified profile and this is where the overall
1:51:30
profile and this is where the overall
1:51:30
profile and this is where the overall view construction of the profile
1:51:32
view construction of the profile
1:51:32
view construction of the profile actually starts let's look at also
1:51:34
actually starts let's look at also
1:51:34
actually starts let's look at also corresponding to the next
1:51:36
corresponding to the next
1:51:36
corresponding to the next slide the architecture of a profile
1:51:40
slide the architecture of a profile
1:51:40
slide the architecture of a profile very similar looking architecture on two
1:51:42
very similar looking architecture on two
1:51:42
very similar looking architecture on two sides and the symmetry is natural here
1:51:44
sides and the symmetry is natural here
1:51:44
sides and the symmetry is natural here if you start thinking about it because
1:51:46
if you start thinking about it because
1:51:46
if you start thinking about it because the data that is going into profile the
1:51:49
the data that is going into profile the
1:51:49
the data that is going into profile the data that is going into identity
1:51:51
data that is going into identity
1:51:51
data that is going into identity they are kind of same data sets they are
1:51:52
they are kind of same data sets they are
1:51:52
they are kind of same data sets they are only not different data sets that are
1:51:54
only not different data sets that are
1:51:54
only not different data sets that are going to different places
1:51:55
going to different places
1:51:55
going to different places the purpose and the computations that
1:51:57
the purpose and the computations that
1:51:57
the purpose and the computations that they do on the same dataset is very
1:51:58
they do on the same dataset is very
1:51:58
they do on the same dataset is very different from each other
1:52:00
different from each other
1:52:00
different from each other so what goes into the underlying data
1:52:02
so what goes into the underlying data
1:52:02
so what goes into the underlying data store is very different
1:52:03
store is very different
1:52:03
store is very different so in case of profile again at the top
1:52:06
so in case of profile again at the top
1:52:06
so in case of profile again at the top level you have same data collection
1:52:08
level you have same data collection
1:52:08
level you have same data collection happening to dc as the data collection
1:52:10
happening to dc as the data collection
1:52:10
happening to dc as the data collection service
1:52:11
service
1:52:11
service underlying we are of course using kafka
1:52:12
underlying we are of course using kafka
1:52:12
underlying we are of course using kafka based pipeline and this is where the
1:52:14
based pipeline and this is where the
1:52:14
based pipeline and this is where the actual injection happens
1:52:16
actual injection happens
1:52:16
actual injection happens now in case of class uh profile however
1:52:19
now in case of class uh profile however
1:52:19
now in case of class uh profile however every event that actually uh happens
1:52:22
every event that actually uh happens
1:52:22
every event that actually uh happens in users history is recorded and stored
1:52:25
in users history is recorded and stored
1:52:25
in users history is recorded and stored in the profile
1:52:27
in the profile
1:52:27
in the profile okay so every web visit you make
1:52:29
okay so every web visit you make
1:52:29
okay so every web visit you make everything that you do inside your
1:52:31
everything that you do inside your
1:52:31
everything that you do inside your in-app
1:52:31
in-app
1:52:31
in-app everything that happens in the store
1:52:33
everything that happens in the store
1:52:33
everything that happens in the store that information is recorded
1:52:35
that information is recorded
1:52:35
that information is recorded for that enterprise in their customer
1:52:37
for that enterprise in their customer
1:52:38
for that enterprise in their customer consumer profile
1:52:39
consumer profile
1:52:39
consumer profile and essentially we call this as a time
1:52:40
and essentially we call this as a time
1:52:40
and essentially we call this as a time series data so if you start looking at
1:52:43
series data so if you start looking at
1:52:43
series data so if you start looking at it the the transactions here is that
1:52:45
it the the transactions here is that
1:52:45
it the the transactions here is that incoming data
1:52:45
incoming data
1:52:46
incoming data maps into an etl etl conversion happens
1:52:48
maps into an etl etl conversion happens
1:52:48
maps into an etl etl conversion happens and
1:52:49
and
1:52:49
and that data essentially goes into ups a
1:52:51
that data essentially goes into ups a
1:52:52
that data essentially goes into ups a single event will result into multiple
1:52:53
single event will result into multiple
1:52:54
single event will result into multiple updates happening or multiple
1:52:55
updates happening or multiple
1:52:55
updates happening or multiple transactions that are happening against
1:52:57
transactions that are happening against
1:52:57
transactions that are happening against cosmos
1:52:57
cosmos
1:52:57
cosmos we look at the scale okay the batch
1:53:00
we look at the scale okay the batch
1:53:00
we look at the scale okay the batch injection actually has a little bit
1:53:01
injection actually has a little bit
1:53:01
injection actually has a little bit more complexities because you're running
1:53:03
more complexities because you're running
1:53:03
more complexities because you're running an entire etl pipeline
1:53:05
an entire etl pipeline
1:53:05
an entire etl pipeline you're talking to catalog services then
1:53:07
you're talking to catalog services then
1:53:08
you're talking to catalog services then the data gets laid down
1:53:09
the data gets laid down
1:53:09
the data gets laid down to azure data lake which is parker files
1:53:11
to azure data lake which is parker files
1:53:11
to azure data lake which is parker files we run compute jobs on top of that which
1:53:13
we run compute jobs on top of that which
1:53:13
we run compute jobs on top of that which is
1:53:13
is
1:53:13
is based on spark you can see again the
1:53:15
based on spark you can see again the
1:53:15
based on spark you can see again the symmetry there similar things are
1:53:17
symmetry there similar things are
1:53:17
symmetry there similar things are happening on both sides
1:53:18
happening on both sides
1:53:18
happening on both sides this is where batch hydration actually
1:53:20
this is where batch hydration actually
1:53:20
this is where batch hydration actually happens we use the term hydration to
1:53:22
happens we use the term hydration to
1:53:22
happens we use the term hydration to mean
1:53:23
mean
1:53:23
mean load data into the profile it's a
1:53:25
load data into the profile it's a
1:53:25
load data into the profile it's a pipeline
1:53:26
pipeline
1:53:26
pipeline and there's a profile so you're just
1:53:27
and there's a profile so you're just
1:53:27
and there's a profile so you're just hydrating the profile that's the
1:53:28
hydrating the profile that's the
1:53:28
hydrating the profile that's the terminology we use
1:53:30
terminology we use
1:53:30
terminology we use once the data is actually in the profile
1:53:32
once the data is actually in the profile
1:53:32
once the data is actually in the profile this is where the next set of loads
1:53:33
this is where the next set of loads
1:53:34
this is where the next set of loads workload starts coming into play
1:53:35
workload starts coming into play
1:53:35
workload starts coming into play the workloads here are essentially about
1:53:40
the workloads here are essentially about
1:53:40
the workloads here are essentially about as i talk about a periodic query
1:53:42
as i talk about a periodic query
1:53:42
as i talk about a periodic query execution which we call the
1:53:44
execution which we call the
1:53:44
execution which we call the baseline segmentation on the bat
1:53:45
baseline segmentation on the bat
1:53:45
baseline segmentation on the bat segmentation jobs that are actually
1:53:47
segmentation jobs that are actually
1:53:47
segmentation jobs that are actually running
1:53:47
running
1:53:47
running streaming segmentation this is this
1:53:49
streaming segmentation this is this
1:53:49
streaming segmentation this is this extreme heavy load that are running
1:53:51
extreme heavy load that are running
1:53:51
extreme heavy load that are running every single
1:53:52
every single
1:53:52
every single second with a very high influx of events
1:53:55
second with a very high influx of events
1:53:55
second with a very high influx of events and then once this data is actually
1:53:57
and then once this data is actually
1:53:57
and then once this data is actually continuously computed you are computing
1:53:59
continuously computed you are computing
1:53:59
continuously computed you are computing audience sets audience membership
1:54:01
audience sets audience membership
1:54:01
audience sets audience membership audience payloads getting them ready for
1:54:03
audience payloads getting them ready for
1:54:03
audience payloads getting them ready for the experience business part of it which
1:54:05
the experience business part of it which
1:54:05
the experience business part of it which is ready for the applications
1:54:06
is ready for the applications
1:54:06
is ready for the applications and these applications you have
1:54:08
and these applications you have
1:54:08
and these applications you have essentially apis again the system is
1:54:10
essentially apis again the system is
1:54:10
essentially apis again the system is based on a micro services api kind of
1:54:13
based on a micro services api kind of
1:54:13
based on a micro services api kind of architecture
1:54:14
architecture
1:54:14
architecture and export jobs essentially provide data
1:54:16
and export jobs essentially provide data
1:54:16
and export jobs essentially provide data for taking care of
1:54:18
for taking care of
1:54:18
for taking care of operational workloads as well as
1:54:20
operational workloads as well as
1:54:20
operational workloads as well as analytical workloads
1:54:21
analytical workloads
1:54:21
analytical workloads because remember another thing that this
1:54:23
because remember another thing that this
1:54:23
because remember another thing that this particular database now has to take care
1:54:25
particular database now has to take care
1:54:25
particular database now has to take care of which is
1:54:26
of which is
1:54:26
of which is the stitching component that we're
1:54:28
the stitching component that we're
1:54:28
the stitching component that we're putting together one single view of
1:54:29
putting together one single view of
1:54:29
putting together one single view of customer
1:54:30
customer
1:54:30
customer is important even in the insights world
1:54:32
is important even in the insights world
1:54:32
is important even in the insights world which is where reporting query
1:54:34
which is where reporting query
1:54:34
which is where reporting query processing uh data science exercise
1:54:36
processing uh data science exercise
1:54:36
processing uh data science exercise playing and they need this particular
1:54:38
playing and they need this particular
1:54:38
playing and they need this particular view to be continuously exported
1:54:40
view to be continuously exported
1:54:40
view to be continuously exported so we are also not only hydrating on one
1:54:42
so we are also not only hydrating on one
1:54:42
so we are also not only hydrating on one end we are also on the
1:54:43
end we are also on the
1:54:43
end we are also on the continuous end exporting the data and
1:54:45
continuous end exporting the data and
1:54:45
continuous end exporting the data and this is again one of the speaker was
1:54:47
this is again one of the speaker was
1:54:47
this is again one of the speaker was talking about cdc's
1:54:48
talking about cdc's
1:54:48
talking about cdc's system heavily internally uses cdc's to
1:54:51
system heavily internally uses cdc's to
1:54:51
system heavily internally uses cdc's to kind of keep on
1:54:52
kind of keep on
1:54:52
kind of keep on hydrating the data on the outbound side
1:54:54
hydrating the data on the outbound side
1:54:54
hydrating the data on the outbound side into data lake on a continuous basis
1:54:56
into data lake on a continuous basis
1:54:56
into data lake on a continuous basis while at the same time ingesting data
1:54:59
while at the same time ingesting data
1:54:59
while at the same time ingesting data let's look at now the scale
1:55:01
let's look at now the scale
1:55:01
let's look at now the scale characteristics of the system so
1:55:02
characteristics of the system so
1:55:02
characteristics of the system so before that let's look at the tech stack
1:55:04
before that let's look at the tech stack
1:55:04
before that let's look at the tech stack itself as
1:55:05
itself as
1:55:05
itself as one of the speaker again said we are not
1:55:07
one of the speaker again said we are not
1:55:08
one of the speaker again said we are not a dot net kind of shop
1:55:09
a dot net kind of shop
1:55:09
a dot net kind of shop we are again a java scala kind of a shop
1:55:11
we are again a java scala kind of a shop
1:55:11
we are again a java scala kind of a shop and it has been perfectly fine working
1:55:14
and it has been perfectly fine working
1:55:14
and it has been perfectly fine working with azure as well as cosmos db no
1:55:16
with azure as well as cosmos db no
1:55:16
with azure as well as cosmos db no problem of
1:55:17
problem of
1:55:17
problem of working with javan scala everything has
1:55:18
working with javan scala everything has
1:55:18
working with javan scala everything has been actually wonderful all along
1:55:21
been actually wonderful all along
1:55:21
been actually wonderful all along our processing essentially we're using
1:55:23
our processing essentially we're using
1:55:23
our processing essentially we're using spark data breaks
1:55:24
spark data breaks
1:55:24
spark data breaks as well as the the aws emr for the
1:55:27
as well as the the aws emr for the
1:55:27
as well as the the aws emr for the legacy reasons
1:55:28
legacy reasons
1:55:28
legacy reasons apache flink actually is getting used i
1:55:30
apache flink actually is getting used i
1:55:30
apache flink actually is getting used i showed you a couple of places
1:55:32
showed you a couple of places
1:55:32
showed you a couple of places uh clouds we are dominantly actually
1:55:35
uh clouds we are dominantly actually
1:55:35
uh clouds we are dominantly actually most of the structure that i actually
1:55:36
most of the structure that i actually
1:55:36
most of the structure that i actually kind of showed you here is based on
1:55:38
kind of showed you here is based on
1:55:38
kind of showed you here is based on azure for the legacy reasons we
1:55:39
azure for the legacy reasons we
1:55:39
azure for the legacy reasons we still have a lot of aws presence uh
1:55:42
still have a lot of aws presence uh
1:55:42
still have a lot of aws presence uh we're using all the kubernetes stack
1:55:44
we're using all the kubernetes stack
1:55:44
we're using all the kubernetes stack and the database side cosmos db gets
1:55:47
and the database side cosmos db gets
1:55:47
and the database side cosmos db gets used
1:55:47
used
1:55:47
used majorly the flavors we are using is not
1:55:50
majorly the flavors we are using is not
1:55:50
majorly the flavors we are using is not cassandra or today we are straight
1:55:52
cassandra or today we are straight
1:55:52
cassandra or today we are straight away
1:55:53
away
1:55:53
away using the native cosmos db and there is
1:55:55
using the native cosmos db and there is
1:55:55
using the native cosmos db and there is uh
1:55:56
uh
1:55:56
uh we are finding actually the scale with
1:55:58
we are finding actually the scale with
1:55:58
we are finding actually the scale with which is running as well as the
1:56:00
which is running as well as the
1:56:00
which is running as well as the reliability that is
1:56:01
reliability that is
1:56:01
reliability that is provided has been enormous for us uh
1:56:03
provided has been enormous for us uh
1:56:03
provided has been enormous for us uh postgresql does get used quite a bit
1:56:06
postgresql does get used quite a bit
1:56:06
postgresql does get used quite a bit for smaller applications the radius is
1:56:08
for smaller applications the radius is
1:56:08
for smaller applications the radius is mostly being the cache kind of
1:56:09
mostly being the cache kind of
1:56:09
mostly being the cache kind of applications
1:56:10
applications
1:56:10
applications now let's look at the scale actually at
1:56:12
now let's look at the scale actually at
1:56:12
now let's look at the scale actually at which the systems are running
1:56:14
which the systems are running
1:56:14
which the systems are running so these are the scales at as our
1:56:16
so these are the scales at as our
1:56:16
so these are the scales at as our customers enterprise customers are
1:56:18
customers enterprise customers are
1:56:18
customers enterprise customers are seeing okay these are not the scales at
1:56:19
seeing okay these are not the scales at
1:56:20
seeing okay these are not the scales at which the cosmos db transactions run
1:56:22
which the cosmos db transactions run
1:56:22
which the cosmos db transactions run individual events here result into large
1:56:25
individual events here result into large
1:56:25
individual events here result into large number of transactions against the
1:56:26
number of transactions against the
1:56:26
number of transactions against the cosmos db
1:56:27
cosmos db
1:56:27
cosmos db and there's a multiplier effect that
1:56:28
and there's a multiplier effect that
1:56:28
and there's a multiplier effect that actually comes into play typically the
1:56:31
actually comes into play typically the
1:56:31
actually comes into play typically the the incoming streaming injection today
1:56:33
the incoming streaming injection today
1:56:33
the incoming streaming injection today runs at about seven and a half billion
1:56:35
runs at about seven and a half billion
1:56:35
runs at about seven and a half billion events
1:56:36
events
1:56:36
events we have two and a half billion events
1:56:37
we have two and a half billion events
1:56:37
we have two and a half billion events actually that are getting ingested
1:56:38
actually that are getting ingested
1:56:38
actually that are getting ingested continuous basis
1:56:40
continuous basis
1:56:40
continuous basis this results into graph updates graph
1:56:42
this results into graph updates graph
1:56:42
this results into graph updates graph updates is a very complex actually
1:56:44
updates is a very complex actually
1:56:44
updates is a very complex actually computation
1:56:44
computation
1:56:44
computation as well large number of transactions and
1:56:46
as well large number of transactions and
1:56:46
as well large number of transactions and updates that happen against
1:56:48
updates that happen against
1:56:48
updates that happen against the database and today the graph
1:56:51
the database and today the graph
1:56:51
the database and today the graph database is a relatively small database
1:56:52
database is a relatively small database
1:56:52
database is a relatively small database about 40 terabytes with 11 billion
1:56:54
about 40 terabytes with 11 billion
1:56:54
about 40 terabytes with 11 billion essentially clusters and 30 billion
1:56:56
essentially clusters and 30 billion
1:56:56
essentially clusters and 30 billion identities
1:56:57
identities
1:56:57
identities now let's look at the the similar scale
1:56:59
now let's look at the the similar scale
1:56:59
now let's look at the the similar scale exactly going towards the profile
1:57:01
exactly going towards the profile
1:57:01
exactly going towards the profile so on the next slide we show essentially
1:57:03
so on the next slide we show essentially
1:57:03
so on the next slide we show essentially the scale for profile
1:57:05
the scale for profile
1:57:05
the scale for profile now injection events as you can see is
1:57:07
now injection events as you can see is
1:57:07
now injection events as you can see is exactly the same the number is not going
1:57:08
exactly the same the number is not going
1:57:08
exactly the same the number is not going to change from looking at the profile
1:57:10
to change from looking at the profile
1:57:10
to change from looking at the profile however storage scale wise
1:57:11
however storage scale wise
1:57:12
however storage scale wise we're looking at a single account has a
1:57:13
we're looking at a single account has a
1:57:13
we're looking at a single account has a 200 terabytes of data actually
1:57:15
200 terabytes of data actually
1:57:15
200 terabytes of data actually sitting in the system with almost 5000
1:57:17
sitting in the system with almost 5000
1:57:18
sitting in the system with almost 5000 plus partitions
1:57:19
plus partitions
1:57:19
plus partitions there are 12 and a billion profiles
1:57:20
there are 12 and a billion profiles
1:57:20
there are 12 and a billion profiles actually located across 600 terabytes of
1:57:22
actually located across 600 terabytes of
1:57:22
actually located across 600 terabytes of data
1:57:23
data
1:57:23
data we are actually handing 50 billion point
1:57:26
we are actually handing 50 billion point
1:57:26
we are actually handing 50 billion point lookups in a given day
1:57:27
lookups in a given day
1:57:27
lookups in a given day okay we are actually handling peak loads
1:57:29
okay we are actually handling peak loads
1:57:29
okay we are actually handling peak loads for a single enterprise in excess of 400
1:57:32
for a single enterprise in excess of 400
1:57:32
for a single enterprise in excess of 400 000 requests
1:57:33
000 requests
1:57:33
000 requests per second at at this rate we are
1:57:35
per second at at this rate we are
1:57:35
per second at at this rate we are running queries against the system
1:57:37
running queries against the system
1:57:37
running queries against the system so this has been a great kind of
1:57:39
so this has been a great kind of
1:57:39
so this has been a great kind of exercise for us to work with microsoft
1:57:41
exercise for us to work with microsoft
1:57:41
exercise for us to work with microsoft um we just want to kind of conclude here
1:57:43
um we just want to kind of conclude here
1:57:43
um we just want to kind of conclude here right now just let's look at uh
1:57:46
right now just let's look at uh
1:57:46
right now just let's look at uh some of this presentation and
1:57:47
some of this presentation and
1:57:47
some of this presentation and information i have not been able to get
1:57:49
information i have not been able to get
1:57:49
information i have not been able to get into details but there's a lot of
1:57:50
into details but there's a lot of
1:57:50
into details but there's a lot of publications and blogs available
1:57:52
publications and blogs available
1:57:52
publications and blogs available i'm reachable at linkedin here as well
1:57:54
i'm reachable at linkedin here as well
1:57:54
i'm reachable at linkedin here as well as some of these blogs are actually
1:57:55
as some of these blogs are actually
1:57:55
as some of these blogs are actually available on adobe site and anybody of
1:57:57
available on adobe site and anybody of
1:57:58
available on adobe site and anybody of you
1:57:58
you
1:57:58
you any of you are interested in actually
1:57:59
any of you are interested in actually
1:57:59
any of you are interested in actually seeing more details behind this because
1:58:01
seeing more details behind this because
1:58:01
seeing more details behind this because a lot of subjects i
1:58:03
a lot of subjects i
1:58:03
a lot of subjects i briefly just touched upon but i would be
1:58:05
briefly just touched upon but i would be
1:58:05
briefly just touched upon but i would be happy to get into details but i want to
1:58:07
happy to get into details but i want to
1:58:07
happy to get into details but i want to stop there for any questions
1:58:11
thank you sandeep it was really amazing
1:58:13
thank you sandeep it was really amazing
1:58:13
thank you sandeep it was really amazing to drill down into adobe's
1:58:15
to drill down into adobe's
1:58:15
to drill down into adobe's infrastructure and
1:58:16
infrastructure and
1:58:16
infrastructure and understand with your help how adobe is
1:58:19
understand with your help how adobe is
1:58:19
understand with your help how adobe is running that infrastructure
1:58:19
running that infrastructure
1:58:20
running that infrastructure at scale we have one question from one
1:58:22
at scale we have one question from one
1:58:22
at scale we have one question from one of our viewers
1:58:23
of our viewers
1:58:23
of our viewers asking how did cosmos db's specific
1:58:27
asking how did cosmos db's specific
1:58:27
asking how did cosmos db's specific characteristics help in building this
1:58:29
characteristics help in building this
1:58:29
characteristics help in building this unified view that you have just
1:58:30
unified view that you have just
1:58:30
unified view that you have just described
1:58:33
these specific characteristics i mean i
1:58:35
these specific characteristics i mean i
1:58:35
these specific characteristics i mean i will tell you from almost simple
1:58:36
will tell you from almost simple
1:58:36
will tell you from almost simple operational aspect we were looking at
1:58:38
operational aspect we were looking at
1:58:38
operational aspect we were looking at essentially database as a service
1:58:41
essentially database as a service
1:58:41
essentially database as a service instant scalability was extremely
1:58:43
instant scalability was extremely
1:58:43
instant scalability was extremely important portion for us
1:58:44
important portion for us
1:58:44
important portion for us so i just want to talk about what kind
1:58:46
so i just want to talk about what kind
1:58:46
so i just want to talk about what kind of scalability patterns that we see when
1:58:48
of scalability patterns that we see when
1:58:48
of scalability patterns that we see when a bad job actually basically starts
1:58:50
a bad job actually basically starts
1:58:50
a bad job actually basically starts so let's say have an email campaign or a
1:58:52
so let's say have an email campaign or a
1:58:52
so let's say have an email campaign or a burst mode campaign that needs to run
1:58:54
burst mode campaign that needs to run
1:58:54
burst mode campaign that needs to run the system is running at actually for
1:58:56
the system is running at actually for
1:58:56
the system is running at actually for that enterprise at a scale
1:58:57
that enterprise at a scale
1:58:57
that enterprise at a scale zero literally no transactions happening
1:58:59
zero literally no transactions happening
1:58:59
zero literally no transactions happening with respect to query processing
1:59:00
with respect to query processing
1:59:00
with respect to query processing and then instantly for next three
1:59:03
and then instantly for next three
1:59:03
and then instantly for next three minutes i need to pump
1:59:04
minutes i need to pump
1:59:04
minutes i need to pump 10 million a minute actually emails and
1:59:07
10 million a minute actually emails and
1:59:07
10 million a minute actually emails and each email that number may look small
1:59:09
each email that number may look small
1:59:09
each email that number may look small but each email requests several look up
1:59:11
but each email requests several look up
1:59:11
but each email requests several look up several transactions and changes that
1:59:13
several transactions and changes that
1:59:13
several transactions and changes that happening in the database
1:59:14
happening in the database
1:59:14
happening in the database and you are moving from the zero to
1:59:16
and you are moving from the zero to
1:59:16
and you are moving from the zero to extreme scales
1:59:17
extreme scales
1:59:17
extreme scales at extremely fast rate so think about
1:59:19
at extremely fast rate so think about
1:59:19
at extremely fast rate so think about this kind of workloads which is
1:59:21
this kind of workloads which is
1:59:21
this kind of workloads which is dynamic characteristics of the workloads
1:59:23
dynamic characteristics of the workloads
1:59:23
dynamic characteristics of the workloads are actually continuously changing
1:59:25
are actually continuously changing
1:59:25
are actually continuously changing we needed a scale characteristics that
1:59:27
we needed a scale characteristics that
1:59:27
we needed a scale characteristics that can actually move so fast based on how
1:59:29
can actually move so fast based on how
1:59:29
can actually move so fast based on how things are actually coming in
1:59:31
things are actually coming in
1:59:31
things are actually coming in i talked about this this aspect where
1:59:32
i talked about this this aspect where
1:59:32
i talked about this this aspect where that we have a continuous load that
1:59:34
that we have a continuous load that
1:59:34
that we have a continuous load that comes in now that's a stable load when i
1:59:36
comes in now that's a stable load when i
1:59:36
comes in now that's a stable load when i say a few hundred thousand hits per
1:59:37
say a few hundred thousand hits per
1:59:37
say a few hundred thousand hits per second that is actually arriving
1:59:39
second that is actually arriving
1:59:39
second that is actually arriving this is the data that's getting actually
1:59:41
this is the data that's getting actually
1:59:41
this is the data that's getting actually ingested and stored again large number
1:59:43
ingested and stored again large number
1:59:43
ingested and stored again large number of transaction continuously running
1:59:44
of transaction continuously running
1:59:44
of transaction continuously running so stable load characteristics managing
1:59:47
so stable load characteristics managing
1:59:47
so stable load characteristics managing that extremely well right
1:59:48
that extremely well right
1:59:48
that extremely well right then as i talked about the other aspect
1:59:51
then as i talked about the other aspect
1:59:51
then as i talked about the other aspect that is important for us is that
1:59:53
that is important for us is that
1:59:53
that is important for us is that the underlying architecture itself is
1:59:55
the underlying architecture itself is
1:59:55
the underlying architecture itself is the architecture of the system is based
1:59:57
the architecture of the system is based
1:59:57
the architecture of the system is based on this notion of fragments and the
1:59:59
on this notion of fragments and the
1:59:59
on this notion of fragments and the graphs and how the data comes together
2:00:01
graphs and how the data comes together
2:00:01
graphs and how the data comes together it's not a sql oriented architecture
2:00:03
it's not a sql oriented architecture
2:00:03
it's not a sql oriented architecture right although we run queries that look
2:00:05
right although we run queries that look
2:00:05
right although we run queries that look like more or less say straight sql we
2:00:07
like more or less say straight sql we
2:00:07
like more or less say straight sql we are actually doing joins and all kinds
2:00:08
are actually doing joins and all kinds
2:00:08
are actually doing joins and all kinds of
2:00:09
of
2:00:09
of time series data processing the
2:00:10
time series data processing the
2:00:10
time series data processing the underlying data layer had to be done in
2:00:12
underlying data layer had to be done in
2:00:12
underlying data layer had to be done in a very different manner
2:00:14
a very different manner
2:00:14
a very different manner and that actually required us a document
2:00:16
and that actually required us a document
2:00:16
and that actually required us a document oriented kind of thinking so
2:00:18
oriented kind of thinking so
2:00:18
oriented kind of thinking so many different flavors that cosmos db
2:00:20
many different flavors that cosmos db
2:00:20
many different flavors that cosmos db supports we became extremely important
2:00:21
supports we became extremely important
2:00:22
supports we became extremely important for us
2:00:22
for us
2:00:22
for us then the partitioning structure
2:00:23
then the partitioning structure
2:00:24
then the partitioning structure underneath i mean i talked about saying
2:00:25
underneath i mean i talked about saying
2:00:25
underneath i mean i talked about saying that we have 5000 partition for one
2:00:27
that we have 5000 partition for one
2:00:27
that we have 5000 partition for one single customer
2:00:28
single customer
2:00:28
single customer this ability to kind of actually
2:00:30
this ability to kind of actually
2:00:30
this ability to kind of actually dynamically scale up the audios and get
2:00:32
dynamically scale up the audios and get
2:00:32
dynamically scale up the audios and get to a point saying
2:00:33
to a point saying
2:00:33
to a point saying i can lay down my my audios and we
2:00:36
i can lay down my my audios and we
2:00:36
i can lay down my my audios and we we think in this way which is i need
2:00:39
we think in this way which is i need
2:00:39
we think in this way which is i need ability to run let's say one terabyte a
2:00:41
ability to run let's say one terabyte a
2:00:41
ability to run let's say one terabyte a minute
2:00:41
minute
2:00:41
minute i o against the entire system and a scan
2:00:44
i o against the entire system and a scan
2:00:44
i o against the entire system and a scan job
2:00:44
job
2:00:44
job ability to get there becomes very
2:00:46
ability to get there becomes very
2:00:46
ability to get there becomes very important now this particular thing may
2:00:47
important now this particular thing may
2:00:47
important now this particular thing may look very simple scan job at a terabyte
2:00:49
look very simple scan job at a terabyte
2:00:49
look very simple scan job at a terabyte in a minute
2:00:50
in a minute
2:00:50
in a minute however with hdfs is not possible to get
2:00:52
however with hdfs is not possible to get
2:00:52
however with hdfs is not possible to get there there were initial trials
2:00:53
there there were initial trials
2:00:54
there there were initial trials we did so i think there were many of
2:00:55
we did so i think there were many of
2:00:55
we did so i think there were many of these considerations that actually kind
2:00:57
these considerations that actually kind
2:00:57
these considerations that actually kind of started getting us towards using
2:00:59
of started getting us towards using
2:00:59
of started getting us towards using cosmos db
2:01:00
cosmos db
2:01:00
cosmos db and this was early in 2017 2018 but has
2:01:03
and this was early in 2017 2018 but has
2:01:03
and this was early in 2017 2018 but has been a very wonderful journey for us
2:01:07
been a very wonderful journey for us
2:01:07
been a very wonderful journey for us that makes sense i think it's just
2:01:08
that makes sense i think it's just
2:01:08
that makes sense i think it's just mind-blowing to understand at which
2:01:10
mind-blowing to understand at which
2:01:10
mind-blowing to understand at which scale you are operating
2:01:11
scale you are operating
2:01:11
scale you are operating your whole platform and how customers be
2:01:13
your whole platform and how customers be
2:01:13
your whole platform and how customers be fits into that platform thanks again
2:01:14
fits into that platform thanks again
2:01:14
fits into that platform thanks again sandeep for joining the conference today
2:01:16
sandeep for joining the conference today
2:01:16
sandeep for joining the conference today thanks for taking the time
2:01:17
thanks for taking the time
2:01:17
thanks for taking the time uh to share all of that uh very precious
2:01:19
uh to share all of that uh very precious
2:01:19
uh to share all of that uh very precious information with us
2:01:21
information with us
2:01:21
information with us next and just before we switch to our
2:01:22
next and just before we switch to our
2:01:22
next and just before we switch to our next speaker we have a quick
2:01:24
next speaker we have a quick
2:01:24
next speaker we have a quick video to show you and team us more
2:01:26
video to show you and team us more
2:01:26
video to show you and team us more details thank you
2:01:29
details thank you
2:01:29
details thank you so cosmos achieve incredible speeds at
2:01:32
so cosmos achieve incredible speeds at
2:01:32
so cosmos achieve incredible speeds at any scale and keep your costs low with a
2:01:34
any scale and keep your costs low with a
2:01:34
any scale and keep your costs low with a well-planned
2:01:39
awesome huh cool i'll give a quick intro
2:01:41
awesome huh cool i'll give a quick intro
2:01:41
awesome huh cool i'll give a quick intro to the video before we
2:01:42
to the video before we
2:01:42
to the video before we we dive into i guess you got a little
2:01:44
we dive into i guess you got a little
2:01:44
we dive into i guess you got a little bit of a sneak preview there
2:01:45
bit of a sneak preview there
2:01:46
bit of a sneak preview there so as a horizontally partitioned nosql
2:01:48
so as a horizontally partitioned nosql
2:01:48
so as a horizontally partitioned nosql database
2:01:49
database
2:01:49
database cosmos tv allows customers to enjoy the
2:01:51
cosmos tv allows customers to enjoy the
2:01:51
cosmos tv allows customers to enjoy the same performance
2:01:52
same performance
2:01:52
same performance in scale and latency slas uh whether
2:01:55
in scale and latency slas uh whether
2:01:55
in scale and latency slas uh whether their database is i mean megabytes in
2:01:57
their database is i mean megabytes in
2:01:57
their database is i mean megabytes in size gigabytes in size or petabytes in
2:01:59
size gigabytes in size or petabytes in
2:01:59
size gigabytes in size or petabytes in size
2:02:00
size
2:02:00
size however to get this kind of performance
2:02:03
however to get this kind of performance
2:02:03
however to get this kind of performance you must take care when picking
2:02:04
you must take care when picking
2:02:04
you must take care when picking your partitioning strategy for your data
2:02:06
your partitioning strategy for your data
2:02:06
your partitioning strategy for your data in other words when picking your
2:02:08
in other words when picking your
2:02:08
in other words when picking your partition key
2:02:09
partition key
2:02:09
partition key so let's take a look at this quick video
2:02:11
so let's take a look at this quick video
2:02:11
so let's take a look at this quick video that helps explain some of the
2:02:12
that helps explain some of the
2:02:12
that helps explain some of the considerations that you should take
2:02:14
considerations that you should take
2:02:14
considerations that you should take when deciding a partitioning strategy
2:02:16
when deciding a partitioning strategy
2:02:16
when deciding a partitioning strategy for your data
2:02:18
for your data
2:02:18
for your data achieve incredible speeds at any scale
2:02:21
achieve incredible speeds at any scale
2:02:21
achieve incredible speeds at any scale and keep your costs low
2:02:22
and keep your costs low
2:02:22
and keep your costs low with a well-planned partitioning
2:02:23
with a well-planned partitioning
2:02:23
with a well-planned partitioning strategy on azure cosmos db
2:02:26
strategy on azure cosmos db
2:02:26
strategy on azure cosmos db partitioning involves writing data to
2:02:28
partitioning involves writing data to
2:02:28
partitioning involves writing data to servers in a way that optimizes both
2:02:30
servers in a way that optimizes both
2:02:30
servers in a way that optimizes both reads and writes it's important for all
2:02:32
reads and writes it's important for all
2:02:32
reads and writes it's important for all applications but becomes critical the
2:02:34
applications but becomes critical the
2:02:34
applications but becomes critical the more you scale
2:02:35
more you scale
2:02:35
more you scale as data spreads to more servers azure
2:02:38
as data spreads to more servers azure
2:02:38
as data spreads to more servers azure cosmos db
2:02:39
cosmos db
2:02:39
cosmos db stores data in virtual buckets called
2:02:41
stores data in virtual buckets called
2:02:41
stores data in virtual buckets called logical partitions
2:02:42
logical partitions
2:02:42
logical partitions it relies on a partition key to
2:02:44
it relies on a partition key to
2:02:44
it relies on a partition key to determine which of these buckets to put
2:02:46
determine which of these buckets to put
2:02:46
determine which of these buckets to put new data in
2:02:46
new data in
2:02:46
new data in and where to look for data during a
2:02:48
and where to look for data during a
2:02:48
and where to look for data during a query
2:02:50
query
2:02:50
query when choosing a key start by testing
2:02:52
when choosing a key start by testing
2:02:52
when choosing a key start by testing your most important requests against the
2:02:54
your most important requests against the
2:02:54
your most important requests against the following three guidelines
2:02:56
following three guidelines
2:02:56
following three guidelines one find the right balance test your
2:02:59
one find the right balance test your
2:02:59
one find the right balance test your partition key to see how it distributes
2:03:01
partition key to see how it distributes
2:03:01
partition key to see how it distributes rights
2:03:02
rights
2:03:02
rights the goal is to avoid hot spots and rate
2:03:04
the goal is to avoid hot spots and rate
2:03:04
the goal is to avoid hot spots and rate limits by achieving even distribution of
2:03:06
limits by achieving even distribution of
2:03:06
limits by achieving even distribution of storage
2:03:06
storage
2:03:06
storage and throughput across logical partitions
2:03:09
and throughput across logical partitions
2:03:10
and throughput across logical partitions two aim for a single partition query
2:03:13
two aim for a single partition query
2:03:13
two aim for a single partition query look to see how many partitions get hit
2:03:15
look to see how many partitions get hit
2:03:15
look to see how many partitions get hit when you run your most frequent queries
2:03:17
when you run your most frequent queries
2:03:17
when you run your most frequent queries ideally you want to avoid the cost and
2:03:19
ideally you want to avoid the cost and
2:03:19
ideally you want to avoid the cost and latency of involving multiple partitions
2:03:21
latency of involving multiple partitions
2:03:21
latency of involving multiple partitions by choosing a key that queries a single
2:03:23
by choosing a key that queries a single
2:03:23
by choosing a key that queries a single partition
2:03:25
partition
2:03:25
partition three understand cross-partition query
2:03:27
three understand cross-partition query
2:03:27
three understand cross-partition query trade-offs
2:03:29
trade-offs
2:03:29
trade-offs if you do run cross-partition queries
2:03:31
if you do run cross-partition queries
2:03:31
if you do run cross-partition queries for less important workloads every once
2:03:33
for less important workloads every once
2:03:33
for less important workloads every once in a while
2:03:33
in a while
2:03:34
in a while it won't impact your overall experience
2:03:36
it won't impact your overall experience
2:03:36
it won't impact your overall experience but if it's more than that
2:03:37
but if it's more than that
2:03:37
but if it's more than that you can use an array of discrete values
2:03:40
you can use an array of discrete values
2:03:40
you can use an array of discrete values for the partition keys in your query to
2:03:42
for the partition keys in your query to
2:03:42
for the partition keys in your query to target a subset of partitions
2:03:44
target a subset of partitions
2:03:44
target a subset of partitions looking to unlock the scale and
2:03:46
looking to unlock the scale and
2:03:46
looking to unlock the scale and performance of azure cosmos db
2:03:48
performance of azure cosmos db
2:03:48
performance of azure cosmos db remember it's all about the partition
2:03:51
remember it's all about the partition
2:03:51
remember it's all about the partition [Music]
2:03:54
[Music]
2:03:54
[Music] key
2:03:57
key
2:03:57
key [Music]
2:04:00
[Music]
2:04:00
[Music] what a great video i think we often say
2:04:02
what a great video i think we often say
2:04:02
what a great video i think we often say that choosing a good partition key
2:04:04
that choosing a good partition key
2:04:04
that choosing a good partition key is key to getting really good
2:04:05
is key to getting really good
2:04:05
is key to getting really good performance when you're doing cosmos cd
2:04:08
performance when you're doing cosmos cd
2:04:08
performance when you're doing cosmos cd so and we'll be hearing more about these
2:04:10
so and we'll be hearing more about these
2:04:10
so and we'll be hearing more about these concepts uh later in the session and
2:04:11
concepts uh later in the session and
2:04:12
concepts uh later in the session and other live streams
2:04:13
other live streams
2:04:13
other live streams so up next excited to introduce alicia
2:04:15
so up next excited to introduce alicia
2:04:15
so up next excited to introduce alicia who's a microsoft mvp
2:04:17
who's a microsoft mvp
2:04:17
who's a microsoft mvp and kafka evangelist at confluence to
2:04:19
and kafka evangelist at confluence to
2:04:19
and kafka evangelist at confluence to tell us how to stream data into azure
2:04:21
tell us how to stream data into azure
2:04:21
tell us how to stream data into azure custom tv
2:04:22
custom tv
2:04:22
custom tv with confluence welcome alicia hi thank
2:04:25
with confluence welcome alicia hi thank
2:04:25
with confluence welcome alicia hi thank you for having me
2:04:28
hi my name is alicia moniz and i'm a
2:04:30
hi my name is alicia moniz and i'm a
2:04:30
hi my name is alicia moniz and i'm a microsoft mvp
2:04:32
microsoft mvp
2:04:32
microsoft mvp as well as a kafka azure evangelista
2:04:35
as well as a kafka azure evangelista
2:04:35
as well as a kafka azure evangelista here at confluent
2:04:36
here at confluent
2:04:36
here at confluent so here at complement i empower and
2:04:38
so here at complement i empower and
2:04:38
so here at complement i empower and educate our team and our customers
2:04:41
educate our team and our customers
2:04:41
educate our team and our customers on azure capabilities and within the
2:04:43
on azure capabilities and within the
2:04:43
on azure capabilities and within the microsoft community i provide helpful
2:04:45
microsoft community i provide helpful
2:04:45
microsoft community i provide helpful tips and tricks for
2:04:46
tips and tricks for
2:04:46
tips and tricks for accelerating azure projects with kafka
2:04:52
accelerating azure projects with kafka
2:04:52
accelerating azure projects with kafka one of my first opportunities here at
2:04:53
one of my first opportunities here at
2:04:54
one of my first opportunities here at confluent was the opportunity to work
2:04:55
confluent was the opportunity to work
2:04:55
confluent was the opportunity to work directly with not only our engineering
2:04:57
directly with not only our engineering
2:04:57
directly with not only our engineering team
2:04:57
team
2:04:57
team but also the microsoft team with
2:05:00
but also the microsoft team with
2:05:00
but also the microsoft team with developing the cosmos db
2:05:01
developing the cosmos db
2:05:01
developing the cosmos db kafka connector so this is a very fun
2:05:03
kafka connector so this is a very fun
2:05:04
kafka connector so this is a very fun project for me
2:05:05
project for me
2:05:05
project for me on our side i worked with niko nam our
2:05:07
on our side i worked with niko nam our
2:05:07
on our side i worked with niko nam our senior product manager and rankesh kumar
2:05:09
senior product manager and rankesh kumar
2:05:09
senior product manager and rankesh kumar who leads our partner integrations
2:05:11
who leads our partner integrations
2:05:11
who leads our partner integrations and on the microsoft side i worked with
2:05:14
and on the microsoft side i worked with
2:05:14
and on the microsoft side i worked with ryan
2:05:15
ryan
2:05:15
ryan krakow and randy brown from microsoft
2:05:18
krakow and randy brown from microsoft
2:05:18
krakow and randy brown from microsoft commercial services
2:05:19
commercial services
2:05:19
commercial services i also had the opportunity to work with
2:05:21
i also had the opportunity to work with
2:05:21
i also had the opportunity to work with siva mullapuity
2:05:23
siva mullapuity
2:05:23
siva mullapuity from development and if you're
2:05:25
from development and if you're
2:05:25
from development and if you're interested steven and
2:05:26
interested steven and
2:05:26
interested steven and siva and i have a blog out there with
2:05:29
siva and i have a blog out there with
2:05:29
siva and i have a blog out there with some helpful hints
2:05:30
some helpful hints
2:05:30
some helpful hints on getting connected and also ryan was a
2:05:33
on getting connected and also ryan was a
2:05:33
on getting connected and also ryan was a guest
2:05:34
guest
2:05:34
guest on the compliment podcast streaming
2:05:36
on the compliment podcast streaming
2:05:36
on the compliment podcast streaming audio
2:05:37
audio
2:05:37
audio with confluent developer advocate tim
2:05:39
with confluent developer advocate tim
2:05:39
with confluent developer advocate tim berglund
2:05:40
berglund
2:05:40
berglund so it was really great for me to see the
2:05:42
so it was really great for me to see the
2:05:42
so it was really great for me to see the developers on
2:05:43
developers on
2:05:43
developers on global teams collaborating so seamlessly
2:05:46
global teams collaborating so seamlessly
2:05:46
global teams collaborating so seamlessly and that's been my experience so far
2:05:49
and that's been my experience so far
2:05:49
and that's been my experience so far working with
2:05:50
working with
2:05:50
working with both confluent and microsoft so thank
2:05:53
both confluent and microsoft so thank
2:05:53
both confluent and microsoft so thank you everyone for
2:05:54
you everyone for
2:05:54
you everyone for being so welcoming
2:05:59
in january of this year confluent
2:06:01
in january of this year confluent
2:06:01
in january of this year confluent announce their partnership with
2:06:02
announce their partnership with
2:06:02
announce their partnership with microsoft as well as integration
2:06:04
microsoft as well as integration
2:06:04
microsoft as well as integration capabilities with azure
2:06:06
capabilities with azure
2:06:06
capabilities with azure so azure customers are now able to
2:06:08
so azure customers are now able to
2:06:08
so azure customers are now able to purchase compliment cloud via the azure
2:06:10
purchase compliment cloud via the azure
2:06:10
purchase compliment cloud via the azure marketplace
2:06:11
marketplace
2:06:11
marketplace and authenticate to conflict cloud using
2:06:13
and authenticate to conflict cloud using
2:06:13
and authenticate to conflict cloud using single sign-on
2:06:14
single sign-on
2:06:14
single sign-on so additionally users will be able to
2:06:17
so additionally users will be able to
2:06:17
so additionally users will be able to draw down on their azure commits
2:06:19
draw down on their azure commits
2:06:19
draw down on their azure commits and they'll be able to view their
2:06:21
and they'll be able to view their
2:06:21
and they'll be able to view their confluent cloud resources
2:06:23
confluent cloud resources
2:06:23
confluent cloud resources via the portal so microsoft has been
2:06:26
via the portal so microsoft has been
2:06:26
via the portal so microsoft has been fantastic and they've truly partnered
2:06:28
fantastic and they've truly partnered
2:06:28
fantastic and they've truly partnered with us on getting this portal
2:06:29
with us on getting this portal
2:06:29
with us on getting this portal integration
2:06:29
integration
2:06:30
integration in place a special thanks to ramya
2:06:33
in place a special thanks to ramya
2:06:33
in place a special thanks to ramya origanti and her team
2:06:35
origanti and her team
2:06:35
origanti and her team and they've done a lot of work with
2:06:37
and they've done a lot of work with
2:06:37
and they've done a lot of work with putting additional
2:06:39
putting additional
2:06:39
putting additional tips and tricks out on the microsoft
2:06:41
tips and tricks out on the microsoft
2:06:41
tips and tricks out on the microsoft docs site
2:06:42
docs site
2:06:42
docs site there's also dedicated space there and
2:06:45
there's also dedicated space there and
2:06:45
there's also dedicated space there and we're
2:06:46
we're
2:06:46
we're part of the partner solutions so keep an
2:06:48
part of the partner solutions so keep an
2:06:48
part of the partner solutions so keep an eye out and we'll be sure to share
2:06:50
eye out and we'll be sure to share
2:06:50
eye out and we'll be sure to share information as we go through the year
2:06:52
information as we go through the year
2:06:52
information as we go through the year with um helpful references between
2:06:56
with um helpful references between
2:06:56
with um helpful references between our two websites
2:07:03
so customers with cop on premises
2:07:06
so customers with cop on premises
2:07:06
so customers with cop on premises are looking to migrate to the cloud or
2:07:09
are looking to migrate to the cloud or
2:07:09
are looking to migrate to the cloud or develop
2:07:09
develop
2:07:09
develop azure native services so they usually
2:07:12
azure native services so they usually
2:07:12
azure native services so they usually follow a journey that often starts with
2:07:14
follow a journey that often starts with
2:07:14
follow a journey that often starts with data migration then progresses to hybrid
2:07:16
data migration then progresses to hybrid
2:07:16
data migration then progresses to hybrid cloud
2:07:17
cloud
2:07:17
cloud followed by azure native app development
2:07:20
followed by azure native app development
2:07:20
followed by azure native app development so today i'll be sharing the reference
2:07:22
so today i'll be sharing the reference
2:07:22
so today i'll be sharing the reference architecture and design pattern
2:07:24
architecture and design pattern
2:07:24
architecture and design pattern for a solution integration with
2:07:26
for a solution integration with
2:07:26
for a solution integration with compliment azure
2:07:27
compliment azure
2:07:27
compliment azure and cosmos db so hopefully this will
2:07:30
and cosmos db so hopefully this will
2:07:30
and cosmos db so hopefully this will illustrate how compliment can
2:07:31
illustrate how compliment can
2:07:32
illustrate how compliment can expedite delivery on azure projects
2:07:41
expedite delivery on azure projects
2:07:41
expedite delivery on azure projects this is what a hybrid design pattern
2:07:43
this is what a hybrid design pattern
2:07:43
this is what a hybrid design pattern with confluent cloud looks like
2:07:45
with confluent cloud looks like
2:07:45
with confluent cloud looks like conflict most likely already exists
2:07:48
conflict most likely already exists
2:07:48
conflict most likely already exists within your customers organization
2:07:50
within your customers organization
2:07:50
within your customers organization and is being used on premises so
2:07:52
and is being used on premises so
2:07:52
and is being used on premises so developers are using
2:07:53
developers are using
2:07:54
developers are using kafka connectors to ingest data from
2:07:56
kafka connectors to ingest data from
2:07:56
kafka connectors to ingest data from mainframe systems
2:07:57
mainframe systems
2:07:57
mainframe systems now the confluent platform easily allows
2:08:00
now the confluent platform easily allows
2:08:00
now the confluent platform easily allows for the extension
2:08:01
for the extension
2:08:01
for the extension of that on-prem data pipeline to the
2:08:04
of that on-prem data pipeline to the
2:08:04
of that on-prem data pipeline to the cloud so step one in azure projects
2:08:06
cloud so step one in azure projects
2:08:06
cloud so step one in azure projects for most enterprise clients usually
2:08:09
for most enterprise clients usually
2:08:09
for most enterprise clients usually involves
2:08:09
involves
2:08:09
involves some type of data or server migration
2:08:12
some type of data or server migration
2:08:12
some type of data or server migration conflint can easily accelerate delivery
2:08:15
conflint can easily accelerate delivery
2:08:15
conflint can easily accelerate delivery on database migration
2:08:17
on database migration
2:08:17
on database migration and consolidation projects with their
2:08:19
and consolidation projects with their
2:08:19
and consolidation projects with their ecosystem and collection of connectors
2:08:22
ecosystem and collection of connectors
2:08:22
ecosystem and collection of connectors so in this scenario not only do our
2:08:24
so in this scenario not only do our
2:08:24
so in this scenario not only do our connectors provide access to open source
2:08:26
connectors provide access to open source
2:08:26
connectors provide access to open source systems
2:08:27
systems
2:08:27
systems but we can also easily flow data from
2:08:30
but we can also easily flow data from
2:08:30
but we can also easily flow data from multi-cloud sources
2:08:32
multi-cloud sources
2:08:32
multi-cloud sources this real-time access to data alleviates
2:08:35
this real-time access to data alleviates
2:08:35
this real-time access to data alleviates the pressure of having to move
2:08:36
the pressure of having to move
2:08:36
the pressure of having to move infrastructure prior to building
2:08:39
infrastructure prior to building
2:08:39
infrastructure prior to building cloud-native applications
2:08:42
cloud-native applications
2:08:42
cloud-native applications on the sync side the confluent team has
2:08:44
on the sync side the confluent team has
2:08:44
on the sync side the confluent team has done excellent work in connecting to the
2:08:46
done excellent work in connecting to the
2:08:46
done excellent work in connecting to the popular azure services
2:08:48
popular azure services
2:08:48
popular azure services such as azure adls blob storage
2:08:52
such as azure adls blob storage
2:08:52
such as azure adls blob storage and azure sql the cosmos cb connector is
2:08:55
and azure sql the cosmos cb connector is
2:08:55
and azure sql the cosmos cb connector is now in beta but is fully supported by
2:08:57
now in beta but is fully supported by
2:08:57
now in beta but is fully supported by microsoft
2:08:59
microsoft
2:08:59
microsoft now you may be wondering what is a
2:09:02
now you may be wondering what is a
2:09:02
now you may be wondering what is a connector
2:09:06
connectors are distributed bits of java
2:09:08
connectors are distributed bits of java
2:09:08
connectors are distributed bits of java code that sit in kafka clusters
2:09:11
code that sit in kafka clusters
2:09:11
code that sit in kafka clusters they allow us to import and export data
2:09:13
they allow us to import and export data
2:09:14
they allow us to import and export data from systems
2:09:15
from systems
2:09:15
from systems a managed connector run on a confluent
2:09:19
a managed connector run on a confluent
2:09:19
a managed connector run on a confluent cloud cluster
2:09:20
cloud cluster
2:09:20
cloud cluster so scaling and throughput is going to be
2:09:22
so scaling and throughput is going to be
2:09:22
so scaling and throughput is going to be managed for the user
2:09:24
managed for the user
2:09:24
managed for the user now a self-managed connector will run on
2:09:26
now a self-managed connector will run on
2:09:26
now a self-managed connector will run on a cluster
2:09:27
a cluster
2:09:27
a cluster where infrastructure is configured and
2:09:29
where infrastructure is configured and
2:09:29
where infrastructure is configured and maintained by the user
2:09:36
while you can communicate with storage
2:09:38
while you can communicate with storage
2:09:38
while you can communicate with storage databases and
2:09:39
databases and
2:09:39
databases and iot services on azure via connectors
2:09:43
iot services on azure via connectors
2:09:43
iot services on azure via connectors you can also interact with kafka data in
2:09:45
you can also interact with kafka data in
2:09:45
you can also interact with kafka data in azure app services
2:09:47
azure app services
2:09:47
azure app services by leveraging kafka apis with languages
2:09:50
by leveraging kafka apis with languages
2:09:50
by leveraging kafka apis with languages such as
2:09:51
such as
2:09:51
such as java python or dotnet
2:10:01
java python or dotnet
2:10:01
java python or dotnet today i'll be showing a walkthrough of
2:10:03
today i'll be showing a walkthrough of
2:10:03
today i'll be showing a walkthrough of configuring
2:10:06
a bandage azure data lake storage
2:10:08
a bandage azure data lake storage
2:10:08
a bandage azure data lake storage connector
2:10:09
connector
2:10:09
connector as well as the cosmos db self-managed
2:10:11
as well as the cosmos db self-managed
2:10:11
as well as the cosmos db self-managed connector
2:10:12
connector
2:10:12
connector so i've already configured a kafka
2:10:15
so i've already configured a kafka
2:10:16
so i've already configured a kafka cluster on the cloud
2:10:18
cluster on the cloud
2:10:18
cluster on the cloud and in the diagram below we can see that
2:10:21
and in the diagram below we can see that
2:10:21
and in the diagram below we can see that confluent cloud
2:10:22
confluent cloud
2:10:22
confluent cloud is readily available without
2:10:24
is readily available without
2:10:24
is readily available without installation on azure
2:10:26
installation on azure
2:10:26
installation on azure we have components such as schema
2:10:28
we have components such as schema
2:10:28
we have components such as schema registry k
2:10:29
registry k
2:10:29
registry k sql db and a variety of managed
2:10:31
sql db and a variety of managed
2:10:31
sql db and a variety of managed connectors available for use
2:10:33
connectors available for use
2:10:34
connectors available for use with dynamic scaling already handled by
2:10:36
with dynamic scaling already handled by
2:10:36
with dynamic scaling already handled by complex
2:10:37
complex
2:10:37
complex clusters so on the left hand side of the
2:10:39
clusters so on the left hand side of the
2:10:39
clusters so on the left hand side of the diagram
2:10:40
diagram
2:10:40
diagram are the azure container service
2:10:42
are the azure container service
2:10:42
are the azure container service components we'll be installing one
2:10:44
components we'll be installing one
2:10:44
components we'll be installing one instance of the coupling control center
2:10:46
instance of the coupling control center
2:10:46
instance of the coupling control center and one instance of complement connect
2:10:49
and one instance of complement connect
2:10:49
and one instance of complement connect so our script will automatically install
2:10:51
so our script will automatically install
2:10:51
so our script will automatically install the library files
2:10:52
the library files
2:10:52
the library files for the azure cosmos cv connector onto
2:10:56
for the azure cosmos cv connector onto
2:10:56
for the azure cosmos cv connector onto the connect instance
2:10:57
the connect instance
2:10:57
the connect instance within azure container services and
2:10:59
within azure container services and
2:10:59
within azure container services and we'll be able to monitor our services
2:11:02
we'll be able to monitor our services
2:11:02
we'll be able to monitor our services via the confluent control center
2:11:10
via the confluent control center
2:11:10
via the confluent control center to get started we'll need an account on
2:11:13
to get started we'll need an account on
2:11:13
to get started we'll need an account on azure
2:11:14
azure
2:11:14
azure azure cli tools and docker desktop
2:11:18
azure cli tools and docker desktop
2:11:18
azure cli tools and docker desktop our getting started script is available
2:11:19
our getting started script is available
2:11:20
our getting started script is available on github
2:11:21
on github
2:11:21
on github if you'd like to follow along you can go
2:11:22
if you'd like to follow along you can go
2:11:22
if you'd like to follow along you can go ahead and clone a repository
2:11:24
ahead and clone a repository
2:11:24
ahead and clone a repository there and within the repo is also a
2:11:27
there and within the repo is also a
2:11:27
there and within the repo is also a sample.emd file with our configurations
2:11:30
sample.emd file with our configurations
2:11:30
sample.emd file with our configurations and of course our yaml file once we're
2:11:33
and of course our yaml file once we're
2:11:33
and of course our yaml file once we're ready we'll need five docker commands to
2:11:35
ready we'll need five docker commands to
2:11:35
ready we'll need five docker commands to get up and running
2:11:38
get up and running
2:11:38
get up and running now before we take we get started let's
2:11:41
now before we take we get started let's
2:11:41
now before we take we get started let's take
2:11:42
take
2:11:42
take a quick look at our emv file so
2:11:45
a quick look at our emv file so
2:11:45
a quick look at our emv file so this file is going to contain the
2:11:46
this file is going to contain the
2:11:46
this file is going to contain the variables that will be referenced by our
2:11:48
variables that will be referenced by our
2:11:48
variables that will be referenced by our docker compose file
2:11:51
docker compose file
2:11:51
docker compose file so all i'm going to do is drop this into
2:11:53
so all i'm going to do is drop this into
2:11:53
so all i'm going to do is drop this into the project folder that docker is
2:11:54
the project folder that docker is
2:11:54
the project folder that docker is utilizing
2:11:56
utilizing
2:11:56
utilizing so of course we'll need to include our
2:11:58
so of course we'll need to include our
2:11:58
so of course we'll need to include our azure account details such as location
2:12:01
azure account details such as location
2:12:01
azure account details such as location subscription
2:12:01
subscription
2:12:02
subscription and resource group and then we'll go
2:12:03
and resource group and then we'll go
2:12:03
and resource group and then we'll go ahead and add in information for our
2:12:05
ahead and add in information for our
2:12:06
ahead and add in information for our cluster we'll need to provide the
2:12:07
cluster we'll need to provide the
2:12:08
cluster we'll need to provide the cluster bootstrap server information
2:12:10
cluster bootstrap server information
2:12:10
cluster bootstrap server information so whether that's on-prem or in the
2:12:13
so whether that's on-prem or in the
2:12:13
so whether that's on-prem or in the cloud
2:12:14
cloud
2:12:14
cloud we are using a confluent cloud cluster
2:12:16
we are using a confluent cloud cluster
2:12:16
we are using a confluent cloud cluster so
2:12:18
so
2:12:18
so we'll go ahead and put that url info
2:12:20
we'll go ahead and put that url info
2:12:20
we'll go ahead and put that url info there
2:12:21
there
2:12:21
there we'll also need to include the api key
2:12:23
we'll also need to include the api key
2:12:23
we'll also need to include the api key in secret
2:12:24
in secret
2:12:24
in secret and because we're using confluent cloud
2:12:26
and because we're using confluent cloud
2:12:26
and because we're using confluent cloud we're also going to be using
2:12:28
we're also going to be using
2:12:28
we're also going to be using schema registry on the cloud so we'll go
2:12:30
schema registry on the cloud so we'll go
2:12:30
schema registry on the cloud so we'll go ahead and
2:12:31
ahead and
2:12:31
ahead and add in the url info for that as well as
2:12:34
add in the url info for that as well as
2:12:34
add in the url info for that as well as the credentials
2:12:41
now to help you get up and running
2:12:42
now to help you get up and running
2:12:42
now to help you get up and running quickly
2:12:44
quickly
2:12:44
quickly we're going to go ahead and use code
2:12:45
we're going to go ahead and use code
2:12:45
we're going to go ahead and use code readily available in the confluence
2:12:47
readily available in the confluence
2:12:48
readily available in the confluence github repo
2:12:49
github repo
2:12:49
github repo so we've set up a separate directory
2:12:52
so we've set up a separate directory
2:12:52
so we've set up a separate directory confluent azure examples and that
2:12:56
confluent azure examples and that
2:12:56
confluent azure examples and that directory is maintained by my associate
2:12:59
directory is maintained by my associate
2:12:59
directory is maintained by my associate jean-luca natalie and he is
2:13:01
jean-luca natalie and he is
2:13:01
jean-luca natalie and he is fantastic he's an associate of mine
2:13:04
fantastic he's an associate of mine
2:13:04
fantastic he's an associate of mine in partner solutions and he's prepared a
2:13:06
in partner solutions and he's prepared a
2:13:06
in partner solutions and he's prepared a quick start to help you get started
2:13:08
quick start to help you get started
2:13:08
quick start to help you get started um he's also got some information out
2:13:10
um he's also got some information out
2:13:10
um he's also got some information out there about integrating with
2:13:12
there about integrating with
2:13:12
there about integrating with databricks and as we continue working
2:13:14
databricks and as we continue working
2:13:14
databricks and as we continue working with
2:13:15
with
2:13:15
with different solutions on the azure stack
2:13:18
different solutions on the azure stack
2:13:18
different solutions on the azure stack we'll be adding to that directory
2:13:30
we'll be adding to that directory
2:13:30
we'll be adding to that directory now we're ready to jump into our demo um
2:13:33
now we're ready to jump into our demo um
2:13:33
now we're ready to jump into our demo um first
2:13:33
first
2:13:33
first we're going to walk through creating a
2:13:35
we're going to walk through creating a
2:13:35
we're going to walk through creating a managed connector on complement cloud
2:13:38
managed connector on complement cloud
2:13:38
managed connector on complement cloud so that we can see how easy it is to
2:13:41
so that we can see how easy it is to
2:13:41
so that we can see how easy it is to interact with the
2:13:42
interact with the
2:13:42
interact with the interface there then we'll jump into
2:13:44
interface there then we'll jump into
2:13:44
interface there then we'll jump into creating the cosmos db connector
2:13:46
creating the cosmos db connector
2:13:46
creating the cosmos db connector on a self-managed instance
2:13:50
on a self-managed instance
2:13:50
on a self-managed instance so at the end of this demo we'll have
2:13:52
so at the end of this demo we'll have
2:13:52
so at the end of this demo we'll have confluent cloud on
2:13:53
confluent cloud on
2:13:53
confluent cloud on two containers and one with container is
2:13:57
two containers and one with container is
2:13:57
two containers and one with container is going to have control center
2:13:58
going to have control center
2:13:58
going to have control center and one container is going to have the
2:14:01
and one container is going to have the
2:14:01
and one container is going to have the connect instance
2:14:17
in this demo we'll go ahead and get
2:14:19
in this demo we'll go ahead and get
2:14:19
in this demo we'll go ahead and get started with creating
2:14:21
started with creating
2:14:21
started with creating a managed connector for azure data lake
2:14:24
a managed connector for azure data lake
2:14:24
a managed connector for azure data lake storage
2:14:26
storage
2:14:26
storage now azure data lake storage is a managed
2:14:28
now azure data lake storage is a managed
2:14:28
now azure data lake storage is a managed connector
2:14:30
connector
2:14:30
connector so we can go ahead and create this
2:14:32
so we can go ahead and create this
2:14:32
so we can go ahead and create this directly in the confluent cloud
2:14:34
directly in the confluent cloud
2:14:34
directly in the confluent cloud interface
2:14:36
interface
2:14:36
interface we can see the tile here for data link
2:14:38
we can see the tile here for data link
2:14:38
we can see the tile here for data link storage
2:14:39
storage
2:14:39
storage so we'll go ahead and select that i've
2:14:42
so we'll go ahead and select that i've
2:14:42
so we'll go ahead and select that i've already got a topic configured for the
2:14:44
already got a topic configured for the
2:14:44
already got a topic configured for the demo
2:14:48
so we'll select our topic
2:14:54
so we'll select our topic
2:14:54
so we'll select our topic a quick rename
2:15:01
the expected values are going to be in
2:15:04
the expected values are going to be in
2:15:04
the expected values are going to be in json
2:15:05
json
2:15:05
json i've also copied out the api
2:15:09
i've also copied out the api
2:15:09
i've also copied out the api key and credentials for my kafka cluster
2:15:14
key and credentials for my kafka cluster
2:15:14
key and credentials for my kafka cluster if you haven't already done so you can
2:15:17
if you haven't already done so you can
2:15:17
if you haven't already done so you can also generate
2:15:18
also generate
2:15:18
also generate this api can seek secret directly
2:15:22
this api can seek secret directly
2:15:22
this api can seek secret directly from the connector
2:15:25
from the connector
2:15:25
from the connector creation ui
2:15:29
i'm going to go ahead and default it to
2:15:32
i'm going to go ahead and default it to
2:15:32
i'm going to go ahead and default it to the topics directory i've also
2:15:36
the topics directory i've also
2:15:36
the topics directory i've also gone into the settings configuration
2:15:39
gone into the settings configuration
2:15:40
gone into the settings configuration for my storage account and
2:15:43
for my storage account and
2:15:43
for my storage account and i'll go ahead and add in my storage
2:15:46
i'll go ahead and add in my storage
2:15:46
i'll go ahead and add in my storage account name
2:15:47
account name
2:15:47
account name as well as the access key
2:15:52
that is needed to authenticate to my
2:15:55
that is needed to authenticate to my
2:15:56
that is needed to authenticate to my storage account
2:16:00
my output messages are also going to be
2:16:03
my output messages are also going to be
2:16:03
my output messages are also going to be in json format
2:16:09
my time interval is the time
2:16:12
my time interval is the time
2:16:12
my time interval is the time frequency that i'll be partitioning my
2:16:15
frequency that i'll be partitioning my
2:16:16
frequency that i'll be partitioning my data
2:16:17
data
2:16:17
data i'll go ahead and select hourly just so
2:16:20
i'll go ahead and select hourly just so
2:16:20
i'll go ahead and select hourly just so that
2:16:20
that
2:16:20
that i can more easily debug the initial set
2:16:24
i can more easily debug the initial set
2:16:24
i can more easily debug the initial set of data i'll go ahead and flush
2:16:29
of data i'll go ahead and flush
2:16:29
of data i'll go ahead and flush by default 1000 records
2:16:40
and i'll go ahead and set this to one
2:16:42
and i'll go ahead and set this to one
2:16:42
and i'll go ahead and set this to one task for this connector
2:16:54
the connector configuration's been
2:16:57
the connector configuration's been
2:16:57
the connector configuration's been validated
2:16:58
validated
2:16:58
validated and i'm good to go and we can see that
2:17:02
and i'm good to go and we can see that
2:17:02
and i'm good to go and we can see that we have
2:17:04
we have
2:17:04
we have the new connector in the provisioning
2:17:07
the new connector in the provisioning
2:17:07
the new connector in the provisioning state
2:17:09
state
2:17:09
state where it's kicking up one task
2:17:13
where it's kicking up one task
2:17:13
where it's kicking up one task now we can go into our storage account
2:17:16
now we can go into our storage account
2:17:16
now we can go into our storage account open up storage explorer
2:17:20
then open up our blob container and
2:17:24
then open up our blob container and
2:17:24
then open up our blob container and after our time interval has passed in
2:17:26
after our time interval has passed in
2:17:26
after our time interval has passed in this case one hour
2:17:28
this case one hour
2:17:28
this case one hour we will see the directory topics
2:17:33
and we will see the partitions for the
2:17:35
and we will see the partitions for the
2:17:35
and we will see the partitions for the data that we are passing in
2:17:37
data that we are passing in
2:17:37
data that we are passing in via the connector
2:17:43
so we can see how easily we can start
2:17:47
so we can see how easily we can start
2:17:47
so we can see how easily we can start taking data that's already sitting
2:17:50
taking data that's already sitting
2:17:50
taking data that's already sitting within the kafka ecosystem and
2:17:54
within the kafka ecosystem and
2:17:54
within the kafka ecosystem and share that data and spread that data
2:17:56
share that data and spread that data
2:17:56
share that data and spread that data across
2:17:57
across
2:17:57
across of systems in azure that we are trying
2:18:00
of systems in azure that we are trying
2:18:00
of systems in azure that we are trying to
2:18:00
to
2:18:00
to to build up and this is something that
2:18:03
to build up and this is something that
2:18:03
to build up and this is something that we can do
2:18:05
we can do
2:18:05
we can do very very dynamically um so that was
2:18:08
very very dynamically um so that was
2:18:08
very very dynamically um so that was a demo of configuring a managed
2:18:11
a demo of configuring a managed
2:18:11
a demo of configuring a managed connector
2:18:12
connector
2:18:12
connector and now that we have a little bit of
2:18:14
and now that we have a little bit of
2:18:14
and now that we have a little bit of exposure
2:18:15
exposure
2:18:15
exposure to the complement cloud ui and
2:18:19
to the complement cloud ui and
2:18:19
to the complement cloud ui and um and how easy it is to work with there
2:18:22
um and how easy it is to work with there
2:18:22
um and how easy it is to work with there we can go ahead and start looking at
2:18:24
we can go ahead and start looking at
2:18:24
we can go ahead and start looking at well what is it going to take
2:18:26
well what is it going to take
2:18:26
well what is it going to take to install a self-managed connector on
2:18:29
to install a self-managed connector on
2:18:29
to install a self-managed connector on azure
2:18:29
azure
2:18:29
azure and in this case we're going to be
2:18:31
and in this case we're going to be
2:18:31
and in this case we're going to be installing the cosmos db connector
2:18:35
installing the cosmos db connector
2:18:35
installing the cosmos db connector so compliment has a fantastic
2:18:38
so compliment has a fantastic
2:18:38
so compliment has a fantastic developer community and there's there's
2:18:41
developer community and there's there's
2:18:41
developer community and there's there's tons
2:18:41
tons
2:18:41
tons of docker scripts out there for
2:18:44
of docker scripts out there for
2:18:44
of docker scripts out there for installing the
2:18:45
installing the
2:18:45
installing the entire compliment platform um in this
2:18:48
entire compliment platform um in this
2:18:48
entire compliment platform um in this scenario what we're doing
2:18:49
scenario what we're doing
2:18:49
scenario what we're doing is we're just installing one instance of
2:18:52
is we're just installing one instance of
2:18:52
is we're just installing one instance of the connect
2:18:53
the connect
2:18:53
the connect and one instance of the control center
2:18:56
and one instance of the control center
2:18:56
and one instance of the control center so we can go ahead and get started
2:19:00
so we can go ahead and get started
2:19:00
so we can go ahead and get started with the github documentation for the cp
2:19:03
with the github documentation for the cp
2:19:03
with the github documentation for the cp all-in-one repo and in there you'll find
2:19:07
all-in-one repo and in there you'll find
2:19:07
all-in-one repo and in there you'll find um the yama files to to do the full
2:19:10
um the yama files to to do the full
2:19:10
um the yama files to to do the full compliment platform install um but in
2:19:13
compliment platform install um but in
2:19:13
compliment platform install um but in this
2:19:13
this
2:19:13
this instance what we did was we went ahead
2:19:15
instance what we did was we went ahead
2:19:15
instance what we did was we went ahead and stripped it down so that we have
2:19:17
and stripped it down so that we have
2:19:17
and stripped it down so that we have configurations for the control center
2:19:23
configurations for the connect
2:19:31
and what kafka wants you to do is it
2:19:33
and what kafka wants you to do is it
2:19:33
and what kafka wants you to do is it wants you to install the
2:19:34
wants you to install the
2:19:34
wants you to install the library files for the
2:19:38
library files for the
2:19:38
library files for the connectors onto the connect instance and
2:19:40
connectors onto the connect instance and
2:19:40
connectors onto the connect instance and do a quick bombs
2:19:41
do a quick bombs
2:19:41
do a quick bombs so instead of doing that manually we
2:19:44
so instead of doing that manually we
2:19:44
so instead of doing that manually we have
2:19:45
have
2:19:45
have a bash script at the bottom of the file
2:19:48
a bash script at the bottom of the file
2:19:48
a bash script at the bottom of the file and this is going to go ahead and pull
2:19:50
and this is going to go ahead and pull
2:19:50
and this is going to go ahead and pull down
2:19:51
down
2:19:51
down all the connectors that we want to
2:19:52
all the connectors that we want to
2:19:52
all the connectors that we want to install on our connect instance
2:19:54
install on our connect instance
2:19:54
install on our connect instance so in this case we're definitely going
2:19:56
so in this case we're definitely going
2:19:56
so in this case we're definitely going to want to install
2:19:58
to want to install
2:19:58
to want to install the cosmos db file
2:20:04
and we are planning on having um
2:20:07
and we are planning on having um
2:20:07
and we are planning on having um a repo where um by default we'll
2:20:11
a repo where um by default we'll
2:20:11
a repo where um by default we'll we'll just go ahead and provide a file
2:20:13
we'll just go ahead and provide a file
2:20:13
we'll just go ahead and provide a file that preloads all the
2:20:14
that preloads all the
2:20:14
that preloads all the the latest um azure service libraries
2:20:18
the latest um azure service libraries
2:20:18
the latest um azure service libraries so that people who are wanting to get
2:20:20
so that people who are wanting to get
2:20:20
so that people who are wanting to get started
2:20:21
started
2:20:21
started on azure really quickly will
2:20:24
on azure really quickly will
2:20:24
on azure really quickly will have this ability to to pre-load this
2:20:27
have this ability to to pre-load this
2:20:27
have this ability to to pre-load this script and launch it in their
2:20:28
script and launch it in their
2:20:28
script and launch it in their subscription
2:20:31
subscription
2:20:31
subscription now we did go ahead and pull out all the
2:20:33
now we did go ahead and pull out all the
2:20:34
now we did go ahead and pull out all the variables from the script so if we take
2:20:35
variables from the script so if we take
2:20:35
variables from the script so if we take a quick look at the emv
2:20:37
a quick look at the emv
2:20:37
a quick look at the emv file
2:20:42
we'll notice that we're passing in our
2:20:45
we'll notice that we're passing in our
2:20:45
we'll notice that we're passing in our subscription information
2:20:46
subscription information
2:20:46
subscription information um our bootstrap servers
2:20:49
um our bootstrap servers
2:20:49
um our bootstrap servers so this is going to be um for our
2:20:52
so this is going to be um for our
2:20:52
so this is going to be um for our compliment cloud instance
2:20:54
compliment cloud instance
2:20:54
compliment cloud instance so this is our kafka clusters running on
2:20:57
so this is our kafka clusters running on
2:20:57
so this is our kafka clusters running on complement cloud
2:21:00
we've got our api keys and our schema
2:21:03
we've got our api keys and our schema
2:21:03
we've got our api keys and our schema registry
2:21:04
registry
2:21:04
registry details as well and yes i will
2:21:08
details as well and yes i will
2:21:08
details as well and yes i will be deleting these api keys after this
2:21:10
be deleting these api keys after this
2:21:10
be deleting these api keys after this demo
2:21:11
demo
2:21:11
demo so um thank you for my data and security
2:21:14
so um thank you for my data and security
2:21:14
so um thank you for my data and security conscious friends
2:21:15
conscious friends
2:21:15
conscious friends out there i know you guys are looking
2:21:16
out there i know you guys are looking
2:21:16
out there i know you guys are looking out for me
2:21:21
so looking at what we have available in
2:21:24
so looking at what we have available in
2:21:24
so looking at what we have available in our portal and i've already gone ahead
2:21:27
our portal and i've already gone ahead
2:21:27
our portal and i've already gone ahead and
2:21:28
and
2:21:28
and run this script once just to make sure
2:21:30
run this script once just to make sure
2:21:30
run this script once just to make sure that we are good to go
2:21:32
that we are good to go
2:21:32
that we are good to go um so what we should see afterwards
2:21:35
um so what we should see afterwards
2:21:35
um so what we should see afterwards is we will see within azure container
2:21:38
is we will see within azure container
2:21:38
is we will see within azure container services
2:21:38
services
2:21:38
services one instance of the connect as well as
2:21:41
one instance of the connect as well as
2:21:41
one instance of the connect as well as one instance of the control center
2:21:44
one instance of the control center
2:21:44
one instance of the control center and what we're communicating with today
2:21:47
and what we're communicating with today
2:21:47
and what we're communicating with today is going to be
2:21:48
is going to be
2:21:48
is going to be our cosmos db instance and
2:21:51
our cosmos db instance and
2:21:51
our cosmos db instance and looking at the settings here we do have
2:21:53
looking at the settings here we do have
2:21:53
looking at the settings here we do have a partition key
2:21:55
a partition key
2:21:55
a partition key on the field id so now that we've taken
2:21:59
on the field id so now that we've taken
2:21:59
on the field id so now that we've taken a look at the components we can go ahead
2:22:02
a look at the components we can go ahead
2:22:02
a look at the components we can go ahead and
2:22:02
and
2:22:02
and get started with the demo
2:22:07
get started with the demo
2:22:08
get started with the demo once our containers are up and running
2:22:09
once our containers are up and running
2:22:09
once our containers are up and running we can access the control center
2:22:12
we can access the control center
2:22:12
we can access the control center let me get that restarted
2:22:18
once our containers are up and running
2:22:20
once our containers are up and running
2:22:20
once our containers are up and running we can access the control center
2:22:22
we can access the control center
2:22:22
we can access the control center via the ip address listed
2:22:26
via the ip address listed
2:22:26
via the ip address listed with port number nine zero two one go
2:22:29
with port number nine zero two one go
2:22:29
with port number nine zero two one go ahead and drop that
2:22:30
ahead and drop that
2:22:30
ahead and drop that into a browser and via the ui
2:22:34
into a browser and via the ui
2:22:34
into a browser and via the ui access our connect instance
2:22:40
now we can see the cosmos db sync
2:22:42
now we can see the cosmos db sync
2:22:42
now we can see the cosmos db sync connector tile here
2:22:44
connector tile here
2:22:44
connector tile here because we've already installed the
2:22:46
because we've already installed the
2:22:46
because we've already installed the library as needed
2:22:48
library as needed
2:22:48
library as needed if you don't see the sync connector tile
2:22:51
if you don't see the sync connector tile
2:22:51
if you don't see the sync connector tile then what you most likely need to do is
2:22:54
then what you most likely need to do is
2:22:54
then what you most likely need to do is install the libraries on your connect
2:22:56
install the libraries on your connect
2:22:56
install the libraries on your connect instance and of course restart connect
2:23:00
instance and of course restart connect
2:23:00
instance and of course restart connect and we're just going to go ahead and
2:23:02
and we're just going to go ahead and
2:23:02
and we're just going to go ahead and walk through these configurations
2:23:05
walk through these configurations
2:23:05
walk through these configurations now we are using data from the product
2:23:08
now we are using data from the product
2:23:08
now we are using data from the product inventory
2:23:09
inventory
2:23:09
inventory topic the name of my connector is cosmos
2:23:12
topic the name of my connector is cosmos
2:23:12
topic the name of my connector is cosmos db demo 3.
2:23:14
db demo 3.
2:23:14
db demo 3. i've configured it for one task and i'm
2:23:17
i've configured it for one task and i'm
2:23:17
i've configured it for one task and i'm using the avro converter
2:23:18
using the avro converter
2:23:18
using the avro converter it's important to note that you will
2:23:20
it's important to note that you will
2:23:20
it's important to note that you will need your cosmos db endpoint and
2:23:22
need your cosmos db endpoint and
2:23:22
need your cosmos db endpoint and connection key
2:23:24
connection key
2:23:24
connection key and then we'll go ahead and enter in our
2:23:27
and then we'll go ahead and enter in our
2:23:27
and then we'll go ahead and enter in our cosmos db database name
2:23:29
cosmos db database name
2:23:29
cosmos db database name which in this case is connect demo
2:23:32
which in this case is connect demo
2:23:32
which in this case is connect demo and the topic container map so what the
2:23:35
and the topic container map so what the
2:23:35
and the topic container map so what the top right container
2:23:36
top right container
2:23:36
top right container map is is the mapping between
2:23:40
map is is the mapping between
2:23:40
map is is the mapping between your topic and in this case
2:23:44
your topic and in this case
2:23:44
your topic and in this case we're pulling data from the streaming
2:23:47
we're pulling data from the streaming
2:23:47
we're pulling data from the streaming product
2:23:47
product
2:23:47
product inventory topic
2:23:51
inventory topic
2:23:51
inventory topic and the container within your cosmos db
2:23:55
and the container within your cosmos db
2:23:55
and the container within your cosmos db where you want the data to be synced
2:23:58
where you want the data to be synced
2:23:58
where you want the data to be synced to so in this case i do have
2:24:01
to so in this case i do have
2:24:02
to so in this case i do have a container called product that sits
2:24:05
a container called product that sits
2:24:05
a container called product that sits in the cosmos db
2:24:09
in the cosmos db
2:24:09
in the cosmos db which is called connect demo so the
2:24:11
which is called connect demo so the
2:24:11
which is called connect demo so the topic container map
2:24:13
topic container map
2:24:13
topic container map is just that linking between the product
2:24:16
is just that linking between the product
2:24:16
is just that linking between the product inventory topic and the product
2:24:19
inventory topic and the product
2:24:19
inventory topic and the product container
2:24:20
container
2:24:20
container within the cosmos db
2:24:23
within the cosmos db
2:24:23
within the cosmos db i'm going to go ahead and set cosmos
2:24:25
i'm going to go ahead and set cosmos
2:24:25
i'm going to go ahead and set cosmos upstart to falls
2:24:30
and these are additional values that
2:24:32
and these are additional values that
2:24:32
and these are additional values that you'll need to
2:24:33
you'll need to
2:24:34
you'll need to configure if you're using a schema
2:24:36
configure if you're using a schema
2:24:36
configure if you're using a schema registry
2:24:37
registry
2:24:37
registry that's cloud hosted
2:24:41
that's cloud hosted
2:24:41
that's cloud hosted now you do need the the url
2:24:44
now you do need the the url
2:24:44
now you do need the the url the username and the
2:24:48
the username and the
2:24:48
the username and the access key and
2:24:51
access key and
2:24:51
access key and we'll need to converter these two times
2:24:54
we'll need to converter these two times
2:24:54
we'll need to converter these two times one time for the value converter
2:24:56
one time for the value converter
2:24:56
one time for the value converter and one time for the key converter so
2:24:59
and one time for the key converter so
2:24:59
and one time for the key converter so good news
2:25:00
good news
2:25:00
good news we have successfully validated
2:25:03
we have successfully validated
2:25:03
we have successfully validated our configurations without any issues
2:25:06
our configurations without any issues
2:25:06
our configurations without any issues so we can go ahead and download a second
2:25:10
so we can go ahead and download a second
2:25:10
so we can go ahead and download a second connector config file here if we've made
2:25:13
connector config file here if we've made
2:25:13
connector config file here if we've made changes
2:25:14
changes
2:25:14
changes to the ui that we would like to preserve
2:25:17
to the ui that we would like to preserve
2:25:17
to the ui that we would like to preserve before we hit go
2:25:20
before we hit go
2:25:20
before we hit go or we can continue forward so we let's
2:25:22
or we can continue forward so we let's
2:25:22
or we can continue forward so we let's go ahead and continue forward
2:25:28
and we have a
2:25:31
now that our connector is running we can
2:25:34
now that our connector is running we can
2:25:34
now that our connector is running we can go ahead and take a look
2:25:35
go ahead and take a look
2:25:35
go ahead and take a look at our cosmos db instance
2:25:40
let's go to the product container and do
2:25:42
let's go to the product container and do
2:25:42
let's go to the product container and do a quick query
2:25:44
a quick query
2:25:44
a quick query against items we see that we now have
2:25:47
against items we see that we now have
2:25:47
against items we see that we now have data flowing in to the items container
2:25:51
data flowing in to the items container
2:25:51
data flowing in to the items container on cosmo cd congratulations
2:25:55
on cosmo cd congratulations
2:25:55
on cosmo cd congratulations you've created a sync connector and
2:25:57
you've created a sync connector and
2:25:57
you've created a sync connector and you're now
2:25:58
you're now
2:25:58
you're now inserting data into your cosmos db
2:26:04
instance
2:26:17
so thank you and as next steps i'd
2:26:20
so thank you and as next steps i'd
2:26:20
so thank you and as next steps i'd encourage you
2:26:21
encourage you
2:26:21
encourage you to visit confluent on microsoft docs and
2:26:24
to visit confluent on microsoft docs and
2:26:24
to visit confluent on microsoft docs and also
2:26:25
also
2:26:25
also you can join our community or visit our
2:26:28
you can join our community or visit our
2:26:28
you can join our community or visit our urslock channel or our forum at
2:26:32
urslock channel or our forum at
2:26:32
urslock channel or our forum at forum.com and
2:26:33
forum.com and
2:26:34
forum.com and you can look for my kaka and azure
2:26:36
you can look for my kaka and azure
2:26:36
you can look for my kaka and azure learning series on youtube
2:26:38
learning series on youtube
2:26:38
learning series on youtube and john lucas natalie's
2:26:41
and john lucas natalie's
2:26:41
and john lucas natalie's repository is at complement inc at
2:26:44
repository is at complement inc at
2:26:44
repository is at complement inc at complement azure
2:26:45
complement azure
2:26:45
complement azure examples and uh thank you for having me
2:26:48
examples and uh thank you for having me
2:26:48
examples and uh thank you for having me today it's been a blast and
2:26:50
today it's been a blast and
2:26:50
today it's been a blast and i love hearing about all the different
2:26:53
i love hearing about all the different
2:26:53
i love hearing about all the different ways that
2:26:54
ways that
2:26:54
ways that you can interact with real-time
2:26:56
you can interact with real-time
2:26:56
you can interact with real-time streaming
2:26:57
streaming
2:26:57
streaming scenarios and cosmos tv it's it's a
2:27:00
scenarios and cosmos tv it's it's a
2:27:00
scenarios and cosmos tv it's it's a great that
2:27:01
great that
2:27:01
great that thank you
2:27:09
thanks so much for the session for the
2:27:10
thanks so much for the session for the
2:27:10
thanks so much for the session for the demo and explaining how our customers
2:27:12
demo and explaining how our customers
2:27:12
demo and explaining how our customers can connect
2:27:13
can connect
2:27:13
can connect cosmo cp and kafka together using
2:27:16
cosmo cp and kafka together using
2:27:16
cosmo cp and kafka together using confluent
2:27:17
confluent
2:27:17
confluent i had a couple questions from our
2:27:18
i had a couple questions from our
2:27:18
i had a couple questions from our viewers here uh one of them is
2:27:21
viewers here uh one of them is
2:27:21
viewers here uh one of them is can you tell us more about some of the
2:27:22
can you tell us more about some of the
2:27:22
can you tell us more about some of the use cases and scenarios that this new
2:27:24
use cases and scenarios that this new
2:27:24
use cases and scenarios that this new connector
2:27:25
connector
2:27:25
connector has enabled that you've seen
2:27:28
has enabled that you've seen
2:27:28
has enabled that you've seen so the the connector just came out
2:27:31
so the the connector just came out
2:27:32
so the the connector just came out as of um last month so i i haven't had
2:27:35
as of um last month so i i haven't had
2:27:36
as of um last month so i i haven't had a lot of field time with it
2:27:39
a lot of field time with it
2:27:39
a lot of field time with it but we we do have a number of people who
2:27:42
but we we do have a number of people who
2:27:42
but we we do have a number of people who are looking
2:27:42
are looking
2:27:42
are looking at getting it into production scenarios
2:27:47
at getting it into production scenarios
2:27:48
at getting it into production scenarios in in the next month or so so i there
2:27:50
in in the next month or so so i there
2:27:50
in in the next month or so so i there are some retail customers
2:27:52
are some retail customers
2:27:52
are some retail customers um but given that it's just so new
2:27:56
um but given that it's just so new
2:27:56
um but given that it's just so new it's uh but um keep an eye out
2:27:59
it's uh but um keep an eye out
2:27:59
it's uh but um keep an eye out we'll be happy to share those success
2:28:01
we'll be happy to share those success
2:28:02
we'll be happy to share those success stories moving forward
2:28:03
stories moving forward
2:28:04
stories moving forward that sounds great we'd be happy to hear
2:28:05
that sounds great we'd be happy to hear
2:28:05
that sounds great we'd be happy to hear that feedback and just got uh
2:28:07
that feedback and just got uh
2:28:07
that feedback and just got uh one more question we're actually curious
2:28:09
one more question we're actually curious
2:28:09
one more question we're actually curious to learn more about what was the
2:28:10
to learn more about what was the
2:28:10
to learn more about what was the collaboration like between
2:28:12
collaboration like between
2:28:12
collaboration like between uh confluent and you mentioned a group
2:28:14
uh confluent and you mentioned a group
2:28:14
uh confluent and you mentioned a group in microsoft engineering to develop
2:28:16
in microsoft engineering to develop
2:28:16
in microsoft engineering to develop uh this connector together yeah so
2:28:19
uh this connector together yeah so
2:28:19
uh this connector together yeah so actually it was
2:28:20
actually it was
2:28:20
actually it was a lot quicker than i thought like i
2:28:22
a lot quicker than i thought like i
2:28:22
a lot quicker than i thought like i didn't realize
2:28:23
didn't realize
2:28:23
didn't realize you know and i should have known better
2:28:26
you know and i should have known better
2:28:26
you know and i should have known better but i didn't realize microsoft could
2:28:27
but i didn't realize microsoft could
2:28:27
but i didn't realize microsoft could move so quickly
2:28:28
move so quickly
2:28:28
move so quickly you know like with developing um just
2:28:31
you know like with developing um just
2:28:31
you know like with developing um just because there's so much going on
2:28:32
because there's so much going on
2:28:32
because there's so much going on in you know in the ecosystem and um i
2:28:36
in you know in the ecosystem and um i
2:28:36
in you know in the ecosystem and um i want to say we initiated
2:28:37
want to say we initiated
2:28:37
want to say we initiated uh calls back and forth between the two
2:28:40
uh calls back and forth between the two
2:28:40
uh calls back and forth between the two teams
2:28:41
teams
2:28:41
teams and um i want to say october and
2:28:44
and um i want to say october and
2:28:44
and um i want to say october and uh october november and
2:28:47
uh october november and
2:28:48
uh october november and and you know and within four months we
2:28:50
and you know and within four months we
2:28:50
and you know and within four months we they had a well not i but they had
2:28:52
they had a well not i but they had
2:28:52
they had a well not i but they had products out there available
2:28:54
products out there available
2:28:54
products out there available for use and in data and fully verified
2:28:56
for use and in data and fully verified
2:28:56
for use and in data and fully verified by somebody on our team
2:28:58
by somebody on our team
2:28:58
by somebody on our team so i i think it's great and i think that
2:29:01
so i i think it's great and i think that
2:29:01
so i i think it's great and i think that um
2:29:01
um
2:29:01
um it's really microsoft's ability to
2:29:03
it's really microsoft's ability to
2:29:03
it's really microsoft's ability to respond to
2:29:04
respond to
2:29:04
respond to to customer needs i think there's a lot
2:29:06
to customer needs i think there's a lot
2:29:06
to customer needs i think there's a lot of people who are looking
2:29:08
of people who are looking
2:29:08
of people who are looking to to work with cosmos tv and
2:29:11
to to work with cosmos tv and
2:29:11
to to work with cosmos tv and a lot of those people are also looking
2:29:13
a lot of those people are also looking
2:29:13
a lot of those people are also looking at leveraging the data that they have
2:29:15
at leveraging the data that they have
2:29:15
at leveraging the data that they have in in kafka so i i think you know
2:29:19
in in kafka so i i think you know
2:29:19
in in kafka so i i think you know to the community it's like kudos to you
2:29:21
to the community it's like kudos to you
2:29:21
to the community it's like kudos to you guys for providing feedback to the dev
2:29:23
guys for providing feedback to the dev
2:29:23
guys for providing feedback to the dev teams
2:29:24
teams
2:29:24
teams um because you know just immediately so
2:29:28
um because you know just immediately so
2:29:28
um because you know just immediately so we've also worked with the adls team and
2:29:31
we've also worked with the adls team and
2:29:31
we've also worked with the adls team and the adls team
2:29:32
the adls team
2:29:32
the adls team also developed their own kafka connector
2:29:36
also developed their own kafka connector
2:29:36
also developed their own kafka connector to be compatible with confluent and it's
2:29:38
to be compatible with confluent and it's
2:29:38
to be compatible with confluent and it's been through the verification process
2:29:41
been through the verification process
2:29:41
been through the verification process so it's it's been great working with the
2:29:44
so it's it's been great working with the
2:29:44
so it's it's been great working with the development teams
2:29:45
development teams
2:29:45
development teams and when we well when complement
2:29:48
and when we well when complement
2:29:48
and when we well when complement builds out the managed connectors that
2:29:51
builds out the managed connectors that
2:29:51
builds out the managed connectors that are managed and complement sources
2:29:53
are managed and complement sources
2:29:53
are managed and complement sources resources and integrating with azure
2:29:56
resources and integrating with azure
2:29:56
resources and integrating with azure services
2:29:57
services
2:29:57
services they work hand-in-hand with the
2:29:59
they work hand-in-hand with the
2:29:59
they work hand-in-hand with the development teams to optimize
2:30:01
development teams to optimize
2:30:01
development teams to optimize the managed service connectors so
2:30:04
the managed service connectors so
2:30:04
the managed service connectors so it's it's actually been like a fantastic
2:30:07
it's it's actually been like a fantastic
2:30:07
it's it's actually been like a fantastic collaboration
2:30:08
collaboration
2:30:08
collaboration and um i'm i'm hoping that we can work
2:30:11
and um i'm i'm hoping that we can work
2:30:11
and um i'm i'm hoping that we can work through as many of the services
2:30:13
through as many of the services
2:30:13
through as many of the services as we can so keep an eye out
2:30:17
that's awesome to hear uh thanks again
2:30:19
that's awesome to hear uh thanks again
2:30:19
that's awesome to hear uh thanks again alicia for being part of our
2:30:21
alicia for being part of our
2:30:21
alicia for being part of our very first azure cosmos db conference uh
2:30:24
very first azure cosmos db conference uh
2:30:24
very first azure cosmos db conference uh next and last we have last but certainly
2:30:26
next and last we have last but certainly
2:30:26
next and last we have last but certainly not least we have another speaker lined
2:30:28
not least we have another speaker lined
2:30:28
not least we have another speaker lined up
2:30:29
up
2:30:29
up and this is uh lenny lobol i'm not even
2:30:32
and this is uh lenny lobol i'm not even
2:30:32
and this is uh lenny lobol i'm not even sure if we need to introduce lenny
2:30:34
sure if we need to introduce lenny
2:30:34
sure if we need to introduce lenny uh he has been he has been around for
2:30:37
uh he has been he has been around for
2:30:37
uh he has been he has been around for such such a long time
2:30:38
such such a long time
2:30:38
such such a long time lenny a long time nbp for how long have
2:30:40
lenny a long time nbp for how long have
2:30:40
lenny a long time nbp for how long have you been an mvp
2:30:42
you been an mvp
2:30:42
you been an mvp uh it's coming up on a decade now thomas
2:30:44
uh it's coming up on a decade now thomas
2:30:44
uh it's coming up on a decade now thomas a decade now wow
2:30:47
a decade now wow
2:30:47
a decade now wow nice to see you too lenny lenny is also
2:30:49
nice to see you too lenny lenny is also
2:30:49
nice to see you too lenny lenny is also the cto and founder of
2:30:51
the cto and founder of
2:30:51
the cto and founder of sleek technologies uh based out of new
2:30:54
sleek technologies uh based out of new
2:30:54
sleek technologies uh based out of new york
2:30:55
york
2:30:55
york a major contributor for multiple
2:30:58
a major contributor for multiple
2:30:58
a major contributor for multiple azure data services the sql and cosmos
2:31:00
azure data services the sql and cosmos
2:31:00
azure data services the sql and cosmos db included
2:31:01
db included
2:31:01
db included the topic for today lenny is a topic
2:31:03
the topic for today lenny is a topic
2:31:03
the topic for today lenny is a topic that i think never gets old
2:31:05
that i think never gets old
2:31:05
that i think never gets old is data partitioning and modeling right
2:31:08
is data partitioning and modeling right
2:31:08
is data partitioning and modeling right indeed
2:31:09
indeed
2:31:09
indeed yeah all right so well thank you so much
2:31:12
yeah all right so well thank you so much
2:31:12
yeah all right so well thank you so much and uh it's great to see you all here at
2:31:14
and uh it's great to see you all here at
2:31:14
and uh it's great to see you all here at our first ever
2:31:15
our first ever
2:31:15
our first ever azure cosmos db conference my name
2:31:19
azure cosmos db conference my name
2:31:19
azure cosmos db conference my name again is lenny lobel i'm a data platform
2:31:21
again is lenny lobel i'm a data platform
2:31:21
again is lenny lobel i'm a data platform mvp
2:31:22
mvp
2:31:22
mvp um as tom has mentioned i've been
2:31:24
um as tom has mentioned i've been
2:31:24
um as tom has mentioned i've been working with cosmos dbo since
2:31:26
working with cosmos dbo since
2:31:26
working with cosmos dbo since the documentdb days back in 2014
2:31:29
the documentdb days back in 2014
2:31:29
the documentdb days back in 2014 and i'm here today to talk to you about
2:31:30
and i'm here today to talk to you about
2:31:30
and i'm here today to talk to you about data modeling and partitioning
2:31:33
data modeling and partitioning
2:31:33
data modeling and partitioning in cosmos db we all know that
2:31:35
in cosmos db we all know that
2:31:35
in cosmos db we all know that partitioning is very critical
2:31:37
partitioning is very critical
2:31:37
partitioning is very critical so we're going to be talking quite a bit
2:31:38
so we're going to be talking quite a bit
2:31:38
so we're going to be talking quite a bit about that of course cosmos db
2:31:40
about that of course cosmos db
2:31:40
about that of course cosmos db is many things um it is horizontally
2:31:43
is many things um it is horizontally
2:31:43
is many things um it is horizontally scalable
2:31:44
scalable
2:31:44
scalable it is non-relational and a bunch of
2:31:47
it is non-relational and a bunch of
2:31:47
it is non-relational and a bunch of other things it's
2:31:47
other things it's
2:31:48
other things it's uh we can define it as being globally
2:31:49
uh we can define it as being globally
2:31:49
uh we can define it as being globally distributed multi-model
2:31:52
distributed multi-model
2:31:52
distributed multi-model um tunable consistency there's lots of
2:31:54
um tunable consistency there's lots of
2:31:54
um tunable consistency there's lots of ways you can describe cosmo cv
2:31:55
ways you can describe cosmo cv
2:31:56
ways you can describe cosmo cv but for the purpose of this session
2:31:58
but for the purpose of this session
2:31:58
but for the purpose of this session let's just focus on these two
2:31:59
let's just focus on these two
2:31:59
let's just focus on these two definitions it is horizontally scalable
2:32:03
definitions it is horizontally scalable
2:32:03
definitions it is horizontally scalable and it is non-relational and drilling
2:32:05
and it is non-relational and drilling
2:32:05
and it is non-relational and drilling into both of those
2:32:07
into both of those
2:32:07
into both of those when you're dealing with cosmos eb right
2:32:09
when you're dealing with cosmos eb right
2:32:09
when you're dealing with cosmos eb right we're dealing with a container
2:32:10
we're dealing with a container
2:32:10
we're dealing with a container and we interact with one container as a
2:32:12
and we interact with one container as a
2:32:12
and we interact with one container as a single logical resource
2:32:14
single logical resource
2:32:14
single logical resource for storing documents and retrieving
2:32:16
for storing documents and retrieving
2:32:16
for storing documents and retrieving documents but of course behind that
2:32:17
documents but of course behind that
2:32:17
documents but of course behind that container there's a cluster
2:32:19
container there's a cluster
2:32:19
container there's a cluster of servers right if you will or physical
2:32:21
of servers right if you will or physical
2:32:21
of servers right if you will or physical partitions for lack of a better term
2:32:23
partitions for lack of a better term
2:32:23
partitions for lack of a better term we'll just think of them as
2:32:25
we'll just think of them as
2:32:25
we'll just think of them as physical machines just like a box that
2:32:28
physical machines just like a box that
2:32:28
physical machines just like a box that has
2:32:29
has
2:32:29
has storage and cpu processing power and
2:32:32
storage and cpu processing power and
2:32:32
storage and cpu processing power and there's been no real upper limit on the
2:32:33
there's been no real upper limit on the
2:32:33
there's been no real upper limit on the number of
2:32:34
number of
2:32:34
number of servers or physical partitions in a
2:32:36
servers or physical partitions in a
2:32:36
servers or physical partitions in a cluster behind a container and therefore
2:32:38
cluster behind a container and therefore
2:32:38
cluster behind a container and therefore if you partition your data model
2:32:40
if you partition your data model
2:32:40
if you partition your data model properly there's really no ability
2:32:42
properly there's really no ability
2:32:42
properly there's really no ability you have a unlimited storage and
2:32:44
you have a unlimited storage and
2:32:44
you have a unlimited storage and unlimited throughput
2:32:45
unlimited throughput
2:32:45
unlimited throughput with this model right because each
2:32:47
with this model right because each
2:32:47
with this model right because each physical partition offers
2:32:48
physical partition offers
2:32:48
physical partition offers more and more storage and throughput
2:32:52
more and more storage and throughput
2:32:52
more and more storage and throughput right what was the other term we were
2:32:54
right what was the other term we were
2:32:54
right what was the other term we were going to drill into
2:32:55
going to drill into
2:32:55
going to drill into non-relational and if we think about how
2:32:56
non-relational and if we think about how
2:32:56
non-relational and if we think about how we do things in the relational world
2:32:58
we do things in the relational world
2:32:58
we do things in the relational world where we're dealing with tables and rows
2:33:00
where we're dealing with tables and rows
2:33:00
where we're dealing with tables and rows what is it that we love to do more than
2:33:02
what is it that we love to do more than
2:33:02
what is it that we love to do more than anything else in this world right
2:33:03
anything else in this world right
2:33:03
anything else in this world right we love to join these things right
2:33:06
we love to join these things right
2:33:06
we love to join these things right that's what makes us happy
2:33:07
that's what makes us happy
2:33:07
that's what makes us happy we like to come up with the perfect
2:33:11
we like to come up with the perfect
2:33:11
we like to come up with the perfect normalized data model we like to craft
2:33:14
normalized data model we like to craft
2:33:14
normalized data model we like to craft our entities such that we have
2:33:16
our entities such that we have
2:33:16
our entities such that we have primary and foreign keys giving us a
2:33:18
primary and foreign keys giving us a
2:33:18
primary and foreign keys giving us a very normalized relational data model
2:33:21
very normalized relational data model
2:33:21
very normalized relational data model and that's the way things work
2:33:22
and that's the way things work
2:33:22
and that's the way things work in the relational world but of course in
2:33:25
in the relational world but of course in
2:33:25
in the relational world but of course in cosmos db and in the nosql world we are
2:33:27
cosmos db and in the nosql world we are
2:33:27
cosmos db and in the nosql world we are storing data inside of json documents
2:33:30
storing data inside of json documents
2:33:30
storing data inside of json documents not rows and while there's certainly
2:33:33
not rows and while there's certainly
2:33:33
not rows and while there's certainly nothing you can store in a row that you
2:33:35
nothing you can store in a row that you
2:33:35
nothing you can store in a row that you can't store in the json document and
2:33:37
can't store in the json document and
2:33:37
can't store in the json document and therefore
2:33:37
therefore
2:33:37
therefore nothing stopping you from designing a
2:33:39
nothing stopping you from designing a
2:33:39
nothing stopping you from designing a data model where you pretty much
2:33:41
data model where you pretty much
2:33:41
data model where you pretty much you know treat json documents as if they
2:33:44
you know treat json documents as if they
2:33:44
you know treat json documents as if they arose
2:33:44
arose
2:33:44
arose and have you know references from
2:33:47
and have you know references from
2:33:48
and have you know references from one related document to another um while
2:33:50
one related document to another um while
2:33:50
one related document to another um while you can do that
2:33:51
you can do that
2:33:52
you can do that such a design would break down in cosmos
2:33:55
such a design would break down in cosmos
2:33:55
such a design would break down in cosmos db this would this would
2:33:56
db this would this would
2:33:56
db this would this would perform very very poorly it could be
2:33:59
perform very very poorly it could be
2:33:59
perform very very poorly it could be made to work but it could not be made to
2:34:00
made to work but it could not be made to
2:34:00
made to work but it could not be made to work
2:34:01
work
2:34:01
work well and why is that it ties back to our
2:34:04
well and why is that it ties back to our
2:34:04
well and why is that it ties back to our first definition of being
2:34:05
first definition of being
2:34:06
first definition of being horizontally scalable the fact is that
2:34:08
horizontally scalable the fact is that
2:34:08
horizontally scalable the fact is that any one of these documents can land on
2:34:09
any one of these documents can land on
2:34:10
any one of these documents can land on any physical partition
2:34:11
any physical partition
2:34:11
any physical partition and in order to maintain the supreme
2:34:14
and in order to maintain the supreme
2:34:14
and in order to maintain the supreme level of performance uh the high
2:34:16
level of performance uh the high
2:34:16
level of performance uh the high performance
2:34:16
performance
2:34:16
performance slas and the single digital second
2:34:19
slas and the single digital second
2:34:19
slas and the single digital second reason rights and
2:34:20
reason rights and
2:34:20
reason rights and and all those guarantees on performance
2:34:21
and all those guarantees on performance
2:34:21
and all those guarantees on performance it's simply not practical or feasible to
2:34:24
it's simply not practical or feasible to
2:34:24
it's simply not practical or feasible to enforce relational constraints between
2:34:26
enforce relational constraints between
2:34:26
enforce relational constraints between related documents or indeed to even
2:34:28
related documents or indeed to even
2:34:28
related documents or indeed to even support
2:34:28
support
2:34:28
support joins on them and that's why we need to
2:34:31
joins on them and that's why we need to
2:34:31
joins on them and that's why we need to think about things differently
2:34:33
think about things differently
2:34:33
think about things differently because of course after that opening you
2:34:35
because of course after that opening you
2:34:35
because of course after that opening you know it certainly begs the question well
2:34:37
know it certainly begs the question well
2:34:37
know it certainly begs the question well then is cosmos to be at all suitable for
2:34:40
then is cosmos to be at all suitable for
2:34:40
then is cosmos to be at all suitable for a relational workload
2:34:41
a relational workload
2:34:41
a relational workload the answer is of course it is otherwise
2:34:43
the answer is of course it is otherwise
2:34:43
the answer is of course it is otherwise the session would be very short it would
2:34:44
the session would be very short it would
2:34:44
the session would be very short it would end here
2:34:45
end here
2:34:45
end here and i would go home uh but in fact uh
2:34:48
and i would go home uh but in fact uh
2:34:48
and i would go home uh but in fact uh you know
2:34:48
you know
2:34:48
you know many workloads are relational not all uh
2:34:51
many workloads are relational not all uh
2:34:51
many workloads are relational not all uh but many are
2:34:52
but many are
2:34:52
but many are and customers b is certainly suitable
2:34:54
and customers b is certainly suitable
2:34:54
and customers b is certainly suitable for those relational workloads it's just
2:34:56
for those relational workloads it's just
2:34:56
for those relational workloads it's just that you have to think about things
2:34:57
that you have to think about things
2:34:57
that you have to think about things differently
2:34:57
differently
2:34:57
differently you can't do things the way that you're
2:34:59
you can't do things the way that you're
2:34:59
you can't do things the way that you're going to cost
2:35:00
going to cost
2:35:00
going to cost become accustomed to doing them in the
2:35:02
become accustomed to doing them in the
2:35:02
become accustomed to doing them in the relational world even if some of those
2:35:04
relational world even if some of those
2:35:04
relational world even if some of those tried and true
2:35:05
tried and true
2:35:05
tried and true practices uh have been around with us
2:35:07
practices uh have been around with us
2:35:07
practices uh have been around with us for decades for some of us
2:35:09
for decades for some of us
2:35:09
for decades for some of us uh we have to kind of go against the
2:35:11
uh we have to kind of go against the
2:35:11
uh we have to kind of go against the grain of what we consider to be best
2:35:12
grain of what we consider to be best
2:35:12
grain of what we consider to be best practices in the relational world
2:35:14
practices in the relational world
2:35:14
practices in the relational world in order to materialize relationships in
2:35:17
in order to materialize relationships in
2:35:17
in order to materialize relationships in an efficient manner
2:35:19
an efficient manner
2:35:19
an efficient manner um in this non-relational world of
2:35:21
um in this non-relational world of
2:35:21
um in this non-relational world of cosmos db
2:35:23
cosmos db
2:35:23
cosmos db and so with that
2:35:26
and so with that
2:35:26
and so with that kind of backdrop set let's just have a
2:35:28
kind of backdrop set let's just have a
2:35:28
kind of backdrop set let's just have a look at this
2:35:30
look at this
2:35:30
look at this this little diagram here this is our web
2:35:32
this little diagram here this is our web
2:35:32
this little diagram here this is our web store relational model think of this as
2:35:33
store relational model think of this as
2:35:33
store relational model think of this as a
2:35:34
a
2:35:34
a a database behind our customer facing
2:35:37
a database behind our customer facing
2:35:37
a database behind our customer facing e-commerce website
2:35:38
e-commerce website
2:35:38
e-commerce website and uh it's a relational workload like a
2:35:40
and uh it's a relational workload like a
2:35:40
and uh it's a relational workload like a mini adventure works it's got
2:35:42
mini adventure works it's got
2:35:42
mini adventure works it's got all the things you would expect to find
2:35:43
all the things you would expect to find
2:35:43
all the things you would expect to find in a typical relational
2:35:45
in a typical relational
2:35:45
in a typical relational data model you've got customers with a
2:35:47
data model you've got customers with a
2:35:47
data model you've got customers with a one too many relationship
2:35:49
one too many relationship
2:35:49
one too many relationship with their addresses a one-to-one
2:35:51
with their addresses a one-to-one
2:35:51
with their addresses a one-to-one relationship with their password
2:35:53
relationship with their password
2:35:53
relationship with their password you've got products which are
2:35:56
you've got products which are
2:35:56
you've got products which are which participates in a parent child
2:35:58
which participates in a parent child
2:35:58
which participates in a parent child relationship with project category as
2:36:00
relationship with project category as
2:36:00
relationship with project category as well
2:36:00
well
2:36:00
well there are product tags in which case
2:36:03
there are product tags in which case
2:36:03
there are product tags in which case we've got a many-to-many relationship
2:36:05
we've got a many-to-many relationship
2:36:05
we've got a many-to-many relationship here
2:36:05
here
2:36:05
here so there's that intermediate table right
2:36:07
so there's that intermediate table right
2:36:07
so there's that intermediate table right this intermediate product
2:36:09
this intermediate product
2:36:09
this intermediate product tags table here
2:36:13
just trying to get the yeah trying to
2:36:15
just trying to get the yeah trying to
2:36:15
just trying to get the yeah trying to get my laser pointer to work right with
2:36:16
get my laser pointer to work right with
2:36:16
get my laser pointer to work right with foreign keys pointing to both sides of
2:36:18
foreign keys pointing to both sides of
2:36:18
foreign keys pointing to both sides of the many too many relationship
2:36:20
the many too many relationship
2:36:20
the many too many relationship and we've got sales orders right uh
2:36:22
and we've got sales orders right uh
2:36:22
and we've got sales orders right uh which are children to customers so a
2:36:24
which are children to customers so a
2:36:24
which are children to customers so a customer will have a set of sales orders
2:36:25
customer will have a set of sales orders
2:36:26
customer will have a set of sales orders each sales order will have a senate
2:36:27
each sales order will have a senate
2:36:27
each sales order will have a senate sales order details and it's a small
2:36:29
sales order details and it's a small
2:36:29
sales order details and it's a small data model but it is
2:36:31
data model but it is
2:36:31
data model but it is fairly representative of the most common
2:36:33
fairly representative of the most common
2:36:33
fairly representative of the most common uh
2:36:34
uh
2:36:34
uh data modeling patterns that you'll find
2:36:36
data modeling patterns that you'll find
2:36:36
data modeling patterns that you'll find in a relational database
2:36:39
in a relational database
2:36:39
in a relational database so if you're coming to cosmos to be
2:36:41
so if you're coming to cosmos to be
2:36:41
so if you're coming to cosmos to be fresh and you're just you know
2:36:42
fresh and you're just you know
2:36:42
fresh and you're just you know and it's new to you your first instinct
2:36:44
and it's new to you your first instinct
2:36:44
and it's new to you your first instinct might be to say hey i've got nine tables
2:36:46
might be to say hey i've got nine tables
2:36:46
might be to say hey i've got nine tables here
2:36:47
here
2:36:47
here maybe i should have non-containers right
2:36:50
maybe i should have non-containers right
2:36:50
maybe i should have non-containers right one container would make sense to store
2:36:52
one container would make sense to store
2:36:52
one container would make sense to store uh to think of a table as a container
2:36:53
uh to think of a table as a container
2:36:53
uh to think of a table as a container when you're first entering the world of
2:36:55
when you're first entering the world of
2:36:55
when you're first entering the world of cosmos eb
2:36:55
cosmos eb
2:36:55
cosmos eb but the question is is this a good
2:36:56
but the question is is this a good
2:36:56
but the question is is this a good design and of course the answer is no as
2:36:59
design and of course the answer is no as
2:36:59
design and of course the answer is no as i started
2:36:59
i started
2:36:59
i started explaining you know that due to the lack
2:37:01
explaining you know that due to the lack
2:37:01
explaining you know that due to the lack of uh
2:37:02
of uh
2:37:02
of uh joins and the fact that there is no
2:37:05
joins and the fact that there is no
2:37:05
joins and the fact that there is no referential integrity and there are no
2:37:06
referential integrity and there are no
2:37:06
referential integrity and there are no relational constraints
2:37:07
relational constraints
2:37:07
relational constraints um this could be made to work but could
2:37:10
um this could be made to work but could
2:37:10
um this could be made to work but could not be made to work well
2:37:11
not be made to work well
2:37:11
not be made to work well it would perform very very poorly would
2:37:13
it would perform very very poorly would
2:37:13
it would perform very very poorly would be actually the worst possible design
2:37:14
be actually the worst possible design
2:37:14
be actually the worst possible design that you could come up with
2:37:16
that you could come up with
2:37:16
that you could come up with so for the rest of this session we're
2:37:17
so for the rest of this session we're
2:37:17
so for the rest of this session we're going to take it step by step and see
2:37:18
going to take it step by step and see
2:37:18
going to take it step by step and see how to
2:37:19
how to
2:37:19
how to implement a data model like this in
2:37:21
implement a data model like this in
2:37:21
implement a data model like this in cosmos db
2:37:22
cosmos db
2:37:22
cosmos db uh for high efficiency high performance
2:37:24
uh for high efficiency high performance
2:37:24
uh for high efficiency high performance and high scale
2:37:26
and high scale
2:37:26
and high scale and we'll start with customers we've got
2:37:29
and we'll start with customers we've got
2:37:29
and we'll start with customers we've got three tables here for the customers
2:37:30
three tables here for the customers
2:37:30
three tables here for the customers the one one-to-many between customer and
2:37:32
the one one-to-many between customer and
2:37:32
the one one-to-many between customer and addresses and the one-to-one customer
2:37:34
addresses and the one-to-one customer
2:37:34
addresses and the one-to-one customer and password
2:37:34
and password
2:37:34
and password step one is you just literally translate
2:37:38
step one is you just literally translate
2:37:38
step one is you just literally translate your
2:37:39
your
2:37:39
your column names to property names as if it
2:37:41
column names to property names as if it
2:37:41
column names to property names as if it were a json document which is all i've
2:37:43
were a json document which is all i've
2:37:43
were a json document which is all i've done here
2:37:44
done here
2:37:44
done here other than camel casings uh the
2:37:47
other than camel casings uh the
2:37:47
other than camel casings uh the identifiers the property names in my
2:37:49
identifiers the property names in my
2:37:49
identifiers the property names in my json document
2:37:50
json document
2:37:50
json document match the column names in my database
2:37:53
match the column names in my database
2:37:53
match the column names in my database uh i had to rename the primary key
2:37:55
uh i had to rename the primary key
2:37:55
uh i had to rename the primary key columns simply as
2:37:56
columns simply as
2:37:56
columns simply as id because of course in cosmos db uh
2:37:59
id because of course in cosmos db uh
2:37:59
id because of course in cosmos db uh that that id property must always be
2:38:01
that that id property must always be
2:38:01
that that id property must always be named id but otherwise
2:38:02
named id but otherwise
2:38:02
named id but otherwise it's a simple translation forward and at
2:38:05
it's a simple translation forward and at
2:38:05
it's a simple translation forward and at that point i could stop there right
2:38:06
that point i could stop there right
2:38:06
that point i could stop there right because i do have these kind of
2:38:08
because i do have these kind of
2:38:08
because i do have these kind of and i don't want to use the word
2:38:09
and i don't want to use the word
2:38:09
and i don't want to use the word relationships here let's just say
2:38:10
relationships here let's just say
2:38:10
relationships here let's just say references i mean i do have ids pointing
2:38:13
references i mean i do have ids pointing
2:38:13
references i mean i do have ids pointing from one document to another um
2:38:16
from one document to another um
2:38:16
from one document to another um but let's just call them references not
2:38:18
but let's just call them references not
2:38:18
but let's just call them references not relationships because this is a
2:38:19
relationships because this is a
2:38:19
relationships because this is a non-relational data model
2:38:20
non-relational data model
2:38:20
non-relational data model and so it begs the question would it be
2:38:22
and so it begs the question would it be
2:38:22
and so it begs the question would it be better perhaps since we are working with
2:38:24
better perhaps since we are working with
2:38:24
better perhaps since we are working with json
2:38:25
json
2:38:25
json and json is semi-structured uh
2:38:28
and json is semi-structured uh
2:38:28
and json is semi-structured uh document format we could embed our
2:38:30
document format we could embed our
2:38:30
document format we could embed our addresses
2:38:31
addresses
2:38:31
addresses as an array as i've done here i only
2:38:33
as an array as i've done here i only
2:38:33
as an array as i've done here i only have space to show one address
2:38:34
have space to show one address
2:38:34
have space to show one address but those square brackets mean i could
2:38:36
but those square brackets mean i could
2:38:36
but those square brackets mean i could have any number of addresses
2:38:38
have any number of addresses
2:38:38
have any number of addresses or let's say a reasonable number of
2:38:39
or let's say a reasonable number of
2:38:40
or let's say a reasonable number of addresses as we'll get to in a moment
2:38:41
addresses as we'll get to in a moment
2:38:41
addresses as we'll get to in a moment and the password itself is embedded as
2:38:43
and the password itself is embedded as
2:38:43
and the password itself is embedded as an object using a curly brace
2:38:45
an object using a curly brace
2:38:45
an object using a curly brace and you know thereby uh you know you
2:38:48
and you know thereby uh you know you
2:38:48
and you know thereby uh you know you eliminate
2:38:49
eliminate
2:38:49
eliminate the problem of no joins if you pre-join
2:38:51
the problem of no joins if you pre-join
2:38:51
the problem of no joins if you pre-join right we have pre-joined
2:38:53
right we have pre-joined
2:38:53
right we have pre-joined effectively our addresses with the
2:38:55
effectively our addresses with the
2:38:55
effectively our addresses with the customers and the password of the
2:38:56
customers and the password of the
2:38:56
customers and the password of the customer
2:38:57
customer
2:38:57
customer and we've also solved the problem of
2:38:59
and we've also solved the problem of
2:38:59
and we've also solved the problem of referential integrity because
2:39:01
referential integrity because
2:39:01
referential integrity because the uh a child array lives
2:39:05
the uh a child array lives
2:39:05
the uh a child array lives nested within his parent and can never
2:39:06
nested within his parent and can never
2:39:06
nested within his parent and can never be broken from his parent right
2:39:08
be broken from his parent right
2:39:08
be broken from his parent right um so the question is is this
2:39:10
um so the question is is this
2:39:10
um so the question is is this appropriate to do this
2:39:12
appropriate to do this
2:39:12
appropriate to do this so do you embed or like so or
2:39:15
so do you embed or like so or
2:39:15
so do you embed or like so or do you reference like so and you need to
2:39:18
do you reference like so and you need to
2:39:18
do you reference like so and you need to you know this is a very typical question
2:39:20
you know this is a very typical question
2:39:20
you know this is a very typical question that you ask during
2:39:21
that you ask during
2:39:21
that you ask during this process so the rules aren't hard
2:39:24
this process so the rules aren't hard
2:39:24
this process so the rules aren't hard and fast but the general rules are
2:39:26
and fast but the general rules are
2:39:26
and fast but the general rules are if you've got a one-to-one relationship
2:39:27
if you've got a one-to-one relationship
2:39:27
if you've got a one-to-one relationship like you have in the case of a password
2:39:29
like you have in the case of a password
2:39:29
like you have in the case of a password or you want to few relationship like you
2:39:32
or you want to few relationship like you
2:39:32
or you want to few relationship like you have in the case of addresses
2:39:33
have in the case of addresses
2:39:33
have in the case of addresses then it would be reasonable to embed one
2:39:35
then it would be reasonable to embed one
2:39:35
then it would be reasonable to embed one just want to few
2:39:37
just want to few
2:39:37
just want to few you know in the relational world there's
2:39:38
you know in the relational world there's
2:39:38
you know in the relational world there's no such thing it's either one to one
2:39:40
no such thing it's either one to one
2:39:40
no such thing it's either one to one or it's one to many and even if it's two
2:39:43
or it's one to many and even if it's two
2:39:43
or it's one to many and even if it's two or more
2:39:43
or more
2:39:43
or more it's considered one to many um in this
2:39:46
it's considered one to many um in this
2:39:46
it's considered one to many um in this world we are thinking of a data model
2:39:48
world we are thinking of a data model
2:39:48
world we are thinking of a data model where we have a json document with an
2:39:49
where we have a json document with an
2:39:49
where we have a json document with an upper limit size
2:39:50
upper limit size
2:39:50
upper limit size of two megabytes whereas it's perfectly
2:39:53
of two megabytes whereas it's perfectly
2:39:53
of two megabytes whereas it's perfectly reasonable to
2:39:54
reasonable to
2:39:54
reasonable to you know cap the number of addresses
2:39:55
you know cap the number of addresses
2:39:55
you know cap the number of addresses that a customer might have in their
2:39:57
that a customer might have in their
2:39:57
that a customer might have in their profile to all fit into megabytes
2:39:59
profile to all fit into megabytes
2:39:59
profile to all fit into megabytes and so it would make that's a one to few
2:40:01
and so it would make that's a one to few
2:40:01
and so it would make that's a one to few think of their orders
2:40:02
think of their orders
2:40:02
think of their orders we want to think of orders as
2:40:03
we want to think of orders as
2:40:03
we want to think of orders as one-to-many don't we we want lots and
2:40:05
one-to-many don't we we want lots and
2:40:05
one-to-many don't we we want lots and lots of orders
2:40:06
lots of orders
2:40:06
lots of orders ideally an unbounded number of orders
2:40:09
ideally an unbounded number of orders
2:40:09
ideally an unbounded number of orders and so it's certainly not feasible to
2:40:12
and so it's certainly not feasible to
2:40:12
and so it's certainly not feasible to embed all of that in a single document
2:40:13
embed all of that in a single document
2:40:13
embed all of that in a single document and so then you'll reference when it's a
2:40:15
and so then you'll reference when it's a
2:40:15
and so then you'll reference when it's a one-to-many relationship particularly
2:40:16
one-to-many relationship particularly
2:40:16
one-to-many relationship particularly unbounded
2:40:17
unbounded
2:40:17
unbounded relationships you'll go with referencing
2:40:19
relationships you'll go with referencing
2:40:19
relationships you'll go with referencing or if you have a many-to-many
2:40:20
or if you have a many-to-many
2:40:20
or if you have a many-to-many relationship as you'll see
2:40:22
relationship as you'll see
2:40:22
relationship as you'll see the way you will handle the product tags
2:40:24
the way you will handle the product tags
2:40:24
the way you will handle the product tags relationship
2:40:25
relationship
2:40:25
relationship you'll also want to reference actually a
2:40:27
you'll also want to reference actually a
2:40:27
you'll also want to reference actually a combination of embedding and referencing
2:40:29
combination of embedding and referencing
2:40:29
combination of embedding and referencing and another kind of general rule is that
2:40:31
and another kind of general rule is that
2:40:32
and another kind of general rule is that you know even if it's just one to one
2:40:33
you know even if it's just one to one
2:40:33
you know even if it's just one to one and even if it's uh or just one to few
2:40:36
and even if it's uh or just one to few
2:40:36
and even if it's uh or just one to few um if you're never really querying for
2:40:38
um if you're never really querying for
2:40:38
um if you're never really querying for these things together and you're never
2:40:39
these things together and you're never
2:40:39
these things together and you're never really updating them together
2:40:40
really updating them together
2:40:40
really updating them together it also makes sense to reference that um
2:40:44
it also makes sense to reference that um
2:40:44
it also makes sense to reference that um if they're being queried or updated
2:40:46
if they're being queried or updated
2:40:46
if they're being queried or updated independently again these aren't hard
2:40:47
independently again these aren't hard
2:40:47
independently again these aren't hard and fast rules
2:40:48
and fast rules
2:40:48
and fast rules but for our scenario here we can
2:40:50
but for our scenario here we can
2:40:50
but for our scenario here we can certainly go with embedding for the
2:40:52
certainly go with embedding for the
2:40:52
certainly go with embedding for the addresses and for the password and turn
2:40:53
addresses and for the password and turn
2:40:53
addresses and for the password and turn three tables into one essentially
2:40:56
three tables into one essentially
2:40:56
three tables into one essentially uh by encapsulating all that information
2:40:58
uh by encapsulating all that information
2:40:58
uh by encapsulating all that information in our customer entity
2:41:00
in our customer entity
2:41:00
in our customer entity and then the question at that point
2:41:01
and then the question at that point
2:41:02
and then the question at that point becomes if we're going to store customer
2:41:03
becomes if we're going to store customer
2:41:04
becomes if we're going to store customer documents like so
2:41:05
documents like so
2:41:05
documents like so inside of a customer container how do we
2:41:07
inside of a customer container how do we
2:41:07
inside of a customer container how do we partition that container what is our
2:41:09
partition that container what is our
2:41:09
partition that container what is our partition key
2:41:10
partition key
2:41:10
partition key the most critical question that you'll
2:41:11
the most critical question that you'll
2:41:11
the most critical question that you'll always have to ask when defining a
2:41:14
always have to ask when defining a
2:41:14
always have to ask when defining a container
2:41:16
container
2:41:16
container well the way that you come up with the
2:41:18
well the way that you come up with the
2:41:18
well the way that you come up with the right partition key is you need to
2:41:19
right partition key is you need to
2:41:19
right partition key is you need to really understand the most common
2:41:21
really understand the most common
2:41:21
really understand the most common querying patterns you need to understand
2:41:22
querying patterns you need to understand
2:41:22
querying patterns you need to understand how your application is going to most
2:41:24
how your application is going to most
2:41:24
how your application is going to most frequently
2:41:25
frequently
2:41:25
frequently query your data how your users are going
2:41:27
query your data how your users are going
2:41:27
query your data how your users are going to most frequently query your data
2:41:29
to most frequently query your data
2:41:29
to most frequently query your data and again sticking with this uh scenario
2:41:31
and again sticking with this uh scenario
2:41:31
and again sticking with this uh scenario of a
2:41:33
of a
2:41:33
of a customer-facing e-commerce website um
2:41:36
customer-facing e-commerce website um
2:41:36
customer-facing e-commerce website um the most common query is that
2:41:37
the most common query is that
2:41:37
the most common query is that we're going to a customer is going to
2:41:38
we're going to a customer is going to
2:41:38
we're going to a customer is going to want to retrieve information about that
2:41:41
want to retrieve information about that
2:41:41
want to retrieve information about that you know their profile information which
2:41:42
you know their profile information which
2:41:42
you know their profile information which is their name address all their
2:41:43
is their name address all their
2:41:44
is their name address all their addresses their password
2:41:45
addresses their password
2:41:45
addresses their password perhaps for rendering on a profile page
2:41:48
perhaps for rendering on a profile page
2:41:48
perhaps for rendering on a profile page and so the most common query is really
2:41:50
and so the most common query is really
2:41:50
and so the most common query is really to get an individual customer buy their
2:41:51
to get an individual customer buy their
2:41:52
to get an individual customer buy their id which would pull in their addresses
2:41:53
id which would pull in their addresses
2:41:53
id which would pull in their addresses right because they're embedded and their
2:41:55
right because they're embedded and their
2:41:55
right because they're embedded and their password because it's embedded
2:41:56
password because it's embedded
2:41:56
password because it's embedded right and so that really means the id
2:41:59
right and so that really means the id
2:41:59
right and so that really means the id property itself
2:42:00
property itself
2:42:00
property itself would be actually the most suitable
2:42:03
would be actually the most suitable
2:42:03
would be actually the most suitable property for the partition key here
2:42:05
property for the partition key here
2:42:05
property for the partition key here now it might seem like a strange choice
2:42:07
now it might seem like a strange choice
2:42:07
now it might seem like a strange choice because as you may or may not know every
2:42:09
because as you may or may not know every
2:42:09
because as you may or may not know every document is unique in a container
2:42:11
document is unique in a container
2:42:11
document is unique in a container based on a combination of this id
2:42:13
based on a combination of this id
2:42:14
based on a combination of this id property here
2:42:14
property here
2:42:14
property here and whatever other property you decide
2:42:17
and whatever other property you decide
2:42:17
and whatever other property you decide to choose
2:42:18
to choose
2:42:18
to choose for the partition key however if you
2:42:21
for the partition key however if you
2:42:21
for the partition key however if you choose the id property
2:42:22
choose the id property
2:42:22
choose the id property itself as the partition key then you get
2:42:25
itself as the partition key then you get
2:42:25
itself as the partition key then you get exact uniqueness
2:42:26
exact uniqueness
2:42:26
exact uniqueness of one document per logical partition
2:42:29
of one document per logical partition
2:42:29
of one document per logical partition and you can't have more
2:42:31
and you can't have more
2:42:31
and you can't have more now again it might seem like an odd
2:42:32
now again it might seem like an odd
2:42:32
now again it might seem like an odd pattern but it is actually a very good
2:42:34
pattern but it is actually a very good
2:42:34
pattern but it is actually a very good approach when you're dealing with
2:42:35
approach when you're dealing with
2:42:35
approach when you're dealing with point operations point reads and point
2:42:37
point operations point reads and point
2:42:37
point operations point reads and point rights where your primary use case is to
2:42:39
rights where your primary use case is to
2:42:39
rights where your primary use case is to read
2:42:39
read
2:42:40
read or write an individual document in time
2:42:41
or write an individual document in time
2:42:42
or write an individual document in time the id property is often a very good
2:42:43
the id property is often a very good
2:42:43
the id property is often a very good choice
2:42:44
choice
2:42:44
choice and it certainly satisfies our most
2:42:45
and it certainly satisfies our most
2:42:45
and it certainly satisfies our most common query so that's what we'll go
2:42:47
common query so that's what we'll go
2:42:47
common query so that's what we'll go with
2:42:48
with
2:42:48
with now let's plow ahead with our next
2:42:50
now let's plow ahead with our next
2:42:50
now let's plow ahead with our next entity in the data model which is
2:42:51
entity in the data model which is
2:42:51
entity in the data model which is product categories this is very very
2:42:53
product categories this is very very
2:42:53
product categories this is very very simple
2:42:53
simple
2:42:54
simple right there's a product category id and
2:42:55
right there's a product category id and
2:42:55
right there's a product category id and a display name so
2:42:57
a display name so
2:42:57
a display name so like before we translate our column
2:43:00
like before we translate our column
2:43:00
like before we translate our column names into
2:43:01
names into
2:43:01
names into json property names and then
2:43:05
json property names and then
2:43:05
json property names and then decide to put that in a container for
2:43:07
decide to put that in a container for
2:43:07
decide to put that in a container for the product category we'll put that in
2:43:08
the product category we'll put that in
2:43:08
the product category we'll put that in the product category container
2:43:10
the product category container
2:43:10
the product category container ask the same question how do we
2:43:11
ask the same question how do we
2:43:11
ask the same question how do we partition it answer it the same way
2:43:13
partition it answer it the same way
2:43:13
partition it answer it the same way what's your most common querying pattern
2:43:15
what's your most common querying pattern
2:43:15
what's your most common querying pattern on that on that website
2:43:17
on that on that website
2:43:17
on that on that website you're going to want to display all the
2:43:19
you're going to want to display all the
2:43:19
you're going to want to display all the product categories right maybe in a
2:43:20
product categories right maybe in a
2:43:20
product categories right maybe in a sidebar on the left or a little menu
2:43:22
sidebar on the left or a little menu
2:43:22
sidebar on the left or a little menu so the user can click a product category
2:43:24
so the user can click a product category
2:43:24
so the user can click a product category to drill in when they find a category of
2:43:26
to drill in when they find a category of
2:43:26
to drill in when they find a category of interest
2:43:27
interest
2:43:27
interest to see the products in that category and
2:43:28
to see the products in that category and
2:43:28
to see the products in that category and that really means a select statement
2:43:30
that really means a select statement
2:43:30
that really means a select statement with no where clause because you want
2:43:31
with no where clause because you want
2:43:31
with no where clause because you want them all
2:43:32
them all
2:43:32
them all and the way that you implement this type
2:43:34
and the way that you implement this type
2:43:34
and the way that you implement this type of pattern in cosmos db
2:43:36
of pattern in cosmos db
2:43:36
of pattern in cosmos db is to add a type property really i want
2:43:39
is to add a type property really i want
2:43:39
is to add a type property really i want all of them inside one logical partition
2:43:42
all of them inside one logical partition
2:43:42
all of them inside one logical partition think of this as a lookup list anything
2:43:43
think of this as a lookup list anything
2:43:43
think of this as a lookup list anything that's a reasonable size when i say
2:43:45
that's a reasonable size when i say
2:43:45
that's a reasonable size when i say reasonable size i mean it'll fit inside
2:43:46
reasonable size i mean it'll fit inside
2:43:46
reasonable size i mean it'll fit inside of a
2:43:47
of a
2:43:47
of a single logical partition which is
2:43:49
single logical partition which is
2:43:49
single logical partition which is limited to 20 gigabytes in size and
2:43:51
limited to 20 gigabytes in size and
2:43:51
limited to 20 gigabytes in size and we certainly don't have a list of
2:43:52
we certainly don't have a list of
2:43:52
we certainly don't have a list of product categories coming anywhere near
2:43:54
product categories coming anywhere near
2:43:54
product categories coming anywhere near that
2:43:55
that
2:43:55
that so there's no problem at all putting
2:43:57
so there's no problem at all putting
2:43:57
so there's no problem at all putting them all inside one single
2:43:58
them all inside one single
2:43:58
them all inside one single logical partition and the way we do this
2:44:00
logical partition and the way we do this
2:44:00
logical partition and the way we do this is we add a type process we add a type
2:44:02
is we add a type process we add a type
2:44:02
is we add a type process we add a type property which is certainly optional
2:44:04
property which is certainly optional
2:44:04
property which is certainly optional you know it's schema-free you're free to
2:44:06
you know it's schema-free you're free to
2:44:06
you know it's schema-free you're free to define any schema the id property is the
2:44:07
define any schema the id property is the
2:44:08
define any schema the id property is the only required property
2:44:09
only required property
2:44:09
only required property but it's always a good practice to throw
2:44:10
but it's always a good practice to throw
2:44:10
but it's always a good practice to throw in that type property
2:44:12
in that type property
2:44:12
in that type property to help unambiguously distinguish one
2:44:15
to help unambiguously distinguish one
2:44:16
to help unambiguously distinguish one type of document from another
2:44:17
type of document from another
2:44:17
type of document from another and if you create a property call type
2:44:20
and if you create a property call type
2:44:20
and if you create a property call type and you set the value of that property
2:44:21
and you set the value of that property
2:44:21
and you set the value of that property to category for every product category
2:44:23
to category for every product category
2:44:23
to category for every product category document
2:44:24
document
2:44:24
document and then partition on the type property
2:44:26
and then partition on the type property
2:44:26
and then partition on the type property of course that's gonna
2:44:27
of course that's gonna
2:44:27
of course that's gonna lump all of them into one logical
2:44:29
lump all of them into one logical
2:44:29
lump all of them into one logical partition and give you a very efficient
2:44:30
partition and give you a very efficient
2:44:30
partition and give you a very efficient result
2:44:31
result
2:44:31
result a very efficient query that results in
2:44:33
a very efficient query that results in
2:44:33
a very efficient query that results in returning all of the categories
2:44:35
returning all of the categories
2:44:35
returning all of the categories that you have in the database all in a
2:44:37
that you have in the database all in a
2:44:37
that you have in the database all in a single
2:44:38
single
2:44:38
single partition query product tags i don't
2:44:41
partition query product tags i don't
2:44:41
partition query product tags i don't rehash that because it's the same
2:44:42
rehash that because it's the same
2:44:42
rehash that because it's the same concept
2:44:43
concept
2:44:43
concept a simple key value pair where the user
2:44:45
a simple key value pair where the user
2:44:45
a simple key value pair where the user might want to see a list of tags
2:44:46
might want to see a list of tags
2:44:46
might want to see a list of tags they see a tag of interest they'll want
2:44:48
they see a tag of interest they'll want
2:44:48
they see a tag of interest they'll want to see all the properties associated
2:44:49
to see all the properties associated
2:44:49
to see all the properties associated with that tag
2:44:50
with that tag
2:44:50
with that tag same concept translate the column names
2:44:53
same concept translate the column names
2:44:53
same concept translate the column names to property names
2:44:54
to property names
2:44:54
to property names put them in a tag container ask yourself
2:44:56
put them in a tag container ask yourself
2:44:56
put them in a tag container ask yourself how to partition it and answer it with
2:44:58
how to partition it and answer it with
2:44:58
how to partition it and answer it with the same answer
2:44:59
the same answer
2:44:59
the same answer we want to get all the product tags it's
2:45:01
we want to get all the product tags it's
2:45:01
we want to get all the product tags it's a select statement that returns them all
2:45:03
a select statement that returns them all
2:45:03
a select statement that returns them all so we'll add that type property only for
2:45:06
so we'll add that type property only for
2:45:06
so we'll add that type property only for the tag documents we'll set the value to
2:45:08
the tag documents we'll set the value to
2:45:08
the tag documents we'll set the value to tag right and then by partitioning on
2:45:10
tag right and then by partitioning on
2:45:10
tag right and then by partitioning on the type property all of the
2:45:12
the type property all of the
2:45:12
the type property all of the product tag documents are stored
2:45:13
product tag documents are stored
2:45:14
product tag documents are stored together in the same logical partition
2:45:15
together in the same logical partition
2:45:15
together in the same logical partition just like categories
2:45:17
just like categories
2:45:17
just like categories products is where it gets more
2:45:18
products is where it gets more
2:45:18
products is where it gets more interesting
2:45:20
interesting
2:45:20
interesting with products we have that many-to-many
2:45:22
with products we have that many-to-many
2:45:22
with products we have that many-to-many relationship right
2:45:24
relationship right
2:45:24
relationship right so first we'll do step one we'll
2:45:25
so first we'll do step one we'll
2:45:25
so first we'll do step one we'll translate our column names the property
2:45:26
translate our column names the property
2:45:26
translate our column names the property names
2:45:27
names
2:45:27
names but what about this many-to-many
2:45:28
but what about this many-to-many
2:45:28
but what about this many-to-many relationship with product tags
2:45:30
relationship with product tags
2:45:30
relationship with product tags the way this is implemented and i
2:45:31
the way this is implemented and i
2:45:31
the way this is implemented and i mentioned this earlier is with a
2:45:32
mentioned this earlier is with a
2:45:32
mentioned this earlier is with a combination of embedding and referencing
2:45:35
combination of embedding and referencing
2:45:35
combination of embedding and referencing i am embedding an array of tag ids
2:45:39
i am embedding an array of tag ids
2:45:39
i am embedding an array of tag ids into the product document but that array
2:45:41
into the product document but that array
2:45:41
into the product document but that array itself is an array of references
2:45:42
itself is an array of references
2:45:42
itself is an array of references to tags and in this case i am embedding
2:45:45
to tags and in this case i am embedding
2:45:45
to tags and in this case i am embedding a an array of tag ids into each product
2:45:47
a an array of tag ids into each product
2:45:47
a an array of tag ids into each product document
2:45:47
document
2:45:48
document this could also be made to work by
2:45:49
this could also be made to work by
2:45:49
this could also be made to work by embedding an array of product ids in
2:45:51
embedding an array of product ids in
2:45:51
embedding an array of product ids in each tag document
2:45:53
each tag document
2:45:53
each tag document depending on your usage patterns one of
2:45:55
depending on your usage patterns one of
2:45:55
depending on your usage patterns one of those may be better
2:45:57
those may be better
2:45:57
those may be better than the other but in either case this
2:45:58
than the other but in either case this
2:45:58
than the other but in either case this is how you implement the many to many
2:46:00
is how you implement the many to many
2:46:00
is how you implement the many to many relationships
2:46:01
relationships
2:46:01
relationships now how do we partition the products
2:46:03
now how do we partition the products
2:46:03
now how do we partition the products we're sticking with our
2:46:04
we're sticking with our
2:46:04
we're sticking with our ecommerce website scenario the user has
2:46:06
ecommerce website scenario the user has
2:46:06
ecommerce website scenario the user has found a category of interest they want
2:46:08
found a category of interest they want
2:46:08
found a category of interest they want to get all the products in that category
2:46:10
to get all the products in that category
2:46:10
to get all the products in that category so the most common query
2:46:11
so the most common query
2:46:11
so the most common query products is going to be get all the
2:46:13
products is going to be get all the
2:46:13
products is going to be get all the products within a given category
2:46:14
products within a given category
2:46:14
products within a given category not all the products in the entire
2:46:16
not all the products in the entire
2:46:16
not all the products in the entire database
2:46:17
database
2:46:17
database so that means we want all of our product
2:46:19
so that means we want all of our product
2:46:19
so that means we want all of our product documents to be kind of grouped together
2:46:21
documents to be kind of grouped together
2:46:21
documents to be kind of grouped together logically in a logical partition by
2:46:23
logically in a logical partition by
2:46:23
logically in a logical partition by category
2:46:23
category
2:46:23
category which you guessed it if you were
2:46:25
which you guessed it if you were
2:46:25
which you guessed it if you were thinking category id is the proper
2:46:27
thinking category id is the proper
2:46:27
thinking category id is the proper property to use here you guessed
2:46:28
property to use here you guessed
2:46:28
property to use here you guessed correctly and that will achieve
2:46:30
correctly and that will achieve
2:46:30
correctly and that will achieve just that so now i've got
2:46:33
just that so now i've got
2:46:33
just that so now i've got my product documents in hand but if you
2:46:36
my product documents in hand but if you
2:46:36
my product documents in hand but if you look at a product document what do you
2:46:37
look at a product document what do you
2:46:37
look at a product document what do you have inside
2:46:37
have inside
2:46:38
have inside of a product document you have a bunch
2:46:39
of a product document you have a bunch
2:46:39
of a product document you have a bunch of ids but you don't really have
2:46:40
of ids but you don't really have
2:46:40
of ids but you don't really have everything that you need
2:46:41
everything that you need
2:46:41
everything that you need to display the product on your web page
2:46:44
to display the product on your web page
2:46:44
to display the product on your web page right you have the category id but not
2:46:45
right you have the category id but not
2:46:45
right you have the category id but not the category name
2:46:46
the category name
2:46:46
the category name you have the tag ids but not the tag
2:46:48
you have the tag ids but not the tag
2:46:48
you have the tag ids but not the tag names
2:46:51
right and um that means that if you want
2:46:53
right and um that means that if you want
2:46:53
right and um that means that if you want to get all the products in a given
2:46:54
to get all the products in a given
2:46:54
to get all the products in a given category and display them on your web
2:46:56
category and display them on your web
2:46:56
category and display them on your web page you would have to run one query to
2:46:58
page you would have to run one query to
2:46:58
page you would have to run one query to get all the products in the given
2:46:59
get all the products in the given
2:46:59
get all the products in the given category this should be fairly efficient
2:47:00
category this should be fairly efficient
2:47:00
category this should be fairly efficient because
2:47:01
because
2:47:01
because it's a single partition query you can
2:47:02
it's a single partition query you can
2:47:02
it's a single partition query you can even further filter on this within a
2:47:04
even further filter on this within a
2:47:04
even further filter on this within a category uh
2:47:06
category uh
2:47:06
category uh if you wanted it would still be a single
2:47:07
if you wanted it would still be a single
2:47:07
if you wanted it would still be a single partition query but you don't have the
2:47:09
partition query but you don't have the
2:47:09
partition query but you don't have the display names so it would be a second
2:47:10
display names so it would be a second
2:47:10
display names so it would be a second query now on the category
2:47:12
query now on the category
2:47:12
query now on the category container by the category i need to get
2:47:14
container by the category i need to get
2:47:14
container by the category i need to get the display name for the category and
2:47:16
the display name for the category and
2:47:16
the display name for the category and then it gets worse because
2:47:17
then it gets worse because
2:47:17
then it gets worse because for each product returned by that first
2:47:19
for each product returned by that first
2:47:20
for each product returned by that first query you would need to run this third
2:47:21
query you would need to run this third
2:47:21
query you would need to run this third query on the tag document
2:47:22
query on the tag document
2:47:22
query on the tag document to get the display names for the tags
2:47:24
to get the display names for the tags
2:47:24
to get the display names for the tags and this is where you know
2:47:25
and this is where you know
2:47:25
and this is where you know you pound your fist on the table and say
2:47:27
you pound your fist on the table and say
2:47:27
you pound your fist on the table and say well where are my joins of course in the
2:47:28
well where are my joins of course in the
2:47:28
well where are my joins of course in the relational world this is a no-brainer
2:47:30
relational world this is a no-brainer
2:47:30
relational world this is a no-brainer you just join these things together on
2:47:32
you just join these things together on
2:47:32
you just join these things together on the ids and you get the display names
2:47:33
the ids and you get the display names
2:47:33
the ids and you get the display names but we're in cosmos db world
2:47:35
but we're in cosmos db world
2:47:35
but we're in cosmos db world there are no joins we need a new way a
2:47:38
there are no joins we need a new way a
2:47:38
there are no joins we need a new way a different way
2:47:39
different way
2:47:39
different way of handling this and and that is going
2:47:42
of handling this and and that is going
2:47:42
of handling this and and that is going to be done with denormalization
2:47:44
to be done with denormalization
2:47:44
to be done with denormalization we're going to simply copy or duplicate
2:47:46
we're going to simply copy or duplicate
2:47:46
we're going to simply copy or duplicate that master data in each products
2:47:48
that master data in each products
2:47:48
that master data in each products document so each product document in
2:47:50
document so each product document in
2:47:50
document so each product document in addition to having the category id
2:47:52
addition to having the category id
2:47:52
addition to having the category id will have a copy of that category name
2:47:54
will have a copy of that category name
2:47:54
will have a copy of that category name that's a copy of the master data
2:47:56
that's a copy of the master data
2:47:56
that's a copy of the master data the tags property is now not just an
2:47:59
the tags property is now not just an
2:47:59
the tags property is now not just an array of ids but an array of objects
2:48:00
array of ids but an array of objects
2:48:00
array of ids but an array of objects that includes both the tag id
2:48:02
that includes both the tag id
2:48:02
that includes both the tag id and a copy of the tag name again from
2:48:05
and a copy of the tag name again from
2:48:05
and a copy of the tag name again from the master data so at the point of
2:48:06
the master data so at the point of
2:48:06
the master data so at the point of insert when i create a new product
2:48:08
insert when i create a new product
2:48:08
insert when i create a new product i'm also copying that master data for
2:48:10
i'm also copying that master data for
2:48:10
i'm also copying that master data for the category name and the tag names
2:48:11
the category name and the tag names
2:48:11
the category name and the tag names which is great meaning when i have a
2:48:13
which is great meaning when i have a
2:48:13
which is great meaning when i have a product document in hand i have
2:48:14
product document in hand i have
2:48:14
product document in hand i have everything i need to display
2:48:16
everything i need to display
2:48:16
everything i need to display my product on the web page but of course
2:48:18
my product on the web page but of course
2:48:18
my product on the web page but of course the obvious question becomes what
2:48:19
the obvious question becomes what
2:48:19
the obvious question becomes what happens when does it change that master
2:48:21
happens when does it change that master
2:48:21
happens when does it change that master data
2:48:21
data
2:48:21
data somebody comes along and changes a
2:48:22
somebody comes along and changes a
2:48:22
somebody comes along and changes a category a name or a tag name
2:48:26
category a name or a tag name
2:48:26
category a name or a tag name and so what we really want to do is
2:48:27
and so what we really want to do is
2:48:27
and so what we really want to do is listen for changes on those two
2:48:29
listen for changes on those two
2:48:29
listen for changes on those two containers
2:48:30
containers
2:48:30
containers and cascade them to the product
2:48:32
and cascade them to the product
2:48:32
and cascade them to the product container and that's where the change
2:48:33
container and that's where the change
2:48:33
container and that's where the change feed comes in which is a very critical
2:48:35
feed comes in which is a very critical
2:48:36
feed comes in which is a very critical component of cosmos to be a bit of an
2:48:37
component of cosmos to be a bit of an
2:48:37
component of cosmos to be a bit of an unsung hero you don't hear enough about
2:48:39
unsung hero you don't hear enough about
2:48:39
unsung hero you don't hear enough about it but it's what drives a lot of
2:48:40
it but it's what drives a lot of
2:48:40
it but it's what drives a lot of microservice architectures
2:48:42
microservice architectures
2:48:42
microservice architectures the change feed gives you a persistent
2:48:44
the change feed gives you a persistent
2:48:44
the change feed gives you a persistent log of changes
2:48:45
log of changes
2:48:45
log of changes so every time a document is added or
2:48:48
so every time a document is added or
2:48:48
so every time a document is added or modified or changed
2:48:49
modified or changed
2:48:49
modified or changed in the in the container it gets emitted
2:48:52
in the in the container it gets emitted
2:48:52
in the in the container it gets emitted through the change feed in
2:48:53
through the change feed in
2:48:53
through the change feed in near real time and also persisted to the
2:48:55
near real time and also persisted to the
2:48:55
near real time and also persisted to the change speed which so it can always be
2:48:57
change speed which so it can always be
2:48:57
change speed which so it can always be rebound and replayed
2:48:58
rebound and replayed
2:48:58
rebound and replayed and you can build microservices um
2:49:01
and you can build microservices um
2:49:01
and you can build microservices um you can use azure functions to do this
2:49:03
you can use azure functions to do this
2:49:03
you can use azure functions to do this very easily to watch the change feed
2:49:05
very easily to watch the change feed
2:49:05
very easily to watch the change feed respond to every change that occurs and
2:49:07
respond to every change that occurs and
2:49:07
respond to every change that occurs and respond and
2:49:08
respond and
2:49:08
respond and process it in some way in this case it
2:49:10
process it in some way in this case it
2:49:10
process it in some way in this case it would be to keep our
2:49:11
would be to keep our
2:49:12
would be to keep our denormalized data model in sync with
2:49:14
denormalized data model in sync with
2:49:14
denormalized data model in sync with master data
2:49:15
master data
2:49:15
master data as it changes so on the product category
2:49:17
as it changes so on the product category
2:49:18
as it changes so on the product category container and on the product tag
2:49:19
container and on the product tag
2:49:19
container and on the product tag container they each have a change feed
2:49:21
container they each have a change feed
2:49:21
container they each have a change feed and when any master data changes we can
2:49:23
and when any master data changes we can
2:49:23
and when any master data changes we can have an azure function wake up
2:49:25
have an azure function wake up
2:49:25
have an azure function wake up and cascade the change to the product
2:49:26
and cascade the change to the product
2:49:26
and cascade the change to the product table here's just some pseudo code
2:49:28
table here's just some pseudo code
2:49:28
table here's just some pseudo code here's my azure function name the
2:49:30
here's my azure function name the
2:49:30
here's my azure function name the cosmodb trigger attribute means the
2:49:32
cosmodb trigger attribute means the
2:49:32
cosmodb trigger attribute means the azure function will wake up
2:49:33
azure function will wake up
2:49:34
azure function will wake up whenever there's a change to the
2:49:35
whenever there's a change to the
2:49:35
whenever there's a change to the collection name which is the container
2:49:36
collection name which is the container
2:49:36
collection name which is the container name that's the product category
2:49:38
name that's the product category
2:49:38
name that's the product category right here that's this guy right here on
2:49:41
right here that's this guy right here on
2:49:41
right here that's this guy right here on the left that we're watching
2:49:42
the left that we're watching
2:49:42
the left that we're watching for changes for this product category
2:49:44
for changes for this product category
2:49:44
for changes for this product category container
2:49:46
container
2:49:46
container and we're handed a read-only list of
2:49:48
and we're handed a read-only list of
2:49:48
and we're handed a read-only list of documents that have changed
2:49:49
documents that have changed
2:49:49
documents that have changed and we simply iterate them and for each
2:49:51
and we simply iterate them and for each
2:49:51
and we simply iterate them and for each one each one of these represents
2:49:53
one each one of these represents
2:49:53
one each one of these represents a change let's say to a product category
2:49:55
a change let's say to a product category
2:49:55
a change let's say to a product category in this case we will just get the
2:49:56
in this case we will just get the
2:49:56
in this case we will just get the category id
2:49:57
category id
2:49:57
category id and name and now we need to update each
2:50:00
and name and now we need to update each
2:50:00
and name and now we need to update each product to cascade and propagate that
2:50:04
product to cascade and propagate that
2:50:04
product to cascade and propagate that master data change
2:50:05
master data change
2:50:05
master data change and that's done first by querying for
2:50:07
and that's done first by querying for
2:50:07
and that's done first by querying for all the products in that category very
2:50:09
all the products in that category very
2:50:09
all the products in that category very efficiently again because
2:50:10
efficiently again because
2:50:10
efficiently again because uh we're partitioned on the category id
2:50:13
uh we're partitioned on the category id
2:50:13
uh we're partitioned on the category id so this will return all the products in
2:50:14
so this will return all the products in
2:50:14
so this will return all the products in one
2:50:15
one
2:50:15
one efficient single partition query and
2:50:18
efficient single partition query and
2:50:18
efficient single partition query and then
2:50:18
then
2:50:18
then set up our um iterator to process those
2:50:22
set up our um iterator to process those
2:50:22
set up our um iterator to process those results and update them one at a time
2:50:23
results and update them one at a time
2:50:23
results and update them one at a time replacing them with point rise updates
2:50:27
replacing them with point rise updates
2:50:27
replacing them with point rise updates that takes us to our last set of tables
2:50:29
that takes us to our last set of tables
2:50:29
that takes us to our last set of tables in our data model
2:50:30
in our data model
2:50:30
in our data model which is our sales order and sales order
2:50:33
which is our sales order and sales order
2:50:33
which is our sales order and sales order details
2:50:34
details
2:50:34
details by now the process should seem a little
2:50:36
by now the process should seem a little
2:50:36
by now the process should seem a little bit familiar to you step one
2:50:38
bit familiar to you step one
2:50:38
bit familiar to you step one translate column names to json property
2:50:39
translate column names to json property
2:50:40
translate column names to json property names step two ask the question
2:50:42
names step two ask the question
2:50:42
names step two ask the question embed or reference this is one to many
2:50:44
embed or reference this is one to many
2:50:44
embed or reference this is one to many or one to few
2:50:46
or one to few
2:50:46
or one to few well hopefully our sales orders are
2:50:48
well hopefully our sales orders are
2:50:48
well hopefully our sales orders are large but i still think that
2:50:49
large but i still think that
2:50:50
large but i still think that um at two megabytes we can accommodate
2:50:52
um at two megabytes we can accommodate
2:50:52
um at two megabytes we can accommodate even a large sales order
2:50:53
even a large sales order
2:50:53
even a large sales order uh so we would definitely be better off
2:50:55
uh so we would definitely be better off
2:50:55
uh so we would definitely be better off by embedding the details
2:50:57
by embedding the details
2:50:57
by embedding the details right so we'll embed the sales order
2:50:58
right so we'll embed the sales order
2:50:58
right so we'll embed the sales order details and essentially pre-joining them
2:51:01
details and essentially pre-joining them
2:51:01
details and essentially pre-joining them and um they're embedded there as an
2:51:03
and um they're embedded there as an
2:51:03
and um they're embedded there as an array of details inside each sales order
2:51:05
array of details inside each sales order
2:51:05
array of details inside each sales order document and then
2:51:06
document and then
2:51:06
document and then we're going to put them in another
2:51:07
we're going to put them in another
2:51:07
we're going to put them in another container called the sales order
2:51:08
container called the sales order
2:51:08
container called the sales order container ask the usual question how do
2:51:10
container ask the usual question how do
2:51:10
container ask the usual question how do i partition this container
2:51:12
i partition this container
2:51:12
i partition this container well answer by are we asking the usual
2:51:15
well answer by are we asking the usual
2:51:15
well answer by are we asking the usual questions what are my most common
2:51:16
questions what are my most common
2:51:16
questions what are my most common queries
2:51:17
queries
2:51:17
queries well we've got this very common query
2:51:19
well we've got this very common query
2:51:19
well we've got this very common query again that the customer is going to be
2:51:21
again that the customer is going to be
2:51:21
again that the customer is going to be uh running all the customers visiting
2:51:24
uh running all the customers visiting
2:51:24
uh running all the customers visiting your website are going to be
2:51:26
your website are going to be
2:51:26
your website are going to be frequently asking to see their sales
2:51:28
frequently asking to see their sales
2:51:28
frequently asking to see their sales orders and then
2:51:29
orders and then
2:51:29
orders and then perhaps less common query that might
2:51:32
perhaps less common query that might
2:51:32
perhaps less common query that might occur let's say in the back office by
2:51:34
occur let's say in the back office by
2:51:34
occur let's say in the back office by some executive
2:51:35
some executive
2:51:35
some executive who's looking at a dashboard perhaps who
2:51:37
who's looking at a dashboard perhaps who
2:51:37
who's looking at a dashboard perhaps who wants to see their best customers might
2:51:38
wants to see their best customers might
2:51:38
wants to see their best customers might want to get uh
2:51:39
want to get uh
2:51:39
want to get uh i might want to query for all the
2:51:41
i might want to query for all the
2:51:41
i might want to query for all the customers uh descending by order account
2:51:43
customers uh descending by order account
2:51:43
customers uh descending by order account to get their best customers at the top
2:51:44
to get their best customers at the top
2:51:44
to get their best customers at the top of the list so let's go with the common
2:51:47
of the list so let's go with the common
2:51:47
of the list so let's go with the common scenario first
2:51:48
scenario first
2:51:48
scenario first getting all sales order for a customer
2:51:50
getting all sales order for a customer
2:51:50
getting all sales order for a customer or would look like something like so
2:51:51
or would look like something like so
2:51:52
or would look like something like so that means we
2:51:52
that means we
2:51:52
that means we you know that's how the query would look
2:51:54
you know that's how the query would look
2:51:54
you know that's how the query would look right give me all the orders
2:51:55
right give me all the orders
2:51:55
right give me all the orders where the customer id matches this
2:51:57
where the customer id matches this
2:51:57
where the customer id matches this customer so if we
2:51:59
customer so if we
2:51:59
customer so if we partition on the customer id it'll do
2:52:01
partition on the customer id it'll do
2:52:01
partition on the customer id it'll do just that right it'll put
2:52:03
just that right it'll put
2:52:03
just that right it'll put all the sales orders for customer a and
2:52:05
all the sales orders for customer a and
2:52:05
all the sales orders for customer a and its own logical partition
2:52:06
its own logical partition
2:52:06
its own logical partition and each other customer in their own
2:52:08
and each other customer in their own
2:52:08
and each other customer in their own separate logical partitions i could get
2:52:09
separate logical partitions i could get
2:52:09
separate logical partitions i could get all the sales orders or a subset of the
2:52:11
all the sales orders or a subset of the
2:52:11
all the sales orders or a subset of the sales orders from a customer uh
2:52:14
sales orders from a customer uh
2:52:14
sales orders from a customer uh with a very efficient single partition
2:52:16
with a very efficient single partition
2:52:16
with a very efficient single partition query
2:52:18
query
2:52:18
query so that means customer id is the correct
2:52:20
so that means customer id is the correct
2:52:20
so that means customer id is the correct uh is the appropriate partition key to
2:52:22
uh is the appropriate partition key to
2:52:22
uh is the appropriate partition key to use here
2:52:23
use here
2:52:23
use here but now let's slam on the breaks here
2:52:25
but now let's slam on the breaks here
2:52:25
but now let's slam on the breaks here and pause and remember i promise we
2:52:27
and pause and remember i promise we
2:52:27
and pause and remember i promise we come back to the customer container
2:52:28
come back to the customer container
2:52:28
come back to the customer container where we partitioned on the id property
2:52:31
where we partitioned on the id property
2:52:31
where we partitioned on the id property because that's the same value that we're
2:52:32
because that's the same value that we're
2:52:32
because that's the same value that we're partitioning on we're partitioning on
2:52:34
partitioning on we're partitioning on
2:52:34
partitioning on we're partitioning on the customer id in both cases
2:52:36
the customer id in both cases
2:52:36
the customer id in both cases in the case of the customer container
2:52:37
in the case of the customer container
2:52:38
in the case of the customer container and in the case of this sales order
2:52:39
and in the case of this sales order
2:52:39
and in the case of this sales order container we determine that customer id
2:52:41
container we determine that customer id
2:52:41
container we determine that customer id is the appropriate value
2:52:42
is the appropriate value
2:52:42
is the appropriate value now granted it's called id in the
2:52:45
now granted it's called id in the
2:52:45
now granted it's called id in the customer
2:52:47
customer
2:52:47
customer uh document and it's called customer id
2:52:49
uh document and it's called customer id
2:52:49
uh document and it's called customer id and the sales order document and we'll
2:52:50
and the sales order document and we'll
2:52:50
and the sales order document and we'll iron out that wrinkle in just a moment
2:52:52
iron out that wrinkle in just a moment
2:52:52
iron out that wrinkle in just a moment but the value is the same it's the same
2:52:55
but the value is the same it's the same
2:52:55
but the value is the same it's the same value that we want to partition on
2:52:57
value that we want to partition on
2:52:57
value that we want to partition on and when you want to uh when your
2:52:59
and when you want to uh when your
2:52:59
and when you want to uh when your partitioning needs are the same between
2:53:01
partitioning needs are the same between
2:53:01
partitioning needs are the same between two different types of
2:53:02
two different types of
2:53:02
two different types of document types you will not create two
2:53:05
document types you will not create two
2:53:05
document types you will not create two different containers we are in a
2:53:07
different containers we are in a
2:53:07
different containers we are in a schema free world here in cosmos tv
2:53:09
schema free world here in cosmos tv
2:53:09
schema free world here in cosmos tv there's no rules on schema there's
2:53:11
there's no rules on schema there's
2:53:11
there's no rules on schema there's nothing preventing you
2:53:12
nothing preventing you
2:53:12
nothing preventing you uh from storing two different entity
2:53:14
uh from storing two different entity
2:53:14
uh from storing two different entity types in the same container and in fact
2:53:16
types in the same container and in fact
2:53:16
types in the same container and in fact it's encouraged to do so uh when you
2:53:18
it's encouraged to do so uh when you
2:53:18
it's encouraged to do so uh when you have two types even though they're
2:53:20
have two types even though they're
2:53:20
have two types even though they're different
2:53:20
different
2:53:20
different types but they share the same
2:53:22
types but they share the same
2:53:22
types but they share the same partitioning needs or they share
2:53:24
partitioning needs or they share
2:53:24
partitioning needs or they share similar throughput needs those are
2:53:27
similar throughput needs those are
2:53:27
similar throughput needs those are reasons
2:53:27
reasons
2:53:27
reasons for storing multiple items of the same
2:53:30
for storing multiple items of the same
2:53:30
for storing multiple items of the same type
2:53:32
type
2:53:32
type in the same container and that applies
2:53:33
in the same container and that applies
2:53:34
in the same container and that applies here because we have the similar
2:53:35
here because we have the similar
2:53:35
here because we have the similar partitioning needs on both
2:53:36
partitioning needs on both
2:53:36
partitioning needs on both these entities and that's the customer
2:53:39
these entities and that's the customer
2:53:39
these entities and that's the customer id so what we'll do is
2:53:40
id so what we'll do is
2:53:40
id so what we'll do is we will not create that new sales order
2:53:42
we will not create that new sales order
2:53:42
we will not create that new sales order container we will stick with the single
2:53:45
container we will stick with the single
2:53:45
container we will stick with the single customer container that we started with
2:53:46
customer container that we started with
2:53:46
customer container that we started with we'll just have to make a minor
2:53:47
we'll just have to make a minor
2:53:47
we'll just have to make a minor adjustment since the id property in each
2:53:49
adjustment since the id property in each
2:53:50
adjustment since the id property in each of these two types is different in the
2:53:52
of these two types is different in the
2:53:52
of these two types is different in the case of the customer
2:53:53
case of the customer
2:53:53
case of the customer document the id is the customer id in
2:53:55
document the id is the customer id in
2:53:55
document the id is the customer id in the case of the sales order document
2:53:57
the case of the sales order document
2:53:57
the case of the sales order document the id is the sales order id and it's
2:54:00
the id is the sales order id and it's
2:54:00
the id is the sales order id and it's the customer id
2:54:01
the customer id
2:54:01
the customer id property that's kind of a foreign key if
2:54:03
property that's kind of a foreign key if
2:54:03
property that's kind of a foreign key if you will
2:54:04
you will
2:54:04
you will so we'll fix that with another little
2:54:05
so we'll fix that with another little
2:54:05
so we'll fix that with another little tiny little bit of denormalization we'll
2:54:07
tiny little bit of denormalization we'll
2:54:07
tiny little bit of denormalization we'll throw in a customer id property on the
2:54:09
throw in a customer id property on the
2:54:09
throw in a customer id property on the customer document
2:54:11
customer document
2:54:11
customer document and essentially store i'm sorry two
2:54:14
and essentially store i'm sorry two
2:54:14
and essentially store i'm sorry two copies
2:54:14
copies
2:54:14
copies i guess i didn't have that call out in
2:54:16
i guess i didn't have that call out in
2:54:16
i guess i didn't have that call out in there we will sort two copies of the
2:54:18
there we will sort two copies of the
2:54:18
there we will sort two copies of the customer id the customer id will be
2:54:20
customer id the customer id will be
2:54:20
customer id the customer id will be stored
2:54:21
stored
2:54:21
stored both here and here
2:54:24
both here and here
2:54:24
both here and here and it's the same name property now in
2:54:26
and it's the same name property now in
2:54:26
and it's the same name property now in both documents
2:54:27
both documents
2:54:27
both documents and so i can partition on the customer
2:54:28
and so i can partition on the customer
2:54:28
and so i can partition on the customer id property
2:54:30
id property
2:54:30
id property it's definitely i mentioned a good idea
2:54:33
it's definitely i mentioned a good idea
2:54:33
it's definitely i mentioned a good idea to always have a type property but
2:54:34
to always have a type property but
2:54:34
to always have a type property but that's especially true when you are
2:54:36
that's especially true when you are
2:54:36
that's especially true when you are combining types in a single container
2:54:37
combining types in a single container
2:54:38
combining types in a single container you want to
2:54:38
you want to
2:54:38
you want to unambiguously distinguish between two
2:54:40
unambiguously distinguish between two
2:54:40
unambiguously distinguish between two types especially
2:54:41
types especially
2:54:41
types especially when they're in the same container but
2:54:43
when they're in the same container but
2:54:43
when they're in the same container but now uh that's our data model at this
2:54:45
now uh that's our data model at this
2:54:45
now uh that's our data model at this point
2:54:46
point
2:54:46
point we now have logical partitions where
2:54:49
we now have logical partitions where
2:54:49
we now have logical partitions where we're mixing types
2:54:50
we're mixing types
2:54:50
we're mixing types one document in a logical partition for
2:54:52
one document in a logical partition for
2:54:52
one document in a logical partition for customer is the
2:54:53
customer is the
2:54:53
customer is the customer document which remember
2:54:55
customer document which remember
2:54:55
customer document which remember embedded inside of that are the
2:54:56
embedded inside of that are the
2:54:56
embedded inside of that are the addresses
2:54:57
addresses
2:54:57
addresses and the password and all the other
2:54:59
and the password and all the other
2:54:59
and the password and all the other documents are sales order documents and
2:55:01
documents are sales order documents and
2:55:01
documents are sales order documents and remember that embedded within those are
2:55:03
remember that embedded within those are
2:55:03
remember that embedded within those are all the sales order details
2:55:05
all the sales order details
2:55:05
all the sales order details and all of which are contained in this
2:55:06
and all of which are contained in this
2:55:06
and all of which are contained in this single logical partition per customer
2:55:08
single logical partition per customer
2:55:08
single logical partition per customer so this is a very nice little design for
2:55:11
so this is a very nice little design for
2:55:11
so this is a very nice little design for uh customers and sales orders
2:55:13
uh customers and sales orders
2:55:13
uh customers and sales orders i can run a query like search to get all
2:55:15
i can run a query like search to get all
2:55:15
i can run a query like search to get all the sales orders for a different for a
2:55:16
the sales orders for a different for a
2:55:16
the sales orders for a different for a given customer
2:55:17
given customer
2:55:17
given customer by querying on the type property um
2:55:20
by querying on the type property um
2:55:20
by querying on the type property um for a given customer in fact if i left
2:55:23
for a given customer in fact if i left
2:55:23
for a given customer in fact if i left out
2:55:24
out
2:55:24
out the cut the the uh the type portion
2:55:27
the cut the the uh the type portion
2:55:27
the cut the the uh the type portion of my query and i just got all of the
2:55:29
of my query and i just got all of the
2:55:29
of my query and i just got all of the documents from a given customer
2:55:30
documents from a given customer
2:55:30
documents from a given customer it would give me the customer and all of
2:55:32
it would give me the customer and all of
2:55:32
it would give me the customer and all of their sales orders along with all of the
2:55:34
their sales orders along with all of the
2:55:34
their sales orders along with all of the embedded information
2:55:35
embedded information
2:55:35
embedded information with a very efficient um with a very
2:55:38
with a very efficient um with a very
2:55:38
with a very efficient um with a very efficient single
2:55:39
efficient single
2:55:39
efficient single partition query and so that therefore we
2:55:41
partition query and so that therefore we
2:55:41
partition query and so that therefore we are getting around the issue of there
2:55:43
are getting around the issue of there
2:55:44
are getting around the issue of there being no relational uh uh
2:55:47
being no relational uh uh
2:55:47
being no relational uh uh integrity no referential integrity no
2:55:48
integrity no referential integrity no
2:55:48
integrity no referential integrity no relational constraints no
2:55:50
relational constraints no
2:55:50
relational constraints no joins because you see it's kind of all
2:55:52
joins because you see it's kind of all
2:55:52
joins because you see it's kind of all joined together for us here isn't it
2:55:54
joined together for us here isn't it
2:55:54
joined together for us here isn't it because we partitioned properly so our
2:55:56
because we partitioned properly so our
2:55:56
because we partitioned properly so our documents kind of join together
2:55:57
documents kind of join together
2:55:57
documents kind of join together naturally
2:55:58
naturally
2:55:58
naturally in an efficient manner through single
2:56:00
in an efficient manner through single
2:56:00
in an efficient manner through single partition queries and within those
2:56:02
partition queries and within those
2:56:02
partition queries and within those documents we've got pre-joined embedded
2:56:04
documents we've got pre-joined embedded
2:56:04
documents we've got pre-joined embedded data as well
2:56:06
data as well
2:56:06
data as well and we've kind of like dealt with these
2:56:09
and we've kind of like dealt with these
2:56:09
and we've kind of like dealt with these radical differences between the
2:56:10
radical differences between the
2:56:10
radical differences between the relational world and the non-relational
2:56:11
relational world and the non-relational
2:56:11
relational world and the non-relational world
2:56:12
world
2:56:12
world and that takes us to our very very last
2:56:14
and that takes us to our very very last
2:56:14
and that takes us to our very very last query example which is that
2:56:15
query example which is that
2:56:15
query example which is that execs sitting in the back office looking
2:56:17
execs sitting in the back office looking
2:56:17
execs sitting in the back office looking at their dashboard who wants to see
2:56:18
at their dashboard who wants to see
2:56:18
at their dashboard who wants to see their top customers
2:56:20
their top customers
2:56:20
their top customers again in the relational world is a
2:56:22
again in the relational world is a
2:56:22
again in the relational world is a no-brainer you just do a select count on
2:56:24
no-brainer you just do a select count on
2:56:24
no-brainer you just do a select count on that order table group by customer id
2:56:25
that order table group by customer id
2:56:25
that order table group by customer id and you're done
2:56:27
and you're done
2:56:27
and you're done that doesn't work here so what we'll do
2:56:29
that doesn't work here so what we'll do
2:56:29
that doesn't work here so what we'll do is a little bit of denormalization
2:56:32
is a little bit of denormalization
2:56:32
is a little bit of denormalization we'll throw in a new column called a new
2:56:34
we'll throw in a new column called a new
2:56:34
we'll throw in a new column called a new accordion slip there a new property
2:56:36
accordion slip there a new property
2:56:36
accordion slip there a new property called sales order account and we're
2:56:39
called sales order account and we're
2:56:39
called sales order account and we're going to
2:56:39
going to
2:56:39
going to maintain we're going to not duplicate
2:56:42
maintain we're going to not duplicate
2:56:42
maintain we're going to not duplicate data in the sense that we're going to
2:56:43
data in the sense that we're going to
2:56:43
data in the sense that we're going to duplicate the count
2:56:44
duplicate the count
2:56:44
duplicate the count so if there are three dots if there are
2:56:46
so if there are three dots if there are
2:56:46
so if there are three dots if there are three sales order documents here
2:56:49
three sales order documents here
2:56:49
three sales order documents here for this customer then the number three
2:56:51
for this customer then the number three
2:56:51
for this customer then the number three will be stored in the customer
2:56:54
will be stored in the customer
2:56:54
will be stored in the customer document as well and that means that if
2:56:56
document as well and that means that if
2:56:56
document as well and that means that if i want to add a new sales order for
2:56:58
i want to add a new sales order for
2:56:58
i want to add a new sales order for customer
2:56:58
customer
2:56:58
customer a i also need to update that customer
2:57:01
a i also need to update that customer
2:57:01
a i also need to update that customer document to increment
2:57:02
document to increment
2:57:02
document to increment the sales order account how can we do
2:57:04
the sales order account how can we do
2:57:04
the sales order account how can we do that if you're thinking change fee
2:57:06
that if you're thinking change fee
2:57:06
that if you're thinking change fee you're on the right you're definitely on
2:57:08
you're on the right you're definitely on
2:57:08
you're on the right you're definitely on one right track change is a wonderful
2:57:10
one right track change is a wonderful
2:57:10
one right track change is a wonderful mechanism
2:57:10
mechanism
2:57:10
mechanism but in this case i'm going to do one
2:57:12
but in this case i'm going to do one
2:57:12
but in this case i'm going to do one better because everything is
2:57:13
better because everything is
2:57:13
better because everything is in the same logical partition we can
2:57:15
in the same logical partition we can
2:57:15
in the same logical partition we can take advantage of transactions
2:57:17
take advantage of transactions
2:57:17
take advantage of transactions using a stored procedure now this is in
2:57:19
using a stored procedure now this is in
2:57:19
using a stored procedure now this is in this case it's a stored procedure with
2:57:20
this case it's a stored procedure with
2:57:20
this case it's a stored procedure with javascript it can be done in c
2:57:22
javascript it can be done in c
2:57:22
javascript it can be done in c sharp or other languages uh or well i
2:57:24
sharp or other languages uh or well i
2:57:24
sharp or other languages uh or well i think at least in c sharp using
2:57:26
think at least in c sharp using
2:57:26
think at least in c sharp using transactional
2:57:27
transactional
2:57:27
transactional bulk of the the transactional batch
2:57:29
bulk of the the transactional batch
2:57:29
bulk of the the transactional batch feature but in either case
2:57:31
feature but in either case
2:57:31
feature but in either case what we're doing here is just a little
2:57:32
what we're doing here is just a little
2:57:32
what we're doing here is just a little bit of code that is going to
2:57:34
bit of code that is going to
2:57:34
bit of code that is going to retrieve the customer document increment
2:57:38
retrieve the customer document increment
2:57:38
retrieve the customer document increment the sales order account of that customer
2:57:39
the sales order account of that customer
2:57:39
the sales order account of that customer document
2:57:41
document
2:57:41
document and then replace it although that
2:57:43
and then replace it although that
2:57:43
and then replace it although that replacement occurs in
2:57:44
replacement occurs in
2:57:44
replacement occurs in in the context of a transaction and is
2:57:47
in the context of a transaction and is
2:57:47
in the context of a transaction and is not actually committed until this
2:57:48
not actually committed until this
2:57:48
not actually committed until this javascript code completes successfully
2:57:51
javascript code completes successfully
2:57:51
javascript code completes successfully so it's been written to the container
2:57:52
so it's been written to the container
2:57:52
so it's been written to the container but not committed and then we can create
2:57:54
but not committed and then we can create
2:57:54
but not committed and then we can create the new sales order document
2:57:56
the new sales order document
2:57:56
the new sales order document and and return let's say the new account
2:57:59
and and return let's say the new account
2:57:59
and and return let's say the new account because we want to return something
2:58:00
because we want to return something
2:58:00
because we want to return something but only then um when
2:58:04
but only then um when
2:58:04
but only then um when the function completes successfully the
2:58:06
the function completes successfully the
2:58:06
the function completes successfully the the does everything get committed the
2:58:08
the does everything get committed the
2:58:08
the does everything get committed the new
2:58:09
new
2:58:09
new order is written and the customer
2:58:12
order is written and the customer
2:58:12
order is written and the customer document is updated all at the same time
2:58:15
document is updated all at the same time
2:58:15
document is updated all at the same time and once you've got that uh store
2:58:17
and once you've got that uh store
2:58:17
and once you've got that uh store procedure in place it doesn't mean that
2:58:18
procedure in place it doesn't mean that
2:58:18
procedure in place it doesn't mean that any time you want to add
2:58:20
any time you want to add
2:58:20
any time you want to add an order document you must do so by
2:58:22
an order document you must do so by
2:58:22
an order document you must do so by calling the stored procedure and not by
2:58:23
calling the stored procedure and not by
2:58:23
calling the stored procedure and not by just dropping it in by directly
2:58:25
just dropping it in by directly
2:58:25
just dropping it in by directly inserting it and if you
2:58:27
inserting it and if you
2:58:27
inserting it and if you uh follow those guidelines you will
2:58:28
uh follow those guidelines you will
2:58:28
uh follow those guidelines you will always be able to run a very efficient
2:58:30
always be able to run a very efficient
2:58:30
always be able to run a very efficient query like so
2:58:31
query like so
2:58:31
query like so relatively efficient query this is a
2:58:33
relatively efficient query this is a
2:58:33
relatively efficient query this is a cross-partition query
2:58:34
cross-partition query
2:58:34
cross-partition query right because there's no where clause in
2:58:36
right because there's no where clause in
2:58:36
right because there's no where clause in here we're ordering by the says on
2:58:37
here we're ordering by the says on
2:58:38
here we're ordering by the says on account descending
2:58:39
account descending
2:58:39
account descending is that evil it's not evil if it's in
2:58:41
is that evil it's not evil if it's in
2:58:41
is that evil it's not evil if it's in the minority of cases like this
2:58:43
the minority of cases like this
2:58:43
the minority of cases like this one exec sitting in the back office
2:58:44
one exec sitting in the back office
2:58:44
one exec sitting in the back office who's hitting refresh on their dashboard
2:58:46
who's hitting refresh on their dashboard
2:58:46
who's hitting refresh on their dashboard once an hour
2:58:47
once an hour
2:58:47
once an hour it is evil if it's the customer-facing
2:58:49
it is evil if it's the customer-facing
2:58:49
it is evil if it's the customer-facing website with all of your customers
2:58:51
website with all of your customers
2:58:51
website with all of your customers hammering away with cross-partition
2:58:52
hammering away with cross-partition
2:58:52
hammering away with cross-partition queries day in and day out
2:58:54
queries day in and day out
2:58:54
queries day in and day out that that is when cross-partitioning
2:58:56
that that is when cross-partitioning
2:58:56
that that is when cross-partitioning queries need to be avoided but here is
2:58:58
queries need to be avoided but here is
2:58:58
queries need to be avoided but here is perfectly fine
2:58:59
perfectly fine
2:58:59
perfectly fine and and it's relatively inexpensive
2:59:01
and and it's relatively inexpensive
2:59:01
and and it's relatively inexpensive because the sales order account has been
2:59:03
because the sales order account has been
2:59:03
because the sales order account has been pre-materialized for us
2:59:06
pre-materialized for us
2:59:06
pre-materialized for us so to wrap up with two minutes to go our
2:59:08
so to wrap up with two minutes to go our
2:59:08
so to wrap up with two minutes to go our final design
2:59:09
final design
2:59:09
final design we have uh a custom we have
2:59:12
we have uh a custom we have
2:59:12
we have uh a custom we have a database with these four containers
2:59:15
a database with these four containers
2:59:15
a database with these four containers not nine containers four containers and
2:59:18
not nine containers four containers and
2:59:18
not nine containers four containers and in the customer container
2:59:19
in the customer container
2:59:19
in the customer container we've got two different types and and in
2:59:21
we've got two different types and and in
2:59:22
we've got two different types and and in case of customer sales orders and
2:59:23
case of customer sales orders and
2:59:23
case of customer sales orders and products
2:59:24
products
2:59:24
products we're also embedding to cut down on the
2:59:26
we're also embedding to cut down on the
2:59:26
we're also embedding to cut down on the number of containers and if you've been
2:59:27
number of containers and if you've been
2:59:28
number of containers and if you've been paying attention and i know you have
2:59:29
paying attention and i know you have
2:59:29
paying attention and i know you have you're probably thinking to yourself
2:59:30
you're probably thinking to yourself
2:59:30
you're probably thinking to yourself product tag and product category they're
2:59:32
product tag and product category they're
2:59:32
product tag and product category they're both partitioned on type
2:59:34
both partitioned on type
2:59:34
both partitioned on type they're both small lookup lists it
2:59:36
they're both small lookup lists it
2:59:36
they're both small lookup lists it doesn't make sense to create a container
2:59:37
doesn't make sense to create a container
2:59:37
doesn't make sense to create a container per small lookup list we can combine
2:59:39
per small lookup list we can combine
2:59:39
per small lookup list we can combine those into one let's say
2:59:40
those into one let's say
2:59:40
those into one let's say metadata container called product meta
2:59:42
metadata container called product meta
2:59:42
metadata container called product meta and combine product tag
2:59:44
and combine product tag
2:59:44
and combine product tag and product category documents and now
2:59:45
and product category documents and now
2:59:45
and product category documents and now bring it down to three containers
2:59:47
bring it down to three containers
2:59:47
bring it down to three containers with a very efficient data model to set
2:59:50
with a very efficient data model to set
2:59:50
with a very efficient data model to set to handle
2:59:51
to handle
2:59:51
to handle a relational data scenario in a
2:59:54
a relational data scenario in a
2:59:54
a relational data scenario in a non-relational database like
2:59:56
non-relational database like
2:59:56
non-relational database like cosmos db the key takeaways for this for
2:59:59
cosmos db the key takeaways for this for
2:59:59
cosmos db the key takeaways for this for the session
2:59:59
the session
2:59:59
the session are you really must know your uh key
3:00:01
are you really must know your uh key
3:00:02
are you really must know your uh key patterns your key access patterns how
3:00:04
patterns your key access patterns how
3:00:04
patterns your key access patterns how your database is going to most be most
3:00:05
your database is going to most be most
3:00:05
your database is going to most be most frequently queried and written to you as
3:00:07
frequently queried and written to you as
3:00:07
frequently queried and written to you as well
3:00:08
well
3:00:08
well and then and you design for those
3:00:09
and then and you design for those
3:00:09
and then and you design for those patterns
3:00:11
patterns
3:00:11
patterns to state the obvious partitioning is
3:00:12
to state the obvious partitioning is
3:00:12
to state the obvious partitioning is critical you're not going to achieve
3:00:13
critical you're not going to achieve
3:00:13
critical you're not going to achieve that performance and scale
3:00:14
that performance and scale
3:00:14
that performance and scale if you pick a poor partition key you
3:00:16
if you pick a poor partition key you
3:00:16
if you pick a poor partition key you want that you want
3:00:18
want that you want
3:00:18
want that you want your uniform distribution as much as
3:00:20
your uniform distribution as much as
3:00:20
your uniform distribution as much as possible both for storage and throughput
3:00:22
possible both for storage and throughput
3:00:22
possible both for storage and throughput using these techniques that i covered to
3:00:24
using these techniques that i covered to
3:00:24
using these techniques that i covered to materialize relationships either by
3:00:26
materialize relationships either by
3:00:26
materialize relationships either by embedding
3:00:26
embedding
3:00:26
embedding or by uh referencing through
3:00:28
or by uh referencing through
3:00:28
or by uh referencing through denormalization pre-aggregating
3:00:30
denormalization pre-aggregating
3:00:30
denormalization pre-aggregating combining uh multiple entities uh
3:00:33
combining uh multiple entities uh
3:00:33
combining uh multiple entities uh different entity types in the same
3:00:34
different entity types in the same
3:00:34
different entity types in the same container if they share similar
3:00:35
container if they share similar
3:00:35
container if they share similar partitioning needs
3:00:36
partitioning needs
3:00:36
partitioning needs and they share similar throughput needs
3:00:39
and they share similar throughput needs
3:00:39
and they share similar throughput needs and using
3:00:40
and using
3:00:40
and using your choice of either the change feed or
3:00:43
your choice of either the change feed or
3:00:43
your choice of either the change feed or transactional
3:00:44
transactional
3:00:44
transactional integrity where when the updates are
3:00:46
integrity where when the updates are
3:00:46
integrity where when the updates are within a single logical partition using
3:00:48
within a single logical partition using
3:00:48
within a single logical partition using the stored
3:00:49
the stored
3:00:49
the stored using stored procedures as well so i
3:00:51
using stored procedures as well so i
3:00:51
using stored procedures as well so i hope you enjoyed the session as much as
3:00:52
hope you enjoyed the session as much as
3:00:52
hope you enjoyed the session as much as i enjoyed preparing it for you
3:00:54
i enjoyed preparing it for you
3:00:54
i enjoyed preparing it for you i want to thank the team for inviting me
3:00:57
i want to thank the team for inviting me
3:00:57
i want to thank the team for inviting me to speak to you today and
3:00:58
to speak to you today and
3:00:58
to speak to you today and uh throw it back to you guys for
3:01:00
uh throw it back to you guys for
3:01:00
uh throw it back to you guys for questions if you have any
3:01:02
questions if you have any
3:01:02
questions if you have any thanks lenny this was an awesome session
3:01:04
thanks lenny this was an awesome session
3:01:04
thanks lenny this was an awesome session i really love the way you
3:01:05
i really love the way you
3:01:05
i really love the way you graphically lay out those design
3:01:07
graphically lay out those design
3:01:07
graphically lay out those design considerations it makes it
3:01:08
considerations it makes it
3:01:08
considerations it makes it so much easier to understand you know
3:01:10
so much easier to understand you know
3:01:10
so much easier to understand you know how to tackle those kind of exercises
3:01:13
how to tackle those kind of exercises
3:01:14
how to tackle those kind of exercises i had some good inspiration there thomas
3:01:16
i had some good inspiration there thomas
3:01:16
i had some good inspiration there thomas yeah well
3:01:17
yeah well
3:01:17
yeah well you you presented very very well uh and
3:01:20
you you presented very very well uh and
3:01:20
you you presented very very well uh and i hope this is going to um to help any
3:01:22
i hope this is going to um to help any
3:01:22
i hope this is going to um to help any of our viewers will need to to tackle
3:01:24
of our viewers will need to to tackle
3:01:24
of our viewers will need to to tackle such such a design exercise in the
3:01:25
such such a design exercise in the
3:01:26
such such a design exercise in the future
3:01:26
future
3:01:26
future we are at time and we are short and time
3:01:27
we are at time and we are short and time
3:01:27
we are at time and we are short and time again but i just wanted to
3:01:29
again but i just wanted to
3:01:29
again but i just wanted to relay one question we got from the many
3:01:31
relay one question we got from the many
3:01:31
relay one question we got from the many questions we got on the chat one of our
3:01:33
questions we got on the chat one of our
3:01:33
questions we got on the chat one of our viewers were was asking
3:01:35
viewers were was asking
3:01:35
viewers were was asking should multiple micro services use the
3:01:37
should multiple micro services use the
3:01:37
should multiple micro services use the same cosmos db container
3:01:39
same cosmos db container
3:01:40
same cosmos db container should multiple micro services use the
3:01:41
should multiple micro services use the
3:01:41
should multiple micro services use the same cosmos db container so that's
3:01:43
same cosmos db container so that's
3:01:44
same cosmos db container so that's that's a that's an explosive question
3:01:46
that's a that's an explosive question
3:01:46
that's a that's an explosive question you know
3:01:47
you know
3:01:48
you know i kept things very simple here sometimes
3:01:49
i kept things very simple here sometimes
3:01:49
i kept things very simple here sometimes you can't just settle on one partition
3:01:51
you can't just settle on one partition
3:01:51
you can't just settle on one partition key and you'll need a microservice to
3:01:53
key and you'll need a microservice to
3:01:53
key and you'll need a microservice to actually
3:01:53
actually
3:01:53
actually replicate data from one container to
3:01:56
replicate data from one container to
3:01:56
replicate data from one container to another
3:01:56
another
3:01:56
another just because you have uh heavy
3:02:00
just because you have uh heavy
3:02:00
just because you have uh heavy uh querying patterns where half the time
3:02:02
uh querying patterns where half the time
3:02:02
uh querying patterns where half the time one property is suitable as the
3:02:04
one property is suitable as the
3:02:04
one property is suitable as the partition key the other half the time
3:02:05
partition key the other half the time
3:02:05
partition key the other half the time another property is suitable as the
3:02:07
another property is suitable as the
3:02:07
another property is suitable as the partition key then you can have one
3:02:09
partition key then you can have one
3:02:09
partition key then you can have one micro service
3:02:10
micro service
3:02:10
micro service whose purpose in life is to just keep
3:02:11
whose purpose in life is to just keep
3:02:11
whose purpose in life is to just keep those two containers in sync
3:02:13
those two containers in sync
3:02:13
those two containers in sync and then uh you will almost certainly
3:02:16
and then uh you will almost certainly
3:02:16
and then uh you will almost certainly need or
3:02:16
need or
3:02:16
need or want to create materialized views for
3:02:19
want to create materialized views for
3:02:19
want to create materialized views for queries that would otherwise be very
3:02:20
queries that would otherwise be very
3:02:20
queries that would otherwise be very expensive
3:02:21
expensive
3:02:21
expensive and that involves more micro services to
3:02:23
and that involves more micro services to
3:02:23
and that involves more micro services to generate to
3:02:24
generate to
3:02:24
generate to to create materialized views in yet
3:02:26
to create materialized views in yet
3:02:26
to create materialized views in yet other containers so you are talking
3:02:28
other containers so you are talking
3:02:28
other containers so you are talking about multiple containers
3:02:29
about multiple containers
3:02:29
about multiple containers and multiple micro services so although
3:02:31
and multiple micro services so although
3:02:31
and multiple micro services so although all those macro services could be all be
3:02:33
all those macro services could be all be
3:02:33
all those macro services could be all be consuming
3:02:34
consuming
3:02:34
consuming the change feed of a single container
3:02:36
the change feed of a single container
3:02:36
the change feed of a single container which is a really nice design
3:02:38
which is a really nice design
3:02:38
which is a really nice design aspect to the way micro services can be
3:02:40
aspect to the way micro services can be
3:02:40
aspect to the way micro services can be implemented you can have a
3:02:42
implemented you can have a
3:02:42
implemented you can have a micro services architecture where you've
3:02:44
micro services architecture where you've
3:02:44
micro services architecture where you've got uh microservices that are
3:02:46
got uh microservices that are
3:02:46
got uh microservices that are completely independent from one another
3:02:48
completely independent from one another
3:02:48
completely independent from one another that they're all sharing the same
3:02:49
that they're all sharing the same
3:02:50
that they're all sharing the same they're consuming the same change speed
3:02:51
they're consuming the same change speed
3:02:51
they're consuming the same change speed and maintaining state independently of
3:02:53
and maintaining state independently of
3:02:53
and maintaining state independently of one another regardless
3:02:54
one another regardless
3:02:54
one another regardless so it's a mix it's a mix and match it
3:02:56
so it's a mix it's a mix and match it
3:02:56
so it's a mix it's a mix and match it will depend on your scenario but you'll
3:02:58
will depend on your scenario but you'll
3:02:58
will depend on your scenario but you'll almost
3:02:58
almost
3:02:58
almost always involve multiple containers and
3:03:00
always involve multiple containers and
3:03:00
always involve multiple containers and multiple microservices
3:03:02
multiple microservices
3:03:02
multiple microservices that makes sense thanks again lenny we
3:03:04
that makes sense thanks again lenny we
3:03:04
that makes sense thanks again lenny we we couldn't have our first cosmos db
3:03:06
we couldn't have our first cosmos db
3:03:06
we couldn't have our first cosmos db conference without you
3:03:07
conference without you
3:03:07
conference without you i really appreciate your participation
3:03:09
i really appreciate your participation
3:03:09
i really appreciate your participation here
3:03:10
here
3:03:10
here of course and uh thanks again everyone
3:03:13
of course and uh thanks again everyone
3:03:13
of course and uh thanks again everyone so much for tuning in so it's been a
3:03:14
so much for tuning in so it's been a
3:03:14
so much for tuning in so it's been a really great live stream it's always
3:03:16
really great live stream it's always
3:03:16
really great live stream it's always great to see how our
3:03:17
great to see how our
3:03:17
great to see how our customers our partners and our community
3:03:19
customers our partners and our community
3:03:19
customers our partners and our community are building their applications with
3:03:21
are building their applications with
3:03:21
are building their applications with cosmos db
3:03:22
cosmos db
3:03:22
cosmos db uh i just wanted to check in tim and
3:03:23
uh i just wanted to check in tim and
3:03:23
uh i just wanted to check in tim and thomas did you have any interesting
3:03:25
thomas did you have any interesting
3:03:25
thomas did you have any interesting takeaways or insights or things that you
3:03:26
takeaways or insights or things that you
3:03:26
takeaways or insights or things that you wanted to
3:03:27
wanted to
3:03:27
wanted to share that you've learned from from what
3:03:29
share that you've learned from from what
3:03:29
share that you've learned from from what we saw today
3:03:31
we saw today
3:03:31
we saw today team i really enjoyed lenny's session
3:03:34
team i really enjoyed lenny's session
3:03:34
team i really enjoyed lenny's session that was awesome
3:03:35
that was awesome
3:03:35
that was awesome um i mean as somebody kind of who knows
3:03:38
um i mean as somebody kind of who knows
3:03:38
um i mean as somebody kind of who knows a lot about cosmos tv
3:03:39
a lot about cosmos tv
3:03:39
a lot about cosmos tv it was great to see a lot of tips and
3:03:40
it was great to see a lot of tips and
3:03:40
it was great to see a lot of tips and tricks that even more advanced users
3:03:42
tricks that even more advanced users
3:03:42
tricks that even more advanced users could uh
3:03:43
could uh
3:03:43
could uh could really use to to kind of become a
3:03:45
could really use to to kind of become a
3:03:46
could really use to to kind of become a power user of the product
3:03:47
power user of the product
3:03:47
power user of the product but thank you uh for watching today it
3:03:49
but thank you uh for watching today it
3:03:49
but thank you uh for watching today it was really great to have you as viewers
3:03:51
was really great to have you as viewers
3:03:51
was really great to have you as viewers in addition to these great sessions in
3:03:53
in addition to these great sessions in
3:03:53
in addition to these great sessions in our live stream we have
3:03:54
our live stream we have
3:03:54
our live stream we have 18 on-demand sessions that you can watch
3:03:56
18 on-demand sessions that you can watch
3:03:56
18 on-demand sessions that you can watch at any time
3:03:58
at any time
3:03:58
at any time so to see this great lineup of on-demand
3:04:00
so to see this great lineup of on-demand
3:04:00
so to see this great lineup of on-demand sessions you can visit gotcosmos.com
3:04:02
sessions you can visit gotcosmos.com
3:04:02
sessions you can visit gotcosmos.com and click on on-demand sessions you can
3:04:05
and click on on-demand sessions you can
3:04:05
and click on on-demand sessions you can also join us for a weekly podcast where
3:04:07
also join us for a weekly podcast where
3:04:07
also join us for a weekly podcast where our host mark brown will be
3:04:09
our host mark brown will be
3:04:09
our host mark brown will be members of the cosmos tv team each week
3:04:11
members of the cosmos tv team each week
3:04:11
members of the cosmos tv team each week uh and talk about things i'm relating to
3:04:13
uh and talk about things i'm relating to
3:04:14
uh and talk about things i'm relating to cosmos db like querying
3:04:15
cosmos db like querying
3:04:15
cosmos db like querying change feed analytics anything you can
3:04:17
change feed analytics anything you can
3:04:17
change feed analytics anything you can do with cosmos tv we'll cover on the
3:04:19
do with cosmos tv we'll cover on the
3:04:19
do with cosmos tv we'll cover on the show
3:04:20
show
3:04:20
show for the show schedule and upcoming
3:04:21
for the show schedule and upcoming
3:04:21
for the show schedule and upcoming episodes also you can go to
3:04:23
episodes also you can go to
3:04:23
episodes also you can go to cosmos.com tv and you'll be able to view
3:04:26
cosmos.com tv and you'll be able to view
3:04:26
cosmos.com tv and you'll be able to view all of that there
3:04:28
all of that there
3:04:28
all of that there and finally if you're new to cosmos tv
3:04:31
and finally if you're new to cosmos tv
3:04:31
and finally if you're new to cosmos tv and want to find new ways you can try it
3:04:33
and want to find new ways you can try it
3:04:33
and want to find new ways you can try it for free
3:04:33
for free
3:04:33
for free read our blog post on the four ways that
3:04:36
read our blog post on the four ways that
3:04:36
read our blog post on the four ways that you can try cosmos tv for free
3:04:38
you can try cosmos tv for free
3:04:38
you can try cosmos tv for free that'll be available at ak dot ms slash
3:04:40
that'll be available at ak dot ms slash
3:04:40
that'll be available at ak dot ms slash cosmos db
3:04:42
cosmos db
3:04:42
cosmos db dash free you can catch our next stream
3:04:46
dash free you can catch our next stream
3:04:46
dash free you can catch our next stream for the pacific time zones starting in
3:04:49
for the pacific time zones starting in
3:04:49
for the pacific time zones starting in eight hours that'll be at 8pm pacific
3:04:51
eight hours that'll be at 8pm pacific
3:04:51
eight hours that'll be at 8pm pacific time
3:04:52
time
3:04:52
time and 11 p.m us eastern time and thank you
3:04:55
and 11 p.m us eastern time and thank you
3:04:55
and 11 p.m us eastern time and thank you again for joining it was so great to
3:04:57
again for joining it was so great to
3:04:57
again for joining it was so great to have you with us
3:04:58
have you with us
3:04:58
have you with us for uh the first part of the cosmos db
3:05:00
for uh the first part of the cosmos db
3:05:00
for uh the first part of the cosmos db conference
3:05:02
conference
3:05:02
conference thank you