Up next in 10
Join this live session with Magnus Mårtensson ft. Clemens Vasters for the next episode of The Cloud Show with Magnus Mårtensson on February 26, at 01:05 PM (EST).
The show is about cloud leadership and all the important questions relating to cloud projects. Certainly, many matters when a company is going to and wants to be successful in the cloud, are about technology. However, there are many additional matters, adjacent to technology, that we also need to tend to regarding business strategy, human resources, organizational change, planning for a technical cloud approach, and many more questions. These conversations are critical for a healthy cloud and for a swift and accurate cloud approach.
GUEST SPEAKER
Clemens Vasters works at Microsoft on the Microsoft Azure product engineering team. As a Principal Architect for Service Bus, Event Hubs, Event Grid, and Relay, he lays the groundwork for future development and drives technical strategy.
He is the co-chair of the OASIS AMQP Technical Committee and a member of the MQTT TC, representing Microsoft in the CNCF CloudEvents Working Group.
Beyond technology, he has a deep passion for military aviation history, particularly the Cold War era. He has curated an extensive public photo archive from some of the world’s most significant aviation museums.
📺 CSharp TV - Dev Streaming Destination http://csharp.tv
🌎 C# Corner - Community of Software and Data Developers https://www.c-sharpcorner.com
#CSharpTV #csharpcorner #TheCloudShow #CSharp #Interview
Show More Show Less View Video Transcript
0:00
welcome to another episode of the cloud
0:02
welcome to another episode of the cloud
0:02
welcome to another episode of the cloud show I am not at home as you can see I
0:04
show I am not at home as you can see I
0:04
show I am not at home as you can see I am here in Belgium at the cloud Brew
0:08
am here in Belgium at the cloud Brew
0:08
am here in Belgium at the cloud Brew Conference standing in one of the
0:10
Conference standing in one of the
0:10
Conference standing in one of the sponsor Booth this is Axis sponsoring
0:12
sponsor Booth this is Axis sponsoring
0:12
sponsor Booth this is Axis sponsoring this Community Conference and we want to
0:14
this Community Conference and we want to
0:14
this Community Conference and we want to give a really good thank you for for
0:16
give a really good thank you for for
0:16
give a really good thank you for for companies that decide to sponsor
0:18
companies that decide to sponsor
0:18
companies that decide to sponsor community events awesome work now I want
0:21
community events awesome work now I want
0:21
community events awesome work now I want to talk to another person another
0:23
to talk to another person another
0:23
to talk to another person another speaker here at the conference we're
0:25
speaker here at the conference we're
0:25
speaker here at the conference we're going to be talking about metadata in
0:28
going to be talking about metadata in
0:28
going to be talking about metadata in messaging and whom better talk about
0:30
messaging and whom better talk about
0:30
messaging and whom better talk about with about messaging than the star of
0:32
with about messaging than the star of
0:32
with about messaging than the star of the show Clemens
0:34
the show Clemens
0:34
the show Clemens [Music]
0:41
[Music]
0:41
[Music] vast hello Clemens how are you doing hi
0:43
vast hello Clemens how are you doing hi
0:43
vast hello Clemens how are you doing hi Magnus fantastic that's great to see you
0:46
Magnus fantastic that's great to see you
0:46
Magnus fantastic that's great to see you man it's finally in the same space again
0:48
man it's finally in the same space again
0:48
man it's finally in the same space again you were saying that it's been a while
0:49
you were saying that it's been a while
0:49
you were saying that it's been a while since we saw each other uh well I don't
0:51
since we saw each other uh well I don't
0:51
since we saw each other uh well I don't know we haven't just haven't been to the
0:53
know we haven't just haven't been to the
0:53
know we haven't just haven't been to the same events I guess exactly yeah like
0:54
same events I guess exactly yeah like
0:54
same events I guess exactly yeah like three years or something or or minus P
0:58
three years or something or or minus P
0:58
three years or something or or minus P minus pandemic whatever right something
1:00
minus pandemic whatever right something
1:00
minus pandemic whatever right something yeah but you have a session here at the
1:02
yeah but you have a session here at the
1:02
yeah but you have a session here at the cloud Brew conference talking about well
1:04
cloud Brew conference talking about well
1:04
cloud Brew conference talking about well as you as you do messaging yes yes but
1:07
as you as you do messaging yes yes but
1:07
as you as you do messaging yes yes but specifically now talking well wait by
1:09
specifically now talking well wait by
1:09
specifically now talking well wait by the way I think everybody knows who
1:11
the way I think everybody knows who
1:11
the way I think everybody knows who Clemens is and I'm so excited to talk to
1:13
Clemens is and I'm so excited to talk to
1:13
Clemens is and I'm so excited to talk to Clemens tell the audience who you are
1:15
Clemens tell the audience who you are
1:15
Clemens tell the audience who you are Clemens um I'm the I'm the lead
1:18
Clemens um I'm the I'm the lead
1:18
Clemens um I'm the I'm the lead architect for the messaging services in
1:20
architect for the messaging services in
1:20
architect for the messaging services in the AZ Cloud so our team owns service
1:22
the AZ Cloud so our team owns service
1:22
the AZ Cloud so our team owns service bus event tubs event grid um stream
1:25
bus event tubs event grid um stream
1:25
bus event tubs event grid um stream analytics the a relay and also the uh
1:28
analytics the a relay and also the uh
1:28
analytics the a relay and also the uh fabric event streams inside of Microsoft
1:31
fabric event streams inside of Microsoft
1:31
fabric event streams inside of Microsoft fabric so a bunch of messaging stuff yes
1:33
fabric so a bunch of messaging stuff yes
1:33
fabric so a bunch of messaging stuff yes essentially that's that's the summary of
1:35
essentially that's that's the summary of
1:35
essentially that's that's the summary of it all messaging all right cool so now
1:38
it all messaging all right cool so now
1:38
it all messaging all right cool so now you said in your session and I just I
1:40
you said in your session and I just I
1:40
you said in your session and I just I just listened in on his session that um
1:43
just listened in on his session that um
1:43
just listened in on his session that um the the problem is not in messaging or
1:45
the the problem is not in messaging or
1:45
the the problem is not in messaging or the infrastructure or any of those
1:47
the infrastructure or any of those
1:47
the infrastructure or any of those things the the challenge remaining is in
1:49
things the the challenge remaining is in
1:49
things the the challenge remaining is in a different space in metadata please
1:51
a different space in metadata please
1:51
a different space in metadata please explain yes um so one of the goals we
1:55
explain yes um so one of the goals we
1:55
explain yes um so one of the goals we have for so let let me start with yeah
1:57
have for so let let me start with yeah
1:57
have for so let let me start with yeah we're building we're Building Services
1:59
we're building we're Building Services
1:59
we're building we're Building Services both in a and in fabric and fabric is a
2:02
both in a and in fabric and fabric is a
2:02
both in a and in fabric and fabric is a very has a very different target
2:03
very has a very different target
2:03
very has a very different target audience from the the developer audience
2:05
audience from the the developer audience
2:05
audience from the the developer audience that we find in in Azure so yeah the the
2:08
that we find in in Azure so yeah the the
2:08
that we find in in Azure so yeah the the the the target audience we have for um
2:11
the the target audience we have for um
2:11
the the target audience we have for um fabric is data analysts maybe data
2:15
fabric is data analysts maybe data
2:15
fabric is data analysts maybe data Engineers but also end users yeah and of
2:17
Engineers but also end users yeah and of
2:17
Engineers but also end users yeah and of course we can't confront them with all
2:19
course we can't confront them with all
2:19
course we can't confront them with all the complexity of building applications
2:22
the complexity of building applications
2:22
the complexity of building applications or even these these pipelines well
2:24
or even these these pipelines well
2:24
or even these these pipelines well they're not developers at court they're
2:25
they're not developers at court they're
2:26
they're not developers at court they're different kinds of people correct so so
2:28
different kinds of people correct so so
2:28
different kinds of people correct so so we need to we need to be able to make
2:30
we need to we need to be able to make
2:30
we need to we need to be able to make very easy for them to kind of plug
2:31
very easy for them to kind of plug
2:31
very easy for them to kind of plug pipelines together and plugging
2:34
pipelines together and plugging
2:34
pipelines together and plugging pipelines together means you need to
2:36
pipelines together means you need to
2:36
pipelines together means you need to have some structure like Lego right
2:38
have some structure like Lego right
2:38
have some structure like Lego right right where you can go and say ah this
2:40
right where you can go and say ah this
2:40
right where you can go and say ah this thing has uh know this thing fits into
2:42
thing has uh know this thing fits into
2:43
thing has uh know this thing fits into this thing this fits into this thing now
2:45
this thing this fits into this thing now
2:45
this thing this fits into this thing now with data streams that is hard because
2:47
with data streams that is hard because
2:47
with data streams that is hard because data streams just move bytes yeah and
2:50
data streams just move bytes yeah and
2:50
data streams just move bytes yeah and and and you were saying very nicely in
2:52
and and you were saying very nicely in
2:52
and and you were saying very nicely in your talk which I I I enjoyed that um we
2:56
your talk which I I I enjoyed that um we
2:56
your talk which I I I enjoyed that um we have to agree on what the message
2:58
have to agree on what the message
2:58
have to agree on what the message structure isre if I'm going to send you
3:00
structure isre if I'm going to send you
3:00
structure isre if I'm going to send you a message through a pipeline and
3:01
a message through a pipeline and
3:01
a message through a pipeline and sometime later you're going to you're
3:03
sometime later you're going to you're
3:03
sometime later you're going to you're going to do something with that we have
3:04
going to do something with that we have
3:05
going to do something with that we have to first agree what's the format of the
3:06
to first agree what's the format of the
3:06
to first agree what's the format of the data we're actually sending here yeah so
3:08
data we're actually sending here yeah so
3:08
data we're actually sending here yeah so there's there's two there's two elements
3:10
there's there's two there's two elements
3:10
there's there's two there's two elements of the is there's there's structure and
3:12
of the is there's there's structure and
3:12
of the is there's there's structure and then there's encoding right um and you
3:15
then there's encoding right um and you
3:15
then there's encoding right um and you know you may want to go and send the
3:16
know you may want to go and send the
3:16
know you may want to go and send the data as Jason you may want to go and
3:17
data as Jason you may want to go and
3:17
data as Jason you may want to go and send the data as Avro you may but then
3:20
send the data as Avro you may but then
3:20
send the data as Avro you may but then you also need to go and Define what the
3:21
you also need to go and Define what the
3:21
you also need to go and Define what the structure of the data is and very often
3:24
structure of the data is and very often
3:24
structure of the data is and very often in practice today yeah people are just
3:27
in practice today yeah people are just
3:27
in practice today yeah people are just building apps that are just putting some
3:29
building apps that are just putting some
3:30
building apps that are just putting some data on The Wire Without Really
3:32
data on The Wire Without Really
3:32
data on The Wire Without Really describing that to everybody else who's
3:34
describing that to everybody else who's
3:34
describing that to everybody else who's collaborating with them and that works
3:37
collaborating with them and that works
3:37
collaborating with them and that works because right now pipeline pipelines
3:39
because right now pipeline pipelines
3:39
because right now pipeline pipelines even down to visualizations are built by
3:42
even down to visualizations are built by
3:42
even down to visualizations are built by just a few people so it's all tribal
3:44
just a few people so it's all tribal
3:44
just a few people so it's all tribal knowledge and they all can all go and
3:46
knowledge and they all can all go and
3:46
knowledge and they all can all go and look at the Cod but that's something
3:47
look at the Cod but that's something
3:47
look at the Cod but that's something that doesn't scale right no it doesn't
3:49
that doesn't scale right no it doesn't
3:49
that doesn't scale right no it doesn't and and also if you're sending if I'm
3:51
and and also if you're sending if I'm
3:51
and and also if you're sending if I'm sending my data to some completely
3:54
sending my data to some completely
3:54
sending my data to some completely different system some some someone else
3:56
different system some some someone else
3:57
different system some some someone else you're not sitting across from me
3:58
you're not sitting across from me
3:58
you're not sitting across from me working together with me you're a
4:00
working together with me you're a
4:00
working together with me you're a different you're a different consumer
4:01
different you're a different consumer
4:01
different you're a different consumer somewhere else corre then I need to
4:02
somewhere else corre then I need to
4:02
somewhere else corre then I need to still like tell you what I intended my
4:05
still like tell you what I intended my
4:05
still like tell you what I intended my data to be like yes so so the obvious
4:08
data to be like yes so so the obvious
4:08
data to be like yes so so the obvious solution to that problem is to um it
4:12
solution to that problem is to um it
4:12
solution to that problem is to um it happens at two layers so first when you
4:14
happens at two layers so first when you
4:14
happens at two layers so first when you when you're sending a message to someone
4:16
when you're sending a message to someone
4:16
when you're sending a message to someone else you need to be able to go and
4:18
else you need to be able to go and
4:18
else you need to be able to go and declare what is the shape of the data
4:20
declare what is the shape of the data
4:20
declare what is the shape of the data that sits inside that message right
4:22
that sits inside that message right
4:22
that sits inside that message right right yep and we have we have Tech
4:24
right yep and we have we have Tech
4:24
right yep and we have we have Tech mechanisms for this we have things like
4:25
mechanisms for this we have things like
4:25
mechanisms for this we have things like Jason schema we have AOS schemas we have
4:28
Jason schema we have AOS schemas we have
4:28
Jason schema we have AOS schemas we have protuff we have all these this
4:30
protuff we have all these this
4:30
protuff we have all these this information and for for encodings like
4:33
information and for for encodings like
4:33
information and for for encodings like Avro and for and like protuff it's
4:35
Avro and for and like protuff it's
4:35
Avro and for and like protuff it's required that you have these schemas
4:37
required that you have these schemas
4:37
required that you have these schemas because otherwise you can't encode or
4:38
because otherwise you can't encode or
4:38
because otherwise you can't encode or decode okay and for for Jason people
4:41
decode okay and for for Jason people
4:41
decode okay and for for Jason people mostly think like ah I probably don't
4:43
mostly think like ah I probably don't
4:43
mostly think like ah I probably don't need this because the data is
4:45
need this because the data is
4:45
need this because the data is self-explanatory right and it carries
4:47
self-explanatory right and it carries
4:47
self-explanatory right and it carries the type and and the uh um and the and
4:50
the type and and the uh um and the and
4:50
the type and and the uh um and the and the data what they forget is that if the
4:55
the data what they forget is that if the
4:55
the data what they forget is that if the person who sits at the other end of the
4:56
person who sits at the other end of the
4:56
person who sits at the other end of the pipeline is not a developer and it's not
4:59
pipeline is not a developer and it's not
4:59
pipeline is not a developer and it's not or but there might be a data analyst or
5:02
or but there might be a data analyst or
5:02
or but there might be a data analyst or someone else who wants to make sense out
5:04
someone else who wants to make sense out
5:04
someone else who wants to make sense out of that data and you have a field that's
5:05
of that data and you have a field that's
5:05
of that data and you have a field that's called temperature right that's related
5:08
called temperature right that's related
5:08
called temperature right that's related to to a light bulb right what does that
5:10
to to a light bulb right what does that
5:10
to to a light bulb right what does that mean what does it mean yeah so well it
5:12
mean what does it mean yeah so well it
5:12
mean what does it mean yeah so well it could be the temperature of the light in
5:16
could be the temperature of the light in
5:16
could be the temperature of the light in kelvin yes it could be the temperature
5:19
kelvin yes it could be the temperature
5:19
kelvin yes it could be the temperature when you when you touch the light bulb
5:21
when you when you touch the light bulb
5:21
when you when you touch the light bulb right in Celsius or in Fahrenheit you
5:23
right in Celsius or in Fahrenheit you
5:23
right in Celsius or in Fahrenheit you you can't know so so the field say that
5:26
you can't know so so the field say that
5:26
you can't know so so the field say that it's temperature yes but what
5:28
it's temperature yes but what
5:28
it's temperature yes but what temperature exactly yeah so and the way
5:31
temperature exactly yeah so and the way
5:31
temperature exactly yeah so and the way how you find that out is by having a
5:32
how you find that out is by having a
5:32
how you find that out is by having a schema that has a description in it I
5:34
schema that has a description in it I
5:34
schema that has a description in it I mean it's not it's not rocket science no
5:36
mean it's not it's not rocket science no
5:36
mean it's not it's not rocket science no no it's not at all like we haven't done
5:38
no it's not at all like we haven't done
5:38
no it's not at all like we haven't done specs before so for so for that for for
5:41
specs before so for so for that for for
5:41
specs before so for so for that for for for managing these sorts of schemas and
5:43
for managing these sorts of schemas and
5:43
for managing these sorts of schemas and messaging we have schema Registries
5:46
messaging we have schema Registries
5:46
messaging we have schema Registries those things exist and they have existed
5:48
those things exist and they have existed
5:48
those things exist and they have existed so you publish your schema say here's my
5:50
so you publish your schema say here's my
5:50
so you publish your schema say here's my schema and then the party that's
5:52
schema and then the party that's
5:52
schema and then the party that's receiving it can receive it but those
5:54
receiving it can receive it but those
5:54
receiving it can receive it but those schema Registries have been effectively
5:56
schema Registries have been effectively
5:56
schema Registries have been effectively limited the scope has been limited to
5:58
limited the scope has been limited to
5:58
limited the scope has been limited to like ca or the event stream okay but
6:01
like ca or the event stream okay but
6:01
like ca or the event stream okay but they could not really be used by by um
6:04
they could not really be used by by um
6:04
they could not really be used by by um you know a database or anything else
6:07
you know a database or anything else
6:07
you know a database or anything else that sits Downstream they could not
6:08
that sits Downstream they could not
6:08
that sits Downstream they could not leverage this okay so we need to have an
6:10
leverage this okay so we need to have an
6:10
leverage this okay so we need to have an open infrastructure we need to have an
6:12
open infrastructure we need to have an
6:12
open infrastructure we need to have an API we need to have a way to go and and
6:14
API we need to have a way to go and and
6:14
API we need to have a way to go and and and you know talk to that schema
6:15
and you know talk to that schema
6:15
and you know talk to that schema registry from all kinds of pieces of
6:17
registry from all kinds of pieces of
6:17
registry from all kinds of pieces of infrastructure okay in addition to this
6:19
infrastructure okay in addition to this
6:19
infrastructure okay in addition to this when you're sending data through an
6:20
when you're sending data through an
6:21
when you're sending data through an event stream you usually don't only send
6:23
event stream you usually don't only send
6:23
event stream you usually don't only send one type of messages messages you send
6:26
one type of messages messages you send
6:26
one type of messages messages you send three four 10 different kinds of
6:28
three four 10 different kinds of
6:28
three four 10 different kinds of messages Okay so multiplexing so now
6:31
messages Okay so multiplexing so now
6:31
messages Okay so multiplexing so now it's not one schema you're actually
6:32
it's not one schema you're actually
6:33
it's not one schema you're actually dealing with 10 schemas okay yeah so how
6:36
dealing with 10 schemas okay yeah so how
6:36
dealing with 10 schemas okay yeah so how do you distinguish those messages well
6:38
do you distinguish those messages well
6:38
do you distinguish those messages well one way to distinguish them is by having
6:40
one way to distinguish them is by having
6:40
one way to distinguish them is by having standardized metadata that says this is
6:42
standardized metadata that says this is
6:42
standardized metadata that says this is this kind of message and this is that
6:44
this kind of message and this is that
6:44
this kind of message and this is that kind of message sure so work we did four
6:46
kind of message sure so work we did four
6:46
kind of message sure so work we did four years we did we finished four years ago
6:48
years we did we finished four years ago
6:48
years we did we finished four years ago and since then has been uh nicely
6:51
and since then has been uh nicely
6:51
and since then has been uh nicely adopted is uh Cloud events in cncf yep
6:55
adopted is uh Cloud events in cncf yep
6:55
adopted is uh Cloud events in cncf yep um and Cloud events gives you
6:56
um and Cloud events gives you
6:56
um and Cloud events gives you effectively a standard way to describe
6:59
effectively a standard way to describe
6:59
effectively a standard way to describe know what in an event what is the type
7:01
know what in an event what is the type
7:01
know what in an event what is the type of the event what is the context of the
7:03
of the event what is the context of the
7:04
of the event what is the context of the event so we're leveraging we're
7:06
event so we're leveraging we're
7:06
event so we're leveraging we're leveraging that to kind of manage M Dem
7:09
leveraging that to kind of manage M Dem
7:09
leveraging that to kind of manage M Dem multiplexing and multiplexing of event
7:11
multiplexing and multiplexing of event
7:11
multiplexing and multiplexing of event streams what we saw in the in the cloud
7:14
streams what we saw in the in the cloud
7:14
streams what we saw in the in the cloud events project then is that we need to
7:15
events project then is that we need to
7:15
events project then is that we need to have a machine readable version if you
7:17
have a machine readable version if you
7:17
have a machine readable version if you will of a declaration of which events
7:20
will of a declaration of which events
7:20
will of a declaration of which events can be associated with a stream okay and
7:22
can be associated with a stream okay and
7:22
can be associated with a stream okay and that's how we started in an effort that
7:25
that's how we started in an effort that
7:25
that's how we started in an effort that was first called uh Cloud Discovery and
7:28
was first called uh Cloud Discovery and
7:28
was first called uh Cloud Discovery and it's now called X registry
7:30
it's now called X registry
7:30
it's now called X registry and what x registry does is gives you um
7:33
and what x registry does is gives you um
7:33
and what x registry does is gives you um first of all an extensible metadata
7:35
first of all an extensible metadata
7:35
first of all an extensible metadata catalog for all kinds of metadata okay
7:37
catalog for all kinds of metadata okay
7:37
catalog for all kinds of metadata okay and then specifically implements three
7:39
and then specifically implements three
7:39
and then specifically implements three metadata Registries one is for endpoints
7:42
metadata Registries one is for endpoints
7:42
metadata Registries one is for endpoints where you can go and register you know I
7:45
where you can go and register you know I
7:45
where you can go and register you know I have this endpoint over there that
7:46
have this endpoint over there that
7:46
have this endpoint over there that speaks Kafka and uh has the following
7:49
speaks Kafka and uh has the following
7:49
speaks Kafka and uh has the following following uh characteristics okay we
7:52
following uh characteristics okay we
7:52
following uh characteristics okay we have a um a registry that can handle
7:56
have a um a registry that can handle
7:56
have a um a registry that can handle message definitions or event definitions
7:57
message definitions or event definitions
7:57
message definitions or event definitions where you can so there's a a set of
7:59
where you can so there's a a set of
7:59
where you can so there's a a set of cloud events M that have these types and
8:03
cloud events M that have these types and
8:03
cloud events M that have these types and you can go and take that group and
8:04
you can go and take that group and
8:04
you can go and take that group and Associate that group with a stream now
8:06
Associate that group with a stream now
8:06
Associate that group with a stream now you have a contract now you can say I
8:08
you have a contract now you can say I
8:08
you have a contract now you can say I can send four events to this to this
8:09
can send four events to this to this
8:10
can send four events to this to this stream and I can expect four these four
8:11
stream and I can expect four these four
8:11
stream and I can expect four these four events coming out of that and then
8:13
events coming out of that and then
8:14
events coming out of that and then there's a schema registry and the schema
8:16
there's a schema registry and the schema
8:16
there's a schema registry and the schema registry then basically gives you the
8:17
registry then basically gives you the
8:17
registry then basically gives you the the the detail in the data inside
8:20
the the detail in the data inside
8:20
the the detail in the data inside effectively into what's carried inside
8:22
effectively into what's carried inside
8:22
effectively into what's carried inside of these these message envelop right so
8:25
of these these message envelop right so
8:25
of these these message envelop right so it's a it's three Registries which
8:28
it's a it's three Registries which
8:28
it's a it's three Registries which follow the same exactly the same
8:30
follow the same exactly the same
8:30
follow the same exactly the same mechanics in terms of interacting with
8:31
mechanics in terms of interacting with
8:31
mechanics in terms of interacting with them through the API and also for the
8:33
them through the API and also for the
8:33
them through the API and also for the document format which is also there um
8:36
document format which is also there um
8:36
document format which is also there um which cover this different aspects
8:37
which cover this different aspects
8:37
which cover this different aspects effectively of the metadata problem and
8:40
effectively of the metadata problem and
8:40
effectively of the metadata problem and what we're um what we have been building
8:42
what we're um what we have been building
8:42
what we're um what we have been building or are
8:43
or are
8:43
or are building kind of as a preview for the
8:46
building kind of as a preview for the
8:46
building kind of as a preview for the product but also as open source
8:48
product but also as open source
8:48
product but also as open source experimentation field is tooling that
8:51
experimentation field is tooling that
8:51
experimentation field is tooling that can now go and take that metadata and um
8:54
can now go and take that metadata and um
8:54
can now go and take that metadata and um so from the schemas um generate code for
8:57
so from the schemas um generate code for
8:57
so from the schemas um generate code for data transfer objects okay
9:00
data transfer objects okay
9:00
data transfer objects okay in a consistent way yeah and then can
9:02
in a consistent way yeah and then can
9:02
in a consistent way yeah and then can also go and take a look at the these
9:04
also go and take a look at the these
9:04
also go and take a look at the these these message definitions basically
9:06
these message definitions basically
9:06
these message definitions basically these interface declarations and can
9:07
these interface declarations and can
9:08
these interface declarations and can create full clients wow so that you can
9:10
create full clients wow so that you can
9:10
create full clients wow so that you can go and create let's say a Kafka um
9:14
go and create let's say a Kafka um
9:14
go and create let's say a Kafka um producer that's great because and you
9:16
producer that's great because and you
9:16
producer that's great because and you can say the Kaa producer needs to go and
9:18
can say the Kaa producer needs to go and
9:18
can say the Kaa producer needs to go and produce these F these seven messages
9:20
produce these F these seven messages
9:20
produce these F these seven messages okay and and the the the send message
9:23
okay and and the the the send message
9:23
okay and and the the the send message only takes your business object
9:25
only takes your business object
9:25
only takes your business object effectively your your your payload
9:26
effectively your your your payload
9:26
effectively your your your payload object as parameter and then the
9:29
object as parameter and then the
9:29
object as parameter and then the creating the right Cloud event with the
9:31
creating the right Cloud event with the
9:31
creating the right Cloud event with the right attributes sticking it in the
9:33
right attributes sticking it in the
9:33
right attributes sticking it in the right place with the right encoding into
9:34
right place with the right encoding into
9:34
right place with the right encoding into the Kafka producer all that all that
9:36
the Kafka producer all that all that
9:36
the Kafka producer all that all that goop that is most repetitive work that's
9:39
goop that is most repetitive work that's
9:39
goop that is most repetitive work that's already done that can already be done by
9:41
already done that can already be done by
9:41
already done that can already be done by a CO generator that's so good yes so so
9:44
a CO generator that's so good yes so so
9:44
a CO generator that's so good yes so so so the goal of this the goal of all of
9:45
so the goal of this the goal of all of
9:45
so the goal of this the goal of all of this is effectively
9:47
this is effectively
9:47
this is effectively to uh enable uh less friction in or
9:51
to uh enable uh less friction in or
9:51
to uh enable uh less friction in or allow less friction in the development
9:53
allow less friction in the development
9:53
allow less friction in the development of these pipelines because we see
9:56
of these pipelines because we see
9:56
of these pipelines because we see massive interest in in real time dat
9:59
massive interest in in real time dat
9:59
massive interest in in real time dat feeds but building them is too hard so
10:02
feeds but building them is too hard so
10:02
feeds but building them is too hard so we try to take the friction out and
10:04
we try to take the friction out and
10:04
we try to take the friction out and ultimately the vision
10:06
ultimately the vision
10:06
ultimately the vision is that you um can uh Define data
10:11
is that you um can uh Define data
10:11
is that you um can uh Define data structures you can describe data
10:12
structures you can describe data
10:12
structures you can describe data structures you can you can make them
10:14
structures you can you can make them
10:14
structures you can you can make them known in through the catalog to an
10:17
known in through the catalog to an
10:17
known in through the catalog to an analyst even without there necessarily
10:19
analyst even without there necessarily
10:19
analyst even without there necessarily being already a data feat that gives you
10:21
being already a data feat that gives you
10:21
being already a data feat that gives you the data okay yeah the data analyst can
10:23
the data okay yeah the data analyst can
10:23
the data okay yeah the data analyst can then go go go to their co-pilot okay and
10:26
then go go go to their co-pilot okay and
10:26
then go go go to their co-pilot okay and say brilliant hey um I I would like to
10:29
say brilliant hey um I I would like to
10:29
say brilliant hey um I I would like to have the following
10:31
have the following
10:31
have the following visualization for the following
10:33
visualization for the following
10:33
visualization for the following speculative data oh nice because the
10:36
speculative data oh nice because the
10:36
speculative data oh nice because the co-pilot can understand the metadata
10:38
co-pilot can understand the metadata
10:38
co-pilot can understand the metadata interpret that's it's the metadata so
10:40
interpret that's it's the metadata so
10:40
interpret that's it's the metadata so the copil can now basically can now
10:42
the copil can now basically can now
10:42
the copil can now basically can now basically figure out it iteratively
10:45
basically figure out it iteratively
10:45
basically figure out it iteratively we're going to go and do a a
10:46
we're going to go and do a a
10:46
we're going to go and do a a visualization for data that you don't
10:48
visualization for data that you don't
10:48
visualization for data that you don't have yet uhuh and then you can basically
10:51
have yet uhuh and then you can basically
10:51
have yet uhuh and then you can basically go and create you can finish your
10:53
go and create you can finish your
10:53
go and create you can finish your visualization we can use effectively
10:55
visualization we can use effectively
10:55
visualization we can use effectively that metadata to also you know create a
10:57
that metadata to also you know create a
10:57
that metadata to also you know create a little data generator which goes and
10:59
little data generator which goes and
10:59
little data generator which goes and creates fake data okay fair yeah um and
11:03
creates fake data okay fair yeah um and
11:03
creates fake data okay fair yeah um and then once you have that once you're
11:06
then once you have that once you're
11:06
then once you have that once you're happy with your sketch of the
11:07
happy with your sketch of the
11:07
happy with your sketch of the visualization you have you now have a
11:10
visualization you have you now have a
11:10
visualization you have you now have a model effectively of the data stream
11:11
model effectively of the data stream
11:11
model effectively of the data stream that you want you can give that to a
11:13
that you want you can give that to a
11:13
that you want you can give that to a developer by ways of just generate an
11:15
developer by ways of just generate an
11:15
developer by ways of just generate an SDK for them and say you go and give me
11:18
SDK for them and say you go and give me
11:18
SDK for them and say you go and give me give me that data and defeat that to me
11:19
give me that data and defeat that to me
11:19
give me that data and defeat that to me so you can enable right to left
11:21
so you can enable right to left
11:21
so you can enable right to left development as I call it that makes
11:22
development as I call it that makes
11:23
development as I call it that makes sense yes very cool all right so that's
11:25
sense yes very cool all right so that's
11:25
sense yes very cool all right so that's what you're heading going towards yeah
11:27
what you're heading going towards yeah
11:27
what you're heading going towards yeah so we're we're effective we're trying to
11:28
so we're we're effective we're trying to
11:28
so we're we're effective we're trying to do so in in a sense it is open API for
11:32
do so in in a sense it is open API for
11:32
do so in in a sense it is open API for streaming right that's the simplest way
11:35
streaming right that's the simplest way
11:35
streaming right that's the simplest way how you can look at it but it's a very
11:38
how you can look at it but it's a very
11:38
how you can look at it but it's a very specific uh metadata model for really
11:41
specific uh metadata model for really
11:41
specific uh metadata model for really for event streams or for messaging which
11:43
for event streams or for messaging which
11:43
for event streams or for messaging which is all
11:44
is all
11:44
is all unidirectional uh U you know pipelines
11:46
unidirectional uh U you know pipelines
11:46
unidirectional uh U you know pipelines for data so there's more weight on the
11:49
for data so there's more weight on the
11:49
for data so there's more weight on the data rather than on the um on this this
11:52
data rather than on the um on this this
11:52
data rather than on the um on this this API interaction AP I forgot and there's
11:54
API interaction AP I forgot and there's
11:54
API interaction AP I forgot and there's a and if I if I look into the future of
11:56
a and if I if I look into the future of
11:56
a and if I if I look into the future of that standardization story
11:59
that standardization story
11:59
that standardization story um once we're done with with X registry
12:01
um once we're done with with X registry
12:02
um once we're done with with X registry which I hope is uh very soon okay um at
12:05
which I hope is uh very soon okay um at
12:05
which I hope is uh very soon okay um at least the the 10 version um the next
12:07
least the the 10 version um the next
12:07
least the the 10 version um the next Frontier is effectively how do we how do
12:10
Frontier is effectively how do we how do
12:10
Frontier is effectively how do we how do we model these interaction shapes so if
12:14
we model these interaction shapes so if
12:14
we model these interaction shapes so if you for instance go
12:15
you for instance go
12:15
you for instance go into financial industry is now switching
12:18
into financial industry is now switching
12:18
into financial industry is now switching to ISO 2022 which is kind of the the the
12:22
to ISO 2022 which is kind of the the the
12:22
to ISO 2022 which is kind of the the the data exchange standards for them all in
12:24
data exchange standards for them all in
12:24
data exchange standards for them all in XML
12:26
XML
12:26
XML yeah and um so
12:30
yeah and um so
12:30
yeah and um so when you look at the the the sets for
12:32
when you look at the the the sets for
12:32
when you look at the the the sets for instance the messages are exchanged in
12:35
instance the messages are exchanged in
12:35
instance the messages are exchanged in um uh a cache machine yeah there's a set
12:39
um uh a cache machine yeah there's a set
12:39
um uh a cache machine yeah there's a set of exchanges that's defined between the
12:40
of exchanges that's defined between the
12:41
of exchanges that's defined between the bank and the cash machine and how they
12:42
bank and the cash machine and how they
12:42
bank and the cash machine and how they how that all happens that is it's
12:45
how that all happens that is it's
12:45
how that all happens that is it's defined as an activity diagram in uml oh
12:49
defined as an activity diagram in uml oh
12:49
defined as an activity diagram in uml oh okay in the documents right and many
12:51
okay in the documents right and many
12:51
okay in the documents right and many people do this as activity diagram so
12:53
people do this as activity diagram so
12:53
people do this as activity diagram so you can have swim lanes and then you
12:57
you can have swim lanes and then you
12:57
you can have swim lanes and then you have so what so I think the next step
13:01
have so what so I think the next step
13:01
have so what so I think the next step better than having an API approach where
13:03
better than having an API approach where
13:03
better than having an API approach where you have like oh input output and you
13:06
you have like oh input output and you
13:06
you have like oh input output and you only look at this from one perspective
13:08
only look at this from one perspective
13:08
only look at this from one perspective have a holistic formal description okay
13:11
have a holistic formal description okay
13:11
have a holistic formal description okay that you can go and then create for
13:12
that you can go and then create for
13:13
that you can go and then create for these flows there there has been
13:14
these flows there there has been
13:14
these flows there there has been priority for this in uh in the itu in
13:17
priority for this in uh in the itu in
13:17
priority for this in uh in the itu in the 90s called
13:19
the 90s called
13:19
the 90s called z120 it's a don't look at it um but but
13:23
z120 it's a don't look at it um but but
13:23
z120 it's a don't look at it um but but they have they had kind of similar
13:25
they have they had kind of similar
13:25
they have they had kind of similar approach that's kind of kind of where we
13:26
approach that's kind of kind of where we
13:26
approach that's kind of kind of where we want to go with all right very cool so
13:29
want to go with all right very cool so
13:29
want to go with all right very cool so big story yeah it's a huge story and and
13:31
big story yeah it's a huge story and and
13:31
big story yeah it's a huge story and and I think from what I saw it's really cool
13:33
I think from what I saw it's really cool
13:33
I think from what I saw it's really cool to be able to get that uh stuff
13:35
to be able to get that uh stuff
13:35
to be able to get that uh stuff generated like the the plumbing code
13:37
generated like the the plumbing code
13:38
generated like the the plumbing code stuff no matter what I'm going to talk
13:39
stuff no matter what I'm going to talk
13:39
stuff no matter what I'm going to talk to which database it's going into it
13:41
to which database it's going into it
13:41
to which database it's going into it doesn't matter all of that is boiler
13:43
doesn't matter all of that is boiler
13:43
doesn't matter all of that is boiler plate and can be generated yes and and
13:46
plate and can be generated yes and and
13:46
plate and can be generated yes and and why would why you know would anyone ever
13:48
why would why you know would anyone ever
13:48
why would why you know would anyone ever build that corre themselves and and it's
13:50
build that corre themselves and and it's
13:51
build that corre themselves and and it's it's the when you when you look at these
13:53
it's the when you when you look at these
13:53
it's the when you when you look at these these these tools so one is the X
13:55
these these tools so one is the X
13:55
these these tools so one is the X registry CLI tool that's in my repo the
13:58
registry CLI tool that's in my repo the
13:58
registry CLI tool that's in my repo the other one is called advertise sure um
14:01
other one is called advertise sure um
14:01
other one is called advertise sure um and when you when you look at those
14:03
and when you when you look at those
14:03
and when you when you look at those tools and when you look at the input
14:05
tools and when you look at the input
14:05
tools and when you look at the input metadata and what comes out of it you
14:07
metadata and what comes out of it you
14:07
metadata and what comes out of it you see that the input metadata is very
14:08
see that the input metadata is very
14:08
see that the input metadata is very minimal it's it's not a super
14:10
minimal it's it's not a super
14:10
minimal it's it's not a super complicated set of things that you need
14:12
complicated set of things that you need
14:12
complicated set of things that you need to go and Define and then you see how
14:14
to go and Define and then you see how
14:14
to go and Define and then you see how much complexity there needs to be to go
14:16
much complexity there needs to be to go
14:16
much complexity there needs to be to go and interact with that messaging
14:18
and interact with that messaging
14:18
and interact with that messaging infrastructure cons consistently and
14:21
infrastructure cons consistently and
14:21
infrastructure cons consistently and that's all stuff that nobody really
14:22
that's all stuff that nobody really
14:22
that's all stuff that nobody really should
14:23
should
14:23
should right um but it's it's required to be
14:26
right um but it's it's required to be
14:26
right um but it's it's required to be interoperable right so there's like
14:28
interoperable right so there's like
14:28
interoperable right so there's like there's there's of complexity that's
14:29
there's there's of complexity that's
14:29
there's there's of complexity that's required for interoperability so that
14:31
required for interoperability so that
14:31
required for interoperability so that everybody can go and agree on those
14:33
everybody can go and agree on those
14:33
everybody can go and agree on those things but at the same time for an
14:34
things but at the same time for an
14:34
things but at the same time for an application developer it's it's too much
14:37
application developer it's it's too much
14:37
application developer it's it's too much so how can we go and bridge that Gap and
14:38
so how can we go and bridge that Gap and
14:38
so how can we go and bridge that Gap and make the application developer not worry
14:40
make the application developer not worry
14:40
make the application developer not worry about all that stuff too much perfect
14:42
about all that stuff too much perfect
14:42
about all that stuff too much perfect well great thank you for being on the
14:44
well great thank you for being on the
14:44
well great thank you for being on the cloud show CL absolutely a great
14:47
cloud show CL absolutely a great
14:47
cloud show CL absolutely a great explanation of of metadata in the
14:49
explanation of of metadata in the
14:49
explanation of of metadata in the messaging world and audience see you
14:51
messaging world and audience see you
14:51
messaging world and audience see you next time on the cloud show
14:54
next time on the cloud show
14:54
next time on the cloud show [Music]
#Business & Industrial
#Computers & Electronics