Up next in 10
In this episode of the Global AI October sessions we focus on Natural Language Processing with industry experts from around the world.
Show More Show Less View Video Transcript
0:11
[Music]
0:37
[Music]
0:56
[Music]
1:29
[Music]
1:35
[Music]
1:35
[Music] [Applause]
1:36
[Applause]
1:36
[Applause] [Music]
1:58
[Music]
2:15
[Music]
2:22
[Music]
2:46
[Music]
2:48
[Music]
2:48
[Music] so
2:51
so
2:51
so [Music]
3:31
[Music]
3:31
[Music] my
3:37
[Music]
4:48
[Music]
5:02
[Music]
5:02
[Music] hi everyone and welcome at another
5:05
hi everyone and welcome at another
5:05
hi everyone and welcome at another episode of the october sessions
5:07
episode of the october sessions
5:07
episode of the october sessions we're at episode 3 of our four episodes
5:10
we're at episode 3 of our four episodes
5:10
we're at episode 3 of our four episodes that we'll have
5:11
that we'll have
5:11
that we'll have we'll be talking about natural language
5:13
we'll be talking about natural language
5:14
we'll be talking about natural language processing
5:15
processing
5:15
processing interesting industry lots of things
5:17
interesting industry lots of things
5:17
interesting industry lots of things going on
5:19
going on
5:19
going on but of course i'm not here alone as a
5:21
but of course i'm not here alone as a
5:21
but of course i'm not here alone as a host we also have
5:22
host we also have
5:22
host we also have amy hey amy hi hiya how are you doing
5:26
amy hey amy hi hiya how are you doing
5:26
amy hey amy hi hiya how are you doing i'm fine i'm very glad to have you as a
5:28
i'm fine i'm very glad to have you as a
5:28
i'm fine i'm very glad to have you as a co-host today
5:30
co-host today
5:30
co-host today we already had quite some nice sessions
5:32
we already had quite some nice sessions
5:32
we already had quite some nice sessions in our previous
5:33
in our previous
5:33
in our previous uh previous uh well october uh sessions
5:39
uh previous uh well october uh sessions
5:39
uh previous uh well october uh sessions well you remember our first one yep so i
5:42
well you remember our first one yep so i
5:42
well you remember our first one yep so i was gonna say
5:43
was gonna say
5:44
was gonna say for anyone that was uh has seen our
5:46
for anyone that was uh has seen our
5:46
for anyone that was uh has seen our previous shows and was in the chat
5:49
previous shows and was in the chat
5:49
previous shows and was in the chat i was amy the moderator at the time that
5:52
i was amy the moderator at the time that
5:52
i was amy the moderator at the time that kept messaging you all the time so
5:54
kept messaging you all the time so
5:54
kept messaging you all the time so um it's nice to see you all this is what
5:56
um it's nice to see you all this is what
5:56
um it's nice to see you all this is what i actually look like
5:58
i actually look like
5:58
i actually look like rather than just typing not just the
6:00
rather than just typing not just the
6:00
rather than just typing not just the name there's a person behind it it's not
6:02
name there's a person behind it it's not
6:02
name there's a person behind it it's not just ai
6:03
just ai
6:03
just ai there's a whole thing going on here i
6:05
there's a whole thing going on here i
6:05
there's a whole thing going on here i know amy the ai actually sounds got a
6:07
know amy the ai actually sounds got a
6:07
know amy the ai actually sounds got a bit of a
6:08
bit of a
6:08
bit of a a swing to it hasn't it but um yeah we
6:12
a swing to it hasn't it but um yeah we
6:12
a swing to it hasn't it but um yeah we are what are we now episode three
6:14
are what are we now episode three
6:14
are what are we now episode three goodness me we're three weeks into
6:16
goodness me we're three weeks into
6:16
goodness me we're three weeks into october out of four episodes
6:19
october out of four episodes
6:19
october out of four episodes um almost the end of the year again
6:22
um almost the end of the year again
6:22
um almost the end of the year again i know i know right how how fast is it
6:25
i know i know right how how fast is it
6:25
i know i know right how how fast is it all going as well i can't believe it
6:27
all going as well i can't believe it
6:27
all going as well i can't believe it we were just talking about so i'm in the
6:29
we were just talking about so i'm in the
6:29
we were just talking about so i'm in the uk
6:30
uk
6:30
uk um and it's getting so dark outside now
6:33
um and it's getting so dark outside now
6:33
um and it's getting so dark outside now here
6:34
here
6:34
here um it gets dark in the winter quite
6:36
um it gets dark in the winter quite
6:36
um it gets dark in the winter quite early and
6:37
early and
6:37
early and yeah like we were doing some testing and
6:39
yeah like we were doing some testing and
6:39
yeah like we were doing some testing and the sun was coming in i was like don't
6:41
the sun was coming in i was like don't
6:41
the sun was coming in i was like don't worry we won't have that problem
6:44
worry we won't have that problem
6:44
worry we won't have that problem we are based here at the studio we are
6:46
we are based here at the studio we are
6:46
we are based here at the studio we are in in holland um
6:48
in in holland um
6:48
in in holland um but we have guests we have amy as a
6:51
but we have guests we have amy as a
6:51
but we have guests we have amy as a guest but i have another surprise guest
6:53
guest but i have another surprise guest
6:54
guest but i have another surprise guest look who we have here is also joining us
7:00
over here he's a little far away he
7:03
over here he's a little far away he
7:03
over here he's a little far away he wants to learn more
7:04
wants to learn more
7:04
wants to learn more about nlp so uh i'm sure he will
7:08
about nlp so uh i'm sure he will
7:08
about nlp so uh i'm sure he will enjoy this session just as everyone else
7:12
enjoy this session just as everyone else
7:12
enjoy this session just as everyone else so we had our session already more
7:15
so we had our session already more
7:15
so we had our session already more basics about getting started with ai
7:19
basics about getting started with ai
7:19
basics about getting started with ai which was very interesting we saw some
7:20
which was very interesting we saw some
7:20
which was very interesting we saw some new features and
7:22
new features and
7:22
new features and azure services like the the metrics
7:25
azure services like the the metrics
7:25
azure services like the the metrics advisor which is really cool
7:28
advisor which is really cool
7:28
advisor which is really cool we had our custom vision last week
7:31
we had our custom vision last week
7:31
we had our custom vision last week where i really enjoyed the session where
7:34
where i really enjoyed the session where
7:34
where i really enjoyed the session where how you could even build
7:35
how you could even build
7:36
how you could even build ai models with a javascript i didn't
7:38
ai models with a javascript i didn't
7:38
ai models with a javascript i didn't know that was possible
7:41
know that was possible
7:41
know that was possible i'm not a huge fan of javascript myself
7:43
i'm not a huge fan of javascript myself
7:43
i'm not a huge fan of javascript myself i am going to have to throw it out there
7:44
i am going to have to throw it out there
7:44
i am going to have to throw it out there however
7:45
however
7:45
however is very impressive um that our amazing
7:49
is very impressive um that our amazing
7:49
is very impressive um that our amazing speaker was able to kind of show us how
7:50
speaker was able to kind of show us how
7:50
speaker was able to kind of show us how that was
7:51
that was
7:51
that was able to be done as well you know we
7:53
able to be done as well you know we
7:53
able to be done as well you know we always maybe think of
7:55
always maybe think of
7:55
always maybe think of maybe two specific languages for data
7:57
maybe two specific languages for data
7:57
maybe two specific languages for data science and so
7:58
science and so
7:58
science and so it was interesting to see that yeah also
8:01
it was interesting to see that yeah also
8:01
it was interesting to see that yeah also the session we had from
8:02
the session we had from
8:02
the session we had from richard campbell was really amazing he's
8:04
richard campbell was really amazing he's
8:04
richard campbell was really amazing he's really a good speaker and
8:06
really a good speaker and
8:06
really a good speaker and he really went from actually what is ai
8:10
he really went from actually what is ai
8:10
he really went from actually what is ai and then where are we now so that was
8:12
and then where are we now so that was
8:12
and then where are we now so that was really cool
8:14
really cool
8:14
really cool yeah the history of ai is such a good
8:16
yeah the history of ai is such a good
8:16
yeah the history of ai is such a good like story if you are
8:18
like story if you are
8:18
like story if you are joining us and you're just getting into
8:20
joining us and you're just getting into
8:20
joining us and you're just getting into ai like i highly recommend
8:22
ai like i highly recommend
8:22
ai like i highly recommend going and doing a little bit of research
8:25
going and doing a little bit of research
8:25
going and doing a little bit of research about
8:25
about
8:25
about where we've come from and how we've got
8:28
where we've come from and how we've got
8:28
where we've come from and how we've got to where we are right now because it's
8:30
to where we are right now because it's
8:30
to where we are right now because it's so
8:30
so
8:30
so interesting to see like a lot of the
8:32
interesting to see like a lot of the
8:32
interesting to see like a lot of the research that has really broke you know
8:35
research that has really broke you know
8:35
research that has really broke you know uh these glass ceilings of things that
8:36
uh these glass ceilings of things that
8:36
uh these glass ceilings of things that we can do
8:38
we can do
8:38
we can do is actually something that might have
8:39
is actually something that might have
8:39
is actually something that might have been quite old or actually in some cases
8:41
been quite old or actually in some cases
8:41
been quite old or actually in some cases was kind of said like
8:42
was kind of said like
8:42
was kind of said like no no that's not the right route to go
8:44
no no that's not the right route to go
8:44
no no that's not the right route to go like you know there are these other
8:46
like you know there are these other
8:46
like you know there are these other routes like
8:46
routes like
8:46
routes like um knowledge bases and and more
8:50
um knowledge bases and and more
8:50
um knowledge bases and and more structured types of machine learning so
8:52
structured types of machine learning so
8:52
structured types of machine learning so it is fascinating so yeah if you're
8:54
it is fascinating so yeah if you're
8:54
it is fascinating so yeah if you're getting into it i can
8:55
getting into it i can
8:55
getting into it i can only recommend going and taking a look
8:57
only recommend going and taking a look
8:57
only recommend going and taking a look at the history i think we can learn a
8:59
at the history i think we can learn a
8:59
at the history i think we can learn a lot
9:00
lot
9:00
lot yeah so today we're going to talk about
9:03
yeah so today we're going to talk about
9:03
yeah so today we're going to talk about nlp natural language processing we have
9:05
nlp natural language processing we have
9:05
nlp natural language processing we have an amazing lineup again of
9:08
an amazing lineup again of
9:08
an amazing lineup again of great speakers we have jos
9:11
great speakers we have jos
9:11
great speakers we have jos josh simmons from vmware we have
9:14
josh simmons from vmware we have
9:14
josh simmons from vmware we have elrond bandel from he's a researcher at
9:17
elrond bandel from he's a researcher at
9:17
elrond bandel from he's a researcher at the bar ilan university
9:19
the bar ilan university
9:19
the bar ilan university we have veret schwartz who works at the
9:22
we have veret schwartz who works at the
9:22
we have veret schwartz who works at the allen institute for ai and we have inez
9:26
allen institute for ai and we have inez
9:26
allen institute for ai and we have inez matoni who's a founder of explosion and
9:28
matoni who's a founder of explosion and
9:28
matoni who's a founder of explosion and actually
9:29
actually
9:29
actually helped building spacey uh well founder
9:31
helped building spacey uh well founder
9:32
helped building spacey uh well founder of spacey
9:33
of spacey
9:33
of spacey library which is a well well-used nlp
9:36
library which is a well well-used nlp
9:36
library which is a well well-used nlp library but before we go on
9:39
library but before we go on
9:39
library but before we go on and there's some small things that we of
9:41
and there's some small things that we of
9:41
and there's some small things that we of course want to share with everyone
9:43
course want to share with everyone
9:43
course want to share with everyone if you have any questions during the
9:45
if you have any questions during the
9:45
if you have any questions during the sessions please don't
9:46
sessions please don't
9:46
sessions please don't uh don't hesitate to use the chat on the
9:49
uh don't hesitate to use the chat on the
9:49
uh don't hesitate to use the chat on the platform that you're using
9:51
platform that you're using
9:51
platform that you're using to just send us the question we will
9:53
to just send us the question we will
9:53
to just send us the question we will receive them
9:54
receive them
9:54
receive them and we will ask the speakers uh well
9:57
and we will ask the speakers uh well
9:57
and we will ask the speakers uh well the questions that you have we will
9:58
the questions that you have we will
9:58
the questions that you have we will moderate them
10:00
moderate them
10:00
moderate them there's a chance to win also a 50 amazon
10:03
there's a chance to win also a 50 amazon
10:03
there's a chance to win also a 50 amazon card which is given away by c
10:05
card which is given away by c
10:05
card which is given away by c sharp corner so thank you very much c
10:07
sharp corner so thank you very much c
10:07
sharp corner so thank you very much c sharp corner for doing this
10:09
sharp corner for doing this
10:09
sharp corner for doing this if you want to win one of those
10:13
if you want to win one of those
10:13
if you want to win one of those amazon vouchers you need to go to
10:17
amazon vouchers you need to go to
10:17
amazon vouchers you need to go to live globalai.life wind dash actions
10:18
live globalai.life wind dash actions
10:18
live globalai.life wind dash actions we'll repeat it again
10:20
we'll repeat it again
10:20
we'll repeat it again global ai dot live slash win
10:23
global ai dot live slash win
10:23
global ai dot live slash win dash action so also
10:27
dash action so also
10:27
dash action so also send out tweets because we will also
10:28
send out tweets because we will also
10:28
send out tweets because we will also moderate through the tweets which
10:30
moderate through the tweets which
10:30
moderate through the tweets which who has most creative uh tweets that are
10:33
who has most creative uh tweets that are
10:33
who has most creative uh tweets that are sensed today
10:35
sensed today
10:35
sensed today we've had some good ones in the past
10:37
we've had some good ones in the past
10:37
we've had some good ones in the past right
10:38
right
10:38
right funny ones
10:42
creativity as creativity
10:45
creativity as creativity
10:45
creativity as creativity yeah and like make sure you put um
10:48
yeah and like make sure you put um
10:48
yeah and like make sure you put um global ai community so that we know what
10:50
global ai community so that we know what
10:50
global ai community so that we know what we're looking for
10:51
we're looking for
10:52
we're looking for um we're always trailing through them in
10:54
um we're always trailing through them in
10:54
um we're always trailing through them in the background
10:55
the background
10:55
the background and taking a look at all your answers on
10:56
and taking a look at all your answers on
10:56
and taking a look at all your answers on the platform so do
10:58
the platform so do
10:58
the platform so do converse with us we don't want this to
11:00
converse with us we don't want this to
11:00
converse with us we don't want this to be a one-way show we want to hear from
11:02
be a one-way show we want to hear from
11:02
be a one-way show we want to hear from you
11:03
you
11:03
you yeah so what do you think amy will we go
11:05
yeah so what do you think amy will we go
11:05
yeah so what do you think amy will we go to our first speaker
11:07
to our first speaker
11:07
to our first speaker yeah definitely definitely let's bring
11:09
yeah definitely definitely let's bring
11:09
yeah definitely definitely let's bring let's bring that office so let's uh
11:11
let's bring that office so let's uh
11:11
let's bring that office so let's uh let's welcome josh hi josh hey there
11:14
let's welcome josh hi josh hey there
11:14
let's welcome josh hi josh hey there how are you doing yeah i'm doing well
11:16
how are you doing yeah i'm doing well
11:16
how are you doing yeah i'm doing well how are you both
11:17
how are you both
11:17
how are you both we are i'm fine how are you amy
11:21
we are i'm fine how are you amy
11:21
we are i'm fine how are you amy ah yeah we're good we're good it's
11:23
ah yeah we're good we're good it's
11:23
ah yeah we're good we're good it's getting a little late so it's 6 p.m here
11:25
getting a little late so it's 6 p.m here
11:25
getting a little late so it's 6 p.m here i do actually have coffee this is
11:27
i do actually have coffee this is
11:27
i do actually have coffee this is against the rules normally
11:28
against the rules normally
11:28
against the rules normally at this kind of time of the day but we
11:30
at this kind of time of the day but we
11:30
at this kind of time of the day but we have three hours of general
11:32
have three hours of general
11:32
have three hours of general agenda oh sammy i'm disappointed if
11:36
agenda oh sammy i'm disappointed if
11:36
agenda oh sammy i'm disappointed if anything
11:36
anything
11:36
anything but josh thank you so much for joining
11:38
but josh thank you so much for joining
11:38
but josh thank you so much for joining us and it's an absolute pleasure to have
11:40
us and it's an absolute pleasure to have
11:40
us and it's an absolute pleasure to have you
11:41
you
11:41
you can you um tell us a little bit more
11:43
can you um tell us a little bit more
11:43
can you um tell us a little bit more about yourself tell us tell our watchers
11:45
about yourself tell us tell our watchers
11:45
about yourself tell us tell our watchers what's uh
11:45
what's uh
11:45
what's uh what's happening yeah sure yeah so
11:48
what's happening yeah sure yeah so
11:48
what's happening yeah sure yeah so uh yeah thanks for the opportunity i i
11:51
uh yeah thanks for the opportunity i i
11:51
uh yeah thanks for the opportunity i i let's see i've been at vmware about 10
11:53
let's see i've been at vmware about 10
11:53
let's see i've been at vmware about 10 years
11:54
years
11:54
years uh i have two roles at vmware i'm the
11:57
uh i have two roles at vmware i'm the
11:57
uh i have two roles at vmware i'm the chief technologist for high performance
11:59
chief technologist for high performance
11:59
chief technologist for high performance computing
11:59
computing
12:00
computing i work in our office of the cto in the
12:02
i work in our office of the cto in the
12:02
i work in our office of the cto in the advanced technology group
12:03
advanced technology group
12:03
advanced technology group and um in that role i spend a lot of
12:06
and um in that role i spend a lot of
12:06
and um in that role i spend a lot of time looking at both high performance
12:08
time looking at both high performance
12:08
time looking at both high performance computing and also
12:09
computing and also
12:09
computing and also machine learning workloads and
12:11
machine learning workloads and
12:11
machine learning workloads and understanding how to get the best
12:12
understanding how to get the best
12:12
understanding how to get the best performance out of those workloads
12:14
performance out of those workloads
12:14
performance out of those workloads because clearly
12:15
because clearly
12:15
because clearly you know the data scientist needs they
12:17
you know the data scientist needs they
12:17
you know the data scientist needs they need stuff underneath them to run on
12:19
need stuff underneath them to run on
12:19
need stuff underneath them to run on right and we spend a lot of time
12:20
right and we spend a lot of time
12:20
right and we spend a lot of time looking at how to make all of those
12:22
looking at how to make all of those
12:22
looking at how to make all of those models et cetera and the training in
12:24
models et cetera and the training in
12:24
models et cetera and the training in both
12:24
both
12:24
both training and inference run uh as well as
12:27
training and inference run uh as well as
12:27
training and inference run uh as well as possible
12:27
possible
12:28
possible um but i also run which is maybe more
12:30
um but i also run which is maybe more
12:30
um but i also run which is maybe more relevant to this discussion i run
12:31
relevant to this discussion i run
12:31
relevant to this discussion i run something called the
12:33
something called the
12:33
something called the vmware machine learning program office
12:35
vmware machine learning program office
12:35
vmware machine learning program office which is a
12:36
which is a
12:36
which is a central a small central team that really
12:39
central a small central team that really
12:39
central a small central team that really works to coordinate
12:40
works to coordinate
12:40
works to coordinate our machine learning activities across
12:42
our machine learning activities across
12:42
our machine learning activities across the company and we get involved in
12:44
the company and we get involved in
12:44
the company and we get involved in things like community building
12:46
things like community building
12:46
things like community building we run internal competitions to help our
12:50
we run internal competitions to help our
12:50
we run internal competitions to help our our employee population uh upskill
12:53
our employee population uh upskill
12:53
our employee population uh upskill themselves uh in fact this week we're
12:55
themselves uh in fact this week we're
12:55
themselves uh in fact this week we're running
12:55
running
12:56
running uh an internal vmware uh machine
12:58
uh an internal vmware uh machine
12:58
uh an internal vmware uh machine learning conference for
12:59
learning conference for
12:59
learning conference for for our employees uh over the over the
13:01
for our employees uh over the over the
13:01
for our employees uh over the over the entire week actually
13:03
entire week actually
13:03
entire week actually nice when we were talking about before
13:06
nice when we were talking about before
13:06
nice when we were talking about before we were really wondering like vmware we
13:09
we were really wondering like vmware we
13:10
we were really wondering like vmware we never would link them to natural
13:11
never would link them to natural
13:11
never would link them to natural language processing and when people say
13:13
language processing and when people say
13:13
language processing and when people say vmware well we always think about
13:15
vmware well we always think about
13:15
vmware well we always think about virtual machines i'm sure there's way
13:17
virtual machines i'm sure there's way
13:17
virtual machines i'm sure there's way more
13:18
more
13:18
more than that behind the screens uh
13:21
than that behind the screens uh
13:21
than that behind the screens uh so we're really looking forward uh to
13:23
so we're really looking forward uh to
13:23
so we're really looking forward uh to your keynotes about
13:25
your keynotes about
13:25
your keynotes about this and what vmware and your
13:28
this and what vmware and your
13:28
this and what vmware and your your department is like exactly doing
13:30
your department is like exactly doing
13:30
your department is like exactly doing with nlp
13:31
with nlp
13:31
with nlp yeah i think with this we would like to
13:33
yeah i think with this we would like to
13:33
yeah i think with this we would like to give you the words
13:34
give you the words
13:34
give you the words and show us what what you're all doing
13:37
and show us what what you're all doing
13:37
and show us what what you're all doing there
13:38
there
13:38
there okay great yeah let me uh let me just
13:40
okay great yeah let me uh let me just
13:40
okay great yeah let me uh let me just share slides here
13:42
share slides here
13:42
share slides here let me know when you can see them
13:47
yeah perfect
13:50
yeah perfect
13:50
yeah perfect okay so um i thought for a while about
13:53
okay so um i thought for a while about
13:53
okay so um i thought for a while about uh what the title of this talk should be
13:55
uh what the title of this talk should be
13:55
uh what the title of this talk should be this is a really boring title
13:56
this is a really boring title
13:56
this is a really boring title unfortunately all the good titles
13:58
unfortunately all the good titles
13:58
unfortunately all the good titles involved using the word bug in some kind
14:01
involved using the word bug in some kind
14:01
involved using the word bug in some kind of a pun
14:02
of a pun
14:02
of a pun and i didn't think that that was going
14:03
and i didn't think that that was going
14:03
and i didn't think that that was going to translate into a global audience so i
14:05
to translate into a global audience so i
14:05
to translate into a global audience so i didn't try
14:06
didn't try
14:06
didn't try um but my intent here i'm going back to
14:09
um but my intent here i'm going back to
14:09
um but my intent here i'm going back to something that amy said
14:10
something that amy said
14:10
something that amy said maybe a little bit earlier is i'm going
14:12
maybe a little bit earlier is i'm going
14:12
maybe a little bit earlier is i'm going to fly at a relatively high level here
14:14
to fly at a relatively high level here
14:14
to fly at a relatively high level here so for people that aren't um you know
14:16
so for people that aren't um you know
14:16
so for people that aren't um you know into the details of nlp i think this
14:18
into the details of nlp i think this
14:18
into the details of nlp i think this will still be very accessible
14:20
will still be very accessible
14:20
will still be very accessible um it's really meant to give you just an
14:22
um it's really meant to give you just an
14:22
um it's really meant to give you just an overview of the a couple of examples of
14:24
overview of the a couple of examples of
14:24
overview of the a couple of examples of the types of things that we do
14:26
the types of things that we do
14:26
the types of things that we do uh inside of vmware around nlp
14:29
uh inside of vmware around nlp
14:29
uh inside of vmware around nlp okay let's see if i can get this to go
14:31
okay let's see if i can get this to go
14:31
okay let's see if i can get this to go forward
14:33
forward
14:33
forward okay so um just really i want to give
14:35
okay so um just really i want to give
14:35
okay so um just really i want to give you enough about vmware just so you
14:37
you enough about vmware just so you
14:37
you enough about vmware just so you understand
14:38
understand
14:38
understand to set context for the rest of the the
14:40
to set context for the rest of the the
14:40
to set context for the rest of the the rest of the presentation
14:41
rest of the presentation
14:42
rest of the presentation you know this isn't about you know
14:43
you know this isn't about you know
14:43
you know this isn't about you know vmware as a business really
14:45
vmware as a business really
14:45
vmware as a business really but really what we focus on um you know
14:47
but really what we focus on um you know
14:47
but really what we focus on um you know for those not familiar we are i
14:49
for those not familiar we are i
14:49
for those not familiar we are i i think at last count something like the
14:51
i think at last count something like the
14:51
i think at last count something like the fifth largest software company in the
14:53
fifth largest software company in the
14:53
fifth largest software company in the world
14:54
world
14:54
world and um we have roughly 500 000 customers
14:57
and um we have roughly 500 000 customers
14:57
and um we have roughly 500 000 customers worldwide
14:58
worldwide
14:58
worldwide and we're really we're in the business
15:00
and we're really we're in the business
15:00
and we're really we're in the business of delivering what we call the digital
15:02
of delivering what we call the digital
15:02
of delivering what we call the digital foundation
15:03
foundation
15:03
foundation that these businesses run their
15:05
that these businesses run their
15:05
that these businesses run their workloads on top of so you can think of
15:08
workloads on top of so you can think of
15:08
workloads on top of so you can think of these their data centers uh running
15:10
these their data centers uh running
15:10
these their data centers uh running vmware software
15:11
vmware software
15:11
vmware software and then that software is being used to
15:13
and then that software is being used to
15:13
and then that software is being used to host all sorts of workloads and
15:14
host all sorts of workloads and
15:14
host all sorts of workloads and applications including
15:16
applications including
15:16
applications including machine learning applications so really
15:19
machine learning applications so really
15:19
machine learning applications so really the most important thing
15:20
the most important thing
15:20
the most important thing to get from from this this slide and
15:23
to get from from this this slide and
15:23
to get from from this this slide and from what i'm saying is that
15:25
from what i'm saying is that
15:25
from what i'm saying is that we are a software company and we write
15:27
we are a software company and we write
15:27
we are a software company and we write prodigious amounts of software
15:29
prodigious amounts of software
15:29
prodigious amounts of software we have millions and millions of lines
15:31
we have millions and millions of lines
15:31
we have millions and millions of lines of code of software
15:33
of code of software
15:33
of code of software in order to enable all of the stuff that
15:34
in order to enable all of the stuff that
15:34
in order to enable all of the stuff that we do
15:37
now if you look at what we do as a
15:39
now if you look at what we do as a
15:39
now if you look at what we do as a company and
15:40
company and
15:40
company and kind of the intersections between what
15:42
kind of the intersections between what
15:42
kind of the intersections between what we do in machine learning
15:44
we do in machine learning
15:44
we do in machine learning it actually intersects with our type of
15:46
it actually intersects with our type of
15:46
it actually intersects with our type of company in three different ways and we
15:48
company in three different ways and we
15:48
company in three different ways and we we call them
15:49
we call them
15:49
we call them as you can see here we call them smarter
15:51
as you can see here we call them smarter
15:51
as you can see here we call them smarter businesses smarter products and services
15:53
businesses smarter products and services
15:53
businesses smarter products and services and smarter vmware
15:54
and smarter vmware
15:54
and smarter vmware and smarter businesses is really about
15:57
and smarter businesses is really about
15:57
and smarter businesses is really about making sure that when customers want to
15:59
making sure that when customers want to
15:59
making sure that when customers want to run their
15:59
run their
15:59
run their ml workloads or their ai workloads in
16:02
ml workloads or their ai workloads in
16:02
ml workloads or their ai workloads in the data center
16:03
the data center
16:04
the data center on top of our software on top of our
16:06
on top of our software on top of our
16:06
on top of our software on top of our virtualization software
16:07
virtualization software
16:08
virtualization software as was mentioned earlier that it runs
16:09
as was mentioned earlier that it runs
16:09
as was mentioned earlier that it runs really well and they have you know full
16:11
really well and they have you know full
16:11
really well and they have you know full access to all the capabilities that
16:13
access to all the capabilities that
16:13
access to all the capabilities that that vmware brings to bear when we're
16:15
that vmware brings to bear when we're
16:15
that vmware brings to bear when we're delivering this sort of digital
16:16
delivering this sort of digital
16:16
delivering this sort of digital infrastructure so that's
16:18
infrastructure so that's
16:18
infrastructure so that's one of the three pillars that we focus
16:20
one of the three pillars that we focus
16:20
one of the three pillars that we focus on and that gets into a lot
16:22
on and that gets into a lot
16:22
on and that gets into a lot of things around application performance
16:24
of things around application performance
16:24
of things around application performance etc
16:25
etc
16:25
etc the second place that we spend a lot of
16:27
the second place that we spend a lot of
16:27
the second place that we spend a lot of time on is what we call smarter products
16:29
time on is what we call smarter products
16:29
time on is what we call smarter products and services and that is
16:31
and services and that is
16:31
and services and that is having expertise internally having data
16:33
having expertise internally having data
16:33
having expertise internally having data scientists and engineers who work
16:34
scientists and engineers who work
16:34
scientists and engineers who work together
16:35
together
16:35
together to add machine learning capabilities
16:37
to add machine learning capabilities
16:37
to add machine learning capabilities into our products and services to make
16:39
into our products and services to make
16:39
into our products and services to make them smarter
16:40
them smarter
16:40
them smarter make them scale better make them more
16:43
make them scale better make them more
16:43
make them scale better make them more you know autonomous et cetera that's a
16:46
you know autonomous et cetera that's a
16:46
you know autonomous et cetera that's a pretty big effort as well
16:47
pretty big effort as well
16:47
pretty big effort as well and then the third one and this i think
16:49
and then the third one and this i think
16:49
and then the third one and this i think would resonate with most organizations
16:51
would resonate with most organizations
16:51
would resonate with most organizations is that we do
16:52
is that we do
16:52
is that we do ml internally for our own benefit right
16:55
ml internally for our own benefit right
16:55
ml internally for our own benefit right so we do it to improve efficiency or
16:57
so we do it to improve efficiency or
16:57
so we do it to improve efficiency or productivity
16:58
productivity
16:58
productivity and the examples that i'm going to give
16:59
and the examples that i'm going to give
16:59
and the examples that i'm going to give you today really kind of come out of
17:01
you today really kind of come out of
17:01
you today really kind of come out of that category like some of the things
17:02
that category like some of the things
17:02
that category like some of the things that we're doing
17:04
that we're doing
17:04
that we're doing internally to make things better and in
17:06
internally to make things better and in
17:06
internally to make things better and in particular we're going to be talking
17:08
particular we're going to be talking
17:08
particular we're going to be talking about
17:08
about
17:08
about the area of bugs and software bugs and
17:10
the area of bugs and software bugs and
17:10
the area of bugs and software bugs and bug management
17:13
so you know very broadly right nlp we
17:16
so you know very broadly right nlp we
17:16
so you know very broadly right nlp we know
17:17
know
17:17
know encompasses a pretty wide array of of um
17:20
encompasses a pretty wide array of of um
17:20
encompasses a pretty wide array of of um of different areas you know within it
17:22
of different areas you know within it
17:22
of different areas you know within it things like question answering and
17:23
things like question answering and
17:23
things like question answering and speech recognition and language
17:25
speech recognition and language
17:25
speech recognition and language translation et cetera
17:26
translation et cetera
17:26
translation et cetera in particular when we look at and this
17:28
in particular when we look at and this
17:28
in particular when we look at and this gets to answering the question of what
17:30
gets to answering the question of what
17:30
gets to answering the question of what kind of nlp are we doing internally
17:33
kind of nlp are we doing internally
17:33
kind of nlp are we doing internally so we have lots of documents lots of
17:35
so we have lots of documents lots of
17:35
so we have lots of documents lots of unstructured text
17:37
unstructured text
17:37
unstructured text things like sr tickets which are support
17:39
things like sr tickets which are support
17:39
things like sr tickets which are support request tickets that would be a customer
17:41
request tickets that would be a customer
17:42
request tickets that would be a customer writing in and saying that they've had a
17:43
writing in and saying that they've had a
17:43
writing in and saying that they've had a problem or some kind of a bug and they
17:45
problem or some kind of a bug and they
17:45
problem or some kind of a bug and they need help with it
17:46
need help with it
17:46
need help with it we have internal tickets that are raised
17:48
we have internal tickets that are raised
17:48
we have internal tickets that are raised we have product documentation
17:50
we have product documentation
17:50
we have product documentation knowledge base articles la our our
17:53
knowledge base articles la our our
17:53
knowledge base articles la our our products
17:53
products
17:53
products generate lots of log files and those log
17:56
generate lots of log files and those log
17:56
generate lots of log files and those log files
17:57
files
17:57
files you wouldn't look at those as being
17:58
you wouldn't look at those as being
17:58
you wouldn't look at those as being english or you know natural language per
18:00
english or you know natural language per
18:00
english or you know natural language per se
18:01
se
18:01
se but they're still amenable to applying
18:03
but they're still amenable to applying
18:03
but they're still amenable to applying nlp techniques to them
18:05
nlp techniques to them
18:05
nlp techniques to them and so we can look at that and even
18:07
and so we can look at that and even
18:07
and so we can look at that and even looking at source code and we haven't
18:09
looking at source code and we haven't
18:09
looking at source code and we haven't done
18:09
done
18:09
done a lot of that to date but as i said we
18:12
a lot of that to date but as i said we
18:12
a lot of that to date but as i said we have you know literally millions and
18:13
have you know literally millions and
18:13
have you know literally millions and millions of lines of code
18:15
millions of lines of code
18:15
millions of lines of code there are some really interesting things
18:16
there are some really interesting things
18:16
there are some really interesting things that we've you know seen in the
18:17
that we've you know seen in the
18:17
that we've you know seen in the literature and that we started to think
18:19
literature and that we started to think
18:19
literature and that we started to think about in terms of
18:20
about in terms of
18:20
about in terms of places we might apply nlp but if you
18:23
places we might apply nlp but if you
18:23
places we might apply nlp but if you sort of think about how you might apply
18:25
sort of think about how you might apply
18:25
sort of think about how you might apply nlp to these sorts of things these sorts
18:27
nlp to these sorts of things these sorts
18:27
nlp to these sorts of things these sorts of
18:28
of
18:28
of document sources these are sort of the
18:31
document sources these are sort of the
18:31
document sources these are sort of the sorts of things that we
18:32
sorts of things that we
18:32
sorts of things that we that we get involved in things like
18:35
that we get involved in things like
18:35
that we get involved in things like assessing customer sentiment analysis
18:37
assessing customer sentiment analysis
18:37
assessing customer sentiment analysis right so being able to look at for
18:38
right so being able to look at for
18:38
right so being able to look at for example conversations that are going on
18:41
example conversations that are going on
18:41
example conversations that are going on in customer forums and trying to
18:43
in customer forums and trying to
18:43
in customer forums and trying to understand whether or not customers are
18:45
understand whether or not customers are
18:45
understand whether or not customers are trending happy or trending
18:46
trending happy or trending
18:46
trending happy or trending unhappy with respect to some of the
18:48
unhappy with respect to some of the
18:48
unhappy with respect to some of the things that you know may be going on
18:50
things that you know may be going on
18:50
things that you know may be going on with respect to say product releases
18:53
with respect to say product releases
18:53
with respect to say product releases i mentioned code analysis and chat
18:54
i mentioned code analysis and chat
18:54
i mentioned code analysis and chat chatbots is another area
18:57
chatbots is another area
18:57
chatbots is another area what we're going to focus on in this
18:58
what we're going to focus on in this
18:58
what we're going to focus on in this short talk is
19:00
short talk is
19:00
short talk is is really around bugs and in particular
19:03
is really around bugs and in particular
19:03
is really around bugs and in particular bug classification and bug deduplication
19:07
bug classification and bug deduplication
19:07
bug classification and bug deduplication okay but before i get there i you know
19:09
okay but before i get there i you know
19:09
okay but before i get there i you know this will be an obvious statement maybe
19:11
this will be an obvious statement maybe
19:11
this will be an obvious statement maybe for the
19:11
for the
19:11
for the nlp folks but i think it's an important
19:13
nlp folks but i think it's an important
19:13
nlp folks but i think it's an important thing to talk uh
19:15
thing to talk uh
19:15
thing to talk uh about at least a little bit so this idea
19:17
about at least a little bit so this idea
19:17
about at least a little bit so this idea of
19:18
of
19:18
of you know what is we're talking about nlp
19:20
you know what is we're talking about nlp
19:20
you know what is we're talking about nlp right so
19:21
right so
19:21
right so natural language well sometimes the
19:23
natural language well sometimes the
19:23
natural language well sometimes the language is not so natural
19:25
language is not so natural
19:26
language is not so natural and what i'm looking at what i'm showing
19:27
and what i'm looking at what i'm showing
19:27
and what i'm looking at what i'm showing you here are some examples of some very
19:30
you here are some examples of some very
19:30
you here are some examples of some very reasonable sentences that
19:32
reasonable sentences that
19:32
reasonable sentences that people at vmware might write or our
19:34
people at vmware might write or our
19:34
people at vmware might write or our customers right might write
19:35
customers right might write
19:35
customers right might write so for example vmc uses nsxt to create
19:39
so for example vmc uses nsxt to create
19:39
so for example vmc uses nsxt to create and manage internal sddc networks
19:41
and manage internal sddc networks
19:41
and manage internal sddc networks that is completely intelligible to me
19:43
that is completely intelligible to me
19:43
that is completely intelligible to me i'm sure that doesn't mean much to
19:45
i'm sure that doesn't mean much to
19:45
i'm sure that doesn't mean much to many of the people that are you know
19:46
many of the people that are you know
19:46
many of the people that are you know listening to this um to this discussion
19:49
listening to this um to this discussion
19:49
listening to this um to this discussion so the point here is that at least in
19:51
so the point here is that at least in
19:51
so the point here is that at least in our industry and i i
19:53
our industry and i i
19:53
our industry and i i expect in many other industries as well
19:55
expect in many other industries as well
19:55
expect in many other industries as well there's a fair amount of jargon
19:57
there's a fair amount of jargon
19:57
there's a fair amount of jargon uh for us it's pretty much everything
19:59
uh for us it's pretty much everything
19:59
uh for us it's pretty much everything begins with a v
20:00
begins with a v
20:00
begins with a v uh at vmware so we have lots of acronyms
20:03
uh at vmware so we have lots of acronyms
20:03
uh at vmware so we have lots of acronyms and product names and just
20:04
and product names and just
20:04
and product names and just jargon generally and so it's really
20:06
jargon generally and so it's really
20:06
jargon generally and so it's really important that when we build nlp systems
20:09
important that when we build nlp systems
20:09
important that when we build nlp systems they're actually able to understand uh
20:12
they're actually able to understand uh
20:12
they're actually able to understand uh the language that we
20:13
the language that we
20:13
the language that we and our customers use and the reason i
20:15
and our customers use and the reason i
20:15
and our customers use and the reason i mention that is because there is
20:16
mention that is because there is
20:16
mention that is because there is obviously a lot of work going on
20:18
obviously a lot of work going on
20:18
obviously a lot of work going on in the community and google for example
20:20
in the community and google for example
20:20
in the community and google for example is a big contributor
20:22
is a big contributor
20:22
is a big contributor using things like bert to build language
20:25
using things like bert to build language
20:25
using things like bert to build language models that people can take as a
20:26
models that people can take as a
20:26
models that people can take as a starting point for
20:27
starting point for
20:27
starting point for doing their own nlp well those starting
20:30
doing their own nlp well those starting
20:30
doing their own nlp well those starting points
20:31
points
20:31
points are based generally speaking on
20:34
are based generally speaking on
20:34
are based generally speaking on relatively
20:35
relatively
20:35
relatively you know standard uh let's say english
20:38
you know standard uh let's say english
20:38
you know standard uh let's say english uh sentences
20:39
uh sentences
20:39
uh sentences without a lot of specialized vocabulary
20:41
without a lot of specialized vocabulary
20:41
without a lot of specialized vocabulary in them necessarily and so
20:42
in them necessarily and so
20:42
in them necessarily and so uh you know we have to take special
20:44
uh you know we have to take special
20:44
uh you know we have to take special action to deal with that and it goes
20:45
action to deal with that and it goes
20:45
action to deal with that and it goes beyond just the jargon
20:47
beyond just the jargon
20:47
beyond just the jargon you know there are also issues around
20:48
you know there are also issues around
20:48
you know there are also issues around usage right so if you look at the word
20:50
usage right so if you look at the word
20:50
usage right so if you look at the word switch
20:50
switch
20:50
switch right so the sentence the switch on my
20:53
right so the sentence the switch on my
20:53
right so the sentence the switch on my iphone is not working
20:55
iphone is not working
20:55
iphone is not working that's a different use of the word
20:57
that's a different use of the word
20:57
that's a different use of the word switch then
20:58
switch then
20:58
switch then vmware vsphere distributed switch
21:01
vmware vsphere distributed switch
21:01
vmware vsphere distributed switch provides a centralized interface
21:03
provides a centralized interface
21:03
provides a centralized interface or how can i switch a raw data mapping
21:05
or how can i switch a raw data mapping
21:05
or how can i switch a raw data mapping between physical and virtual
21:07
between physical and virtual
21:07
between physical and virtual compatibility modes right so
21:08
compatibility modes right so
21:08
compatibility modes right so it's the same word but obviously in
21:10
it's the same word but obviously in
21:10
it's the same word but obviously in english it has different meanings
21:12
english it has different meanings
21:12
english it has different meanings i mean it can even be a different part
21:14
i mean it can even be a different part
21:14
i mean it can even be a different part of speech right it could be a verb
21:15
of speech right it could be a verb
21:15
of speech right it could be a verb versus
21:15
versus
21:16
versus a noun so the context really matters
21:19
a noun so the context really matters
21:19
a noun so the context really matters and so why am i talking about all this
21:21
and so why am i talking about all this
21:21
and so why am i talking about all this well
21:22
well
21:22
well you need to understand the language that
21:25
you need to understand the language that
21:25
you need to understand the language that you're
21:26
you're
21:26
you're you're you need to understand your input
21:27
you're you need to understand your input
21:27
you're you need to understand your input language and then you need to have a way
21:29
language and then you need to have a way
21:29
language and then you need to have a way to
21:30
to
21:30
to map that input language into some kind
21:32
map that input language into some kind
21:32
map that input language into some kind of numerical
21:33
of numerical
21:33
of numerical uh representation that is more amenable
21:35
uh representation that is more amenable
21:36
uh representation that is more amenable to actually
21:36
to actually
21:36
to actually applying um nlp and just ml techniques
21:39
applying um nlp and just ml techniques
21:39
applying um nlp and just ml techniques to it
21:40
to it
21:40
to it generally right so so this idea of word
21:42
generally right so so this idea of word
21:42
generally right so so this idea of word embedding um
21:44
embedding um
21:44
embedding um is is a key concept in nlp as i'm sure
21:47
is is a key concept in nlp as i'm sure
21:47
is is a key concept in nlp as i'm sure many people are aware
21:48
many people are aware
21:48
many people are aware so initially at vmware we started off as
21:51
so initially at vmware we started off as
21:51
so initially at vmware we started off as i think many people did
21:53
i think many people did
21:53
i think many people did with um you know what were sort of state
21:56
with um you know what were sort of state
21:56
with um you know what were sort of state of the art approaches at the time things
21:57
of the art approaches at the time things
21:57
of the art approaches at the time things like glove and word to vec
22:00
like glove and word to vec
22:00
like glove and word to vec in order to build a set of vmware word
22:02
in order to build a set of vmware word
22:02
in order to build a set of vmware word vectors
22:03
vectors
22:03
vectors and that actually was done within my
22:05
and that actually was done within my
22:05
and that actually was done within my team uh to build a standardized set that
22:07
team uh to build a standardized set that
22:07
team uh to build a standardized set that we could then share with other teams
22:09
we could then share with other teams
22:09
we could then share with other teams inside of vmware so that they didn't
22:10
inside of vmware so that they didn't
22:10
inside of vmware so that they didn't have to go to the trouble
22:12
have to go to the trouble
22:12
have to go to the trouble of building their own sets of word
22:13
of building their own sets of word
22:13
of building their own sets of word embeddings to do their own nlp
22:16
embeddings to do their own nlp
22:16
embeddings to do their own nlp now since that time we've moved on and i
22:19
now since that time we've moved on and i
22:19
now since that time we've moved on and i guess
22:19
guess
22:19
guess you could say we've become more
22:21
you could say we've become more
22:21
you could say we've become more sophisticated and i'll i'll talk about
22:23
sophisticated and i'll i'll talk about
22:23
sophisticated and i'll i'll talk about why i'm being a little hesitant about
22:25
why i'm being a little hesitant about
22:25
why i'm being a little hesitant about using that word in a minute um to to
22:28
using that word in a minute um to to
22:28
using that word in a minute um to to something like a bert model
22:29
something like a bert model
22:29
something like a bert model a language model where what we're doing
22:31
a language model where what we're doing
22:31
a language model where what we're doing and it's sort of shown schematically on
22:33
and it's sort of shown schematically on
22:33
and it's sort of shown schematically on the bottom there is that we're taking
22:35
the bottom there is that we're taking
22:35
the bottom there is that we're taking the sort of pre-trained bert models that
22:37
the sort of pre-trained bert models that
22:37
the sort of pre-trained bert models that google makes available so that you know
22:39
google makes available so that you know
22:39
google makes available so that you know have been built
22:40
have been built
22:40
have been built with relatively generic vocabularies and
22:42
with relatively generic vocabularies and
22:42
with relatively generic vocabularies and then essentially doing transfer learning
22:44
then essentially doing transfer learning
22:44
then essentially doing transfer learning on top of those
22:45
on top of those
22:45
on top of those by bringing our own copora like the
22:48
by bringing our own copora like the
22:48
by bringing our own copora like the vmware documentation set
22:49
vmware documentation set
22:49
vmware documentation set uh knowledge base articles and even
22:52
uh knowledge base articles and even
22:52
uh knowledge base articles and even reddit discussions
22:54
reddit discussions
22:54
reddit discussions uh that are about you know what happens
22:56
uh that are about you know what happens
22:56
uh that are about you know what happens what's going on at vmware and vmware
22:58
what's going on at vmware and vmware
22:58
what's going on at vmware and vmware products by
22:59
products by
22:59
products by applying that to the pre-trained burp
23:01
applying that to the pre-trained burp
23:01
applying that to the pre-trained burp model and then
23:02
model and then
23:02
model and then building what we call v-bird so we now
23:04
building what we call v-bird so we now
23:04
building what we call v-bird so we now have a vmware specific language model
23:06
have a vmware specific language model
23:06
have a vmware specific language model from which we can extract embeddings
23:08
from which we can extract embeddings
23:08
from which we can extract embeddings that we can then use for our own
23:10
that we can then use for our own
23:10
that we can then use for our own and uh our own nlp and i guess i do want
23:13
and uh our own nlp and i guess i do want
23:13
and uh our own nlp and i guess i do want to make a comment here that
23:15
to make a comment here that
23:15
to make a comment here that uh and maybe this goes back to the
23:16
uh and maybe this goes back to the
23:16
uh and maybe this goes back to the sophistication comment i was making a
23:18
sophistication comment i was making a
23:18
sophistication comment i was making a second ago
23:19
second ago
23:19
second ago is that you know very clearly it's been
23:21
is that you know very clearly it's been
23:21
is that you know very clearly it's been shown that you can do some
23:22
shown that you can do some
23:22
shown that you can do some very amazing things with bert the fact
23:24
very amazing things with bert the fact
23:24
very amazing things with bert the fact that it captures context and if you
23:26
that it captures context and if you
23:26
that it captures context and if you think about the previous slide
23:28
think about the previous slide
23:28
think about the previous slide right i showed the word switch well with
23:31
right i showed the word switch well with
23:31
right i showed the word switch well with a
23:31
a
23:31
a word vector-based approach you would
23:33
word vector-based approach you would
23:34
word vector-based approach you would have a single
23:34
have a single
23:34
have a single uh embedding representation for that
23:37
uh embedding representation for that
23:37
uh embedding representation for that word switch
23:38
word switch
23:38
word switch regardless of the context within which
23:40
regardless of the context within which
23:40
regardless of the context within which it was used in its underlying meaning
23:42
it was used in its underlying meaning
23:42
it was used in its underlying meaning whereas invert it has the sophistication
23:44
whereas invert it has the sophistication
23:44
whereas invert it has the sophistication to be able to create
23:45
to be able to create
23:46
to be able to create different word embeddings to represent
23:48
different word embeddings to represent
23:48
different word embeddings to represent that word in different contexts right so
23:50
that word in different contexts right so
23:50
that word in different contexts right so it clearly adds value
23:51
it clearly adds value
23:52
it clearly adds value but it does come at a cost right so
23:54
but it does come at a cost right so
23:54
but it does come at a cost right so these uh
23:55
these uh
23:55
these uh large language models are extremely
23:58
large language models are extremely
23:58
large language models are extremely compute intensive requiring very large
24:00
compute intensive requiring very large
24:00
compute intensive requiring very large uh amounts of uh hardware acceleration
24:03
uh amounts of uh hardware acceleration
24:03
uh amounts of uh hardware acceleration for example to run them in any
24:04
for example to run them in any
24:04
for example to run them in any reasonable period of time
24:06
reasonable period of time
24:06
reasonable period of time uh from an eco perspective they're
24:08
uh from an eco perspective they're
24:08
uh from an eco perspective they're pretty unfriendly right they
24:09
pretty unfriendly right they
24:09
pretty unfriendly right they they consume uh the carbon footprints
24:11
they consume uh the carbon footprints
24:11
they consume uh the carbon footprints are quite large
24:12
are quite large
24:12
are quite large uh despite you know of course you are
24:14
uh despite you know of course you are
24:14
uh despite you know of course you are getting something for it but from
24:16
getting something for it but from
24:16
getting something for it but from from our perspective when we think about
24:18
from our perspective when we think about
24:18
from our perspective when we think about sustainability
24:19
sustainability
24:19
sustainability we're not so happy about uh about the
24:21
we're not so happy about uh about the
24:21
we're not so happy about uh about the eco footprint
24:23
eco footprint
24:23
eco footprint of these really really heavy heavyweight
24:25
of these really really heavy heavyweight
24:25
of these really really heavy heavyweight approaches despite the fact that they
24:27
approaches despite the fact that they
24:27
approaches despite the fact that they deliver value and so
24:28
deliver value and so
24:28
deliver value and so in fact as i mentioned we're running a
24:30
in fact as i mentioned we're running a
24:30
in fact as i mentioned we're running a conference this week within vmware
24:32
conference this week within vmware
24:32
conference this week within vmware and we had a talk uh by a team that
24:35
and we had a talk uh by a team that
24:35
and we had a talk uh by a team that actually is still using word vectors so
24:37
actually is still using word vectors so
24:37
actually is still using word vectors so they're using a much much simpler
24:39
they're using a much much simpler
24:39
they're using a much much simpler approach
24:39
approach
24:39
approach and delivering excellent results really
24:41
and delivering excellent results really
24:42
and delivering excellent results really really high accuracies on what they're
24:43
really high accuracies on what they're
24:43
really high accuracies on what they're on
24:43
on
24:43
on on what they're trying to achieve and so
24:45
on what they're trying to achieve and so
24:45
on what they're trying to achieve and so one of the lessons i would share here is
24:47
one of the lessons i would share here is
24:47
one of the lessons i would share here is that it's not
24:48
that it's not
24:48
that it's not always necessary to jump forward to the
24:50
always necessary to jump forward to the
24:50
always necessary to jump forward to the latest and greatest
24:51
latest and greatest
24:51
latest and greatest and uh in particular i think it's worth
24:54
and uh in particular i think it's worth
24:54
and uh in particular i think it's worth looking at some of the more
24:55
looking at some of the more
24:55
looking at some of the more you know basic and maybe even some of
24:57
you know basic and maybe even some of
24:57
you know basic and maybe even some of the more classic ml approaches
24:59
the more classic ml approaches
24:59
the more classic ml approaches when trying to understand how to solve
25:02
when trying to understand how to solve
25:02
when trying to understand how to solve not just an nlp problem but an ml
25:04
not just an nlp problem but an ml
25:04
not just an nlp problem but an ml problem generally
25:05
problem generally
25:05
problem generally don't necessarily jump to the latest and
25:07
don't necessarily jump to the latest and
25:07
don't necessarily jump to the latest and greatest is really what i'm saying
25:11
greatest is really what i'm saying
25:11
greatest is really what i'm saying so okay so bugzilla i want to now we'll
25:13
so okay so bugzilla i want to now we'll
25:13
so okay so bugzilla i want to now we'll start diving down a little bit into bugs
25:15
start diving down a little bit into bugs
25:15
start diving down a little bit into bugs right so bugzilla is the
25:17
right so bugzilla is the
25:17
right so bugzilla is the product that we use internally for uh
25:19
product that we use internally for uh
25:19
product that we use internally for uh all
25:20
all
25:20
all everything to do with tracking bugs that
25:22
everything to do with tracking bugs that
25:22
everything to do with tracking bugs that engineers
25:23
engineers
25:23
engineers and others find in our products and
25:25
and others find in our products and
25:25
and others find in our products and really what this slide is really meant
25:27
really what this slide is really meant
25:27
really what this slide is really meant to
25:27
to
25:27
to demonstrate if you look at the usage
25:29
demonstrate if you look at the usage
25:29
demonstrate if you look at the usage statistics on the left hand side there
25:31
statistics on the left hand side there
25:31
statistics on the left hand side there you can see that this is a very heavily
25:33
you can see that this is a very heavily
25:33
you can see that this is a very heavily used piece of software
25:34
used piece of software
25:34
used piece of software we have like any any software company
25:38
we have like any any software company
25:38
we have like any any software company has we have a fair number of bugs that
25:40
has we have a fair number of bugs that
25:40
has we have a fair number of bugs that you know need to be tracked down and
25:41
you know need to be tracked down and
25:41
you know need to be tracked down and fixed
25:42
fixed
25:42
fixed you know based on either exploration
25:45
you know based on either exploration
25:45
you know based on either exploration that we've done internally or on
25:46
that we've done internally or on
25:46
that we've done internally or on customer reports
25:48
customer reports
25:48
customer reports and what i want to do again really
25:50
and what i want to do again really
25:50
and what i want to do again really briefly is touch a little bit on two
25:52
briefly is touch a little bit on two
25:52
briefly is touch a little bit on two particular
25:53
particular
25:53
particular areas that we've spent some time on
25:55
areas that we've spent some time on
25:55
areas that we've spent some time on applying nlp techniques uh
25:57
applying nlp techniques uh
25:57
applying nlp techniques uh bug duplication uh in bug routing
26:01
bug duplication uh in bug routing
26:01
bug duplication uh in bug routing okay so let's talk about uh bug
26:03
okay so let's talk about uh bug
26:03
okay so let's talk about uh bug duplication
26:05
duplication
26:05
duplication so the the basic first observation is
26:08
so the the basic first observation is
26:08
so the the basic first observation is that
26:09
that
26:09
that it's actually relatively common that you
26:11
it's actually relatively common that you
26:12
it's actually relatively common that you could end up with
26:13
could end up with
26:13
could end up with uh the same bug being represented more
26:15
uh the same bug being represented more
26:15
uh the same bug being represented more than once
26:16
than once
26:16
than once uh in the bug database and that's
26:19
uh in the bug database and that's
26:19
uh in the bug database and that's problematic
26:20
problematic
26:20
problematic right um you know as we get and the
26:23
right um you know as we get and the
26:23
right um you know as we get and the reason that that happens is that
26:24
reason that that happens is that
26:24
reason that that happens is that you know partly at least that because
26:26
you know partly at least that because
26:26
you know partly at least that because we're releasing
26:27
we're releasing
26:27
we're releasing products so quickly um and and the pace
26:30
products so quickly um and and the pace
26:30
products so quickly um and and the pace is so quick
26:31
is so quick
26:31
is so quick it's sometimes uh just you know but by
26:34
it's sometimes uh just you know but by
26:34
it's sometimes uh just you know but by by its nature
26:35
by its nature
26:35
by its nature we end up with these duplicates it also
26:37
we end up with these duplicates it also
26:37
we end up with these duplicates it also can be the case that we have newer
26:38
can be the case that we have newer
26:38
can be the case that we have newer engineers
26:39
engineers
26:39
engineers who are in the company who may not be
26:41
who are in the company who may not be
26:41
who are in the company who may not be aware that there's already a bug
26:42
aware that there's already a bug
26:42
aware that there's already a bug filed and the problem with um the fact
26:45
filed and the problem with um the fact
26:45
filed and the problem with um the fact that you have these uh duplicated
26:47
that you have these uh duplicated
26:47
that you have these uh duplicated uh bugs is that you as i said you end up
26:50
uh bugs is that you as i said you end up
26:50
uh bugs is that you as i said you end up with
26:51
with
26:51
with the the duplications and they can even
26:53
the the duplications and they can even
26:53
the the duplications and they can even be um you know multiple duplicates
26:54
be um you know multiple duplicates
26:54
be um you know multiple duplicates coming in in a single day
26:56
coming in in a single day
26:56
coming in in a single day and so the problem with that is that it
26:58
and so the problem with that is that it
26:58
and so the problem with that is that it leads to duplication of effort to
27:00
leads to duplication of effort to
27:00
leads to duplication of effort to produce bug fixes right so
27:02
produce bug fixes right so
27:02
produce bug fixes right so if i'm working on bug a and there's
27:05
if i'm working on bug a and there's
27:05
if i'm working on bug a and there's another
27:05
another
27:06
another bug that's essentially the same bug and
27:08
bug that's essentially the same bug and
27:08
bug that's essentially the same bug and it's been filed as bug b and another
27:10
it's been filed as bug b and another
27:10
it's been filed as bug b and another engineer is independently working on
27:12
engineer is independently working on
27:12
engineer is independently working on that
27:13
that
27:13
that uh that's that is wasted a wasted
27:15
uh that's that is wasted a wasted
27:15
uh that's that is wasted a wasted opportunity what we really want to be
27:17
opportunity what we really want to be
27:17
opportunity what we really want to be able to do
27:17
able to do
27:17
able to do is notice that a and b actually really
27:20
is notice that a and b actually really
27:20
is notice that a and b actually really are very
27:21
are very
27:21
are very closely related and then merge them
27:23
closely related and then merge them
27:23
closely related and then merge them together so that we have full
27:24
together so that we have full
27:24
together so that we have full information about what that bug is
27:26
information about what that bug is
27:26
information about what that bug is actually about and we can
27:27
actually about and we can
27:27
actually about and we can we can deal with it more quickly so it's
27:30
we can deal with it more quickly so it's
27:30
we can deal with it more quickly so it's really about
27:30
really about
27:30
really about um reducing the the the time to
27:34
um reducing the the the time to
27:34
um reducing the the the time to to fix the problem and increasing uh
27:36
to fix the problem and increasing uh
27:36
to fix the problem and increasing uh experience the customer experience
27:38
experience the customer experience
27:38
experience the customer experience uh you know in in the customer base
27:41
uh you know in in the customer base
27:41
uh you know in in the customer base so uh those are the reasons for looking
27:43
so uh those are the reasons for looking
27:43
so uh those are the reasons for looking at it and nlp can actually really help
27:45
at it and nlp can actually really help
27:45
at it and nlp can actually really help solve this problem
27:47
solve this problem
27:47
solve this problem and so just to give you an example so
27:49
and so just to give you an example so
27:49
and so just to give you an example so this is not an example of a vmware bug
27:51
this is not an example of a vmware bug
27:51
this is not an example of a vmware bug this is a bug
27:52
this is a bug
27:52
this is a bug from the mozilla firefox public database
27:54
from the mozilla firefox public database
27:54
from the mozilla firefox public database right so
27:55
right so
27:55
right so here's bug134649
27:58
here's bug134649
27:58
here's bug134649 and here's another bug and if you look
28:00
and here's another bug and if you look
28:00
and here's another bug and if you look at this and we've gone through and
28:01
at this and we've gone through and
28:01
at this and we've gone through and highlighted
28:02
highlighted
28:02
highlighted some of the the phrases here these are
28:04
some of the the phrases here these are
28:04
some of the the phrases here these are not written
28:05
not written
28:06
not written they're not identical they're certainly
28:07
they're not identical they're certainly
28:07
they're not identical they're certainly not identical but in fact they actually
28:09
not identical but in fact they actually
28:09
not identical but in fact they actually are
28:09
are
28:09
are reporting the same problem and so what
28:12
reporting the same problem and so what
28:12
reporting the same problem and so what we would really like to do is to be able
28:13
we would really like to do is to be able
28:13
we would really like to do is to be able to notice
28:14
to notice
28:14
to notice that these are the same problem and to
28:16
that these are the same problem and to
28:16
that these are the same problem and to be able to merge those bugs together
28:20
be able to merge those bugs together
28:20
be able to merge those bugs together so again at a high level the the first
28:23
so again at a high level the the first
28:23
so again at a high level the the first part of this
28:24
part of this
28:24
part of this is you know using those embeddings and
28:27
is you know using those embeddings and
28:27
is you know using those embeddings and reducing
28:28
reducing
28:28
reducing these bugs into uh a numeric form that
28:31
these bugs into uh a numeric form that
28:31
these bugs into uh a numeric form that actually can be used for doing text
28:32
actually can be used for doing text
28:32
actually can be used for doing text similarity and the tech similarity
28:34
similarity and the tech similarity
28:34
similarity and the tech similarity metrics that we're using are not complex
28:36
metrics that we're using are not complex
28:36
metrics that we're using are not complex i i believe
28:38
i i believe
28:38
i i believe uh cosine similarity is is is probably
28:40
uh cosine similarity is is is probably
28:40
uh cosine similarity is is is probably the main one that we're using
28:42
the main one that we're using
28:42
the main one that we're using at this point uh in order to determine
28:44
at this point uh in order to determine
28:44
at this point uh in order to determine whether or not a particular bug that has
28:46
whether or not a particular bug that has
28:46
whether or not a particular bug that has just been filed
28:47
just been filed
28:47
just been filed uh is actually a duplicate of a bug that
28:50
uh is actually a duplicate of a bug that
28:50
uh is actually a duplicate of a bug that we've already seen
28:51
we've already seen
28:51
we've already seen right so you need to do some kind of a
28:52
right so you need to do some kind of a
28:52
right so you need to do some kind of a document similarity between them
28:54
document similarity between them
28:54
document similarity between them uh in order to uh to do that and we look
28:58
uh in order to uh to do that and we look
28:58
uh in order to uh to do that and we look at the
28:59
at the
28:59
at the the um the the text description that the
29:03
the um the the text description that the
29:03
the um the the text description that the engineer has entered and we also look at
29:05
engineer has entered and we also look at
29:05
engineer has entered and we also look at some of the metadata as well
29:06
some of the metadata as well
29:06
some of the metadata as well uh and then we do uh this text
29:08
uh and then we do uh this text
29:08
uh and then we do uh this text similarity comparison
29:10
similarity comparison
29:10
similarity comparison uh between that bug and existing bugs
29:13
uh between that bug and existing bugs
29:13
uh between that bug and existing bugs now a problem with that
29:14
now a problem with that
29:14
now a problem with that is that let's say we have you know
29:16
is that let's say we have you know
29:16
is that let's say we have you know several million bugs in the database
29:17
several million bugs in the database
29:17
several million bugs in the database where you know a 20 plus year old
29:19
where you know a 20 plus year old
29:19
where you know a 20 plus year old company
29:20
company
29:20
company um you know over the lifetime of the
29:21
um you know over the lifetime of the
29:22
um you know over the lifetime of the company there have been a lot of bugs
29:23
company there have been a lot of bugs
29:23
company there have been a lot of bugs so how are we actually going to do that
29:25
so how are we actually going to do that
29:25
so how are we actually going to do that in some kind of a reasonable way so it
29:26
in some kind of a reasonable way so it
29:26
in some kind of a reasonable way so it turns out
29:27
turns out
29:27
turns out that uh you know by doing analysis what
29:30
that uh you know by doing analysis what
29:30
that uh you know by doing analysis what we're able to
29:32
we're able to
29:32
we're able to sort of empirically determine is that 95
29:35
sort of empirically determine is that 95
29:35
sort of empirically determine is that 95 of those duplicates that are entered
29:37
of those duplicates that are entered
29:37
of those duplicates that are entered occur around within about 56 days
29:40
occur around within about 56 days
29:40
occur around within about 56 days of um of when the other bug was actually
29:43
of um of when the other bug was actually
29:43
of um of when the other bug was actually filed
29:44
filed
29:44
filed and so we can actually take that number
29:46
and so we can actually take that number
29:46
and so we can actually take that number maybe move out a couple of standard
29:47
maybe move out a couple of standard
29:47
maybe move out a couple of standard deviations and
29:49
deviations and
29:49
deviations and give ourselves a time window uh within
29:51
give ourselves a time window uh within
29:52
give ourselves a time window uh within which we should look uh for
29:53
which we should look uh for
29:53
which we should look uh for bug duplicates rather than looking back
29:55
bug duplicates rather than looking back
29:55
bug duplicates rather than looking back say 20 years which
29:57
say 20 years which
29:57
say 20 years which isn't a useful use of time and this is
29:59
isn't a useful use of time and this is
29:59
isn't a useful use of time and this is important because especially if you're
30:01
important because especially if you're
30:01
important because especially if you're trying to do this
30:02
trying to do this
30:02
trying to do this in any kind of an interactive fashion
30:04
in any kind of an interactive fashion
30:04
in any kind of an interactive fashion looking through a million bugs is just
30:06
looking through a million bugs is just
30:06
looking through a million bugs is just not going to work right so this this
30:08
not going to work right so this this
30:08
not going to work right so this this this windowing this temporal windowing
30:10
this windowing this temporal windowing
30:10
this windowing this temporal windowing is is one approach that we take
30:12
is is one approach that we take
30:12
is is one approach that we take and then the other thing that we do is
30:14
and then the other thing that we do is
30:14
and then the other thing that we do is we've also noticed um through analysis
30:16
we've also noticed um through analysis
30:16
we've also noticed um through analysis that
30:17
that
30:17
that that it tends to be so we have a lot of
30:19
that it tends to be so we have a lot of
30:19
that it tends to be so we have a lot of products
30:20
products
30:20
products and it tends to be that these um
30:24
and it tends to be that these um
30:24
and it tends to be that these um these duplicates appear either in the
30:26
these duplicates appear either in the
30:26
these duplicates appear either in the same product or
30:28
same product or
30:28
same product or within maybe a small family of products
30:30
within maybe a small family of products
30:30
within maybe a small family of products so this is another way we can actually
30:32
so this is another way we can actually
30:32
so this is another way we can actually reduce the search space
30:34
reduce the search space
30:34
reduce the search space in order to make this uh a tenable
30:36
in order to make this uh a tenable
30:36
in order to make this uh a tenable solution
30:39
and so from a kind of high-level
30:41
and so from a kind of high-level
30:41
and so from a kind of high-level perspective you know how how does this
30:42
perspective you know how how does this
30:42
perspective you know how how does this actually work right so
30:44
actually work right so
30:44
actually work right so we have the bug database and in the bug
30:46
we have the bug database and in the bug
30:46
we have the bug database and in the bug database
30:47
database
30:48
database uh we we essentially have a labeled
30:49
uh we we essentially have a labeled
30:49
uh we we essentially have a labeled training set because any time
30:51
training set because any time
30:51
training set because any time an engineer determines that a bug is a
30:54
an engineer determines that a bug is a
30:54
an engineer determines that a bug is a duplicate of another existing bug
30:56
duplicate of another existing bug
30:56
duplicate of another existing bug they mark it as such right so we will
30:58
they mark it as such right so we will
30:58
they mark it as such right so we will know which ones historically are
31:00
know which ones historically are
31:00
know which ones historically are actually
31:01
actually
31:01
actually duplicated bugs and we know you know
31:03
duplicated bugs and we know you know
31:03
duplicated bugs and we know you know what the duplicates are or multiple
31:05
what the duplicates are or multiple
31:05
what the duplicates are or multiple duplicates in some cases so
31:07
duplicates in some cases so
31:07
duplicates in some cases so we have that and we can use that to
31:09
we have that and we can use that to
31:09
we have that and we can use that to build a training set of
31:11
build a training set of
31:11
build a training set of both duplicated and non-duplicated bug
31:13
both duplicated and non-duplicated bug
31:13
both duplicated and non-duplicated bug pairs
31:14
pairs
31:14
pairs so once we've done that we take the bugs
31:17
so once we've done that we take the bugs
31:17
so once we've done that we take the bugs we create the word embedding using
31:19
we create the word embedding using
31:19
we create the word embedding using transformer in this particular case
31:21
transformer in this particular case
31:21
transformer in this particular case to do this the document scoring uh we
31:24
to do this the document scoring uh we
31:24
to do this the document scoring uh we extract the features so the time
31:25
extract the features so the time
31:25
extract the features so the time distribution that i mentioned
31:27
distribution that i mentioned
31:27
distribution that i mentioned and then we're using two different
31:28
and then we're using two different
31:28
and then we're using two different approaches we're sort of benchmarking
31:30
approaches we're sort of benchmarking
31:30
approaches we're sort of benchmarking them against each other
31:32
them against each other
31:32
them against each other uh we used a deep learning based
31:33
uh we used a deep learning based
31:33
uh we used a deep learning based approach and we also used
31:35
approach and we also used
31:35
approach and we also used xgboost in order to to try different
31:38
xgboost in order to to try different
31:38
xgboost in order to to try different techniques
31:40
techniques
31:40
techniques and then you basically once you built
31:42
and then you basically once you built
31:42
and then you basically once you built the model you now apply the model on a
31:44
the model you now apply the model on a
31:44
the model you now apply the model on a new bug report
31:45
new bug report
31:46
new bug report and then generate reports for the
31:49
and then generate reports for the
31:49
and then generate reports for the duplicates that you find
31:50
duplicates that you find
31:50
duplicates that you find and then of course evaluate the model
31:51
and then of course evaluate the model
31:51
and then of course evaluate the model performance based on users feedback and
31:53
performance based on users feedback and
31:53
performance based on users feedback and then
31:54
then
31:54
then feed that back refresh the data set and
31:56
feed that back refresh the data set and
31:56
feed that back refresh the data set and use that to retrain the model is really
31:58
use that to retrain the model is really
31:58
use that to retrain the model is really the idea so
32:01
the idea so
32:01
the idea so when we've done that um you know here's
32:03
when we've done that um you know here's
32:03
when we've done that um you know here's here are the results that we have
32:05
here are the results that we have
32:05
here are the results that we have uh so v bird plus deep neural network is
32:08
uh so v bird plus deep neural network is
32:08
uh so v bird plus deep neural network is uh as it sounds it's using the
32:09
uh as it sounds it's using the
32:09
uh as it sounds it's using the embeddings um
32:11
embeddings um
32:11
embeddings um from the burt based uh transfer learning
32:13
from the burt based uh transfer learning
32:14
from the burt based uh transfer learning that we did
32:14
that we did
32:14
that we did uh and then v-bert versus xg boost and
32:18
uh and then v-bert versus xg boost and
32:18
uh and then v-bert versus xg boost and a comment i would make it goes back to
32:19
a comment i would make it goes back to
32:19
a comment i would make it goes back to what i said earlier um
32:21
what i said earlier um
32:21
what i said earlier um you know about using sort of the latest
32:23
you know about using sort of the latest
32:23
you know about using sort of the latest and greatest so
32:24
and greatest so
32:24
and greatest so xgboost uh is doing pretty well
32:27
xgboost uh is doing pretty well
32:27
xgboost uh is doing pretty well actually compared to the deep neural
32:29
actually compared to the deep neural
32:29
actually compared to the deep neural network based approach
32:30
network based approach
32:30
network based approach uh very very close uh in terms of
32:32
uh very very close uh in terms of
32:32
uh very very close uh in terms of performance just a little bit lower
32:34
performance just a little bit lower
32:34
performance just a little bit lower now the the deep neural network takes
32:36
now the the deep neural network takes
32:36
now the the deep neural network takes around eight hours to train with an
32:38
around eight hours to train with an
32:38
around eight hours to train with an nvidia v100 gpu
32:40
nvidia v100 gpu
32:40
nvidia v100 gpu whereas the xg boost uh took only around
32:43
whereas the xg boost uh took only around
32:43
whereas the xg boost uh took only around two hours on the same
32:44
two hours on the same
32:44
two hours on the same on the same gpu so here's a case where
32:47
on the same gpu so here's a case where
32:47
on the same gpu so here's a case where we could potentially decide to take a
32:49
we could potentially decide to take a
32:49
we could potentially decide to take a slight hit in
32:50
slight hit in
32:50
slight hit in say accuracy in order to embrace a model
32:54
say accuracy in order to embrace a model
32:54
say accuracy in order to embrace a model that
32:54
that
32:54
that you know is not quite as resource
32:56
you know is not quite as resource
32:56
you know is not quite as resource intensive as
32:58
intensive as
32:58
intensive as as the as the bird model or the deep
33:00
as the as the bird model or the deep
33:00
as the as the bird model or the deep neural networking model
33:04
okay um the other case i wanted to
33:07
okay um the other case i wanted to
33:07
okay um the other case i wanted to briefly touch on
33:08
briefly touch on
33:08
briefly touch on uh is bug routing and so the idea here
33:11
uh is bug routing and so the idea here
33:11
uh is bug routing and so the idea here is that every time
33:12
is that every time
33:12
is that every time an engineer submits a bug into the bug
33:15
an engineer submits a bug into the bug
33:15
an engineer submits a bug into the bug database
33:16
database
33:16
database we require that they categorize that bug
33:18
we require that they categorize that bug
33:18
we require that they categorize that bug that they classify it
33:20
that they classify it
33:20
that they classify it into a particular product and then
33:22
into a particular product and then
33:22
into a particular product and then within each product at vmware
33:23
within each product at vmware
33:24
within each product at vmware there are several categories that they
33:25
there are several categories that they
33:25
there are several categories that they could file that bug under sort of think
33:27
could file that bug under sort of think
33:27
could file that bug under sort of think of those as subsystems within the
33:29
of those as subsystems within the
33:29
of those as subsystems within the product
33:29
product
33:29
product and then within that subsystem there are
33:32
and then within that subsystem there are
33:32
and then within that subsystem there are components
33:33
components
33:33
components uh that that could also be uh specified
33:36
uh that that could also be uh specified
33:36
uh that that could also be uh specified to get even closer to the correct
33:37
to get even closer to the correct
33:37
to get even closer to the correct engineering team
33:38
engineering team
33:38
engineering team that should be working on that problem
33:41
that should be working on that problem
33:41
that should be working on that problem and just to give you a sense i think
33:42
and just to give you a sense i think
33:42
and just to give you a sense i think there's something like uh
33:44
there's something like uh
33:44
there's something like uh 6 500
33:48
unique components within the database
33:51
unique components within the database
33:51
unique components within the database across all the products that we have so
33:53
across all the products that we have so
33:53
across all the products that we have so the idea is we would love to be able to
33:55
the idea is we would love to be able to
33:55
the idea is we would love to be able to help uh
33:56
help uh
33:56
help uh our engineers in some way do a better
33:58
our engineers in some way do a better
33:58
our engineers in some way do a better job of
33:59
job of
33:59
job of correctly classifying and categorizing
34:01
correctly classifying and categorizing
34:01
correctly classifying and categorizing their their bugs
34:03
their their bugs
34:03
their their bugs because if you put them into the wrong
34:04
because if you put them into the wrong
34:04
because if you put them into the wrong category as you can see here right so so
34:06
category as you can see here right so so
34:06
category as you can see here right so so there are thousands of product
34:08
there are thousands of product
34:08
there are thousands of product uh category component combinations
34:11
uh category component combinations
34:11
uh category component combinations something like 20 to 28
34:12
something like 20 to 28
34:12
something like 20 to 28 of those bugs are misclassified at least
34:14
of those bugs are misclassified at least
34:14
of those bugs are misclassified at least once and again we can tell that by
34:16
once and again we can tell that by
34:16
once and again we can tell that by looking at the bug database and when an
34:18
looking at the bug database and when an
34:18
looking at the bug database and when an engineer finds that a bug
34:20
engineer finds that a bug
34:20
engineer finds that a bug is in the wrong product or category
34:22
is in the wrong product or category
34:22
is in the wrong product or category component they reclassify it so we have
34:24
component they reclassify it so we have
34:24
component they reclassify it so we have full visibility into that for building a
34:26
full visibility into that for building a
34:26
full visibility into that for building a labeled
34:26
labeled
34:26
labeled training set um now the cost of this is
34:29
training set um now the cost of this is
34:29
training set um now the cost of this is a bug that's misclassified
34:31
a bug that's misclassified
34:31
a bug that's misclassified could potentially take up to 1.6 x
34:33
could potentially take up to 1.6 x
34:33
could potentially take up to 1.6 x longer to actually
34:35
longer to actually
34:35
longer to actually close which is a problem right because
34:37
close which is a problem right because
34:37
close which is a problem right because that prolongs the x uh the resolution
34:39
that prolongs the x uh the resolution
34:39
that prolongs the x uh the resolution time by anywhere from
34:40
time by anywhere from
34:40
time by anywhere from two to twenty days which means that
34:42
two to twenty days which means that
34:42
two to twenty days which means that we're taking longer than we need to in
34:44
we're taking longer than we need to in
34:44
we're taking longer than we need to in order to get
34:45
order to get
34:45
order to get uh fixes out to our customers which is
34:47
uh fixes out to our customers which is
34:48
uh fixes out to our customers which is you know always uh
34:49
you know always uh
34:49
you know always uh always an issue in terms of keeping
34:50
always an issue in terms of keeping
34:50
always an issue in terms of keeping customers uh happy
34:52
customers uh happy
34:52
customers uh happy so so for us this is kind of high stakes
34:54
so so for us this is kind of high stakes
34:54
so so for us this is kind of high stakes can we actually figure out a way to do
34:55
can we actually figure out a way to do
34:55
can we actually figure out a way to do this
34:57
this
34:57
this and so we we actually have solved this
34:59
and so we we actually have solved this
34:59
and so we we actually have solved this problem
35:00
problem
35:00
problem three times the first time that we did
35:03
three times the first time that we did
35:03
three times the first time that we did actually the first two times that we did
35:04
actually the first two times that we did
35:04
actually the first two times that we did it
35:05
it
35:05
it we really focused on actually going back
35:07
we really focused on actually going back
35:07
we really focused on actually going back for a second
35:08
for a second
35:08
for a second we focused on assigning to the category
35:10
we focused on assigning to the category
35:10
we focused on assigning to the category level
35:11
level
35:11
level um and didn't weren't able to actually
35:14
um and didn't weren't able to actually
35:14
um and didn't weren't able to actually assign all the way to the component
35:15
assign all the way to the component
35:16
assign all the way to the component level so we had an
35:19
level so we had an
35:19
level so we had an intern come in two summers ago and
35:22
intern come in two summers ago and
35:22
intern come in two summers ago and we ended up adopting an approach and you
35:24
we ended up adopting an approach and you
35:24
we ended up adopting an approach and you can see the reference at the bottom here
35:25
can see the reference at the bottom here
35:25
can see the reference at the bottom here this is an approach that
35:27
this is an approach that
35:27
this is an approach that is very similar to the approach that
35:29
is very similar to the approach that
35:29
is very similar to the approach that uber use uses for their own bug system
35:32
uber use uses for their own bug system
35:32
uber use uses for their own bug system where we have uh sort of a sequential
35:34
where we have uh sort of a sequential
35:34
where we have uh sort of a sequential dependency
35:35
dependency
35:35
dependency in a hierarchical classification
35:37
in a hierarchical classification
35:37
in a hierarchical classification happening here where we're using
35:38
happening here where we're using
35:38
happening here where we're using a set of neural networks to first
35:40
a set of neural networks to first
35:40
a set of neural networks to first classify uh by product
35:43
classify uh by product
35:43
classify uh by product then classified by category and then
35:44
then classified by category and then
35:44
then classified by category and then classified by component
35:46
classified by component
35:46
classified by component and here we're not using uh the v-bert
35:49
and here we're not using uh the v-bert
35:49
and here we're not using uh the v-bert uh you know language model and those
35:51
uh you know language model and those
35:51
uh you know language model and those embeddings we're using the much simpler
35:53
embeddings we're using the much simpler
35:53
embeddings we're using the much simpler uh word defect and glove uh word vector
35:56
uh word defect and glove uh word vector
35:56
uh word defect and glove uh word vector here
35:59
here
35:59
here and so you know key takeaway from this
36:00
and so you know key takeaway from this
36:00
and so you know key takeaway from this one from a business perspective
36:02
one from a business perspective
36:02
one from a business perspective is that we have deployed this um you
36:04
is that we have deployed this um you
36:04
is that we have deployed this um you know it's still early days in terms of
36:06
know it's still early days in terms of
36:06
know it's still early days in terms of the deployment
36:07
the deployment
36:07
the deployment in terms of prediction accuracy it's a
36:09
in terms of prediction accuracy it's a
36:09
in terms of prediction accuracy it's a little hard to explain exactly
36:11
little hard to explain exactly
36:11
little hard to explain exactly uh what the value is but let me let me
36:14
uh what the value is but let me let me
36:14
uh what the value is but let me let me give you a just a quick sense here
36:16
give you a just a quick sense here
36:16
give you a just a quick sense here so at a product level uh it's able to
36:19
so at a product level uh it's able to
36:19
so at a product level uh it's able to get
36:19
get
36:19
get a 95 accuracy in terms of telling the
36:22
a 95 accuracy in terms of telling the
36:22
a 95 accuracy in terms of telling the engine
36:23
engine
36:23
engine let's imagine you're an engineer a new
36:24
let's imagine you're an engineer a new
36:24
let's imagine you're an engineer a new engineer to vmware you're filing a bug
36:27
engineer to vmware you're filing a bug
36:27
engineer to vmware you're filing a bug and you really have no clue
36:28
and you really have no clue
36:28
and you really have no clue where to put this in terms of product
36:30
where to put this in terms of product
36:30
where to put this in terms of product category or or component
36:32
category or or component
36:32
category or or component so with about a 95 accuracy we can tell
36:35
so with about a 95 accuracy we can tell
36:35
so with about a 95 accuracy we can tell that engineer
36:36
that engineer
36:36
that engineer which product it needs to be assigned
36:38
which product it needs to be assigned
36:38
which product it needs to be assigned under uh and then we can get about a
36:41
under uh and then we can get about a
36:41
under uh and then we can get about a 57 accuracy uh taking that to the
36:44
57 accuracy uh taking that to the
36:44
57 accuracy uh taking that to the category level
36:46
category level
36:46
category level now you'll notice that the component
36:47
now you'll notice that the component
36:47
now you'll notice that the component level at least currently is quite low
36:49
level at least currently is quite low
36:49
level at least currently is quite low just around 20 percent although that 20
36:52
just around 20 percent although that 20
36:52
just around 20 percent although that 20 is still probably better than what the
36:54
is still probably better than what the
36:54
is still probably better than what the engineer is actually going to be able to
36:55
engineer is actually going to be able to
36:55
engineer is actually going to be able to do
36:56
do
36:56
do unless they have um you know reasonable
36:59
unless they have um you know reasonable
36:59
unless they have um you know reasonable knowledge about the products
37:00
knowledge about the products
37:00
knowledge about the products and so even if we get them just to the
37:02
and so even if we get them just to the
37:02
and so even if we get them just to the category level then it's much better
37:04
category level then it's much better
37:04
category level then it's much better than
37:05
than
37:05
than um you know just being stuck at the
37:06
um you know just being stuck at the
37:06
um you know just being stuck at the product level and not being able to
37:09
product level and not being able to
37:09
product level and not being able to correctly predict a category level and i
37:11
correctly predict a category level and i
37:11
correctly predict a category level and i should say that
37:12
should say that
37:12
should say that in the cases where they are a skilled
37:14
in the cases where they are a skilled
37:14
in the cases where they are a skilled engineer and for example they know what
37:16
engineer and for example they know what
37:16
engineer and for example they know what the product is
37:17
the product is
37:17
the product is or they know what the category level is
37:19
or they know what the category level is
37:19
or they know what the category level is we can do much higher
37:20
we can do much higher
37:20
we can do much higher levels of accuracy in terms of giving
37:22
levels of accuracy in terms of giving
37:22
levels of accuracy in terms of giving them recommendations about what the
37:23
them recommendations about what the
37:23
them recommendations about what the appropriate
37:24
appropriate
37:24
appropriate component or category should be so the
37:26
component or category should be so the
37:26
component or category should be so the benefit of this approach from our
37:28
benefit of this approach from our
37:28
benefit of this approach from our perspective internally
37:29
perspective internally
37:29
perspective internally is that yes we're starting to solve this
37:31
is that yes we're starting to solve this
37:31
is that yes we're starting to solve this this bug classification problem
37:34
this bug classification problem
37:34
this bug classification problem but we're also uh building these sort of
37:36
but we're also uh building these sort of
37:36
but we're also uh building these sort of reusable models that
37:37
reusable models that
37:37
reusable models that uh with with which which demonstrate
37:40
uh with with which which demonstrate
37:40
uh with with which which demonstrate sort of a hierarchical multi-class
37:42
sort of a hierarchical multi-class
37:42
sort of a hierarchical multi-class text classification that could be
37:44
text classification that could be
37:44
text classification that could be potentially repurposed inside of vmware
37:47
potentially repurposed inside of vmware
37:47
potentially repurposed inside of vmware uh for other other activities now the
37:49
uh for other other activities now the
37:49
uh for other other activities now the last thing i just wanted to quickly
37:51
last thing i just wanted to quickly
37:51
last thing i just wanted to quickly mention
37:51
mention
37:51
mention is that if you if you think about the
37:54
is that if you if you think about the
37:54
is that if you if you think about the sort of overall steps of
37:56
sort of overall steps of
37:56
sort of overall steps of you know conceiving your your ml or nlp
37:59
you know conceiving your your ml or nlp
37:59
you know conceiving your your ml or nlp project and you look at the steps here
38:01
project and you look at the steps here
38:01
project and you look at the steps here it all ends with deploying and
38:02
it all ends with deploying and
38:02
it all ends with deploying and monitoring right and i think
38:04
monitoring right and i think
38:04
monitoring right and i think you know as as the data scientists tend
38:06
you know as as the data scientists tend
38:06
you know as as the data scientists tend to focus a lot on
38:08
to focus a lot on
38:08
to focus a lot on the the actual training and the models
38:10
the the actual training and the models
38:10
the the actual training and the models and the techniques etc and of course
38:12
and the techniques etc and of course
38:12
and the techniques etc and of course that that that makes total sense
38:14
that that that makes total sense
38:14
that that that makes total sense uh of course it's also true that a
38:16
uh of course it's also true that a
38:16
uh of course it's also true that a significant amount of time is spent
38:17
significant amount of time is spent
38:18
significant amount of time is spent at the front of this in data collection
38:20
at the front of this in data collection
38:20
at the front of this in data collection labeling and pre-processing but
38:22
labeling and pre-processing but
38:22
labeling and pre-processing but we often get the impression that people
38:23
we often get the impression that people
38:24
we often get the impression that people don't spend
38:25
don't spend
38:25
don't spend enough time thinking about the deploying
38:27
enough time thinking about the deploying
38:27
enough time thinking about the deploying and the monitoring aspects
38:28
and the monitoring aspects
38:28
and the monitoring aspects there's actually a lot of complexity
38:30
there's actually a lot of complexity
38:30
there's actually a lot of complexity here in terms of integrating
38:32
here in terms of integrating
38:32
here in terms of integrating these models into existing workflows uh
38:35
these models into existing workflows uh
38:35
these models into existing workflows uh in order to
38:36
in order to
38:36
in order to to to make them functional and put them
38:39
to to make them functional and put them
38:39
to to make them functional and put them in place in a way
38:40
in place in a way
38:40
in place in a way where they can be the models can be
38:42
where they can be the models can be
38:42
where they can be the models can be monitored where they can be
38:44
monitored where they can be
38:44
monitored where they can be updated ideally automatically so that
38:46
updated ideally automatically so that
38:46
updated ideally automatically so that you don't
38:47
you don't
38:47
you don't experience uh model drift etc and then
38:50
experience uh model drift etc and then
38:50
experience uh model drift etc and then just you know from an inference
38:53
just you know from an inference
38:53
just you know from an inference perspective
38:53
perspective
38:54
perspective there's some things to think about right
38:55
there's some things to think about right
38:56
there's some things to think about right so again going back to this
38:57
so again going back to this
38:57
so again going back to this this this theme about you know what
39:00
this this theme about you know what
39:00
this this theme about you know what what's the
39:01
what's the
39:01
what's the cost of the model that you're that
39:03
cost of the model that you're that
39:03
cost of the model that you're that you're actually using right are you
39:04
you're actually using right are you
39:04
you're actually using right are you using a big neural network or have you
39:06
using a big neural network or have you
39:06
using a big neural network or have you found
39:06
found
39:06
found some kind of a simple model right well
39:08
some kind of a simple model right well
39:08
some kind of a simple model right well that's going to have an impact on
39:10
that's going to have an impact on
39:10
that's going to have an impact on your ability to do really fast
39:12
your ability to do really fast
39:12
your ability to do really fast inferencing if you need really fast
39:13
inferencing if you need really fast
39:14
inferencing if you need really fast inferencing right so these really
39:15
inferencing right so these really
39:15
inferencing right so these really complex models it's quite difficult to
39:17
complex models it's quite difficult to
39:17
complex models it's quite difficult to get fast inferencing
39:19
get fast inferencing
39:19
get fast inferencing uh so you have to keep that in mind you
39:21
uh so you have to keep that in mind you
39:21
uh so you have to keep that in mind you also need to keep in mind
39:22
also need to keep in mind
39:22
also need to keep in mind it comes down to sort of a both a cost
39:24
it comes down to sort of a both a cost
39:24
it comes down to sort of a both a cost and a performance
39:25
and a performance
39:25
and a performance trade-off is are you going to be using
39:28
trade-off is are you going to be using
39:28
trade-off is are you going to be using cpus or gpus or even say fpgas
39:31
cpus or gpus or even say fpgas
39:31
cpus or gpus or even say fpgas for doing that inference and you know
39:32
for doing that inference and you know
39:32
for doing that inference and you know what's that deployment environment going
39:34
what's that deployment environment going
39:34
what's that deployment environment going to look like in order to accomplish that
39:37
to look like in order to accomplish that
39:37
to look like in order to accomplish that and then you know you're not done once
39:39
and then you know you're not done once
39:39
and then you know you're not done once you've created that model
39:40
you've created that model
39:40
you've created that model right especially if you're very
39:41
right especially if you're very
39:41
right especially if you're very interested in getting high performance
39:43
interested in getting high performance
39:43
interested in getting high performance so
39:43
so
39:44
so there are certainly inference
39:45
there are certainly inference
39:45
there are certainly inference optimizations that can be applied
39:47
optimizations that can be applied
39:47
optimizations that can be applied things like quant quantity quantization
39:49
things like quant quantity quantization
39:49
things like quant quantity quantization uh binarization pruning etc
39:52
uh binarization pruning etc
39:52
uh binarization pruning etc which all need to be looked at in terms
39:55
which all need to be looked at in terms
39:55
which all need to be looked at in terms of
39:55
of
39:55
of a potential accuracy trade-off right so
39:57
a potential accuracy trade-off right so
39:57
a potential accuracy trade-off right so maybe you take a slight hit in accuracy
39:59
maybe you take a slight hit in accuracy
39:59
maybe you take a slight hit in accuracy but you're able to drive up uh you know
40:01
but you're able to drive up uh you know
40:02
but you're able to drive up uh you know drive down the complexity the runtime
40:03
drive down the complexity the runtime
40:03
drive down the complexity the runtime complexity of the model
40:05
complexity of the model
40:05
complexity of the model and then drive up your ability to do
40:06
and then drive up your ability to do
40:06
and then drive up your ability to do faster inferencing so this is just
40:08
faster inferencing so this is just
40:08
faster inferencing so this is just another area where where
40:09
another area where where
40:10
another area where where i think um from an nlp perspective it's
40:12
i think um from an nlp perspective it's
40:12
i think um from an nlp perspective it's important
40:13
important
40:13
important to think about i guess the lesson is
40:15
to think about i guess the lesson is
40:15
to think about i guess the lesson is it's important to think about the entire
40:17
it's important to think about the entire
40:17
it's important to think about the entire pipeline
40:17
pipeline
40:17
pipeline right not just the the sort of fun data
40:19
right not just the the sort of fun data
40:20
right not just the the sort of fun data science stuff in the middle but
40:21
science stuff in the middle but
40:21
science stuff in the middle but all the hard work at the beginning
40:22
all the hard work at the beginning
40:22
all the hard work at the beginning dealing with data and all the
40:24
dealing with data and all the
40:24
dealing with data and all the the difficult work at the end dealing
40:26
the difficult work at the end dealing
40:26
the difficult work at the end dealing with deploying and monitoring
40:29
with deploying and monitoring
40:29
with deploying and monitoring and with that i'm going to stop
40:35
cool that was really interesting josh
40:39
cool that was really interesting josh
40:39
cool that was really interesting josh um it's it's something i never thought
40:41
um it's it's something i never thought
40:41
um it's it's something i never thought about using nlp for mbox
40:44
about using nlp for mbox
40:44
about using nlp for mbox um but once you're thinking about it
40:46
um but once you're thinking about it
40:46
um but once you're thinking about it maybe it could also be used like for
40:48
maybe it could also be used like for
40:48
maybe it could also be used like for customer feedback which feedback comes
40:50
customer feedback which feedback comes
40:50
customer feedback which feedback comes all together
40:51
all together
40:52
all together where are people where are people happy
40:53
where are people where are people happy
40:53
where are people where are people happy about where are people
40:55
about where are people
40:55
about where are people uh not happy about and so on very
40:58
uh not happy about and so on very
40:58
uh not happy about and so on very interesting
40:59
interesting
40:59
interesting cool yeah um you were talking
41:02
cool yeah um you were talking
41:02
cool yeah um you were talking also about deployments
41:05
also about deployments
41:05
also about deployments are do you guys have specific processes
41:08
are do you guys have specific processes
41:08
are do you guys have specific processes in place
41:09
in place
41:09
in place to do your full deployment uh
41:12
to do your full deployment uh
41:12
to do your full deployment uh or yeah that's that's a great question
41:15
or yeah that's that's a great question
41:15
or yeah that's that's a great question we do not
41:16
we do not
41:16
we do not and this is something that we are
41:18
and this is something that we are
41:18
and this is something that we are actively uh discussing because
41:20
actively uh discussing because
41:20
actively uh discussing because as your question implies it's a really
41:22
as your question implies it's a really
41:22
as your question implies it's a really really good idea to have a standardized
41:24
really good idea to have a standardized
41:24
really good idea to have a standardized deployment mechanism
41:25
deployment mechanism
41:26
deployment mechanism right so regardless of what tools you're
41:28
right so regardless of what tools you're
41:28
right so regardless of what tools you're using to actually create the models
41:30
using to actually create the models
41:30
using to actually create the models why not have a standardized deployment
41:32
why not have a standardized deployment
41:32
why not have a standardized deployment mechanism and we're working towards it
41:33
mechanism and we're working towards it
41:33
mechanism and we're working towards it but i think we still have a little bit
41:35
but i think we still have a little bit
41:35
but i think we still have a little bit of distance to go
41:36
of distance to go
41:36
of distance to go this particular project around bug
41:38
this particular project around bug
41:38
this particular project around bug classification
41:39
classification
41:39
classification really was intended to be an end-to-end
41:42
really was intended to be an end-to-end
41:42
really was intended to be an end-to-end process taking things all the way
41:43
process taking things all the way
41:43
process taking things all the way through to production
41:44
through to production
41:44
through to production and use it as a learning experience so
41:46
and use it as a learning experience so
41:46
and use it as a learning experience so that we can understand what the
41:48
that we can understand what the
41:48
that we can understand what the difficulties are and actually
41:49
difficulties are and actually
41:49
difficulties are and actually um you know doing that deployment in a
41:51
um you know doing that deployment in a
41:51
um you know doing that deployment in a good way and in fact
41:52
good way and in fact
41:52
good way and in fact just before um getting on the on the
41:54
just before um getting on the on the
41:54
just before um getting on the on the call here today i was talking with
41:56
call here today i was talking with
41:56
call here today i was talking with one of the lead engineers on the project
41:58
one of the lead engineers on the project
41:58
one of the lead engineers on the project and you know she was
41:59
and you know she was
41:59
and you know she was we were talking about this idea of have
42:01
we were talking about this idea of have
42:01
we were talking about this idea of have we really closed the loop
42:03
we really closed the loop
42:03
we really closed the loop uh in the deployment environment to the
42:05
uh in the deployment environment to the
42:05
uh in the deployment environment to the to the to the extent that
42:07
to the to the extent that
42:07
to the to the extent that if you think about that bug problem
42:10
if you think about that bug problem
42:10
if you think about that bug problem vmware adds new products all the time
42:12
vmware adds new products all the time
42:12
vmware adds new products all the time and we add capabilities to those
42:14
and we add capabilities to those
42:14
and we add capabilities to those products all the time as well so that
42:15
products all the time as well so that
42:15
products all the time as well so that list of products and that list of
42:17
list of products and that list of
42:17
list of products and that list of um of um
42:20
um of um
42:20
um of um categories and components changes over
42:23
categories and components changes over
42:23
categories and components changes over time
42:23
time
42:24
time so there really needs to be a mechanism
42:25
so there really needs to be a mechanism
42:25
so there really needs to be a mechanism in place so that
42:27
in place so that
42:27
in place so that we can automatically rebuild and retrain
42:29
we can automatically rebuild and retrain
42:29
we can automatically rebuild and retrain those models and redeploy them ideally
42:31
those models and redeploy them ideally
42:31
those models and redeploy them ideally automatically
42:32
automatically
42:32
automatically that's not something that's not a loop
42:34
that's not something that's not a loop
42:34
that's not something that's not a loop that we have automated at this point it
42:35
that we have automated at this point it
42:35
that we have automated at this point it still requires
42:37
still requires
42:37
still requires a human in the loop to actually go
42:38
a human in the loop to actually go
42:38
a human in the loop to actually go rebuild that model and that clearly is
42:39
rebuild that model and that clearly is
42:40
rebuild that model and that clearly is something that
42:40
something that
42:40
something that you know needs to be worked on
42:44
you know needs to be worked on
42:44
you know needs to be worked on that's very very cool and also a lot of
42:46
that's very very cool and also a lot of
42:46
that's very very cool and also a lot of people are commenting
42:48
people are commenting
42:48
people are commenting in our chat that they can just see this
42:51
in our chat that they can just see this
42:51
in our chat that they can just see this being so useful in all of the companies
42:53
being so useful in all of the companies
42:53
being so useful in all of the companies they work for
42:54
they work for
42:54
they work for and all of their github projects so
42:56
and all of their github projects so
42:56
and all of their github projects so being able to actually
42:58
being able to actually
42:58
being able to actually um merge different issues actually i'm
43:01
um merge different issues actually i'm
43:01
um merge different issues actually i'm gonna dive in if possible with a quick
43:03
gonna dive in if possible with a quick
43:03
gonna dive in if possible with a quick question from myself how selfish
43:05
question from myself how selfish
43:05
question from myself how selfish um one of the things that first came to
43:07
um one of the things that first came to
43:07
um one of the things that first came to mind when you were saying oh yeah like
43:09
mind when you were saying oh yeah like
43:09
mind when you were saying oh yeah like what we do
43:10
what we do
43:10
what we do is we look for similarity sounds like
43:12
is we look for similarity sounds like
43:12
is we look for similarity sounds like perfect and then you merge them
43:14
perfect and then you merge them
43:14
perfect and then you merge them does the amount that you merge
43:18
does the amount that you merge
43:18
does the amount that you merge have some kind of weight in say like the
43:20
have some kind of weight in say like the
43:20
have some kind of weight in say like the importance of the book
43:22
importance of the book
43:22
importance of the book so i'm thinking you release new software
43:25
so i'm thinking you release new software
43:25
so i'm thinking you release new software and
43:25
and
43:25
and everyone has this issue and it's
43:27
everyone has this issue and it's
43:27
everyone has this issue and it's obviously like really high priority
43:29
obviously like really high priority
43:29
obviously like really high priority because you know it's on like the home
43:30
because you know it's on like the home
43:30
because you know it's on like the home screen or something like that
43:32
screen or something like that
43:32
screen or something like that is there anything like that in your
43:33
is there anything like that in your
43:33
is there anything like that in your model or you mainly just using it to say
43:35
model or you mainly just using it to say
43:35
model or you mainly just using it to say oh
43:36
oh
43:36
oh we know we have this problem and here's
43:37
we know we have this problem and here's
43:37
we know we have this problem and here's all the detail
43:39
all the detail
43:39
all the detail yeah so if i understand that question
43:41
yeah so if i understand that question
43:41
yeah so if i understand that question amy um yeah we're focused on
43:43
amy um yeah we're focused on
43:43
amy um yeah we're focused on on finding the duplicates there's no
43:45
on finding the duplicates there's no
43:45
on finding the duplicates there's no concept in
43:47
concept in
43:47
concept in this particular project around um sort
43:49
this particular project around um sort
43:49
this particular project around um sort of the
43:51
of the
43:51
of the you know fixing the you know oiling the
43:53
you know fixing the you know oiling the
43:53
you know fixing the you know oiling the squeaky wheel in a sense right knowing
43:54
squeaky wheel in a sense right knowing
43:54
squeaky wheel in a sense right knowing which bugs
43:55
which bugs
43:55
which bugs um are the ones to fix first it's funny
43:57
um are the ones to fix first it's funny
43:58
um are the ones to fix first it's funny that you mentioned that because
43:59
that you mentioned that because
43:59
that you mentioned that because um we had a talk uh this week by our cto
44:03
um we had a talk uh this week by our cto
44:03
um we had a talk uh this week by our cto greg lavender and one of the things he
44:04
greg lavender and one of the things he
44:04
greg lavender and one of the things he brought up was actually that very point
44:07
brought up was actually that very point
44:07
brought up was actually that very point he said that you can go into the bug
44:09
he said that you can go into the bug
44:09
he said that you can go into the bug database and you can look at the bug
44:11
database and you can look at the bug
44:11
database and you can look at the bug database and you will see that
44:12
database and you will see that
44:12
database and you will see that priorities have been assigned to those
44:13
priorities have been assigned to those
44:13
priorities have been assigned to those bugs
44:14
bugs
44:14
bugs but that's different than for example
44:17
but that's different than for example
44:17
but that's different than for example let's say we mine
44:19
let's say we mine
44:19
let's say we mine an external forum and we see that lots
44:22
an external forum and we see that lots
44:22
an external forum and we see that lots of people are complaining about a bug
44:23
of people are complaining about a bug
44:23
of people are complaining about a bug that's labeled at like severity three in
44:25
that's labeled at like severity three in
44:25
that's labeled at like severity three in the bug database
44:26
the bug database
44:26
the bug database well geez maybe we should go fix that
44:28
well geez maybe we should go fix that
44:28
well geez maybe we should go fix that severity three bug even though it's not
44:30
severity three bug even though it's not
44:30
severity three bug even though it's not as high priority as a bunch of other
44:32
as high priority as a bunch of other
44:32
as high priority as a bunch of other bugs but it would deliver
44:34
bugs but it would deliver
44:34
bugs but it would deliver much better satisfaction to end users
44:36
much better satisfaction to end users
44:36
much better satisfaction to end users and that's a
44:37
and that's a
44:37
and that's a an awesome idea and you're implying the
44:38
an awesome idea and you're implying the
44:38
an awesome idea and you're implying the same thing and it's something we should
44:40
same thing and it's something we should
44:40
same thing and it's something we should definitely look at
44:41
definitely look at
44:41
definitely look at ah well the global air community we're
44:43
ah well the global air community we're
44:43
ah well the global air community we're all here as idea generators so all of
44:46
all here as idea generators so all of
44:46
all here as idea generators so all of our
44:47
our
44:47
our people online submit your ideas as well
44:50
people online submit your ideas as well
44:50
people online submit your ideas as well yeah i think another question we got
44:53
yeah i think another question we got
44:53
yeah i think another question we got from the public which is
44:54
from the public which is
44:54
from the public which is quite interesting also is like do you
44:56
quite interesting also is like do you
44:56
quite interesting also is like do you assume that your existing bug
44:58
assume that your existing bug
44:58
assume that your existing bug knowledge base is always correct or have
45:01
knowledge base is always correct or have
45:01
knowledge base is always correct or have you used your model to
45:02
you used your model to
45:02
you used your model to merge duplications in your existing
45:05
merge duplications in your existing
45:05
merge duplications in your existing knowledge base or you only use it on
45:06
knowledge base or you only use it on
45:06
knowledge base or you only use it on your new box
45:09
your new box
45:09
your new box yeah so we do assume that the ground
45:11
yeah so we do assume that the ground
45:11
yeah so we do assume that the ground truth is in the bug database
45:13
truth is in the bug database
45:13
truth is in the bug database that's true um that that if an engineer
45:16
that's true um that that if an engineer
45:16
that's true um that that if an engineer i mean it's really it's akin it's
45:17
i mean it's really it's akin it's
45:18
i mean it's really it's akin it's essentially equivalent to hand labeling
45:19
essentially equivalent to hand labeling
45:19
essentially equivalent to hand labeling right somebody has gone in and said
45:21
right somebody has gone in and said
45:21
right somebody has gone in and said this bug and this bug yeah definitely
45:23
this bug and this bug yeah definitely
45:23
this bug and this bug yeah definitely duplicates and so
45:24
duplicates and so
45:24
duplicates and so we're relying on the expertise of our
45:26
we're relying on the expertise of our
45:26
we're relying on the expertise of our engineers to make that determination and
45:28
engineers to make that determination and
45:28
engineers to make that determination and we and
45:29
we and
45:29
we and and we and we proceed on that assumption
45:33
and we and we proceed on that assumption
45:33
and we and we proceed on that assumption sorry i think there was a second part to
45:34
sorry i think there was a second part to
45:34
sorry i think there was a second part to that question that i might have missed
45:36
that question that i might have missed
45:36
that question that i might have missed no well it i think you've answered
45:38
no well it i think you've answered
45:38
no well it i think you've answered everything it was about
45:39
everything it was about
45:39
everything it was about the new ones or or based on what you
45:41
the new ones or or based on what you
45:41
the new ones or or based on what you already had but
45:42
already had but
45:42
already had but with that in mind you're grouping by
45:44
with that in mind you're grouping by
45:44
with that in mind you're grouping by you're grouping all your books together
45:46
you're grouping all your books together
45:46
you're grouping all your books together so are you saying one specific book is
45:49
so are you saying one specific book is
45:49
so are you saying one specific book is the parents of all of them
45:50
the parents of all of them
45:50
the parents of all of them or are you saying which one is more
45:53
or are you saying which one is more
45:53
or are you saying which one is more important than the other
45:55
important than the other
45:55
important than the other yeah wow that's a great question so um i
45:58
yeah wow that's a great question so um i
45:58
yeah wow that's a great question so um i i don't think i mentioned that this
46:00
i don't think i mentioned that this
46:00
i don't think i mentioned that this project we also had an intern in this
46:02
project we also had an intern in this
46:02
project we also had an intern in this past summer a phd student working on
46:04
past summer a phd student working on
46:04
past summer a phd student working on this with us
46:05
this with us
46:05
this with us and um it's not an exaggeration to say i
46:08
and um it's not an exaggeration to say i
46:08
and um it's not an exaggeration to say i think the internship was 12 weeks long
46:10
think the internship was 12 weeks long
46:10
think the internship was 12 weeks long he spent at least six weeks working on
46:14
he spent at least six weeks working on
46:14
he spent at least six weeks working on uh questions just like the one that you
46:16
uh questions just like the one that you
46:16
uh questions just like the one that you raised right trying to understand
46:18
raised right trying to understand
46:18
raised right trying to understand what the right way is to actually
46:20
what the right way is to actually
46:20
what the right way is to actually represent the data
46:21
represent the data
46:21
represent the data um that that actually exists in the bug
46:23
um that that actually exists in the bug
46:24
um that that actually exists in the bug database so you're right
46:25
database so you're right
46:25
database so you're right i mean in a sense the parent one the
46:27
i mean in a sense the parent one the
46:27
i mean in a sense the parent one the parent bug
46:28
parent bug
46:28
parent bug is the one that was filed earliest so we
46:30
is the one that was filed earliest so we
46:30
is the one that was filed earliest so we start with that and then bugs that were
46:32
start with that and then bugs that were
46:32
start with that and then bugs that were filed later
46:33
filed later
46:33
filed later uh uh you know are sort of attached to
46:37
uh uh you know are sort of attached to
46:37
uh uh you know are sort of attached to it
46:37
it
46:37
it in our in our world is i guess the way
46:39
in our in our world is i guess the way
46:39
in our in our world is i guess the way to think of it and there's another
46:40
to think of it and there's another
46:40
to think of it and there's another interesting thing here it's you know
46:41
interesting thing here it's you know
46:41
interesting thing here it's you know it's
46:41
it's
46:42
it's it's easy looking at the database and
46:44
it's easy looking at the database and
46:44
it's easy looking at the database and the ground truth to understand
46:46
the ground truth to understand
46:46
the ground truth to understand which of the duplicates right because
46:47
which of the duplicates right because
46:47
which of the duplicates right because the engineers have said these two are
46:49
the engineers have said these two are
46:49
the engineers have said these two are duplicates
46:49
duplicates
46:49
duplicates but if you're building a training set
46:51
but if you're building a training set
46:51
but if you're building a training set it's really interesting to think about
46:53
it's really interesting to think about
46:53
it's really interesting to think about remember we have to
46:54
remember we have to
46:54
remember we have to pick bug pairs that are both duplicates
46:56
pick bug pairs that are both duplicates
46:56
pick bug pairs that are both duplicates that's easy
46:57
that's easy
46:57
that's easy but duplicates that are bug pairs that
46:59
but duplicates that are bug pairs that
46:59
but duplicates that are bug pairs that are not duplicates
47:00
are not duplicates
47:00
are not duplicates right and that's really interesting how
47:02
right and that's really interesting how
47:02
right and that's really interesting how you make those selections right because
47:04
you make those selections right because
47:04
you make those selections right because let's say
47:04
let's say
47:04
let's say all the pairs that i select i pick a bug
47:07
all the pairs that i select i pick a bug
47:07
all the pairs that i select i pick a bug from product a and a bug from product b
47:09
from product a and a bug from product b
47:09
from product a and a bug from product b so they're very very different
47:11
so they're very very different
47:11
so they're very very different and i give it to the model and i say
47:13
and i give it to the model and i say
47:13
and i give it to the model and i say these bugs are not
47:14
these bugs are not
47:14
these bugs are not duplicates well okay it's kind of
47:16
duplicates well okay it's kind of
47:16
duplicates well okay it's kind of obvious that they're not duplicates what
47:18
obvious that they're not duplicates what
47:18
obvious that they're not duplicates what i really should do
47:19
i really should do
47:19
i really should do when i'm picking bug pairs that are not
47:22
when i'm picking bug pairs that are not
47:22
when i'm picking bug pairs that are not duplicates is pick bugs that are
47:24
duplicates is pick bugs that are
47:24
duplicates is pick bugs that are in the same product and maybe in the
47:25
in the same product and maybe in the
47:25
in the same product and maybe in the same component so that they're in the
47:27
same component so that they're in the
47:28
same component so that they're in the same area and yet they're still
47:29
same area and yet they're still
47:29
same area and yet they're still different because you want the
47:30
different because you want the
47:30
different because you want the model to be able to differentiate find
47:33
model to be able to differentiate find
47:33
model to be able to differentiate find differences between those
47:34
differences between those
47:34
differences between those um between those bugs if that makes
47:37
um between those bugs if that makes
47:37
um between those bugs if that makes sense
47:38
sense
47:38
sense because if you don't do that you're not
47:39
because if you don't do that you're not
47:39
because if you don't do that you're not going to be able to find bugs that are
47:40
going to be able to find bugs that are
47:40
going to be able to find bugs that are you know subtly different
47:42
you know subtly different
47:42
you know subtly different yeah oh i know thank you so much for
47:46
yeah oh i know thank you so much for
47:46
yeah oh i know thank you so much for telling us all about this as sami said
47:48
telling us all about this as sami said
47:48
telling us all about this as sami said right one of the
47:49
right one of the
47:49
right one of the most interesting things is just hearing
47:51
most interesting things is just hearing
47:51
most interesting things is just hearing about these projects that do
47:52
about these projects that do
47:52
about these projects that do happen um in all these different
47:54
happen um in all these different
47:54
happen um in all these different companies um
47:56
companies um
47:56
companies um so one person i would like to bring on
47:58
so one person i would like to bring on
47:58
so one person i would like to bring on is actually our second speaker
48:00
is actually our second speaker
48:00
is actually our second speaker and who is elrond vandal and he has an
48:03
and who is elrond vandal and he has an
48:03
and who is elrond vandal and he has an incredible um business chatbot
48:07
incredible um business chatbot
48:07
incredible um business chatbot that he has been working on um for
48:10
that he has been working on um for
48:10
that he has been working on um for quite a while and he's going to be
48:11
quite a while and he's going to be
48:11
quite a while and he's going to be giving us an amazing session on so hi
48:13
giving us an amazing session on so hi
48:13
giving us an amazing session on so hi elrond thank you for joining us
48:15
elrond thank you for joining us
48:15
elrond thank you for joining us hi everyone thank you for having me
48:18
hi everyone thank you for having me
48:18
hi everyone thank you for having me um so we thought we would um we tend to
48:21
um so we thought we would um we tend to
48:21
um so we thought we would um we tend to do this
48:21
do this
48:22
do this in our previous sessions we bring
48:23
in our previous sessions we bring
48:23
in our previous sessions we bring multiple people on the panel to get kind
48:25
multiple people on the panel to get kind
48:25
multiple people on the panel to get kind of these different perspectives
48:27
of these different perspectives
48:27
of these different perspectives on the same um idea
48:30
on the same um idea
48:30
on the same um idea and so yeah alvon can you just very
48:32
and so yeah alvon can you just very
48:32
and so yeah alvon can you just very briefly introduce yourself and tell us a
48:34
briefly introduce yourself and tell us a
48:34
briefly introduce yourself and tell us a little bit about
48:35
little bit about
48:35
little bit about this spot okay so i'm
48:38
this spot okay so i'm
48:38
this spot okay so i'm a researcher at baillon university and
48:41
a researcher at baillon university and
48:41
a researcher at baillon university and during my bachelor degree i
48:45
during my bachelor degree i
48:45
during my bachelor degree i very enjoyed my studies but some of them
48:47
very enjoyed my studies but some of them
48:47
very enjoyed my studies but some of them some of the first
48:48
some of the first
48:48
some of the first courses were quite boring and
48:51
courses were quite boring and
48:51
courses were quite boring and i started to think like how do you do
48:54
i started to think like how do you do
48:54
i started to think like how do you do actually the real thing that uses
48:55
actually the real thing that uses
48:56
actually the real thing that uses computer science for example chatbot
48:58
computer science for example chatbot
48:58
computer science for example chatbot so i started to dive into this and
49:02
so i started to dive into this and
49:02
so i started to dive into this and i found i started to find it extremely
49:04
i found i started to find it extremely
49:04
i found i started to find it extremely interesting
49:05
interesting
49:05
interesting and then i started to create first one
49:08
and then i started to create first one
49:08
and then i started to create first one which is a
49:09
which is a
49:09
which is a new kind of platform for making chatbots
49:13
new kind of platform for making chatbots
49:13
new kind of platform for making chatbots for small businesses only by using
49:17
for small businesses only by using
49:17
for small businesses only by using textual description of the business
49:20
textual description of the business
49:20
textual description of the business very cool and um yeah i've had a really
49:23
very cool and um yeah i've had a really
49:23
very cool and um yeah i've had a really good nosy around your website and it is
49:25
good nosy around your website and it is
49:25
good nosy around your website and it is actually and much like
49:26
actually and much like
49:26
actually and much like the videos on youtube and it does look
49:28
the videos on youtube and it does look
49:28
the videos on youtube and it does look like a super simple process to set up
49:30
like a super simple process to set up
49:30
like a super simple process to set up and so i thought there was some kind of
49:33
and so i thought there was some kind of
49:33
and so i thought there was some kind of consistency here between josh what
49:35
consistency here between josh what
49:35
consistency here between josh what you've been speaking about and how
49:36
you've been speaking about and how
49:36
you've been speaking about and how you've all been using it um internally
49:39
you've all been using it um internally
49:39
you've all been using it um internally as part of your bug process to be more
49:41
as part of your bug process to be more
49:41
as part of your bug process to be more efficient and elon it felt like when
49:43
efficient and elon it felt like when
49:43
efficient and elon it felt like when you're
49:43
you're
49:44
you're building these business chat bots um for
49:46
building these business chat bots um for
49:46
building these business chat bots um for others so
49:47
others so
49:47
others so i'm a customer a and i can go on i don't
49:50
i'm a customer a and i can go on i don't
49:50
i'm a customer a and i can go on i don't need to know anything about
49:52
need to know anything about
49:52
need to know anything about machine learning and actually i can go
49:53
machine learning and actually i can go
49:54
machine learning and actually i can go ahead uh and create a chat bot
49:56
ahead uh and create a chat bot
49:56
ahead uh and create a chat bot uh we thought it would be quite an
49:58
uh we thought it would be quite an
49:58
uh we thought it would be quite an interesting perspective
50:00
interesting perspective
50:00
interesting perspective um and so i guess one of the first
50:02
um and so i guess one of the first
50:02
um and so i guess one of the first questions
50:03
questions
50:03
questions i kind of wanted to ask you both was a
50:05
i kind of wanted to ask you both was a
50:05
i kind of wanted to ask you both was a little bit
50:06
little bit
50:06
little bit um more around what are some of the
50:09
um more around what are some of the
50:09
um more around what are some of the challenges
50:11
challenges
50:11
challenges around infrastructure as well as
50:13
around infrastructure as well as
50:13
around infrastructure as well as architectural design when it comes to
50:15
architectural design when it comes to
50:15
architectural design when it comes to implementing these things so
50:17
implementing these things so
50:17
implementing these things so you're both doing things that extreme
50:19
you're both doing things that extreme
50:19
you're both doing things that extreme scale
50:20
scale
50:20
scale is there certain challenges you've seen
50:22
is there certain challenges you've seen
50:22
is there certain challenges you've seen um
50:23
um
50:23
um so i'm going to pick one of you oh josh
50:26
so i'm going to pick one of you oh josh
50:26
so i'm going to pick one of you oh josh i'm gonna put you in the hot seat
50:27
i'm gonna put you in the hot seat
50:27
i'm gonna put you in the hot seat is there um specific challenges you've
50:29
is there um specific challenges you've
50:30
is there um specific challenges you've seen when
50:31
seen when
50:31
seen when uh experimenting and deploying these
50:33
uh experimenting and deploying these
50:33
uh experimenting and deploying these different bots
50:35
different bots
50:35
different bots yeah it's a good question um so i
50:38
yeah it's a good question um so i
50:38
yeah it's a good question um so i mentioned that we're using burt
50:40
mentioned that we're using burt
50:40
mentioned that we're using burt um for the for the language models um
50:42
um for the for the language models um
50:42
um for the for the language models um you know that that does require a
50:44
you know that that does require a
50:44
you know that that does require a certain amount of computing horsepower
50:46
certain amount of computing horsepower
50:46
certain amount of computing horsepower but luckily we you know we do have a
50:48
but luckily we you know we do have a
50:48
but luckily we you know we do have a reasonably significant number of gpus
50:51
reasonably significant number of gpus
50:51
reasonably significant number of gpus available internally uh within the
50:53
available internally uh within the
50:53
available internally uh within the company so that
50:54
company so that
50:54
company so that that's not too much of a problem but i
50:57
that's not too much of a problem but i
50:57
that's not too much of a problem but i didn't mention that we have probably
50:58
didn't mention that we have probably
50:58
didn't mention that we have probably something like 100
51:00
something like 100
51:00
something like 100 ml projects running internally within
51:02
ml projects running internally within
51:02
ml projects running internally within vmware and so in aggregate
51:04
vmware and so in aggregate
51:04
vmware and so in aggregate it turns out that you know getting
51:05
it turns out that you know getting
51:05
it turns out that you know getting access to resources can be somewhat
51:07
access to resources can be somewhat
51:07
access to resources can be somewhat challenging
51:08
challenging
51:08
challenging of course i happen to be a little bit
51:10
of course i happen to be a little bit
51:10
of course i happen to be a little bit lucky because i run that central team
51:12
lucky because i run that central team
51:12
lucky because i run that central team and we have our own private resources
51:13
and we have our own private resources
51:14
and we have our own private resources that we can go and use so
51:15
that we can go and use so
51:16
that we can go and use so that's cheating a little bit but yeah
51:19
that's cheating a little bit but yeah
51:19
that's cheating a little bit but yeah you're right i mean getting access to
51:21
you're right i mean getting access to
51:21
you're right i mean getting access to those resources can be somewhat
51:23
those resources can be somewhat
51:23
those resources can be somewhat challenging for sure
51:25
challenging for sure
51:25
challenging for sure how is that for you eldran
51:29
what exactly sorry how is that for you
51:32
what exactly sorry how is that for you
51:32
what exactly sorry how is that for you how do you have do you have issues with
51:33
how do you have do you have issues with
51:33
how do you have do you have issues with infrastructure for
51:35
infrastructure for
51:35
infrastructure for for your project so for me
51:38
for your project so for me
51:38
for your project so for me i find the most challenging part is that
51:42
i find the most challenging part is that
51:42
i find the most challenging part is that basically if you want to use cutting
51:43
basically if you want to use cutting
51:44
basically if you want to use cutting edge technologies
51:45
edge technologies
51:45
edge technologies things are coming out every day every
51:47
things are coming out every day every
51:48
things are coming out every day every day you read cool papers which you would
51:50
day you read cool papers which you would
51:50
day you read cool papers which you would like to try on your system and if you
51:54
like to try on your system and if you
51:54
like to try on your system and if you basically have all your code um
51:57
basically have all your code um
51:57
basically have all your code um like together in one piece it's very
52:00
like together in one piece it's very
52:00
like together in one piece it's very hard to try and change
52:01
hard to try and change
52:01
hard to try and change things especially if you want to try
52:03
things especially if you want to try
52:03
things especially if you want to try some of them in production with
52:05
some of them in production with
52:05
some of them in production with real people that are actually using your
52:07
real people that are actually using your
52:07
real people that are actually using your models and i find it very useful
52:10
models and i find it very useful
52:10
models and i find it very useful in terms of architecture and
52:11
in terms of architecture and
52:11
in terms of architecture and infrastructure to separate
52:13
infrastructure to separate
52:13
infrastructure to separate every every model from the rest and
52:16
every every model from the rest and
52:16
every every model from the rest and actually deploy it on completely
52:18
actually deploy it on completely
52:18
actually deploy it on completely different server
52:19
different server
52:19
different server and as much as possible to separate it
52:22
and as much as possible to separate it
52:22
and as much as possible to separate it actually from the
52:24
actually from the
52:24
actually from the specific machine it's running on and to
52:27
specific machine it's running on and to
52:27
specific machine it's running on and to use
52:28
use
52:28
use some virtual machine that can actually
52:31
some virtual machine that can actually
52:31
some virtual machine that can actually make it possible thank you
52:34
make it possible thank you
52:34
make it possible thank you make it possible for you to emigrate
52:36
make it possible for you to emigrate
52:36
make it possible for you to emigrate from one cloud service to another
52:38
from one cloud service to another
52:38
from one cloud service to another or to try and use different models that
52:41
or to try and use different models that
52:41
or to try and use different models that use
52:42
use
52:42
use um specific um features
52:45
um specific um features
52:45
um specific um features of specific machine you really want to
52:47
of specific machine you really want to
52:47
of specific machine you really want to try and
52:48
try and
52:48
try and it's extremely useful to separate the
52:50
it's extremely useful to separate the
52:50
it's extremely useful to separate the models and make sure you can run them
52:52
models and make sure you can run them
52:52
models and make sure you can run them and try
52:52
and try
52:52
and try different ones very quickly
52:56
different ones very quickly
52:56
different ones very quickly i mean i have to jump in and say yeah of
52:58
i mean i have to jump in and say yeah of
52:58
i mean i have to jump in and say yeah of course i think it's awesome you're
52:59
course i think it's awesome you're
52:59
course i think it's awesome you're running in vms that that's
53:00
running in vms that that's
53:00
running in vms that that's that's something that we see internally
53:02
that's something that we see internally
53:02
that's something that we see internally and also our customers here at this
53:03
and also our customers here at this
53:04
and also our customers here at this uh what you're implying is this ability
53:06
uh what you're implying is this ability
53:06
uh what you're implying is this ability to encapsulate the entire software
53:08
to encapsulate the entire software
53:08
to encapsulate the entire software environment in that vm
53:09
environment in that vm
53:09
environment in that vm and be able to do that for different
53:11
and be able to do that for different
53:11
and be able to do that for different environments simultaneously
53:13
environments simultaneously
53:13
environments simultaneously and then be able to schedule those onto
53:14
and then be able to schedule those onto
53:14
and then be able to schedule those onto your hardware is a pretty powerful
53:16
your hardware is a pretty powerful
53:16
your hardware is a pretty powerful concept right
53:17
concept right
53:17
concept right yeah i think that's the beauty of a
53:19
yeah i think that's the beauty of a
53:19
yeah i think that's the beauty of a virtualization
53:23
and um thinking about that so you've
53:27
and um thinking about that so you've
53:27
and um thinking about that so you've obviously i know you've been on the call
53:29
obviously i know you've been on the call
53:29
obviously i know you've been on the call and so you've heard josh speak about
53:30
and so you've heard josh speak about
53:30
and so you've heard josh speak about some of the stuff they're doing around
53:31
some of the stuff they're doing around
53:31
some of the stuff they're doing around bug tracking
53:32
bug tracking
53:32
bug tracking would that help you with your business
53:34
would that help you with your business
53:34
would that help you with your business is it something do you think many
53:36
is it something do you think many
53:36
is it something do you think many software businesses are going to find
53:37
software businesses are going to find
53:37
software businesses are going to find something like that useful actually
53:39
something like that useful actually
53:39
something like that useful actually while
53:39
while
53:39
while just with talking i was thinking that we
53:42
just with talking i was thinking that we
53:42
just with talking i was thinking that we we sometimes
53:43
we sometimes
53:44
we sometimes like when we're building bot we need to
53:46
like when we're building bot we need to
53:46
like when we're building bot we need to use some
53:47
use some
53:47
use some data in order to train the model of the
53:50
data in order to train the model of the
53:50
data in order to train the model of the dialogue manager of the board for
53:51
dialogue manager of the board for
53:51
dialogue manager of the board for example
53:52
example
53:52
example and sometimes we have bugs basically in
53:55
and sometimes we have bugs basically in
53:55
and sometimes we have bugs basically in the data
53:56
the data
53:56
the data so if we have um two different stories
54:00
so if we have um two different stories
54:00
so if we have um two different stories that we want to
54:01
that we want to
54:02
that we want to teach our dialogue manager but they
54:04
teach our dialogue manager but they
54:04
teach our dialogue manager but they actually contradict each other
54:06
actually contradict each other
54:06
actually contradict each other they're very similar but there is small
54:08
they're very similar but there is small
54:08
they're very similar but there is small difference we find that the models are
54:10
difference we find that the models are
54:10
difference we find that the models are find it very hard to learn and i think
54:13
find it very hard to learn and i think
54:13
find it very hard to learn and i think it could be extremely useful for us to
54:15
it could be extremely useful for us to
54:15
it could be extremely useful for us to find this kind of bugs
54:17
find this kind of bugs
54:17
find this kind of bugs or kind of duplicates in our data and
54:20
or kind of duplicates in our data and
54:20
or kind of duplicates in our data and not
54:20
not
54:20
not selling a code interesting that's much
54:23
selling a code interesting that's much
54:23
selling a code interesting that's much more of a pure nlp task right where
54:25
more of a pure nlp task right where
54:25
more of a pure nlp task right where you're
54:26
you're
54:26
you're you're actually looking more at english
54:27
you're actually looking more at english
54:28
you're actually looking more at english or you know natural language
54:29
or you know natural language
54:30
or you know natural language yeah i was gonna say and so many people
54:33
yeah i was gonna say and so many people
54:33
yeah i was gonna say and so many people um in our comments have been talking
54:35
um in our comments have been talking
54:35
um in our comments have been talking about the data set i always feel like
54:36
about the data set i always feel like
54:36
about the data set i always feel like with so many
54:38
with so many
54:38
with so many different um uh solutions that we're
54:41
different um uh solutions that we're
54:41
different um uh solutions that we're doing with machine learning one of the
54:43
doing with machine learning one of the
54:43
doing with machine learning one of the interesting pieces is kind of like oh
54:45
interesting pieces is kind of like oh
54:45
interesting pieces is kind of like oh you're using all these amazing models
54:47
you're using all these amazing models
54:47
you're using all these amazing models and josh thank you for talking about the
54:49
and josh thank you for talking about the
54:49
and josh thank you for talking about the trade-offs as well between
54:51
trade-offs as well between
54:51
trade-offs as well between using you know the latest and greatest
54:53
using you know the latest and greatest
54:53
using you know the latest and greatest and the new shiny versus
54:55
and the new shiny versus
54:55
and the new shiny versus you know what do we get when we when we
54:56
you know what do we get when we when we
54:56
you know what do we get when we when we look at something a little bit more uh
54:58
look at something a little bit more uh
54:58
look at something a little bit more uh traditional machine learning um but yeah
55:01
traditional machine learning um but yeah
55:01
traditional machine learning um but yeah no it's it's super
55:02
no it's it's super
55:02
no it's it's super interesting to see everyone is focusing
55:03
interesting to see everyone is focusing
55:04
interesting to see everyone is focusing on the data set right and that the
55:06
on the data set right and that the
55:06
on the data set right and that the it it always feels like it comes back to
55:08
it it always feels like it comes back to
55:08
it it always feels like it comes back to that um
55:09
that um
55:09
that um sometimes when i i speak to people when
55:11
sometimes when i i speak to people when
55:11
sometimes when i i speak to people when they're getting into this space
55:12
they're getting into this space
55:12
they're getting into this space that's one of the things that's maybe
55:14
that's one of the things that's maybe
55:14
that's one of the things that's maybe more confusing where i'm like oh
55:16
more confusing where i'm like oh
55:16
more confusing where i'm like oh you know you might experiment with this
55:18
you know you might experiment with this
55:18
you know you might experiment with this first version one of the data set and
55:20
first version one of the data set and
55:20
first version one of the data set and then
55:21
then
55:21
then you almost go back to square one um and
55:24
you almost go back to square one um and
55:24
you almost go back to square one um and josh you brought it up when you brought
55:26
josh you brought it up when you brought
55:26
josh you brought it up when you brought up the
55:26
up the
55:26
up the uh kind of diagram of the life cycle
55:30
uh kind of diagram of the life cycle
55:30
uh kind of diagram of the life cycle process
55:30
process
55:30
process and saying you know the exciting part
55:32
and saying you know the exciting part
55:32
and saying you know the exciting part unfortunately isn't always the bit that
55:34
unfortunately isn't always the bit that
55:34
unfortunately isn't always the bit that you get to spend the most time on
55:36
you get to spend the most time on
55:36
you get to spend the most time on um so yeah do you do your um
55:39
um so yeah do you do your um
55:39
um so yeah do you do your um research teams have to spend a lot of
55:41
research teams have to spend a lot of
55:42
research teams have to spend a lot of time cleaning
55:43
time cleaning
55:43
time cleaning up the book data we had someone who
55:45
up the book data we had someone who
55:45
up the book data we had someone who asked oh you know
55:46
asked oh you know
55:46
asked oh you know there's so many people that ask uh that
55:48
there's so many people that ask uh that
55:48
there's so many people that ask uh that maybe submit the same issue but they all
55:51
maybe submit the same issue but they all
55:51
maybe submit the same issue but they all word it very very differently like is
55:53
word it very very differently like is
55:53
word it very very differently like is there a lot of cleaning that has to
55:54
there a lot of cleaning that has to
55:54
there a lot of cleaning that has to happen
55:55
happen
55:56
happen yeah so yes there the short answer is
55:58
yeah so yes there the short answer is
55:58
yeah so yes there the short answer is yes
55:59
yes
55:59
yes a lot of cleaning for sure so there's
56:01
a lot of cleaning for sure so there's
56:01
a lot of cleaning for sure so there's the issue that i brought up earlier of
56:02
the issue that i brought up earlier of
56:02
the issue that i brought up earlier of of making sure that you have embeddings
56:04
of making sure that you have embeddings
56:04
of making sure that you have embeddings that are appropriate for the vocabulary
56:06
that are appropriate for the vocabulary
56:06
that are appropriate for the vocabulary that you're dealing with right so
56:08
that you're dealing with right so
56:08
that you're dealing with right so the those bugs talk about vmware
56:11
the those bugs talk about vmware
56:11
the those bugs talk about vmware products
56:11
products
56:12
products you know and it definitely does not look
56:14
you know and it definitely does not look
56:14
you know and it definitely does not look like
56:15
like
56:15
like yes yes it's all english syntax but
56:18
yes yes it's all english syntax but
56:18
yes yes it's all english syntax but what they're talking about is stuff
56:19
what they're talking about is stuff
56:19
what they're talking about is stuff that's very foreign uh compared to a
56:21
that's very foreign uh compared to a
56:22
that's very foreign uh compared to a sort of generic vocabulary so that has
56:24
sort of generic vocabulary so that has
56:24
sort of generic vocabulary so that has to be dealt with
56:25
to be dealt with
56:25
to be dealt with uh but then yeah absolutely things like
56:27
uh but then yeah absolutely things like
56:27
uh but then yeah absolutely things like limitation and stemming and all of the
56:29
limitation and stemming and all of the
56:29
limitation and stemming and all of the things that you need to do
56:31
things that you need to do
56:31
things that you need to do in order to reduce the the um the text
56:34
in order to reduce the the um the text
56:34
in order to reduce the the um the text to some kind of a standard form
56:36
to some kind of a standard form
56:36
to some kind of a standard form is really important now the the comment
56:38
is really important now the the comment
56:38
is really important now the the comment about things being worded differently
56:40
about things being worded differently
56:40
about things being worded differently you know that's okay right because the
56:42
you know that's okay right because the
56:42
you know that's okay right because the embedding is going to help somewhat
56:43
embedding is going to help somewhat
56:44
embedding is going to help somewhat with that if the sentence level
56:45
with that if the sentence level
56:45
with that if the sentence level embeddings should capture some of that
56:47
embeddings should capture some of that
56:48
embeddings should capture some of that and then you're doing the document
56:49
and then you're doing the document
56:49
and then you're doing the document similarity between the
56:51
similarity between the
56:51
similarity between the the two bugs and that's kind of the
56:52
the two bugs and that's kind of the
56:52
the two bugs and that's kind of the point you want to you we recognize the
56:54
point you want to you we recognize the
56:54
point you want to you we recognize the fact that
56:55
fact that
56:56
fact that individual engineers may describe
56:58
individual engineers may describe
56:58
individual engineers may describe something differently
56:59
something differently
56:59
something differently but they're presumably describing the
57:01
but they're presumably describing the
57:01
but they're presumably describing the same thing
57:02
same thing
57:02
same thing and so what you're trying to do with the
57:04
and so what you're trying to do with the
57:04
and so what you're trying to do with the combination of the embeddings
57:05
combination of the embeddings
57:05
combination of the embeddings and the symbol the distance you know the
57:07
and the symbol the distance you know the
57:07
and the symbol the distance you know the similarity competition
57:09
similarity competition
57:09
similarity competition is fine those ones that are actually you
57:11
is fine those ones that are actually you
57:11
is fine those ones that are actually you know similar enough to be called
57:13
know similar enough to be called
57:13
know similar enough to be called duplicates
57:14
duplicates
57:14
duplicates yeah and just something then that
57:16
yeah and just something then that
57:16
yeah and just something then that triggered me actually as well and
57:18
triggered me actually as well and
57:18
triggered me actually as well and someone kind of mentioned it in our
57:19
someone kind of mentioned it in our
57:19
someone kind of mentioned it in our comments here
57:20
comments here
57:20
comments here um the cold start problem so the idea
57:24
um the cold start problem so the idea
57:24
um the cold start problem so the idea that vmware must create new products and
57:26
that vmware must create new products and
57:26
that vmware must create new products and so
57:27
so
57:27
so what what happens when we get to that
57:29
what what happens when we get to that
57:29
what what happens when we get to that cold start in some senses where we might
57:31
cold start in some senses where we might
57:31
cold start in some senses where we might not have heard of that product before
57:32
not have heard of that product before
57:32
not have heard of that product before and elrond will come to you for that as
57:34
and elrond will come to you for that as
57:34
and elrond will come to you for that as well because i think that's probably
57:35
well because i think that's probably
57:35
well because i think that's probably relevant for your area
57:39
oh sorry josh you you go first oh yeah
57:41
oh sorry josh you you go first oh yeah
57:42
oh sorry josh you you go first oh yeah sure
57:42
sure
57:42
sure yeah that's that's actually an awesome
57:44
yeah that's that's actually an awesome
57:44
yeah that's that's actually an awesome question so
57:45
question so
57:45
question so um and i'm really glad you asked it so
57:48
um and i'm really glad you asked it so
57:48
um and i'm really glad you asked it so so the bug classification proj project
57:51
so the bug classification proj project
57:51
so the bug classification proj project that i mentioned
57:52
that i mentioned
57:52
that i mentioned we are focusing on so as i said we have
57:55
we are focusing on so as i said we have
57:55
we are focusing on so as i said we have a we have hundreds of products
57:57
a we have hundreds of products
57:57
a we have hundreds of products you know that pcc we have hundreds of
57:59
you know that pcc we have hundreds of
57:59
you know that pcc we have hundreds of products we focus
58:01
products we focus
58:01
products we focus primarily on the top 10 of those
58:03
primarily on the top 10 of those
58:03
primarily on the top 10 of those products right now
58:04
products right now
58:04
products right now from a bug um quantity perspective
58:08
from a bug um quantity perspective
58:08
from a bug um quantity perspective for exactly the reason that you bring up
58:10
for exactly the reason that you bring up
58:10
for exactly the reason that you bring up is that the
58:11
is that the
58:11
is that the the corpus of examples that we have
58:14
the corpus of examples that we have
58:14
the corpus of examples that we have for say a new product it's in the cold
58:16
for say a new product it's in the cold
58:16
for say a new product it's in the cold start phase right we just don't have
58:18
start phase right we just don't have
58:18
start phase right we just don't have enough data
58:19
enough data
58:19
enough data to actually build a credible model to
58:21
to actually build a credible model to
58:21
to actually build a credible model to actually do good predictions
58:23
actually do good predictions
58:23
actually do good predictions for that particular product uh and
58:25
for that particular product uh and
58:25
for that particular product uh and unfortunately it can be the case because
58:28
unfortunately it can be the case because
58:28
unfortunately it can be the case because we have a pretty far-reaching product
58:30
we have a pretty far-reaching product
58:30
we have a pretty far-reaching product set that
58:32
set that
58:32
set that the duplicate that what you're learning
58:34
the duplicate that what you're learning
58:34
the duplicate that what you're learning from other products
58:35
from other products
58:35
from other products may not necessarily transfer all that
58:37
may not necessarily transfer all that
58:37
may not necessarily transfer all that strongly to what's going on within that
58:39
strongly to what's going on within that
58:39
strongly to what's going on within that new product
58:40
new product
58:40
new product and even there could be a different
58:42
and even there could be a different
58:42
and even there could be a different vocabulary which brings up another issue
58:44
vocabulary which brings up another issue
58:44
vocabulary which brings up another issue our vocabulary changes over time if you
58:47
our vocabulary changes over time if you
58:47
our vocabulary changes over time if you look at what we were talking about 10
58:49
look at what we were talking about 10
58:49
look at what we were talking about 10 years ago as a company internally
58:50
years ago as a company internally
58:50
years ago as a company internally and what we talk about now in terms of
58:52
and what we talk about now in terms of
58:52
and what we talk about now in terms of products et cetera it's very different
58:54
products et cetera it's very different
58:54
products et cetera it's very different so that means that those word embeddings
58:56
so that means that those word embeddings
58:56
so that means that those word embeddings and the in the language models those
58:58
and the in the language models those
58:58
and the in the language models those also need to be
58:59
also need to be
58:59
also need to be rebuilt over time as our as our as our
59:02
rebuilt over time as our as our as our
59:02
rebuilt over time as our as our as our vocabulary and our language changes over
59:04
vocabulary and our language changes over
59:04
vocabulary and our language changes over time
59:05
time
59:05
time very very good and elven yeah oh sorry
59:08
very very good and elven yeah oh sorry
59:08
very very good and elven yeah oh sorry sammy you go
59:08
sammy you go
59:08
sammy you go ah no problem go ahead first eldridge
59:11
ah no problem go ahead first eldridge
59:11
ah no problem go ahead first eldridge yeah ellen do you
59:13
yeah ellen do you
59:13
yeah ellen do you do you have cold start problems i'm i
59:16
do you have cold start problems i'm i
59:16
do you have cold start problems i'm i like feel like that might be because
59:18
like feel like that might be because
59:18
like feel like that might be because like a chatbot would be quite young to
59:19
like a chatbot would be quite young to
59:19
like a chatbot would be quite young to start off with until someone adds more
59:21
start off with until someone adds more
59:21
start off with until someone adds more data
59:23
data
59:23
data explaining what's called stuff sorry ah
59:25
explaining what's called stuff sorry ah
59:26
explaining what's called stuff sorry ah sorry yeah so cold saw oh good
59:27
sorry yeah so cold saw oh good
59:27
sorry yeah so cold saw oh good this is a test um how do you expect that
59:30
this is a test um how do you expect that
59:30
this is a test um how do you expect that so it's like
59:31
so it's like
59:31
so it's like when you start building a chat bot and
59:32
when you start building a chat bot and
59:32
when you start building a chat bot and there's not really any data yet
59:34
there's not really any data yet
59:34
there's not really any data yet that you can use to train your model on
59:37
that you can use to train your model on
59:38
that you can use to train your model on oh wow so it's actually very common with
59:40
oh wow so it's actually very common with
59:40
oh wow so it's actually very common with chatbots
59:41
chatbots
59:41
chatbots usually what you're trying to do is to
59:44
usually what you're trying to do is to
59:44
usually what you're trying to do is to get something to run
59:45
get something to run
59:45
get something to run and to actually get real data from the
59:47
and to actually get real data from the
59:47
and to actually get real data from the users because
59:48
users because
59:48
users because if you're constructing all your data by
59:50
if you're constructing all your data by
59:50
if you're constructing all your data by yourself it's it's not going to be
59:52
yourself it's it's not going to be
59:52
yourself it's it's not going to be useful in the real world so the best
59:54
useful in the real world so the best
59:54
useful in the real world so the best thing to do is basically to create a
59:56
thing to do is basically to create a
59:56
thing to do is basically to create a initial initial data set that you can
59:58
initial initial data set that you can
59:58
initial initial data set that you can work with and then get data from your
1:00:00
work with and then get data from your
1:00:00
work with and then get data from your your actual users and use this data
1:00:04
your actual users and use this data
1:00:04
your actual users and use this data to actually teach your systems how it
1:00:07
to actually teach your systems how it
1:00:07
to actually teach your systems how it works
1:00:08
works
1:00:08
works more accurately cool
1:00:14
more accurately cool
1:00:14
more accurately cool well now i wanted to still ask a
1:00:16
well now i wanted to still ask a
1:00:16
well now i wanted to still ask a question and shoot me it's gone
1:00:18
question and shoot me it's gone
1:00:18
question and shoot me it's gone how bad is that it's always like that
1:00:21
how bad is that it's always like that
1:00:21
how bad is that it's always like that let's have a look what else have we got
1:00:23
let's have a look what else have we got
1:00:23
let's have a look what else have we got coming in um
1:00:25
coming in um
1:00:25
coming in um well i think one other question like the
1:00:27
well i think one other question like the
1:00:27
well i think one other question like the models you are building
1:00:29
models you are building
1:00:29
models you are building uh josh at vmware you think it would be
1:00:32
uh josh at vmware you think it would be
1:00:32
uh josh at vmware you think it would be ever something that you would productize
1:00:34
ever something that you would productize
1:00:34
ever something that you would productize to bring public because i'm sure a lot
1:00:36
to bring public because i'm sure a lot
1:00:36
to bring public because i'm sure a lot of people would like to use it
1:00:39
of people would like to use it
1:00:39
of people would like to use it huh well that's an interesting thought
1:00:40
huh well that's an interesting thought
1:00:40
huh well that's an interesting thought um
1:00:42
um
1:00:42
um you know no promises here but um
1:00:46
you know no promises here but um
1:00:46
you know no promises here but um i think we you know for those sorts of
1:00:47
i think we you know for those sorts of
1:00:47
i think we you know for those sorts of things um
1:00:49
things um
1:00:49
things um we actually have a track record of
1:00:51
we actually have a track record of
1:00:51
we actually have a track record of releasing things
1:00:53
releasing things
1:00:53
releasing things um as flings what we call flings which
1:00:55
um as flings what we call flings which
1:00:55
um as flings what we call flings which are sort of
1:00:56
are sort of
1:00:56
are sort of um you know not products but things that
1:00:58
um you know not products but things that
1:00:58
um you know not products but things that we just put out for people
1:01:00
we just put out for people
1:01:00
we just put out for people uh who'd like to take advantage of them
1:01:01
uh who'd like to take advantage of them
1:01:02
uh who'd like to take advantage of them and also we open source things as well
1:01:04
and also we open source things as well
1:01:04
and also we open source things as well so it you know it it's conceivable that
1:01:06
so it you know it it's conceivable that
1:01:06
so it you know it it's conceivable that we could just decide yeah this is an
1:01:08
we could just decide yeah this is an
1:01:08
we could just decide yeah this is an interesting enough approach here
1:01:09
interesting enough approach here
1:01:09
interesting enough approach here maybe we should just put it out for the
1:01:11
maybe we should just put it out for the
1:01:11
maybe we should just put it out for the community um clearly we would benefit
1:01:13
community um clearly we would benefit
1:01:13
community um clearly we would benefit from people
1:01:14
from people
1:01:14
from people you know helping to improve it over time
1:01:15
you know helping to improve it over time
1:01:16
you know helping to improve it over time as well it's that it's the model that
1:01:17
as well it's that it's the model that
1:01:17
as well it's that it's the model that many you know many
1:01:18
many you know many
1:01:18
many you know many certainly larger companies like google
1:01:21
certainly larger companies like google
1:01:21
certainly larger companies like google and facebook and uber etcetera
1:01:22
and facebook and uber etcetera
1:01:22
and facebook and uber etcetera follow uh putting things open out in
1:01:25
follow uh putting things open out in
1:01:25
follow uh putting things open out in open source so we wouldn't necessarily
1:01:27
open source so we wouldn't necessarily
1:01:27
open source so we wouldn't necessarily productize it but the idea of sharing it
1:01:29
productize it but the idea of sharing it
1:01:29
productize it but the idea of sharing it i think is a is a pretty compelling one
1:01:32
i think is a is a pretty compelling one
1:01:32
i think is a is a pretty compelling one okay if we go back to your your
1:01:35
okay if we go back to your your
1:01:35
okay if we go back to your your your session you were talking about the
1:01:36
your session you were talking about the
1:01:36
your session you were talking about the amount of time it took to
1:01:38
amount of time it took to
1:01:38
amount of time it took to train your model and you talked about
1:01:40
train your model and you talked about
1:01:40
train your model and you talked about vbirth xgboost versus v-birds
1:01:44
vbirth xgboost versus v-birds
1:01:44
vbirth xgboost versus v-birds neural network yeah is it something you
1:01:47
neural network yeah is it something you
1:01:47
neural network yeah is it something you you retrain your models like daily
1:01:49
you retrain your models like daily
1:01:49
you retrain your models like daily or in what how many times do you retrain
1:01:53
or in what how many times do you retrain
1:01:53
or in what how many times do you retrain it
1:01:54
it
1:01:54
it yeah it's a good question so that
1:01:55
yeah it's a good question so that
1:01:55
yeah it's a good question so that particular project has not gone into
1:01:57
particular project has not gone into
1:01:57
particular project has not gone into production yet the classification one
1:01:59
production yet the classification one
1:01:59
production yet the classification one is in production uh it's a good question
1:02:02
is in production uh it's a good question
1:02:02
is in production uh it's a good question what the frequency needs to be
1:02:04
what the frequency needs to be
1:02:04
what the frequency needs to be uh for sure i don't think we we have a
1:02:06
uh for sure i don't think we we have a
1:02:06
uh for sure i don't think we we have a really good handle on that at this point
1:02:08
really good handle on that at this point
1:02:08
really good handle on that at this point okay well i think i can ask the same
1:02:11
okay well i think i can ask the same
1:02:11
okay well i think i can ask the same question to elrond
1:02:12
question to elrond
1:02:12
question to elrond for a chat bots if you want to build
1:02:13
for a chat bots if you want to build
1:02:13
for a chat bots if you want to build chat bots the the model behind the nlp
1:02:16
chat bots the the model behind the nlp
1:02:16
chat bots the the model behind the nlp model
1:02:17
model
1:02:17
model how many times do you need to refresh it
1:02:20
how many times do you need to refresh it
1:02:20
how many times do you need to refresh it oh wow
1:02:20
oh wow
1:02:20
oh wow so things are changing very fast
1:02:23
so things are changing very fast
1:02:23
so things are changing very fast especially lately
1:02:25
especially lately
1:02:25
especially lately and yeah i feel like as
1:02:29
and yeah i feel like as
1:02:29
and yeah i feel like as someone that like wants to like your
1:02:31
someone that like wants to like your
1:02:32
someone that like wants to like your products work the best
1:02:33
products work the best
1:02:33
products work the best possible you're always trying to get to
1:02:36
possible you're always trying to get to
1:02:36
possible you're always trying to get to new
1:02:36
new
1:02:36
new papers written about it and trying new
1:02:38
papers written about it and trying new
1:02:38
papers written about it and trying new things and
1:02:40
things and
1:02:40
things and we actually often found that little
1:02:43
we actually often found that little
1:02:43
we actually often found that little tricks we read on papers
1:02:44
tricks we read on papers
1:02:44
tricks we read on papers or even completely different way of
1:02:47
or even completely different way of
1:02:48
or even completely different way of thinking about things help us to improve
1:02:50
thinking about things help us to improve
1:02:50
thinking about things help us to improve things dramatically
1:02:52
things dramatically
1:02:52
things dramatically okay cool we have another question from
1:02:55
okay cool we have another question from
1:02:55
okay cool we have another question from the public
1:02:57
the public
1:02:57
the public it's concerning if you know of any
1:03:00
it's concerning if you know of any
1:03:00
it's concerning if you know of any nlp tool or application that might come
1:03:03
nlp tool or application that might come
1:03:03
nlp tool or application that might come in the future it might be even an id
1:03:05
in the future it might be even an id
1:03:06
in the future it might be even an id that could well that could change
1:03:07
that could well that could change
1:03:07
that could well that could change education
1:03:10
education
1:03:10
education change education what does that mean to
1:03:12
change education what does that mean to
1:03:12
change education what does that mean to change education
1:03:13
change education
1:03:13
change education the way of teaching children for example
1:03:19
around nlp yeah i guess yeah they get
1:03:22
around nlp yeah i guess yeah they get
1:03:22
around nlp yeah i guess yeah they get just anything being able to
1:03:23
just anything being able to
1:03:23
just anything being able to automatically interpret or like mark
1:03:26
automatically interpret or like mark
1:03:26
automatically interpret or like mark mark homework
1:03:27
mark homework
1:03:27
mark homework type thing maybe and because it feels
1:03:30
type thing maybe and because it feels
1:03:30
type thing maybe and because it feels like
1:03:30
like
1:03:30
like the bug stuff it's like automation isn't
1:03:33
the bug stuff it's like automation isn't
1:03:33
the bug stuff it's like automation isn't it of it and so i think with most
1:03:35
it of it and so i think with most
1:03:35
it of it and so i think with most sectors now p
1:03:37
sectors now p
1:03:37
sectors now p it feels like there's a lot to how can
1:03:39
it feels like there's a lot to how can
1:03:39
it feels like there's a lot to how can we very
1:03:40
we very
1:03:40
we very quickly understand context and kind of
1:03:42
quickly understand context and kind of
1:03:42
quickly understand context and kind of like aggregate context in some senses
1:03:46
like aggregate context in some senses
1:03:46
like aggregate context in some senses aren't universities also using some kind
1:03:48
aren't universities also using some kind
1:03:48
aren't universities also using some kind of tools to double check
1:03:50
of tools to double check
1:03:50
of tools to double check if new papers are not copies of
1:03:53
if new papers are not copies of
1:03:53
if new papers are not copies of existing papers yeah
1:03:58
existing papers yeah
1:03:58
existing papers yeah now that um amy has made that comment it
1:03:59
now that um amy has made that comment it
1:04:00
now that um amy has made that comment it makes me think about um
1:04:01
makes me think about um
1:04:01
makes me think about um another instance and that is if for
1:04:03
another instance and that is if for
1:04:03
another instance and that is if for example you look at um
1:04:05
example you look at um
1:04:05
example you look at um something like coursera or any of the
1:04:07
something like coursera or any of the
1:04:07
something like coursera or any of the online
1:04:08
online
1:04:08
online learning platforms right it's not
1:04:10
learning platforms right it's not
1:04:10
learning platforms right it's not unusual that when you join one of those
1:04:11
unusual that when you join one of those
1:04:11
unusual that when you join one of those courses there may be
1:04:13
courses there may be
1:04:13
courses there may be you know 30 000 or 50 000 people taking
1:04:15
you know 30 000 or 50 000 people taking
1:04:16
you know 30 000 or 50 000 people taking the same course
1:04:17
the same course
1:04:17
the same course and if you think about sure you know how
1:04:19
and if you think about sure you know how
1:04:19
and if you think about sure you know how do you score things there
1:04:20
do you score things there
1:04:20
do you score things there well it's one thing to have multiple
1:04:22
well it's one thing to have multiple
1:04:22
well it's one thing to have multiple choice questions and they tend to have a
1:04:24
choice questions and they tend to have a
1:04:24
choice questions and they tend to have a high reliance on that but if you want to
1:04:26
high reliance on that but if you want to
1:04:26
high reliance on that but if you want to allow people to have maybe more nuanced
1:04:28
allow people to have maybe more nuanced
1:04:28
allow people to have maybe more nuanced you know maybe essay type answers to
1:04:30
you know maybe essay type answers to
1:04:30
you know maybe essay type answers to questions
1:04:31
questions
1:04:31
questions you can imagine using an lp to do some
1:04:33
you can imagine using an lp to do some
1:04:33
you can imagine using an lp to do some kind of analysis
1:04:34
kind of analysis
1:04:34
kind of analysis of of you know student
1:04:38
of of you know student
1:04:38
of of you know student answers to try to understand whether or
1:04:40
answers to try to understand whether or
1:04:40
answers to try to understand whether or not maybe they have particular aspects
1:04:42
not maybe they have particular aspects
1:04:42
not maybe they have particular aspects of the
1:04:43
of the
1:04:43
of the answer that the instructor is looking
1:04:44
answer that the instructor is looking
1:04:44
answer that the instructor is looking for kind of embedded in some way in
1:04:46
for kind of embedded in some way in
1:04:46
for kind of embedded in some way in their response
1:04:47
their response
1:04:47
their response would be one thing you could do and you
1:04:48
would be one thing you could do and you
1:04:48
would be one thing you could do and you would need something like that because
1:04:49
would need something like that because
1:04:49
would need something like that because there's no way
1:04:51
there's no way
1:04:51
there's no way you're going to be able to do that
1:04:52
you're going to be able to do that
1:04:52
you're going to be able to do that manually with that many people in the
1:04:53
manually with that many people in the
1:04:53
manually with that many people in the class
1:04:55
class
1:04:55
class absolutely well
1:04:59
absolutely well
1:04:59
absolutely well uh josh thank you very much for joining
1:05:01
uh josh thank you very much for joining
1:05:01
uh josh thank you very much for joining us today we were very happy to have you
1:05:03
us today we were very happy to have you
1:05:03
us today we were very happy to have you on board
1:05:04
on board
1:05:04
on board we still wish you a very nice pleasant
1:05:06
we still wish you a very nice pleasant
1:05:06
we still wish you a very nice pleasant afternoon i think you're lazy based in
1:05:08
afternoon i think you're lazy based in
1:05:08
afternoon i think you're lazy based in boston so enjoy your afternoon and
1:05:11
boston so enjoy your afternoon and
1:05:11
boston so enjoy your afternoon and we hope to speak to you another time
1:05:13
we hope to speak to you another time
1:05:13
we hope to speak to you another time again thank you very much
1:05:15
again thank you very much
1:05:15
again thank you very much for joining us thanks so much thanks so
1:05:17
for joining us thanks so much thanks so
1:05:17
for joining us thanks so much thanks so much for inviting me i really enjoyed
1:05:18
much for inviting me i really enjoyed
1:05:18
much for inviting me i really enjoyed myself
1:05:19
myself
1:05:19
myself okay cool bye-bye see you later josh
1:05:25
okay so we are over to elrond elrond
1:05:28
okay so we are over to elrond elrond
1:05:28
okay so we are over to elrond elrond the floor is over to you you are gonna
1:05:31
the floor is over to you you are gonna
1:05:31
the floor is over to you you are gonna tell us
1:05:32
tell us
1:05:32
tell us uh i i made up the title of your session
1:05:34
uh i i made up the title of your session
1:05:34
uh i i made up the title of your session so uh the story behind first part
1:05:37
so uh the story behind first part
1:05:37
so uh the story behind first part talking about um some of the challenges
1:05:39
talking about um some of the challenges
1:05:39
talking about um some of the challenges some of the models and some of the
1:05:41
some of the models and some of the
1:05:41
some of the models and some of the um different architecture choices you
1:05:43
um different architecture choices you
1:05:43
um different architecture choices you made and i think this is going to be
1:05:45
made and i think this is going to be
1:05:45
made and i think this is going to be so exciting so yes over to you please
1:05:47
so exciting so yes over to you please
1:05:48
so exciting so yes over to you please take it away
1:05:50
take it away
1:05:50
take it away all right so
1:05:51
all right so
1:05:51
all right so [Music]
1:05:54
[Music]
1:05:54
[Music] can you see my screen we can
1:05:57
can you see my screen we can
1:05:57
can you see my screen we can yes we can this is a good start so
1:06:00
yes we can this is a good start so
1:06:00
yes we can this is a good start so again about it yeah a bit a bit more
1:06:04
again about it yeah a bit a bit more
1:06:04
again about it yeah a bit a bit more about myself
1:06:04
about myself
1:06:04
about myself um my name is lauren bender i'm an nlp
1:06:08
um my name is lauren bender i'm an nlp
1:06:08
um my name is lauren bender i'm an nlp researcher at biolan university
1:06:10
researcher at biolan university
1:06:10
researcher at biolan university and i'm very excited to be here today
1:06:12
and i'm very excited to be here today
1:06:12
and i'm very excited to be here today and tell you about
1:06:13
and tell you about
1:06:13
and tell you about best spot which is a project that i
1:06:15
best spot which is a project that i
1:06:15
best spot which is a project that i really like
1:06:16
really like
1:06:16
really like and instead of explaining to you what
1:06:19
and instead of explaining to you what
1:06:19
and instead of explaining to you what this what is i would like to show you a
1:06:21
this what is i would like to show you a
1:06:21
this what is i would like to show you a short
1:06:30
video
1:06:33
video
1:06:33
video sorry you might need to stop sharing
1:06:35
sorry you might need to stop sharing
1:06:36
sorry you might need to stop sharing your screen
1:06:37
your screen
1:06:37
your screen yeah a second that's right sorry
1:06:40
yeah a second that's right sorry
1:06:40
yeah a second that's right sorry you want to connect with your online
1:06:42
you want to connect with your online
1:06:42
you want to connect with your online customers
1:06:44
customers
1:06:44
customers a program that interacts with clients
1:06:46
a program that interacts with clients
1:06:46
a program that interacts with clients could be
1:06:47
could be
1:06:47
could be complex and expensive so let us
1:06:50
complex and expensive so let us
1:06:50
complex and expensive so let us introduce you
1:06:51
introduce you
1:06:51
introduce you to verify a zipper platform for genius
1:06:55
to verify a zipper platform for genius
1:06:55
to verify a zipper platform for genius business chatbots
1:07:00
business chatbots
1:07:00
business chatbots sorry serve
1:07:04
sorry serve
1:07:04
sorry serve bestbot.com to create your own business
1:07:07
bestbot.com to create your own business
1:07:07
bestbot.com to create your own business chatbot
1:07:09
sign up and tell us about yourself with
1:07:12
sign up and tell us about yourself with
1:07:12
sign up and tell us about yourself with just a few
1:07:13
just a few
1:07:13
just a few details
1:07:15
[Music]
1:07:20
now you're ready to create your first
1:07:23
now you're ready to create your first
1:07:23
now you're ready to create your first friendly
1:07:23
friendly
1:07:24
friendly business chatbot feed your chapper with
1:07:27
business chatbot feed your chapper with
1:07:27
business chatbot feed your chapper with a business description
1:07:29
a business description
1:07:29
a business description including information customers
1:07:32
including information customers
1:07:32
including information customers frequently asked
1:07:34
frequently asked
1:07:34
frequently asked the chat but then uses this description
1:07:36
the chat but then uses this description
1:07:36
the chat but then uses this description to instantaneously assemble your own
1:07:38
to instantaneously assemble your own
1:07:38
to instantaneously assemble your own chatbot
1:07:40
chatbot
1:07:40
chatbot it gets running at the speed of light
1:07:44
it gets running at the speed of light
1:07:44
it gets running at the speed of light the chatbot can then be placed on your
1:07:47
the chatbot can then be placed on your
1:07:47
the chatbot can then be placed on your business website for use
1:07:52
now try interacting with your bot and
1:07:54
now try interacting with your bot and
1:07:54
now try interacting with your bot and see what needs a little tidying up
1:07:58
see what needs a little tidying up
1:07:58
see what needs a little tidying up [Music]
1:08:08
[Music]
1:08:08
[Music] make changes and see how your chatbot
1:08:11
make changes and see how your chatbot
1:08:11
make changes and see how your chatbot grows
1:08:12
grows
1:08:12
grows you can edit your business description
1:08:14
you can edit your business description
1:08:14
you can edit your business description at any time
1:08:21
your chat bot will know when human
1:08:23
your chat bot will know when human
1:08:23
your chat bot will know when human assistance is necessary
1:08:33
it then collects relevant contact
1:08:36
it then collects relevant contact
1:08:36
it then collects relevant contact information
1:08:37
information
1:08:37
information so you can get back to your customers
1:08:39
so you can get back to your customers
1:08:39
so you can get back to your customers inquiries in a timely manner
1:08:44
your chatbot keeps and records all
1:08:47
your chatbot keeps and records all
1:08:47
your chatbot keeps and records all connections
1:08:48
connections
1:08:48
connections with your customers
1:08:52
with your customers
1:08:52
with your customers visit us at vespart.com to learn more
1:09:06
visit us at vespart.com to learn more
1:09:06
visit us at vespart.com to learn more [Music]
1:09:10
okay
1:09:14
sorry okay so why is
1:09:17
sorry okay so why is
1:09:17
sorry okay so why is the thought even necessary
1:09:21
the thought even necessary
1:09:21
the thought even necessary so current conversational assistance
1:09:23
so current conversational assistance
1:09:23
so current conversational assistance system
1:09:24
system
1:09:24
system basically used large data sets
1:09:27
basically used large data sets
1:09:27
basically used large data sets containing
1:09:28
containing
1:09:28
containing questions and and suitable answers and
1:09:31
questions and and suitable answers and
1:09:31
questions and and suitable answers and plenty of stories that
1:09:32
plenty of stories that
1:09:32
plenty of stories that might come up as conversation between um
1:09:35
might come up as conversation between um
1:09:35
might come up as conversation between um the chatbot and your clients a problem
1:09:38
the chatbot and your clients a problem
1:09:38
the chatbot and your clients a problem with this process
1:09:39
with this process
1:09:39
with this process can be extremely expensive and
1:09:42
can be extremely expensive and
1:09:42
can be extremely expensive and long and we wanted to make it a bit
1:09:45
long and we wanted to make it a bit
1:09:45
long and we wanted to make it a bit shorter
1:09:46
shorter
1:09:46
shorter and more than that a bit more natural so
1:09:49
and more than that a bit more natural so
1:09:49
and more than that a bit more natural so for someone that have a small business
1:09:52
for someone that have a small business
1:09:52
for someone that have a small business doesn't necessarily have the right
1:09:55
doesn't necessarily have the right
1:09:55
doesn't necessarily have the right knowledge of
1:09:56
knowledge of
1:09:56
knowledge of what kind of question answers should
1:09:58
what kind of question answers should
1:09:58
what kind of question answers should choose in order to
1:09:59
choose in order to
1:09:59
choose in order to train a good model so what we try to do
1:10:01
train a good model so what we try to do
1:10:02
train a good model so what we try to do is actually to decouple the business
1:10:03
is actually to decouple the business
1:10:03
is actually to decouple the business knowledge
1:10:04
knowledge
1:10:04
knowledge from the qa expertise
1:10:08
from the qa expertise
1:10:08
from the qa expertise so then the the business owner can just
1:10:11
so then the the business owner can just
1:10:11
so then the the business owner can just bring the business knowledge as a
1:10:13
bring the business knowledge as a
1:10:13
bring the business knowledge as a as a just plain text and we will bring
1:10:15
as a just plain text and we will bring
1:10:16
as a just plain text and we will bring the qax 30 and we will extract the right
1:10:18
the qax 30 and we will extract the right
1:10:18
the qax 30 and we will extract the right answers from this
1:10:19
answers from this
1:10:19
answers from this from this text based on the questions
1:10:21
from this text based on the questions
1:10:21
from this text based on the questions being asked from
1:10:23
being asked from
1:10:23
being asked from their clients
1:10:26
so what we are trying to do is
1:10:29
so what we are trying to do is
1:10:30
so what we are trying to do is basically to create a chatbot just from
1:10:32
basically to create a chatbot just from
1:10:32
basically to create a chatbot just from text
1:10:35
so the problem with this method is
1:10:38
so the problem with this method is
1:10:38
so the problem with this method is basically that you can't really get all
1:10:40
basically that you can't really get all
1:10:40
basically that you can't really get all the information needed from
1:10:42
the information needed from
1:10:42
the information needed from a text tax if you think about it some of
1:10:45
a text tax if you think about it some of
1:10:45
a text tax if you think about it some of the
1:10:46
the
1:10:46
the chats with businesses can be very
1:10:48
chats with businesses can be very
1:10:48
chats with businesses can be very complicated for example if you want to
1:10:49
complicated for example if you want to
1:10:50
complicated for example if you want to call to your bank and ask for a loan
1:10:52
call to your bank and ask for a loan
1:10:52
call to your bank and ask for a loan then your your bank will might ask you
1:10:55
then your your bank will might ask you
1:10:55
then your your bank will might ask you about your risk profile or about your
1:10:58
about your risk profile or about your
1:10:58
about your risk profile or about your credit
1:10:58
credit
1:10:58
credit or or whatever and the thing is
1:11:02
or or whatever and the thing is
1:11:02
or or whatever and the thing is we can't really learn everything from
1:11:03
we can't really learn everything from
1:11:04
we can't really learn everything from tax so we try to focus on small
1:11:05
tax so we try to focus on small
1:11:05
tax so we try to focus on small businesses
1:11:06
businesses
1:11:06
businesses that need um small operations so
1:11:10
that need um small operations so
1:11:10
that need um small operations so the three main goals we saw in front of
1:11:13
the three main goals we saw in front of
1:11:13
the three main goals we saw in front of eyes while developing passport is that
1:11:15
eyes while developing passport is that
1:11:15
eyes while developing passport is that it will be extremely easy
1:11:16
it will be extremely easy
1:11:16
it will be extremely easy and cheap to create it will be able to
1:11:18
and cheap to create it will be able to
1:11:18
and cheap to create it will be able to answer informative questions about the
1:11:20
answer informative questions about the
1:11:20
answer informative questions about the business very well
1:11:22
business very well
1:11:22
business very well and the most important part whenever the
1:11:24
and the most important part whenever the
1:11:24
and the most important part whenever the bot feel
1:11:25
bot feel
1:11:25
bot feel like it can't give you a good answer or
1:11:27
like it can't give you a good answer or
1:11:27
like it can't give you a good answer or it can't handle a situation it will
1:11:28
it can't handle a situation it will
1:11:28
it can't handle a situation it will redirect you to
1:11:29
redirect you to
1:11:29
redirect you to human we think it's very important part
1:11:32
human we think it's very important part
1:11:32
human we think it's very important part of it
1:11:38
so a bit about the
1:11:41
so a bit about the
1:11:41
so a bit about the what's happening investment behind the
1:11:43
what's happening investment behind the
1:11:43
what's happening investment behind the scene so basically we have
1:11:45
scene so basically we have
1:11:46
scene so basically we have two flaws here the first floor is the
1:11:48
two flaws here the first floor is the
1:11:48
two flaws here the first floor is the floor of the business donor which you
1:11:49
floor of the business donor which you
1:11:50
floor of the business donor which you can see is the
1:11:50
can see is the
1:11:50
can see is the is the green floor whenever the business
1:11:53
is the green floor whenever the business
1:11:53
is the green floor whenever the business owner goes on our website they can
1:11:55
owner goes on our website they can
1:11:55
owner goes on our website they can they can input our system with a
1:11:57
they can input our system with a
1:11:58
they can input our system with a description of their business
1:12:00
description of their business
1:12:00
description of their business then this description goes to our
1:12:01
then this description goes to our
1:12:01
then this description goes to our servers and stored in our database
1:12:05
servers and stored in our database
1:12:05
servers and stored in our database and the most interesting floor is
1:12:06
and the most interesting floor is
1:12:06
and the most interesting floor is actually the floor of the customer of
1:12:08
actually the floor of the customer of
1:12:08
actually the floor of the customer of this business
1:12:09
this business
1:12:09
this business the one which will actually interact
1:12:12
the one which will actually interact
1:12:12
the one which will actually interact with the chatbot
1:12:13
with the chatbot
1:12:13
with the chatbot so this is the red flow so another one
1:12:17
so this is the red flow so another one
1:12:17
so this is the red flow so another one is a question
1:12:20
is a question
1:12:20
is a question being asked the question that the client
1:12:23
being asked the question that the client
1:12:23
being asked the question that the client asked the
1:12:23
asked the
1:12:24
asked the chatbot it goes number one to our server
1:12:27
chatbot it goes number one to our server
1:12:27
chatbot it goes number one to our server and then straight to the dialog manager
1:12:29
and then straight to the dialog manager
1:12:29
and then straight to the dialog manager the role of the dialog manager
1:12:31
the role of the dialog manager
1:12:31
the role of the dialog manager is to keep the state of what happened in
1:12:33
is to keep the state of what happened in
1:12:33
is to keep the state of what happened in the conversation before
1:12:35
the conversation before
1:12:35
the conversation before and then decide what will be the
1:12:39
and then decide what will be the
1:12:39
and then decide what will be the the the right way to answer
1:12:42
the the right way to answer
1:12:42
the the right way to answer this conversation so
1:12:45
this conversation so
1:12:45
this conversation so if this conversation has nothing to do
1:12:47
if this conversation has nothing to do
1:12:47
if this conversation has nothing to do with a specific business for example if
1:12:49
with a specific business for example if
1:12:49
with a specific business for example if someone asks
1:12:49
someone asks
1:12:50
someone asks you are you a real person or a chatbot
1:12:53
you are you a real person or a chatbot
1:12:53
you are you a real person or a chatbot then you can answer this question even
1:12:55
then you can answer this question even
1:12:55
then you can answer this question even if you don't know anything about a
1:12:56
if you don't know anything about a
1:12:56
if you don't know anything about a business so in this kind of cases the
1:12:58
business so in this kind of cases the
1:12:58
business so in this kind of cases the dialogue manager can handle this
1:13:01
dialogue manager can handle this
1:13:01
dialogue manager can handle this this conversation entirely but then
1:13:03
this conversation entirely but then
1:13:03
this conversation entirely but then whenever
1:13:04
whenever
1:13:04
whenever someone asking about the specifics of
1:13:07
someone asking about the specifics of
1:13:07
someone asking about the specifics of the business something that has to do
1:13:08
the business something that has to do
1:13:08
the business something that has to do with
1:13:09
with
1:13:09
with the specific information of the business
1:13:11
the specific information of the business
1:13:11
the specific information of the business then
1:13:12
then
1:13:12
then we will take the information of the
1:13:15
we will take the information of the
1:13:15
we will take the information of the business
1:13:16
business
1:13:16
business in number three from the database then
1:13:19
in number three from the database then
1:13:19
in number three from the database then we'll apply some pre-processing
1:13:21
we'll apply some pre-processing
1:13:21
we'll apply some pre-processing on this on the question and the text of
1:13:24
on this on the question and the text of
1:13:24
on this on the question and the text of the business
1:13:25
the business
1:13:25
the business and then we'll use the neural network to
1:13:27
and then we'll use the neural network to
1:13:27
and then we'll use the neural network to extract
1:13:28
extract
1:13:28
extract the answer for the question
1:13:31
the answer for the question
1:13:31
the answer for the question from the business tax and
1:13:35
from the business tax and
1:13:35
from the business tax and i will now go over every component here
1:13:39
i will now go over every component here
1:13:39
i will now go over every component here separately and i'll just add that every
1:13:41
separately and i'll just add that every
1:13:41
separately and i'll just add that every one of them is basically
1:13:42
one of them is basically
1:13:42
one of them is basically operating on a completely different
1:13:45
operating on a completely different
1:13:45
operating on a completely different servers and microservice
1:13:46
servers and microservice
1:13:46
servers and microservice completely independent from another
1:13:48
completely independent from another
1:13:48
completely independent from another that's what helped us to try and
1:13:51
that's what helped us to try and
1:13:51
that's what helped us to try and use completely different methods and
1:13:54
use completely different methods and
1:13:54
use completely different methods and models and i will speak about it later
1:13:56
models and i will speak about it later
1:13:56
models and i will speak about it later on
1:13:57
on
1:13:57
on so the first part is the dialog manager
1:14:01
so the first part is the dialog manager
1:14:01
so the first part is the dialog manager the dialog manager see that it
1:14:04
the dialog manager see that it
1:14:04
the dialog manager see that it has the um the history of the
1:14:08
has the um the history of the
1:14:08
has the um the history of the chat with the client and then now he
1:14:10
chat with the client and then now he
1:14:10
chat with the client and then now he needs to decide what will be
1:14:12
needs to decide what will be
1:14:12
needs to decide what will be the next part of the conv of the
1:14:14
the next part of the conv of the
1:14:14
the next part of the conv of the conversation what will be the right
1:14:16
conversation what will be the right
1:14:16
conversation what will be the right response
1:14:17
response
1:14:17
response so how do you actually do it because
1:14:19
so how do you actually do it because
1:14:19
so how do you actually do it because basically all everything the dialogue
1:14:20
basically all everything the dialogue
1:14:20
basically all everything the dialogue manager have
1:14:21
manager have
1:14:22
manager have is um just sequence of words
1:14:25
is um just sequence of words
1:14:25
is um just sequence of words so how do you get sequence of words into
1:14:28
so how do you get sequence of words into
1:14:28
so how do you get sequence of words into something meaningful you can actually
1:14:30
something meaningful you can actually
1:14:30
something meaningful you can actually work with in order to find the right
1:14:32
work with in order to find the right
1:14:32
work with in order to find the right response
1:14:33
response
1:14:33
response so the first step is actually to take
1:14:36
so the first step is actually to take
1:14:36
so the first step is actually to take every part of this
1:14:37
every part of this
1:14:37
every part of this every sentence in this conversation
1:14:39
every sentence in this conversation
1:14:39
every sentence in this conversation classify it
1:14:40
classify it
1:14:40
classify it as an object with a where we did
1:14:43
as an object with a where we did
1:14:43
as an object with a where we did distinct meaning
1:14:44
distinct meaning
1:14:44
distinct meaning for example if the user said hi we we
1:14:47
for example if the user said hi we we
1:14:47
for example if the user said hi we we classified as
1:14:48
classified as
1:14:48
classified as grip if the user said how are you we
1:14:51
grip if the user said how are you we
1:14:51
grip if the user said how are you we classified as an object we call mod
1:14:55
classified as an object we call mod
1:14:55
classified as an object we call mod mood cube sorry and then after we have
1:14:58
mood cube sorry and then after we have
1:14:58
mood cube sorry and then after we have all these objects
1:15:00
all these objects
1:15:00
all these objects we can actually have a series of objects
1:15:01
we can actually have a series of objects
1:15:02
we can actually have a series of objects and what we need to
1:15:03
and what we need to
1:15:03
and what we need to clap now what we need to predict is just
1:15:05
clap now what we need to predict is just
1:15:05
clap now what we need to predict is just the next
1:15:06
the next
1:15:06
the next object so then what
1:15:10
object so then what
1:15:10
object so then what what we're doing in this case that we
1:15:11
what we're doing in this case that we
1:15:11
what we're doing in this case that we have large reserve
1:15:13
have large reserve
1:15:14
have large reserve large data set of stories which are
1:15:17
large data set of stories which are
1:15:17
large data set of stories which are basically sequences of objects and then
1:15:19
basically sequences of objects and then
1:15:19
basically sequences of objects and then all we have to do
1:15:21
all we have to do
1:15:21
all we have to do is to find the most similar story and
1:15:25
is to find the most similar story and
1:15:25
is to find the most similar story and use this
1:15:26
use this
1:15:26
use this this story in order to predict the next
1:15:29
this story in order to predict the next
1:15:29
this story in order to predict the next um the next response
1:15:33
um the next response
1:15:33
um the next response so how do how did we actually do all
1:15:36
so how do how did we actually do all
1:15:36
so how do how did we actually do all that
1:15:36
that
1:15:36
that we actually used a open source
1:15:40
we actually used a open source
1:15:40
we actually used a open source python library called rasa and
1:15:44
python library called rasa and
1:15:44
python library called rasa and rasa allows you basically to train all
1:15:47
rasa allows you basically to train all
1:15:47
rasa allows you basically to train all the models
1:15:48
the models
1:15:48
the models need needed for a dialog manager in one
1:15:50
need needed for a dialog manager in one
1:15:50
need needed for a dialog manager in one place so all you need to do
1:15:53
place so all you need to do
1:15:53
place so all you need to do is actually to construct your own data
1:15:55
is actually to construct your own data
1:15:55
is actually to construct your own data set
1:15:56
set
1:15:56
set that tells like the models how they
1:15:59
that tells like the models how they
1:15:59
that tells like the models how they like every kind of objects should look
1:16:02
like every kind of objects should look
1:16:02
like every kind of objects should look like
1:16:03
like
1:16:03
like in a natural language form and then all
1:16:05
in a natural language form and then all
1:16:05
in a natural language form and then all the stores you expect to see
1:16:06
the stores you expect to see
1:16:06
the stores you expect to see in in your conversations and then choose
1:16:09
in in your conversations and then choose
1:16:09
in in your conversations and then choose from
1:16:10
from
1:16:10
from the various models that rasa offer you
1:16:13
the various models that rasa offer you
1:16:13
the various models that rasa offer you and train it and in iterative process
1:16:17
and train it and in iterative process
1:16:17
and train it and in iterative process try
1:16:18
try
1:16:18
try try and improve your dialog manager
1:16:21
try and improve your dialog manager
1:16:21
try and improve your dialog manager and as i mentioned before it's very
1:16:23
and as i mentioned before it's very
1:16:23
and as i mentioned before it's very important to try and come up with a
1:16:25
important to try and come up with a
1:16:25
important to try and come up with a simple dialog manager and then try to
1:16:28
simple dialog manager and then try to
1:16:28
simple dialog manager and then try to get
1:16:29
get
1:16:29
get and interact with real real users and
1:16:32
and interact with real real users and
1:16:32
and interact with real real users and see
1:16:32
see
1:16:32
see what works and what not and then improve
1:16:35
what works and what not and then improve
1:16:35
what works and what not and then improve every part
1:16:36
every part
1:16:36
every part of of the dialogue manager
1:16:40
of of the dialogue manager
1:16:40
of of the dialogue manager so the most important object in the
1:16:43
so the most important object in the
1:16:43
so the most important object in the first book domain is the
1:16:46
first book domain is the
1:16:46
first book domain is the object we called org info which is
1:16:50
object we called org info which is
1:16:50
object we called org info which is questions about the specifics of the
1:16:52
questions about the specifics of the
1:16:52
questions about the specifics of the business it's a question that you can
1:16:54
business it's a question that you can
1:16:54
business it's a question that you can answer only
1:16:55
answer only
1:16:55
answer only if you have information about a business
1:16:57
if you have information about a business
1:16:57
if you have information about a business for example what services do you offer
1:16:59
for example what services do you offer
1:16:59
for example what services do you offer what is your address and so on
1:17:03
what is your address and so on
1:17:03
what is your address and so on so after we recognize this
1:17:06
so after we recognize this
1:17:06
so after we recognize this um after we recognize this object we
1:17:09
um after we recognize this object we
1:17:09
um after we recognize this object we know that this is the right time
1:17:11
know that this is the right time
1:17:11
know that this is the right time to take the question and
1:17:14
to take the question and
1:17:14
to take the question and redirect it to the neural model that
1:17:16
redirect it to the neural model that
1:17:16
redirect it to the neural model that will help us to find
1:17:18
will help us to find
1:17:18
will help us to find the answer from the text the
1:17:21
the answer from the text the
1:17:21
the answer from the text the business of the owner wrote in order to
1:17:23
business of the owner wrote in order to
1:17:23
business of the owner wrote in order to describe his business
1:17:26
so so by now
1:17:30
so so by now
1:17:30
so so by now what we have is a dialog manager that
1:17:32
what we have is a dialog manager that
1:17:32
what we have is a dialog manager that can handle conversation with no
1:17:34
can handle conversation with no
1:17:34
can handle conversation with no organization context
1:17:35
organization context
1:17:35
organization context and now we know how to recognize the
1:17:37
and now we know how to recognize the
1:17:37
and now we know how to recognize the questions that need
1:17:39
questions that need
1:17:39
questions that need the organization contacts and what we
1:17:41
the organization contacts and what we
1:17:41
the organization contacts and what we need
1:17:42
need
1:17:42
need to do is to find the answer from the
1:17:45
to do is to find the answer from the
1:17:45
to do is to find the answer from the text describing the business
1:17:47
text describing the business
1:17:47
text describing the business so basically you can go here in many
1:17:50
so basically you can go here in many
1:17:50
so basically you can go here in many ways you can
1:17:51
ways you can
1:17:51
ways you can you can train a seek to seek neural
1:17:54
you can train a seek to seek neural
1:17:54
you can train a seek to seek neural model
1:17:55
model
1:17:55
model you can um
1:17:58
you can um
1:17:58
you can um you can truly go in in many ways with
1:18:00
you can truly go in in many ways with
1:18:00
you can truly go in in many ways with prime anyways i'm going to describe
1:18:02
prime anyways i'm going to describe
1:18:02
prime anyways i'm going to describe one which was the most successful
1:18:05
one which was the most successful
1:18:05
one which was the most successful and obviously if you find
1:18:09
and obviously if you find
1:18:09
and obviously if you find find it interesting to want to know
1:18:10
find it interesting to want to know
1:18:10
find it interesting to want to know about the other options you can
1:18:12
about the other options you can
1:18:12
about the other options you can contact me through linkedin or through
1:18:14
contact me through linkedin or through
1:18:14
contact me through linkedin or through versebot.com
1:18:15
versebot.com
1:18:15
versebot.com so the solution we chose is extractive
1:18:18
so the solution we chose is extractive
1:18:18
so the solution we chose is extractive question answering which is a
1:18:21
question answering which is a
1:18:21
question answering which is a subfield subfield of question answering
1:18:24
subfield subfield of question answering
1:18:24
subfield subfield of question answering which is the subfield of nlp so what are
1:18:27
which is the subfield of nlp so what are
1:18:27
which is the subfield of nlp so what are you doing
1:18:28
you doing
1:18:28
you doing extractive question answering basically
1:18:29
extractive question answering basically
1:18:29
extractive question answering basically you have input which is a
1:18:31
you have input which is a
1:18:31
you have input which is a text and a question and what you want to
1:18:33
text and a question and what you want to
1:18:33
text and a question and what you want to do is to find
1:18:34
do is to find
1:18:34
do is to find the most promising span in this text
1:18:37
the most promising span in this text
1:18:38
the most promising span in this text it will be an answer for this question
1:18:42
it will be an answer for this question
1:18:42
it will be an answer for this question so there are actually plenty data sets
1:18:45
so there are actually plenty data sets
1:18:45
so there are actually plenty data sets that
1:18:46
that
1:18:46
that um constructed just for extractive
1:18:49
um constructed just for extractive
1:18:49
um constructed just for extractive question answering so
1:18:52
question answering so
1:18:52
question answering so we tried many of them and
1:18:55
we tried many of them and
1:18:55
we tried many of them and at the very beginning beginning we
1:18:57
at the very beginning beginning we
1:18:57
at the very beginning beginning we started to notice
1:18:58
started to notice
1:18:58
started to notice that the ones trained on wikipedia
1:19:02
that the ones trained on wikipedia
1:19:02
that the ones trained on wikipedia gave us the best accuracy
1:19:07
gave us the best accuracy
1:19:07
gave us the best accuracy while trying to predict the answers and
1:19:10
while trying to predict the answers and
1:19:10
while trying to predict the answers and at a very very early stage we felt like
1:19:13
at a very very early stage we felt like
1:19:13
at a very very early stage we felt like that the fact that wikipedia contains
1:19:15
that the fact that wikipedia contains
1:19:15
that the fact that wikipedia contains information about
1:19:17
information about
1:19:17
information about so many businesses and completely
1:19:19
so many businesses and completely
1:19:19
so many businesses and completely different domains
1:19:21
different domains
1:19:21
different domains was very good for our for our problem
1:19:24
was very good for our for our problem
1:19:24
was very good for our for our problem because
1:19:25
because
1:19:25
because basically we can't really know what
1:19:27
basically we can't really know what
1:19:27
basically we can't really know what business will use our service
1:19:28
business will use our service
1:19:28
business will use our service and we wanted it to be flexible and
1:19:31
and we wanted it to be flexible and
1:19:31
and we wanted it to be flexible and usable for
1:19:32
usable for
1:19:32
usable for businesses coming from completely
1:19:34
businesses coming from completely
1:19:34
businesses coming from completely different domains
1:19:36
different domains
1:19:36
different domains so after trying many
1:19:40
so after trying many
1:19:40
so after trying many data set trend on wikipedia we found
1:19:42
data set trend on wikipedia we found
1:19:42
data set trend on wikipedia we found that squad gave us the best results on a
1:19:45
that squad gave us the best results on a
1:19:45
that squad gave us the best results on a small data set of um real business
1:19:48
small data set of um real business
1:19:48
small data set of um real business description
1:19:50
description
1:19:50
description we we used for evaluation
1:19:54
we we used for evaluation
1:19:54
we we used for evaluation and what we
1:19:57
and what we
1:19:57
and what we basically had to do after we had the
1:19:59
basically had to do after we had the
1:19:59
basically had to do after we had the data set of questions
1:20:01
data set of questions
1:20:01
data set of questions um texts and answers inside of stacks is
1:20:03
um texts and answers inside of stacks is
1:20:04
um texts and answers inside of stacks is to train models and see
1:20:05
to train models and see
1:20:06
to train models and see what what will be the best model to go
1:20:08
what what will be the best model to go
1:20:08
what what will be the best model to go with in order to find answers
1:20:10
with in order to find answers
1:20:10
with in order to find answers in our text so what we did we took the
1:20:16
in our text so what we did we took the
1:20:16
in our text so what we did we took the practically almost every
1:20:18
practically almost every
1:20:18
practically almost every state-of-the-art
1:20:19
state-of-the-art
1:20:19
state-of-the-art model neural model you can find you can
1:20:21
model neural model you can find you can
1:20:21
model neural model you can find you can find today
1:20:22
find today
1:20:22
find today most of them are um transformer
1:20:25
most of them are um transformer
1:20:25
most of them are um transformer neural models just because they had very
1:20:28
neural models just because they had very
1:20:28
neural models just because they had very good results
1:20:29
good results
1:20:29
good results and just to show you
1:20:32
and just to show you
1:20:32
and just to show you one of them for example span that has f1
1:20:35
one of them for example span that has f1
1:20:35
one of them for example span that has f1 of
1:20:36
of
1:20:36
of 91.9 which means that
1:20:39
91.9 which means that
1:20:40
91.9 which means that out of the 10 000 questions in the um
1:20:43
out of the 10 000 questions in the um
1:20:43
out of the 10 000 questions in the um squad test
1:20:44
squad test
1:20:44
squad test set um it succeeded
1:20:47
set um it succeeded
1:20:47
set um it succeeded to find the area of the right answer
1:20:51
to find the area of the right answer
1:20:51
to find the area of the right answer in 90 of the cases
1:20:54
in 90 of the cases
1:20:54
in 90 of the cases and then we had to think about another
1:20:58
and then we had to think about another
1:20:58
and then we had to think about another um another part of using those models
1:21:01
um another part of using those models
1:21:01
um another part of using those models which is
1:21:03
which is
1:21:03
which is the production side of it and if you
1:21:05
the production side of it and if you
1:21:05
the production side of it and if you want to use any of those model and
1:21:07
want to use any of those model and
1:21:07
want to use any of those model and production you need
1:21:08
production you need
1:21:08
production you need to actually deploy it on a machine and
1:21:11
to actually deploy it on a machine and
1:21:11
to actually deploy it on a machine and we wanted to deploy it on a cpu
1:21:13
we wanted to deploy it on a cpu
1:21:13
we wanted to deploy it on a cpu we could have deployed on gpus etc but
1:21:16
we could have deployed on gpus etc but
1:21:16
we could have deployed on gpus etc but we felt like it's
1:21:17
we felt like it's
1:21:17
we felt like it's it's a bit of an overkill try to um use
1:21:20
it's a bit of an overkill try to um use
1:21:20
it's a bit of an overkill try to um use the best one
1:21:20
the best one
1:21:20
the best one the model that will work the best on um
1:21:24
the model that will work the best on um
1:21:24
the model that will work the best on um cpu and will still give good results so
1:21:27
cpu and will still give good results so
1:21:28
cpu and will still give good results so um as you can see here you can see the
1:21:32
um as you can see here you can see the
1:21:32
um as you can see here you can see the sizes and the speeds
1:21:33
sizes and the speeds
1:21:33
sizes and the speeds of those models and we
1:21:36
of those models and we
1:21:36
of those models and we eventually we chose um a
1:21:40
eventually we chose um a
1:21:40
eventually we chose um a an assembly of an assemble of
1:21:43
an assembly of an assemble of
1:21:43
an assembly of an assemble of models that part of them were trained on
1:21:46
models that part of them were trained on
1:21:46
models that part of them were trained on squad one part of them were trained on
1:21:48
squad one part of them were trained on
1:21:48
squad one part of them were trained on squad two
1:21:49
squad two
1:21:49
squad two and just it's worth to mention that
1:21:51
and just it's worth to mention that
1:21:51
and just it's worth to mention that squad two also
1:21:53
squad two also
1:21:53
squad two also introduced the notion of um
1:21:56
introduced the notion of um
1:21:56
introduced the notion of um question questions with with no answers
1:21:58
question questions with with no answers
1:21:58
question questions with with no answers so
1:21:59
so
1:21:59
so it's actually desirable in our domain
1:22:01
it's actually desirable in our domain
1:22:01
it's actually desirable in our domain sometimes you say okay there's no
1:22:03
sometimes you say okay there's no
1:22:03
sometimes you say okay there's no answer in this text so then we'll want
1:22:06
answer in this text so then we'll want
1:22:06
answer in this text so then we'll want to give a suitable response instead
1:22:09
to give a suitable response instead
1:22:10
to give a suitable response instead of giving that answer
1:22:14
of giving that answer
1:22:14
of giving that answer but then we have to face a new challenge
1:22:17
but then we have to face a new challenge
1:22:17
but then we have to face a new challenge which is squad data set is different
1:22:20
which is squad data set is different
1:22:20
which is squad data set is different than our domain
1:22:21
than our domain
1:22:21
than our domain in actually few ways so the first way
1:22:24
in actually few ways so the first way
1:22:24
in actually few ways so the first way the attacks in squad are wikipedia texts
1:22:28
the attacks in squad are wikipedia texts
1:22:28
the attacks in squad are wikipedia texts but when
1:22:29
but when
1:22:29
but when someone describing his business is not
1:22:31
someone describing his business is not
1:22:31
someone describing his business is not necessarily using the same
1:22:33
necessarily using the same
1:22:33
necessarily using the same language as wikipedia and
1:22:37
language as wikipedia and
1:22:37
language as wikipedia and as you probably know if you trained um
1:22:40
as you probably know if you trained um
1:22:40
as you probably know if you trained um neural models the more the the actual
1:22:43
neural models the more the the actual
1:22:43
neural models the more the the actual data using the small one
1:22:46
data using the small one
1:22:46
data using the small one similar to the data you trended them on
1:22:49
similar to the data you trended them on
1:22:49
similar to the data you trended them on then the models were more accurately
1:22:53
then the models were more accurately
1:22:53
then the models were more accurately so it was very important for us to find
1:22:55
so it was very important for us to find
1:22:55
so it was very important for us to find a way
1:22:56
a way
1:22:56
a way to get our data to look a bit more like
1:23:00
to get our data to look a bit more like
1:23:00
to get our data to look a bit more like the data or that our models were trained
1:23:03
the data or that our models were trained
1:23:03
the data or that our models were trained on
1:23:04
on
1:23:04
on so we actually
1:23:07
so we actually
1:23:07
so we actually use two different ways to to
1:23:10
use two different ways to to
1:23:10
use two different ways to to deal with it the first one is domain
1:23:13
deal with it the first one is domain
1:23:13
deal with it the first one is domain shift we try to change the
1:23:16
shift we try to change the
1:23:16
shift we try to change the the questions a bit to look a bit more
1:23:18
the questions a bit to look a bit more
1:23:18
the questions a bit to look a bit more like
1:23:19
like
1:23:19
like the the questions of squad
1:23:24
the the questions of squad
1:23:24
the the questions of squad and the other solution is to try and
1:23:27
and the other solution is to try and
1:23:27
and the other solution is to try and make the users of those models to
1:23:31
make the users of those models to
1:23:31
make the users of those models to understand better how the model work and
1:23:34
understand better how the model work and
1:23:34
understand better how the model work and what will make
1:23:35
what will make
1:23:35
what will make the the
1:23:38
mod the checkbox to use the the model in
1:23:41
mod the checkbox to use the the model in
1:23:42
mod the checkbox to use the the model in the best
1:23:42
the best
1:23:42
the best way possible and
1:23:46
way possible and
1:23:46
way possible and okay so let's move to the next to the
1:23:49
okay so let's move to the next to the
1:23:49
okay so let's move to the next to the first solution so domain shift
1:23:53
first solution so domain shift
1:23:53
first solution so domain shift so let's look a bit at how questions
1:23:56
so let's look a bit at how questions
1:23:56
so let's look a bit at how questions come in squad look like so for example
1:23:59
come in squad look like so for example
1:23:59
come in squad look like so for example the first question here is you can see
1:24:01
the first question here is you can see
1:24:01
the first question here is you can see from the wikipedia value of abc
1:24:04
from the wikipedia value of abc
1:24:04
from the wikipedia value of abc how many affiliated stations does abcd
1:24:06
how many affiliated stations does abcd
1:24:06
how many affiliated stations does abcd currently have
1:24:08
currently have
1:24:08
currently have so the subject of this question is
1:24:10
so the subject of this question is
1:24:10
so the subject of this question is stated
1:24:11
stated
1:24:11
stated explicitly but not necessarily
1:24:15
explicitly but not necessarily
1:24:15
explicitly but not necessarily it will be that way when when someone
1:24:17
it will be that way when when someone
1:24:17
it will be that way when when someone asks questions
1:24:18
asks questions
1:24:18
asks questions in our chat but for example if someone
1:24:20
in our chat but for example if someone
1:24:20
in our chat but for example if someone asks will you ship it to me
1:24:22
asks will you ship it to me
1:24:22
asks will you ship it to me if i buy it what is the subject who is
1:24:25
if i buy it what is the subject who is
1:24:25
if i buy it what is the subject who is you
1:24:25
you
1:24:25
you and who is me and
1:24:29
and who is me and
1:24:29
and who is me and more important than that the grammatical
1:24:32
more important than that the grammatical
1:24:32
more important than that the grammatical person
1:24:32
person
1:24:32
person is completely different and for example
1:24:35
is completely different and for example
1:24:35
is completely different and for example the word
1:24:36
the word
1:24:36
the word the word me will never be in a
1:24:39
the word me will never be in a
1:24:39
the word me will never be in a text like wikipedia but in our time in
1:24:42
text like wikipedia but in our time in
1:24:42
text like wikipedia but in our time in our type of texas
1:24:43
our type of texas
1:24:43
our type of texas it can actually appear quite often
1:24:47
it can actually appear quite often
1:24:47
it can actually appear quite often so what we did in order to solve it
1:24:51
so what we did in order to solve it
1:24:51
so what we did in order to solve it so basically what we
1:24:54
so basically what we
1:24:54
so basically what we we first tried to um
1:24:58
we first tried to um
1:24:58
we first tried to um train a simple neural model to change
1:25:01
train a simple neural model to change
1:25:01
train a simple neural model to change the questions
1:25:01
the questions
1:25:02
the questions to make them look more like squad and it
1:25:05
to make them look more like squad and it
1:25:05
to make them look more like squad and it didn't work
1:25:06
didn't work
1:25:06
didn't work good enough and sometimes the mistakes
1:25:09
good enough and sometimes the mistakes
1:25:09
good enough and sometimes the mistakes they did make it
1:25:10
they did make it
1:25:10
they did make it not possible to actually use so we
1:25:12
not possible to actually use so we
1:25:12
not possible to actually use so we actually
1:25:13
actually
1:25:13
actually went with more um traditional way
1:25:16
went with more um traditional way
1:25:16
went with more um traditional way so what we did we actually created set
1:25:19
so what we did we actually created set
1:25:19
so what we did we actually created set of rules of how to change
1:25:21
of rules of how to change
1:25:21
of rules of how to change questions for example questions like
1:25:24
questions for example questions like
1:25:24
questions for example questions like can you send it to me we change it can
1:25:28
can you send it to me we change it can
1:25:28
can you send it to me we change it can org send it to you and org will be
1:25:31
org send it to you and org will be
1:25:31
org send it to you and org will be whatever organization it is
1:25:33
whatever organization it is
1:25:33
whatever organization it is so this is the uh so now go over a real
1:25:36
so this is the uh so now go over a real
1:25:36
so this is the uh so now go over a real example from
1:25:37
example from
1:25:37
example from actually a real question someone asked
1:25:39
actually a real question someone asked
1:25:39
actually a real question someone asked archer about in our website
1:25:41
archer about in our website
1:25:42
archer about in our website and it goes this way so the person asked
1:25:45
and it goes this way so the person asked
1:25:45
and it goes this way so the person asked what can you do for me
1:25:47
what can you do for me
1:25:47
what can you do for me and it's very interesting here because
1:25:51
and it's very interesting here because
1:25:51
and it's very interesting here because every time it's written in our text the
1:25:53
every time it's written in our text the
1:25:53
every time it's written in our text the word you
1:25:54
word you
1:25:54
word you will refer to the customers but here
1:25:58
will refer to the customers but here
1:25:58
will refer to the customers but here the word you refers to us
1:26:01
the word you refers to us
1:26:01
the word you refers to us so obviously the answer was something
1:26:04
so obviously the answer was something
1:26:04
so obviously the answer was something related to what the
1:26:07
related to what the
1:26:08
related to what the customer should do and not what we can
1:26:09
customer should do and not what we can
1:26:10
customer should do and not what we can do for it so
1:26:11
do for it so
1:26:11
do for it so then when we change the question using
1:26:13
then when we change the question using
1:26:13
then when we change the question using those rules
1:26:14
those rules
1:26:14
those rules to what converse would do for you the
1:26:17
to what converse would do for you the
1:26:17
to what converse would do for you the answer
1:26:18
answer
1:26:18
answer was um the right answer accordingly
1:26:22
was um the right answer accordingly
1:26:22
was um the right answer accordingly so the
1:26:25
so the
1:26:25
so the pipeline for actually doing it is to
1:26:28
pipeline for actually doing it is to
1:26:28
pipeline for actually doing it is to break the
1:26:29
break the
1:26:29
break the the questions into words to sentences
1:26:32
the questions into words to sentences
1:26:32
the questions into words to sentences first and then two words
1:26:33
first and then two words
1:26:34
first and then two words and then to find out the words you want
1:26:35
and then to find out the words you want
1:26:35
and then to find out the words you want to replace and see if they suit
1:26:37
to replace and see if they suit
1:26:38
to replace and see if they suit one of the rules you predefined and then
1:26:40
one of the rules you predefined and then
1:26:40
one of the rules you predefined and then if this
1:26:41
if this
1:26:41
if this if they fit one of those rules just
1:26:42
if they fit one of those rules just
1:26:42
if they fit one of those rules just replace them and it raised
1:26:44
replace them and it raised
1:26:44
replace them and it raised uh accuracy dramatically
1:26:47
uh accuracy dramatically
1:26:47
uh accuracy dramatically because now our questions look a bit
1:26:49
because now our questions look a bit
1:26:49
because now our questions look a bit more the question of squad and they're
1:26:51
more the question of squad and they're
1:26:51
more the question of squad and they're more related to
1:26:53
more related to
1:26:53
more related to the they use the same grammatical person
1:26:55
the they use the same grammatical person
1:26:55
the they use the same grammatical person as the text
1:26:56
as the text
1:26:56
as the text we were using so
1:27:00
the next solution the solution we're
1:27:04
the next solution the solution we're
1:27:04
the next solution the solution we're actually
1:27:05
actually
1:27:05
actually uh very excited about and i'm very
1:27:08
uh very excited about and i'm very
1:27:08
uh very excited about and i'm very excited to tell you about so they use
1:27:10
excited to tell you about so they use
1:27:10
excited to tell you about so they use the model integration
1:27:12
the model integration
1:27:12
the model integration we at the beginning when we
1:27:15
we at the beginning when we
1:27:15
we at the beginning when we tried to go with this project we thought
1:27:18
tried to go with this project we thought
1:27:18
tried to go with this project we thought that someone will just
1:27:20
that someone will just
1:27:20
that someone will just input a system within stacks and then
1:27:23
input a system within stacks and then
1:27:23
input a system within stacks and then just hope that the chatbot will be good
1:27:26
just hope that the chatbot will be good
1:27:26
just hope that the chatbot will be good but then we realize that people change
1:27:28
but then we realize that people change
1:27:28
but then we realize that people change it quite often
1:27:30
it quite often
1:27:30
it quite often and the reason people change it quite
1:27:32
and the reason people change it quite
1:27:32
and the reason people change it quite often is because they're starting to
1:27:34
often is because they're starting to
1:27:34
often is because they're starting to understand how the model work
1:27:36
understand how the model work
1:27:36
understand how the model work and they're changing the text
1:27:38
and they're changing the text
1:27:38
and they're changing the text accordingly
1:27:39
accordingly
1:27:39
accordingly and the reason that they're changing the
1:27:43
and the reason that they're changing the
1:27:44
and the reason that they're changing the text is because they truly understand
1:27:47
text is because they truly understand
1:27:47
text is because they truly understand some of some parts of the workings of
1:27:50
some of some parts of the workings of
1:27:50
some of some parts of the workings of those models
1:27:52
those models
1:27:52
those models and we find this concept quite
1:27:54
and we find this concept quite
1:27:54
and we find this concept quite interesting
1:27:55
interesting
1:27:55
interesting because usually we're thinking we're
1:27:58
because usually we're thinking we're
1:27:58
because usually we're thinking we're thinking about
1:27:59
thinking about
1:27:59
thinking about neural models something that is running
1:28:01
neural models something that is running
1:28:02
neural models something that is running on a server
1:28:03
on a server
1:28:03
on a server and the user doesn't really get to touch
1:28:06
and the user doesn't really get to touch
1:28:06
and the user doesn't really get to touch or
1:28:07
or
1:28:07
or um to experience and here we had
1:28:10
um to experience and here we had
1:28:10
um to experience and here we had sort of an opportunity to give users to
1:28:12
sort of an opportunity to give users to
1:28:12
sort of an opportunity to give users to experience um
1:28:15
experience um
1:28:15
experience um with a with their own hands um
1:28:18
with a with their own hands um
1:28:18
with a with their own hands um how neural models feel like and
1:28:21
how neural models feel like and
1:28:21
how neural models feel like and we found it to be very helpful
1:28:24
we found it to be very helpful
1:28:24
we found it to be very helpful so the first way of of helping the uh
1:28:28
so the first way of of helping the uh
1:28:28
so the first way of of helping the uh users to get a text to be more similar
1:28:31
users to get a text to be more similar
1:28:31
users to get a text to be more similar to our
1:28:32
to our
1:28:32
to our um training data was interaction with
1:28:35
um training data was interaction with
1:28:35
um training data was interaction with the chatbot
1:28:35
the chatbot
1:28:36
the chatbot then expert advice is a set of tips we
1:28:38
then expert advice is a set of tips we
1:28:38
then expert advice is a set of tips we supply
1:28:39
supply
1:28:39
supply our users to actually
1:28:42
our users to actually
1:28:42
our users to actually make them construct that text in a way
1:28:45
make them construct that text in a way
1:28:45
make them construct that text in a way that will be
1:28:45
that will be
1:28:45
that will be more easy for the models to work with
1:28:48
more easy for the models to work with
1:28:48
more easy for the models to work with and then retrospective which is um the
1:28:51
and then retrospective which is um the
1:28:51
and then retrospective which is um the history of the
1:28:52
history of the
1:28:52
history of the chat that the clients had with a
1:28:57
chat that the clients had with a
1:28:57
chat that the clients had with a wii with a chatbot so they can actually
1:28:59
wii with a chatbot so they can actually
1:28:59
wii with a chatbot so they can actually see what worked and what didn't
1:29:03
see what worked and what didn't
1:29:03
see what worked and what didn't and this is
1:29:07
and this is
1:29:07
and this is actually it brings me to the last point
1:29:09
actually it brings me to the last point
1:29:09
actually it brings me to the last point i'm going to talk about today
1:29:11
i'm going to talk about today
1:29:11
i'm going to talk about today and it's the concept of model
1:29:15
and it's the concept of model
1:29:15
and it's the concept of model programming by
1:29:16
programming by
1:29:16
programming by natural language you all probably heard
1:29:18
natural language you all probably heard
1:29:18
natural language you all probably heard about gpt3
1:29:19
about gpt3
1:29:20
about gpt3 and how amazing it works just by feeding
1:29:23
and how amazing it works just by feeding
1:29:23
and how amazing it works just by feeding the um fitting a
1:29:26
the um fitting a
1:29:26
the um fitting a prompt to define the problem give a few
1:29:29
prompt to define the problem give a few
1:29:29
prompt to define the problem give a few examples and then gpt3
1:29:31
examples and then gpt3
1:29:31
examples and then gpt3 does its medi magic and give you the
1:29:33
does its medi magic and give you the
1:29:33
does its medi magic and give you the right answer
1:29:34
right answer
1:29:34
right answer and we actually found the same sort of
1:29:37
and we actually found the same sort of
1:29:37
and we actually found the same sort of process
1:29:38
process
1:29:38
process right here so our customers giving
1:29:41
right here so our customers giving
1:29:41
right here so our customers giving the model um basically the knowledge and
1:29:45
the model um basically the knowledge and
1:29:45
the model um basically the knowledge and you can you can look at this knowledge
1:29:47
you can you can look at this knowledge
1:29:47
you can you can look at this knowledge of
1:29:48
of
1:29:48
of this text as input to the neural model
1:29:52
this text as input to the neural model
1:29:52
this text as input to the neural model but you can you can actually look at it
1:29:54
but you can you can actually look at it
1:29:54
but you can you can actually look at it from a different perspective because
1:29:56
from a different perspective because
1:29:56
from a different perspective because this text
1:29:57
this text
1:29:57
this text actually being transferred into a set of
1:30:03
actually being transferred into a set of
1:30:03
actually being transferred into a set of set of vectors representing those words
1:30:05
set of vectors representing those words
1:30:05
set of vectors representing those words and those vectors
1:30:06
and those vectors
1:30:06
and those vectors basically were interacting with the
1:30:09
basically were interacting with the
1:30:09
basically were interacting with the weights of the
1:30:09
weights of the
1:30:09
weights of the neural network so you can look at the
1:30:13
neural network so you can look at the
1:30:13
neural network so you can look at the process of what they
1:30:14
process of what they
1:30:14
process of what they what our users do while they're changing
1:30:17
what our users do while they're changing
1:30:17
what our users do while they're changing the text
1:30:19
the text
1:30:19
the text is not only changing the input for the
1:30:21
is not only changing the input for the
1:30:21
is not only changing the input for the model but actually
1:30:22
model but actually
1:30:22
model but actually changing the inner workings of the
1:30:24
changing the inner workings of the
1:30:24
changing the inner workings of the models
1:30:26
models
1:30:26
models basically just but just by changing the
1:30:30
basically just but just by changing the
1:30:30
basically just but just by changing the um the natural language um
1:30:33
um the natural language um
1:30:33
um the natural language um knowledge that they giving to this model
1:30:35
knowledge that they giving to this model
1:30:36
knowledge that they giving to this model because
1:30:37
because
1:30:37
because as you can see they for from every
1:30:40
as you can see they for from every
1:30:40
as you can see they for from every business perspective after they find a
1:30:43
business perspective after they find a
1:30:43
business perspective after they find a good tax representative business the
1:30:45
good tax representative business the
1:30:45
good tax representative business the only thing that changes is the question
1:30:47
only thing that changes is the question
1:30:47
only thing that changes is the question and the knowledge stays the same so it's
1:30:50
and the knowledge stays the same so it's
1:30:50
and the knowledge stays the same so it's practically
1:30:50
practically
1:30:50
practically part of the practically it's part of the
1:30:54
part of the practically it's part of the
1:30:54
part of the practically it's part of the neural network and we find this concept
1:30:58
neural network and we find this concept
1:30:58
neural network and we find this concept of
1:30:59
of
1:30:59
of programming the neural network and its
1:31:01
programming the neural network and its
1:31:01
programming the neural network and its weights
1:31:03
weights
1:31:03
weights using natural language quite exciting
1:31:06
using natural language quite exciting
1:31:06
using natural language quite exciting concept and if you also feel
1:31:10
concept and if you also feel
1:31:10
concept and if you also feel you excited about it can interact us
1:31:12
you excited about it can interact us
1:31:12
you excited about it can interact us then we have many
1:31:14
then we have many
1:31:14
then we have many more things to say about it so you can
1:31:16
more things to say about it so you can
1:31:16
more things to say about it so you can find us actually at versebot.com
1:31:18
find us actually at versebot.com
1:31:18
find us actually at versebot.com you can create your own chatbot now it's
1:31:21
you can create your own chatbot now it's
1:31:21
you can create your own chatbot now it's not
1:31:21
not
1:31:22
not a it's not a productive way
1:31:25
a it's not a productive way
1:31:25
a it's not a productive way product we actually coming up out with
1:31:28
product we actually coming up out with
1:31:28
product we actually coming up out with it just um experimental product and we
1:31:32
it just um experimental product and we
1:31:32
it just um experimental product and we highly invite you to go to vespa.com and
1:31:35
highly invite you to go to vespa.com and
1:31:35
highly invite you to go to vespa.com and try and use it and see
1:31:36
try and use it and see
1:31:36
try and use it and see how it works and hopefully you'll find
1:31:39
how it works and hopefully you'll find
1:31:40
how it works and hopefully you'll find it as exciting as we find it
1:31:45
it as exciting as we find it
1:31:45
it as exciting as we find it thank you great thank you very much
1:31:48
thank you great thank you very much
1:31:48
thank you great thank you very much elron
1:31:49
elron
1:31:49
elron well in the meanwhile we already see in
1:31:51
well in the meanwhile we already see in
1:31:51
well in the meanwhile we already see in our comments that there are people
1:31:52
our comments that there are people
1:31:52
our comments that there are people already trying out
1:31:53
already trying out
1:31:54
already trying out at the moment uh the verse but very
1:31:56
at the moment uh the verse but very
1:31:56
at the moment uh the verse but very interesting product
1:31:57
interesting product
1:31:58
interesting product and approach of building a chat bot it's
1:32:00
and approach of building a chat bot it's
1:32:00
and approach of building a chat bot it's something totally different than the
1:32:01
something totally different than the
1:32:02
something totally different than the most
1:32:02
most
1:32:02
most frameworks you see so really nice
1:32:06
frameworks you see so really nice
1:32:06
frameworks you see so really nice one question we had um is around support
1:32:09
one question we had um is around support
1:32:09
one question we had um is around support of
1:32:10
of
1:32:10
of languages
1:32:13
i'm sure while building a chat bots in
1:32:15
i'm sure while building a chat bots in
1:32:15
i'm sure while building a chat bots in one specific language
1:32:17
one specific language
1:32:17
one specific language okay but making it available in multiple
1:32:20
okay but making it available in multiple
1:32:20
okay but making it available in multiple languages how do you handle that
1:32:23
languages how do you handle that
1:32:23
languages how do you handle that actually it's quite a sad problem of
1:32:26
actually it's quite a sad problem of
1:32:26
actually it's quite a sad problem of nlp these days that most of the work is
1:32:29
nlp these days that most of the work is
1:32:29
nlp these days that most of the work is being done on english
1:32:31
being done on english
1:32:31
being done on english and i'm i'm actually a
1:32:34
and i'm i'm actually a
1:32:34
and i'm i'm actually a hebrew native speaker and it's quite sad
1:32:36
hebrew native speaker and it's quite sad
1:32:36
hebrew native speaker and it's quite sad that like
1:32:37
that like
1:32:37
that like we're not there yet in lower source
1:32:40
we're not there yet in lower source
1:32:40
we're not there yet in lower source languages
1:32:41
languages
1:32:41
languages and hopefully many people actually work
1:32:45
and hopefully many people actually work
1:32:45
and hopefully many people actually work in or work on bringing those
1:32:48
in or work on bringing those
1:32:48
in or work on bringing those amazing new models to um
1:32:51
amazing new models to um
1:32:51
amazing new models to um low resource languages and it's still in
1:32:54
low resource languages and it's still in
1:32:54
low resource languages and it's still in progress and
1:32:55
progress and
1:32:55
progress and would love to see more progress in it
1:32:58
would love to see more progress in it
1:32:58
would love to see more progress in it and
1:32:59
and
1:32:59
and combine many languages as much as
1:33:01
combine many languages as much as
1:33:01
combine many languages as much as possible
1:33:02
possible
1:33:02
possible okay now
1:33:06
okay now
1:33:06
okay now if you're writing or even when you're
1:33:08
if you're writing or even when you're
1:33:08
if you're writing or even when you're speaking i'm not a native
1:33:10
speaking i'm not a native
1:33:10
speaking i'm not a native english speaker as you might hear um we
1:33:12
english speaker as you might hear um we
1:33:12
english speaker as you might hear um we make mistakes
1:33:14
make mistakes
1:33:14
make mistakes we make writing mistakes we you we speak
1:33:17
we make writing mistakes we you we speak
1:33:17
we make writing mistakes we you we speak out
1:33:17
out
1:33:17
out wrong things um how do you handle that
1:33:20
wrong things um how do you handle that
1:33:20
wrong things um how do you handle that and uh
1:33:20
and uh
1:33:20
and uh in your project for example so
1:33:24
in your project for example so
1:33:24
in your project for example so actually it's very good question because
1:33:27
actually it's very good question because
1:33:27
actually it's very good question because we
1:33:27
we
1:33:28
we tried many type of um
1:33:31
tried many type of um
1:33:31
tried many type of um grammatical mistakes and even um
1:33:35
grammatical mistakes and even um
1:33:35
grammatical mistakes and even um spelling mistakes and trying to see how
1:33:37
spelling mistakes and trying to see how
1:33:37
spelling mistakes and trying to see how our system deals with it
1:33:39
our system deals with it
1:33:39
our system deals with it and the very interesting thing we found
1:33:41
and the very interesting thing we found
1:33:41
and the very interesting thing we found is that
1:33:42
is that
1:33:42
is that models that are not trying to detect if
1:33:45
models that are not trying to detect if
1:33:45
models that are not trying to detect if they
1:33:46
they
1:33:46
they if the question is um answerable
1:33:50
if the question is um answerable
1:33:50
if the question is um answerable which means that it has an answer in the
1:33:52
which means that it has an answer in the
1:33:52
which means that it has an answer in the text they
1:33:53
text they
1:33:53
text they do better in um environment
1:33:57
do better in um environment
1:33:57
do better in um environment which things are not very clear so they
1:33:59
which things are not very clear so they
1:33:59
which things are not very clear so they actually can generalize better
1:34:01
actually can generalize better
1:34:01
actually can generalize better to poor language or to
1:34:06
to poor language or to
1:34:06
to poor language or to different domains than the training the
1:34:09
different domains than the training the
1:34:09
different domains than the training the training data
1:34:11
training data
1:34:12
training data okay cool now when you're training
1:34:15
okay cool now when you're training
1:34:15
okay cool now when you're training different
1:34:16
different
1:34:16
different models you'll have specific data sets
1:34:18
models you'll have specific data sets
1:34:18
models you'll have specific data sets that you're using
1:34:19
that you're using
1:34:19
that you're using um how do you manage a versioning of
1:34:21
um how do you manage a versioning of
1:34:21
um how do you manage a versioning of that
1:34:23
that
1:34:23
that to make sure you have to make a pro
1:34:24
to make sure you have to make a pro
1:34:24
to make sure you have to make a pro improvements of it for a b
1:34:26
improvements of it for a b
1:34:26
improvements of it for a b testing and so on so
1:34:29
testing and so on so
1:34:30
testing and so on so it was in terms of our data we tried to
1:34:33
it was in terms of our data we tried to
1:34:33
it was in terms of our data we tried to actually um keep records of what we did
1:34:37
actually um keep records of what we did
1:34:37
actually um keep records of what we did in every stage and
1:34:38
in every stage and
1:34:38
in every stage and we we just used a simple um
1:34:42
we we just used a simple um
1:34:42
we we just used a simple um git system to do it and we find it very
1:34:46
git system to do it and we find it very
1:34:46
git system to do it and we find it very useful and in terms of our academic data
1:34:50
useful and in terms of our academic data
1:34:50
useful and in terms of our academic data sets
1:34:51
sets
1:34:52
sets we had lots of insights about them that
1:34:54
we had lots of insights about them that
1:34:54
we had lots of insights about them that we also documented
1:34:56
we also documented
1:34:56
we also documented in order to get the most out of them
1:35:02
that's really cool i have a quick
1:35:04
that's really cool i have a quick
1:35:04
that's really cool i have a quick question for you alright um
1:35:06
question for you alright um
1:35:06
question for you alright um you might have mentioned that i feel but
1:35:07
you might have mentioned that i feel but
1:35:08
you might have mentioned that i feel but i just want to clarify so
1:35:09
i just want to clarify so
1:35:09
i just want to clarify so and you've added in the language models
1:35:12
and you've added in the language models
1:35:12
and you've added in the language models the knowledge base is kind of the core
1:35:14
the knowledge base is kind of the core
1:35:14
the knowledge base is kind of the core and it's the question that is going to
1:35:15
and it's the question that is going to
1:35:16
and it's the question that is going to kind of frequently change
1:35:17
kind of frequently change
1:35:18
kind of frequently change but what if someone just asks something
1:35:20
but what if someone just asks something
1:35:20
but what if someone just asks something completely out of context and we know
1:35:22
completely out of context and we know
1:35:22
completely out of context and we know that people do that with chat bots
1:35:25
that people do that with chat bots
1:35:25
that people do that with chat bots yes so um we handle it in
1:35:30
yes so um we handle it in
1:35:30
yes so um we handle it in like in two different levels so first in
1:35:32
like in two different levels so first in
1:35:32
like in two different levels so first in the level of the dialog manager
1:35:34
the level of the dialog manager
1:35:34
the level of the dialog manager we have a we have basically a data set
1:35:37
we have a we have basically a data set
1:35:38
we have a we have basically a data set of questions which
1:35:39
of questions which
1:35:39
of questions which are completely irrelevant chit-chat
1:35:42
are completely irrelevant chit-chat
1:35:42
are completely irrelevant chit-chat bad language or everything you can think
1:35:44
bad language or everything you can think
1:35:44
bad language or everything you can think about and it's supposed to detect this
1:35:46
about and it's supposed to detect this
1:35:46
about and it's supposed to detect this kind of language and give
1:35:48
kind of language and give
1:35:48
kind of language and give responses accordingly and
1:35:51
responses accordingly and
1:35:51
responses accordingly and the other part is when someone is asking
1:35:54
the other part is when someone is asking
1:35:54
the other part is when someone is asking questions that cannot be answered by the
1:35:56
questions that cannot be answered by the
1:35:56
questions that cannot be answered by the text
1:35:57
text
1:35:57
text then we our newer model actually um
1:36:00
then we our newer model actually um
1:36:00
then we our newer model actually um produce uh certainty probability that
1:36:04
produce uh certainty probability that
1:36:04
produce uh certainty probability that help us to know how
1:36:07
help us to know how
1:36:07
help us to know how accurate the question is and how um
1:36:10
accurate the question is and how um
1:36:10
accurate the question is and how um how much the question can be answered by
1:36:12
how much the question can be answered by
1:36:12
how much the question can be answered by the text
1:36:14
the text
1:36:14
the text very nice no that's that's interesting
1:36:16
very nice no that's that's interesting
1:36:16
very nice no that's that's interesting and yeah kind of thinking about how to
1:36:18
and yeah kind of thinking about how to
1:36:18
and yeah kind of thinking about how to handle
1:36:19
handle
1:36:19
handle all those uh it's almost like exceptions
1:36:21
all those uh it's almost like exceptions
1:36:22
all those uh it's almost like exceptions it feels like in code like how are you
1:36:23
it feels like in code like how are you
1:36:23
it feels like in code like how are you going to handle them when they hit the
1:36:25
going to handle them when they hit the
1:36:25
going to handle them when they hit the not the actual design of how you think
1:36:28
not the actual design of how you think
1:36:28
not the actual design of how you think people will use chat bots that's
1:36:29
people will use chat bots that's
1:36:29
people will use chat bots that's uh that's super interesting now in the
1:36:32
uh that's super interesting now in the
1:36:32
uh that's super interesting now in the parts we've seen
1:36:33
parts we've seen
1:36:33
parts we've seen already it's mostly you have a question
1:36:36
already it's mostly you have a question
1:36:36
already it's mostly you have a question and you give an answer back
1:36:38
and you give an answer back
1:36:38
and you give an answer back any plans to also add actions to it
1:36:42
any plans to also add actions to it
1:36:42
any plans to also add actions to it so the only actions we have now is to
1:36:44
so the only actions we have now is to
1:36:44
so the only actions we have now is to collect
1:36:45
collect
1:36:45
collect clients information when we feel like
1:36:48
clients information when we feel like
1:36:48
clients information when we feel like the body is not doing well or
1:36:50
the body is not doing well or
1:36:50
the body is not doing well or there is something the board can't
1:36:52
there is something the board can't
1:36:52
there is something the board can't really do so
1:36:54
really do so
1:36:54
really do so the actions we do there is to collect
1:36:56
the actions we do there is to collect
1:36:56
the actions we do there is to collect the client information
1:36:58
the client information
1:36:58
the client information and pass it on to the board owner
1:37:01
and pass it on to the board owner
1:37:01
and pass it on to the board owner but obviously we we thought about it and
1:37:04
but obviously we we thought about it and
1:37:04
but obviously we we thought about it and we're still thinking about
1:37:06
we're still thinking about
1:37:06
we're still thinking about how to combine new actions and new
1:37:08
how to combine new actions and new
1:37:08
how to combine new actions and new skills to this chatbot
1:37:10
skills to this chatbot
1:37:10
skills to this chatbot and at the same time we want it still to
1:37:13
and at the same time we want it still to
1:37:13
and at the same time we want it still to be
1:37:13
be
1:37:13
be generic and to suit many kind of
1:37:15
generic and to suit many kind of
1:37:15
generic and to suit many kind of businesses and
1:37:18
businesses and
1:37:18
businesses and you truly need to think about it it's
1:37:20
you truly need to think about it it's
1:37:20
you truly need to think about it it's not a it's not an easy
1:37:22
not a it's not an easy
1:37:22
not a it's not an easy problem okay cool elrond thank you very
1:37:26
problem okay cool elrond thank you very
1:37:26
problem okay cool elrond thank you very much for your time today thank you for
1:37:27
much for your time today thank you for
1:37:27
much for your time today thank you for having me
1:37:28
having me
1:37:28
having me all the all the best with your product
1:37:31
all the all the best with your product
1:37:31
all the all the best with your product versa bots
1:37:31
versa bots
1:37:31
versa bots we're looking forward to the next
1:37:33
we're looking forward to the next
1:37:33
we're looking forward to the next iterations of it what all the things
1:37:35
iterations of it what all the things
1:37:35
iterations of it what all the things that we'll do
1:37:36
that we'll do
1:37:36
that we'll do um so enjoy the rest of your evening for
1:37:40
um so enjoy the rest of your evening for
1:37:40
um so enjoy the rest of your evening for you it's still an evening if i'm correct
1:37:43
you it's still an evening if i'm correct
1:37:43
you it's still an evening if i'm correct um and we will invite our second
1:37:46
um and we will invite our second
1:37:46
um and we will invite our second or second or third guest for today
1:37:48
or second or third guest for today
1:37:48
or second or third guest for today already
1:37:50
already
1:37:50
already veret schwartz
1:37:57
and we'll have a little issue but that
1:37:59
and we'll have a little issue but that
1:37:59
and we'll have a little issue but that will be fixed
1:38:00
will be fixed
1:38:00
will be fixed very soon hi hi
1:38:05
we're we're not seeing you yet on the
1:38:07
we're we're not seeing you yet on the
1:38:07
we're we're not seeing you yet on the screen
1:38:10
great great how are you i'm good how are
1:38:13
great great how are you i'm good how are
1:38:13
great great how are you i'm good how are you
1:38:13
you
1:38:13
you i'm fine i'm fine we're very glad to
1:38:15
i'm fine i'm fine we're very glad to
1:38:15
i'm fine i'm fine we're very glad to have you also on board today for the
1:38:17
have you also on board today for the
1:38:17
have you also on board today for the october sessions
1:38:19
october sessions
1:38:19
october sessions um please introduce yourself to the
1:38:22
um please introduce yourself to the
1:38:22
um please introduce yourself to the public who are you
1:38:24
public who are you
1:38:24
public who are you um so i'm a post-doctoral researcher at
1:38:27
um so i'm a post-doctoral researcher at
1:38:27
um so i'm a post-doctoral researcher at the
1:38:28
the
1:38:28
the allen institute for ai and um
1:38:31
allen institute for ai and um
1:38:31
allen institute for ai and um in the also in the university of
1:38:33
in the also in the university of
1:38:33
in the also in the university of washington
1:38:35
washington
1:38:35
washington uh i'm working on nlp or
1:38:38
uh i'm working on nlp or
1:38:38
uh i'm working on nlp or more specifically recently working on
1:38:41
more specifically recently working on
1:38:41
more specifically recently working on teaching machines common sense reasoning
1:38:44
teaching machines common sense reasoning
1:38:44
teaching machines common sense reasoning okay cool can you tell us something
1:38:46
okay cool can you tell us something
1:38:46
okay cool can you tell us something short about
1:38:48
short about
1:38:48
short about the allen institute sure so
1:38:51
the allen institute sure so
1:38:51
the allen institute sure so it's a non-profit organization for
1:38:54
it's a non-profit organization for
1:38:54
it's a non-profit organization for doing ai for good mainly
1:38:57
doing ai for good mainly
1:38:58
doing ai for good mainly doing nlp but we also have a vision team
1:39:00
doing nlp but we also have a vision team
1:39:00
doing nlp but we also have a vision team and
1:39:01
and
1:39:01
and yeah doing a lot of very cool work
1:39:05
yeah doing a lot of very cool work
1:39:05
yeah doing a lot of very cool work so you're glad you can be there
1:39:08
cool you are what are you going to
1:39:11
cool you are what are you going to
1:39:11
cool you are what are you going to discuss with
1:39:11
discuss with
1:39:12
discuss with us today uh so i'm going to talk about
1:39:15
us today uh so i'm going to talk about
1:39:15
us today uh so i'm going to talk about the
1:39:16
the
1:39:16
the um the things that work or don't yet
1:39:18
um the things that work or don't yet
1:39:18
um the things that work or don't yet work in an
1:39:19
work in an
1:39:19
work in an lp today which is mostly
1:39:23
lp today which is mostly
1:39:23
lp today which is mostly deep learning based okay cool well with
1:39:26
deep learning based okay cool well with
1:39:26
deep learning based okay cool well with that
1:39:27
that
1:39:27
that i give you the platform and we are
1:39:30
i give you the platform and we are
1:39:30
i give you the platform and we are listening with open ears
1:39:36
okay can you see my screen
1:39:40
okay can you see my screen
1:39:40
okay can you see my screen just give it one moment and i'm sure we
1:39:43
just give it one moment and i'm sure we
1:39:44
just give it one moment and i'm sure we will
1:39:45
will
1:39:45
will um oh people are asking um what is the
1:39:49
um oh people are asking um what is the
1:39:50
um oh people are asking um what is the raccoon called the raccoon
1:39:51
raccoon called the raccoon
1:39:51
raccoon called the raccoon on next to the is it's called
1:39:54
on next to the is it's called
1:39:54
on next to the is it's called bit um it's in like wrong cameras
1:39:59
bit um it's in like wrong cameras
1:39:59
bit um it's in like wrong cameras um yeah it's a it's a bit of a a
1:40:02
um yeah it's a it's a bit of a a
1:40:02
um yeah it's a it's a bit of a a friendly mascot
1:40:03
friendly mascot
1:40:03
friendly mascot of this community which is a tree
1:40:06
of this community which is a tree
1:40:06
of this community which is a tree exactly and before we forget also
1:40:08
exactly and before we forget also
1:40:08
exactly and before we forget also uh please send out some tweets
1:40:11
uh please send out some tweets
1:40:11
uh please send out some tweets um if you go to globalai
1:40:14
um if you go to globalai
1:40:14
um if you go to globalai dot live slash photos link again
1:40:18
dot live slash photos link again
1:40:18
dot live slash photos link again win dash action it will be added in the
1:40:21
win dash action it will be added in the
1:40:21
win dash action it will be added in the chat also
1:40:21
chat also
1:40:22
chat also if you add a link to your tweet we will
1:40:24
if you add a link to your tweet we will
1:40:24
if you add a link to your tweet we will pick someone
1:40:25
pick someone
1:40:25
pick someone uh randomly and then you can win a
1:40:27
uh randomly and then you can win a
1:40:27
uh randomly and then you can win a voucher of fifty dollars for amazon so
1:40:30
voucher of fifty dollars for amazon so
1:40:30
voucher of fifty dollars for amazon so go ahead and try to win it
1:40:34
okay well i'm in a little prominent
1:40:36
okay well i'm in a little prominent
1:40:36
okay well i'm in a little prominent ferret could you try and restare your
1:40:38
ferret could you try and restare your
1:40:38
ferret could you try and restare your screen
1:40:38
screen
1:40:38
screen for us sure um
1:40:42
for us sure um
1:40:42
for us sure um let's see if we can this is uh i mean
1:40:45
let's see if we can this is uh i mean
1:40:45
let's see if we can this is uh i mean we're doing pretty well uh what are we
1:40:48
we're doing pretty well uh what are we
1:40:48
we're doing pretty well uh what are we like an hour and a half in and this has
1:40:50
like an hour and a half in and this has
1:40:50
like an hour and a half in and this has been the first technical glitch here we
1:40:51
been the first technical glitch here we
1:40:52
been the first technical glitch here we go
1:40:52
go
1:40:52
go is that it right so i i can see it hank
1:40:55
is that it right so i i can see it hank
1:40:55
is that it right so i i can see it hank are you able to do some magic
1:40:57
are you able to do some magic
1:40:57
are you able to do some magic and so that everyone else can see it
1:41:04
and so that everyone else can see it
1:41:04
and so that everyone else can see it yes there we go okay we all see it there
1:41:08
yes there we go okay we all see it there
1:41:08
yes there we go okay we all see it there we go
1:41:08
we go
1:41:08
we go uh please take it away great thank you
1:41:11
uh please take it away great thank you
1:41:11
uh please take it away great thank you okay so i'm going to talk about recent
1:41:13
okay so i'm going to talk about recent
1:41:14
okay so i'm going to talk about recent breakthroughs and uphill battles in
1:41:17
breakthroughs and uphill battles in
1:41:17
breakthroughs and uphill battles in um so if you get most of your uh
1:41:21
um so if you get most of your uh
1:41:21
um so if you get most of your uh ai updates from popular media then you
1:41:24
ai updates from popular media then you
1:41:24
ai updates from popular media then you might get the
1:41:25
might get the
1:41:25
might get the wrong idea that languages is already
1:41:27
wrong idea that languages is already
1:41:27
wrong idea that languages is already solved and
1:41:28
solved and
1:41:28
solved and that moreover uh robots are pretty
1:41:31
that moreover uh robots are pretty
1:41:31
that moreover uh robots are pretty dangerous
1:41:32
dangerous
1:41:32
dangerous and you would get that from headlines
1:41:34
and you would get that from headlines
1:41:34
and you would get that from headlines like uh google's birth
1:41:35
like uh google's birth
1:41:36
like uh google's birth understands language better than humans
1:41:38
understands language better than humans
1:41:38
understands language better than humans or
1:41:39
or
1:41:39
or um various paraphrases on ai beats
1:41:42
um various paraphrases on ai beats
1:41:42
um various paraphrases on ai beats humans
1:41:43
humans
1:41:43
humans and some famous stories like
1:41:46
and some famous stories like
1:41:46
and some famous stories like uh facebook had to shut down ai that
1:41:49
uh facebook had to shut down ai that
1:41:49
uh facebook had to shut down ai that invented its own language
1:41:51
invented its own language
1:41:51
invented its own language or open ai refusing to release its model
1:41:54
or open ai refusing to release its model
1:41:54
or open ai refusing to release its model claiming it's too dangerous to release
1:41:57
claiming it's too dangerous to release
1:41:57
claiming it's too dangerous to release there's also
1:41:58
there's also
1:41:58
there's also some funny examples like this one from
1:42:01
some funny examples like this one from
1:42:01
some funny examples like this one from the end of last year
1:42:03
the end of last year
1:42:03
the end of last year where gpt2 openai's model
1:42:06
where gpt2 openai's model
1:42:06
where gpt2 openai's model was used to predict the future or the
1:42:09
was used to predict the future or the
1:42:09
was used to predict the future or the world in 2020
1:42:11
world in 2020
1:42:11
world in 2020 and it was unfortunately very wrong
1:42:13
and it was unfortunately very wrong
1:42:13
and it was unfortunately very wrong otherwise it would have been a much
1:42:14
otherwise it would have been a much
1:42:14
otherwise it would have been a much nicer year
1:42:19
inside the field we measure progress
1:42:21
inside the field we measure progress
1:42:21
inside the field we measure progress using leaderboards
1:42:22
using leaderboards
1:42:22
using leaderboards that contain various nlp tasks
1:42:26
that contain various nlp tasks
1:42:26
that contain various nlp tasks and they also um are a bit misleading
1:42:29
and they also um are a bit misleading
1:42:29
and they also um are a bit misleading because
1:42:30
because
1:42:30
because uh it seems like we are at human level
1:42:32
uh it seems like we are at human level
1:42:32
uh it seems like we are at human level or even above human level on
1:42:34
or even above human level on
1:42:34
or even above human level on uh various complex complex tasks uh but
1:42:37
uh various complex complex tasks uh but
1:42:37
uh various complex complex tasks uh but that's not quite the k
1:42:38
that's not quite the k
1:42:38
that's not quite the k the case and that's what i'm going to
1:42:40
the case and that's what i'm going to
1:42:40
the case and that's what i'm going to talk about today
1:42:42
talk about today
1:42:42
talk about today hey verad sorry to interrupt i think we
1:42:45
hey verad sorry to interrupt i think we
1:42:45
hey verad sorry to interrupt i think we might be seeing
1:42:46
might be seeing
1:42:46
might be seeing a different screen are you advancing
1:42:48
a different screen are you advancing
1:42:48
a different screen are you advancing slides
1:42:51
are you are you advancing are you
1:42:53
are you are you advancing are you
1:42:53
are you are you advancing are you changing to different slides
1:42:55
changing to different slides
1:42:55
changing to different slides it's changed now but that was the first
1:42:57
it's changed now but that was the first
1:42:57
it's changed now but that was the first time it moved
1:42:58
time it moved
1:42:58
time it moved so i think we might have missed the epic
1:43:00
so i think we might have missed the epic
1:43:00
so i think we might have missed the epic i'm just going to share the entire
1:43:02
i'm just going to share the entire
1:43:02
i'm just going to share the entire screen sorry about it
1:43:03
screen sorry about it
1:43:03
screen sorry about it no worries they're going
1:43:08
um do you see it now
1:43:12
um do you see it now
1:43:12
um do you see it now yes yeah okay i believe yeah yeah it's
1:43:15
yes yeah okay i believe yeah yeah it's
1:43:15
yes yeah okay i believe yeah yeah it's better
1:43:16
better
1:43:16
better hopefully it's gonna work now um
1:43:19
hopefully it's gonna work now um
1:43:19
hopefully it's gonna work now um okay um and
1:43:22
okay um and
1:43:22
okay um and if you work in on nlp in industry or
1:43:25
if you work in on nlp in industry or
1:43:25
if you work in on nlp in industry or if you're just the user of nlp and
1:43:28
if you're just the user of nlp and
1:43:28
if you're just the user of nlp and most of you are then you should know
1:43:31
most of you are then you should know
1:43:31
most of you are then you should know that some
1:43:31
that some
1:43:32
that some things work reasonably well in practice
1:43:35
things work reasonably well in practice
1:43:35
things work reasonably well in practice and specifically for english but not
1:43:38
and specifically for english but not
1:43:38
and specifically for english but not always
1:43:40
always
1:43:40
always so for other languages so i personally
1:43:43
so for other languages so i personally
1:43:43
so for other languages so i personally use
1:43:43
use
1:43:43
use autocomplete grammar corrections search
1:43:45
autocomplete grammar corrections search
1:43:45
autocomplete grammar corrections search and translation
1:43:47
and translation
1:43:47
and translation which work very well for or reasonably
1:43:49
which work very well for or reasonably
1:43:50
which work very well for or reasonably well for me in english uh but i i'm
1:43:52
well for me in english uh but i i'm
1:43:52
well for me in english uh but i i'm also a native hebrew speaker and these
1:43:54
also a native hebrew speaker and these
1:43:54
also a native hebrew speaker and these things don't really work well in hebrew
1:43:57
things don't really work well in hebrew
1:43:57
things don't really work well in hebrew or not as well as they work for english
1:44:01
or not as well as they work for english
1:44:01
or not as well as they work for english um so the the reason that these
1:44:05
um so the the reason that these
1:44:05
um so the the reason that these models work well today is mostly uh
1:44:08
models work well today is mostly uh
1:44:08
models work well today is mostly uh due to the encoder art and decoder
1:44:10
due to the encoder art and decoder
1:44:10
due to the encoder art and decoder architecture
1:44:11
architecture
1:44:11
architecture or um stick to sick as it was referred
1:44:14
or um stick to sick as it was referred
1:44:14
or um stick to sick as it was referred to in one of the earlier talks
1:44:17
to in one of the earlier talks
1:44:17
to in one of the earlier talks where you can take a task that requires
1:44:19
where you can take a task that requires
1:44:20
where you can take a task that requires translating an
1:44:21
translating an
1:44:21
translating an input sequence into an output sequence
1:44:24
input sequence into an output sequence
1:44:24
input sequence into an output sequence such as let's say for example
1:44:26
such as let's say for example
1:44:26
such as let's say for example translating for
1:44:27
translating for
1:44:27
translating for from english to spanish so you would
1:44:29
from english to spanish so you would
1:44:29
from english to spanish so you would take a sequence of english words like
1:44:32
take a sequence of english words like
1:44:32
take a sequence of english words like the white cat
1:44:33
the white cat
1:44:33
the white cat and then encode it using the encoder uh
1:44:36
and then encode it using the encoder uh
1:44:36
and then encode it using the encoder uh into some vector representation
1:44:38
into some vector representation
1:44:38
into some vector representation and then the decoder decodes this um
1:44:41
and then the decoder decodes this um
1:44:41
and then the decoder decodes this um word by word output in the spanish
1:44:44
word by word output in the spanish
1:44:44
word by word output in the spanish translation would
1:44:45
translation would
1:44:45
translation would which would be elgato blanco
1:44:48
which would be elgato blanco
1:44:48
which would be elgato blanco uh and so this is not just limited to
1:44:50
uh and so this is not just limited to
1:44:50
uh and so this is not just limited to translation it works well for
1:44:52
translation it works well for
1:44:52
translation it works well for other tasks that are in a similar nature
1:44:54
other tasks that are in a similar nature
1:44:54
other tasks that are in a similar nature um
1:44:55
um
1:44:55
um such as grammatical error correction uh
1:44:57
such as grammatical error correction uh
1:44:57
such as grammatical error correction uh email response suggestions
1:44:59
email response suggestions
1:44:59
email response suggestions uh which i really like and use a lot
1:45:02
uh which i really like and use a lot
1:45:02
uh which i really like and use a lot or um image captioning where the the
1:45:05
or um image captioning where the the
1:45:05
or um image captioning where the the input sequence is not necessarily it's
1:45:07
input sequence is not necessarily it's
1:45:07
input sequence is not necessarily it's not a text
1:45:08
not a text
1:45:08
not a text it's an image and with partial success
1:45:12
it's an image and with partial success
1:45:12
it's an image and with partial success also for summarization
1:45:16
also for summarization
1:45:16
also for summarization so um encoder recorder models are very
1:45:18
so um encoder recorder models are very
1:45:18
so um encoder recorder models are very efficient for tests that require
1:45:20
efficient for tests that require
1:45:20
efficient for tests that require learning some mapping between input
1:45:22
learning some mapping between input
1:45:22
learning some mapping between input sequence and output sequences
1:45:25
sequence and output sequences
1:45:25
sequence and output sequences but that's given that they have enough
1:45:27
but that's given that they have enough
1:45:28
but that's given that they have enough training data
1:45:30
training data
1:45:30
training data so let's talk about these enough
1:45:32
so let's talk about these enough
1:45:32
so let's talk about these enough training examples and
1:45:34
training examples and
1:45:34
training examples and i'm going to get a bit hand wavy here
1:45:36
i'm going to get a bit hand wavy here
1:45:36
i'm going to get a bit hand wavy here and i want to make the
1:45:37
and i want to make the
1:45:37
and i want to make the claim that humans generalize from few
1:45:39
claim that humans generalize from few
1:45:39
claim that humans generalize from few examples
1:45:41
examples
1:45:41
examples so for example if you show a child a
1:45:44
so for example if you show a child a
1:45:44
so for example if you show a child a picture of a cat and you tell them that
1:45:47
picture of a cat and you tell them that
1:45:47
picture of a cat and you tell them that the cat eats
1:45:48
the cat eats
1:45:48
the cat eats then they're going to build some
1:45:50
then they're going to build some
1:45:50
then they're going to build some abstract cat representation in their
1:45:52
abstract cat representation in their
1:45:52
abstract cat representation in their head
1:45:53
head
1:45:53
head and then upon seeing the cat doing other
1:45:55
and then upon seeing the cat doing other
1:45:55
and then upon seeing the cat doing other things they can generalize and say okay
1:45:56
things they can generalize and say okay
1:45:56
things they can generalize and say okay here the cat drinks
1:45:58
here the cat drinks
1:45:58
here the cat drinks or the cat sleeps or maybe seeing a
1:46:00
or the cat sleeps or maybe seeing a
1:46:00
or the cat sleeps or maybe seeing a different cat and still being able to
1:46:02
different cat and still being able to
1:46:02
different cat and still being able to tell that the cat eats
1:46:04
tell that the cat eats
1:46:04
tell that the cat eats this is not quite the case with machines
1:46:06
this is not quite the case with machines
1:46:06
this is not quite the case with machines where if you just trained uh
1:46:08
where if you just trained uh
1:46:08
where if you just trained uh an image captioning model with one
1:46:10
an image captioning model with one
1:46:10
an image captioning model with one example it's going to be pretty
1:46:11
example it's going to be pretty
1:46:12
example it's going to be pretty helpless in captioning similar images
1:46:16
helpless in captioning similar images
1:46:16
helpless in captioning similar images what you would do instead is you would
1:46:18
what you would do instead is you would
1:46:18
what you would do instead is you would provide it with multiple examples
1:46:20
provide it with multiple examples
1:46:20
provide it with multiple examples so in this case it's going to be
1:46:22
so in this case it's going to be
1:46:22
so in this case it's going to be multiple images
1:46:24
multiple images
1:46:24
multiple images of different cats and eating in
1:46:26
of different cats and eating in
1:46:26
of different cats and eating in different angles
1:46:27
different angles
1:46:27
different angles and then these models can do pretty well
1:46:31
so uh the problem is that models are
1:46:34
so uh the problem is that models are
1:46:34
so uh the problem is that models are data hungry
1:46:37
and um for for this reason uh
1:46:39
and um for for this reason uh
1:46:40
and um for for this reason uh specifically
1:46:40
specifically
1:46:40
specifically um these uh applications mostly work
1:46:43
um these uh applications mostly work
1:46:44
um these uh applications mostly work well in production for english which is
1:46:45
well in production for english which is
1:46:45
well in production for english which is a high resource language
1:46:47
a high resource language
1:46:47
a high resource language uh but they don't always work as well
1:46:50
uh but they don't always work as well
1:46:50
uh but they don't always work as well for languages with enough training data
1:46:53
for languages with enough training data
1:46:53
for languages with enough training data so uh one good example for that is
1:46:55
so uh one good example for that is
1:46:55
so uh one good example for that is translation
1:46:56
translation
1:46:56
translation and uh there's here's where you can see
1:46:59
and uh there's here's where you can see
1:46:59
and uh there's here's where you can see uh where lp for other languages is even
1:47:01
uh where lp for other languages is even
1:47:01
uh where lp for other languages is even less sold than for english
1:47:03
less sold than for english
1:47:03
less sold than for english um so here's a chart from one of
1:47:06
um so here's a chart from one of
1:47:06
um so here's a chart from one of google's papers
1:47:08
google's papers
1:47:08
google's papers that shows the number of training
1:47:10
that shows the number of training
1:47:10
that shows the number of training examples for
1:47:11
examples for
1:47:11
examples for different pairs of languages so the
1:47:15
different pairs of languages so the
1:47:15
different pairs of languages so the higher resource languages they have
1:47:17
higher resource languages they have
1:47:17
higher resource languages they have almost 2 billion examples
1:47:19
almost 2 billion examples
1:47:19
almost 2 billion examples where where is the lower resource
1:47:22
where where is the lower resource
1:47:22
where where is the lower resource languages
1:47:22
languages
1:47:22
languages they can have even just around tens of
1:47:25
they can have even just around tens of
1:47:25
they can have even just around tens of thousands
1:47:26
thousands
1:47:26
thousands so it's a major difference
1:47:29
so it's a major difference
1:47:30
so it's a major difference so what happens when there's little
1:47:31
so what happens when there's little
1:47:31
so what happens when there's little training data so here's something that
1:47:33
training data so here's something that
1:47:33
training data so here's something that happened two years ago which
1:47:35
happened two years ago which
1:47:35
happened two years ago which has been fixed by uh by now but uh i
1:47:37
has been fixed by uh by now but uh i
1:47:37
has been fixed by uh by now but uh i think it's a good demonstration
1:47:39
think it's a good demonstration
1:47:39
think it's a good demonstration uh nevertheless um so google translate
1:47:42
uh nevertheless um so google translate
1:47:42
uh nevertheless um so google translate was generating religious prophecies
1:47:44
was generating religious prophecies
1:47:44
was generating religious prophecies uh and here's an example that i um that
1:47:47
uh and here's an example that i um that
1:47:47
uh and here's an example that i um that i kept from
1:47:48
i kept from
1:47:48
i kept from um 2018 where before this bug was fixed
1:47:52
um 2018 where before this bug was fixed
1:47:52
um 2018 where before this bug was fixed um and what i did here is i set the the
1:47:55
um and what i did here is i set the the
1:47:55
um and what i did here is i set the the input language to igbo which is a
1:47:57
input language to igbo which is a
1:47:57
input language to igbo which is a low resource language that i i'm not a
1:47:59
low resource language that i i'm not a
1:47:59
low resource language that i i'm not a speaker of
1:48:01
speaker of
1:48:01
speaker of and i just uh inputted some nonsensical
1:48:05
and i just uh inputted some nonsensical
1:48:05
and i just uh inputted some nonsensical uh sequence of eyes and spaces which i'm
1:48:07
uh sequence of eyes and spaces which i'm
1:48:07
uh sequence of eyes and spaces which i'm pretty sure is not
1:48:09
pretty sure is not
1:48:09
pretty sure is not an igbo utterance um by setting the
1:48:12
an igbo utterance um by setting the
1:48:12
an igbo utterance um by setting the translation to english what you would
1:48:14
translation to english what you would
1:48:14
translation to english what you would expect to happen is the model to tell
1:48:16
expect to happen is the model to tell
1:48:16
expect to happen is the model to tell you
1:48:16
you
1:48:16
you this doesn't look like a valid igbo
1:48:19
this doesn't look like a valid igbo
1:48:19
this doesn't look like a valid igbo input i don't know what to do with it
1:48:21
input i don't know what to do with it
1:48:21
input i don't know what to do with it but this is not how a neural model works
1:48:24
but this is not how a neural model works
1:48:24
but this is not how a neural model works neural models work
1:48:25
neural models work
1:48:25
neural models work and uh what they do instead when they
1:48:27
and uh what they do instead when they
1:48:27
and uh what they do instead when they don't know the answer is they just try
1:48:29
don't know the answer is they just try
1:48:29
don't know the answer is they just try to
1:48:29
to
1:48:30
to still answer somehow by generating
1:48:33
still answer somehow by generating
1:48:33
still answer somehow by generating something that's
1:48:34
something that's
1:48:34
something that's similar to their training data and what
1:48:36
similar to their training data and what
1:48:36
similar to their training data and what was generated here was
1:48:38
was generated here was
1:48:38
was generated here was um the english translation was
1:48:41
um the english translation was
1:48:41
um the english translation was as it is written in the book of the law
1:48:43
as it is written in the book of the law
1:48:43
as it is written in the book of the law of moses which was in
1:48:44
of moses which was in
1:48:44
of moses which was in the wilderness which was before the man
1:48:46
the wilderness which was before the man
1:48:46
the wilderness which was before the man who did the work in the kingdom of
1:48:47
who did the work in the kingdom of
1:48:47
who did the work in the kingdom of israel
1:48:48
israel
1:48:48
israel which i'm pretty sure is not what i
1:48:50
which i'm pretty sure is not what i
1:48:50
which i'm pretty sure is not what i wrote um
1:48:52
wrote um
1:48:52
wrote um so the reason that this is uh
1:48:55
so the reason that this is uh
1:48:56
so the reason that this is uh biblical is likely because
1:48:59
biblical is likely because
1:48:59
biblical is likely because the way that machine translation works
1:49:01
the way that machine translation works
1:49:01
the way that machine translation works is that it's
1:49:02
is that it's
1:49:02
is that it's trained on parallel texts saying the
1:49:05
trained on parallel texts saying the
1:49:05
trained on parallel texts saying the same thing but in different languages
1:49:07
same thing but in different languages
1:49:07
same thing but in different languages which are usually obtained by human
1:49:09
which are usually obtained by human
1:49:09
which are usually obtained by human translations
1:49:11
translations
1:49:11
translations and it's likely that there is not enough
1:49:13
and it's likely that there is not enough
1:49:13
and it's likely that there is not enough data for igbo
1:49:14
data for igbo
1:49:14
data for igbo english translations but something that
1:49:18
english translations but something that
1:49:18
english translations but something that is often available in
1:49:19
is often available in
1:49:19
is often available in in across multiple languages is the
1:49:21
in across multiple languages is the
1:49:21
in across multiple languages is the bible uh this is
1:49:23
bible uh this is
1:49:23
bible uh this is pretty much the lowest common
1:49:24
pretty much the lowest common
1:49:24
pretty much the lowest common denominator of parallel data
1:49:27
denominator of parallel data
1:49:27
denominator of parallel data so it's likely that the vast majority of
1:49:29
so it's likely that the vast majority of
1:49:30
so it's likely that the vast majority of the
1:49:30
the
1:49:30
the igbo english model training data came
1:49:33
igbo english model training data came
1:49:33
igbo english model training data came from the bible
1:49:36
so uh models fail to generalize to out
1:49:39
so uh models fail to generalize to out
1:49:39
so uh models fail to generalize to out of the main examples and even worse than
1:49:41
of the main examples and even worse than
1:49:41
of the main examples and even worse than that they
1:49:42
that they
1:49:42
that they don't know to tell you when something is
1:49:44
don't know to tell you when something is
1:49:44
don't know to tell you when something is out of domain but they still
1:49:45
out of domain but they still
1:49:45
out of domain but they still try to answer the question
1:49:50
so the solution is that when it's
1:49:51
so the solution is that when it's
1:49:52
so the solution is that when it's possible you don't train these model
1:49:53
possible you don't train these model
1:49:53
possible you don't train these model from scratch
1:49:54
from scratch
1:49:54
from scratch but instead you you continue
1:49:57
but instead you you continue
1:49:57
but instead you you continue training a pre-trained model so this has
1:50:00
training a pre-trained model so this has
1:50:00
training a pre-trained model so this has been happening in the vision community
1:50:02
been happening in the vision community
1:50:02
been happening in the vision community for quite a while now
1:50:03
for quite a while now
1:50:04
for quite a while now and in the last two and a half years
1:50:06
and in the last two and a half years
1:50:06
and in the last two and a half years we've been doing that in nlp as well
1:50:08
we've been doing that in nlp as well
1:50:08
we've been doing that in nlp as well and if you've heard about all these
1:50:10
and if you've heard about all these
1:50:10
and if you've heard about all these models with the sesame street character
1:50:12
models with the sesame street character
1:50:12
models with the sesame street character names
1:50:13
names
1:50:13
names this is what we're talking about this is
1:50:15
this is what we're talking about this is
1:50:15
this is what we're talking about this is the these are the pre-trained language
1:50:17
the these are the pre-trained language
1:50:17
the these are the pre-trained language models
1:50:19
models
1:50:19
models so what is the pre-training language
1:50:21
so what is the pre-training language
1:50:21
so what is the pre-training language model a language model
1:50:24
model a language model
1:50:24
model a language model standard left to right language model
1:50:26
standard left to right language model
1:50:26
standard left to right language model traditionally is um
1:50:28
traditionally is um
1:50:28
traditionally is um so language models in general are
1:50:30
so language models in general are
1:50:30
so language models in general are trained uh in a self-supervised manner
1:50:33
trained uh in a self-supervised manner
1:50:33
trained uh in a self-supervised manner on a
1:50:33
on a
1:50:33
on a large text corporates and standard left
1:50:36
large text corporates and standard left
1:50:36
large text corporates and standard left or right language models are trained by
1:50:39
or right language models are trained by
1:50:39
or right language models are trained by um predicting the next word in a
1:50:41
um predicting the next word in a
1:50:41
um predicting the next word in a sequence
1:50:42
sequence
1:50:42
sequence so for example um in in this sentence
1:50:46
so for example um in in this sentence
1:50:46
so for example um in in this sentence parrots are among the most intelligent
1:50:48
parrots are among the most intelligent
1:50:48
parrots are among the most intelligent birds and the ability of some species to
1:50:50
birds and the ability of some species to
1:50:50
birds and the ability of some species to imitate human speech enhances their
1:50:52
imitate human speech enhances their
1:50:52
imitate human speech enhances their popularity as
1:50:54
popularity as
1:50:54
popularity as so we need the model to predict pets
1:50:57
so we need the model to predict pets
1:50:57
so we need the model to predict pets there's a second variant called mask
1:50:59
there's a second variant called mask
1:50:59
there's a second variant called mask language model
1:51:00
language model
1:51:00
language model in which you don't have to predict the
1:51:02
in which you don't have to predict the
1:51:02
in which you don't have to predict the last word but you can instead just mask
1:51:05
last word but you can instead just mask
1:51:05
last word but you can instead just mask out some word in the sequence
1:51:07
out some word in the sequence
1:51:07
out some word in the sequence and try to restore it so this is what
1:51:10
and try to restore it so this is what
1:51:10
and try to restore it so this is what happens during the pre-training step
1:51:11
happens during the pre-training step
1:51:12
happens during the pre-training step and what you typically would do is that
1:51:14
and what you typically would do is that
1:51:14
and what you typically would do is that you would take these
1:51:15
you would take these
1:51:15
you would take these pre-trained models and you would
1:51:16
pre-trained models and you would
1:51:16
pre-trained models and you would fine-tune them
1:51:18
fine-tune them
1:51:18
fine-tune them so um keep training them on a different
1:51:22
so um keep training them on a different
1:51:22
so um keep training them on a different task or your task of interest and
1:51:25
task or your task of interest and
1:51:25
task or your task of interest and um you would um compute the test
1:51:28
um you would um compute the test
1:51:28
um you would um compute the test specific class and then back propagate
1:51:31
specific class and then back propagate
1:51:31
specific class and then back propagate it to
1:51:31
it to
1:51:32
it to uh update the representation
1:51:35
uh update the representation
1:51:35
uh update the representation so pre-training language models are
1:51:37
so pre-training language models are
1:51:37
so pre-training language models are trained once and then
1:51:38
trained once and then
1:51:38
trained once and then fine-tuned for each task with pretty
1:51:41
fine-tuned for each task with pretty
1:51:41
fine-tuned for each task with pretty tremendous success across
1:51:43
tremendous success across
1:51:43
tremendous success across nlp tasks with potentially
1:51:46
nlp tasks with potentially
1:51:46
nlp tasks with potentially needing fewer training examples for the
1:51:48
needing fewer training examples for the
1:51:48
needing fewer training examples for the specific tasks because
1:51:49
specific tasks because
1:51:50
specific tasks because the model already has a lot of knowledge
1:51:54
the problem is not it's that it's not
1:51:56
the problem is not it's that it's not
1:51:56
the problem is not it's that it's not like um
1:51:57
like um
1:51:57
like um someone trained one language model and
1:51:59
someone trained one language model and
1:51:59
someone trained one language model and then we all just used it and moved on to
1:52:02
then we all just used it and moved on to
1:52:02
then we all just used it and moved on to other research problems but instead this
1:52:05
other research problems but instead this
1:52:05
other research problems but instead this has become this competition between the
1:52:07
has become this competition between the
1:52:08
has become this competition between the large research research groups
1:52:11
large research research groups
1:52:11
large research research groups on which one of them would train the
1:52:14
on which one of them would train the
1:52:14
on which one of them would train the best language model
1:52:15
best language model
1:52:15
best language model and the best language model correlates
1:52:17
and the best language model correlates
1:52:17
and the best language model correlates with um
1:52:18
with um
1:52:18
with um an increased number of parameters so um
1:52:22
an increased number of parameters so um
1:52:22
an increased number of parameters so um just two and a half years ago the elmo
1:52:24
just two and a half years ago the elmo
1:52:24
just two and a half years ago the elmo model contained
1:52:25
model contained
1:52:25
model contained 94 million parameters which was also
1:52:27
94 million parameters which was also
1:52:28
94 million parameters which was also already
1:52:28
already
1:52:28
already quite big for the for the time but
1:52:31
quite big for the for the time but
1:52:31
quite big for the for the time but now recently google announced its
1:52:33
now recently google announced its
1:52:33
now recently google announced its g-shard model which has
1:52:35
g-shard model which has
1:52:35
g-shard model which has 600 billion parameters and
1:52:39
600 billion parameters and
1:52:39
600 billion parameters and consequently this has become very
1:52:42
consequently this has become very
1:52:42
consequently this has become very expensive to train such models
1:52:44
expensive to train such models
1:52:44
expensive to train such models where um open ai
1:52:48
where um open ai
1:52:48
where um open ai latest model gt3 is estimated to have
1:52:50
latest model gt3 is estimated to have
1:52:50
latest model gt3 is estimated to have cost 12 million dollars to train
1:52:53
cost 12 million dollars to train
1:52:53
cost 12 million dollars to train and this is obviously not a cost that
1:52:55
and this is obviously not a cost that
1:52:55
and this is obviously not a cost that all research groups
1:52:57
all research groups
1:52:57
all research groups want or are capable of investing
1:53:01
want or are capable of investing
1:53:01
want or are capable of investing and it's also not very clear that the
1:53:04
and it's also not very clear that the
1:53:04
and it's also not very clear that the effort
1:53:04
effort
1:53:04
effort the the cost and the effort is really
1:53:07
the the cost and the effort is really
1:53:07
the the cost and the effort is really worth it because if you look at the
1:53:10
worth it because if you look at the
1:53:10
worth it because if you look at the standard leaderboards you can see that
1:53:12
standard leaderboards you can see that
1:53:12
standard leaderboards you can see that there is
1:53:13
there is
1:53:13
there is still slight improvement from one model
1:53:16
still slight improvement from one model
1:53:16
still slight improvement from one model to the other but
1:53:17
to the other but
1:53:17
to the other but it's pretty much uh reaching the point
1:53:20
it's pretty much uh reaching the point
1:53:20
it's pretty much uh reaching the point of diminishing returns uh
1:53:21
of diminishing returns uh
1:53:21
of diminishing returns uh where the the gaps between the different
1:53:24
where the the gaps between the different
1:53:24
where the the gaps between the different models the accuracy gaps are pretty
1:53:26
models the accuracy gaps are pretty
1:53:26
models the accuracy gaps are pretty small
1:53:27
small
1:53:27
small and definitely not proportional to the
1:53:30
and definitely not proportional to the
1:53:30
and definitely not proportional to the increase in parameters or
1:53:31
increase in parameters or
1:53:31
increase in parameters or or in training cost and it's it's
1:53:34
or in training cost and it's it's
1:53:34
or in training cost and it's it's already beating humans on
1:53:35
already beating humans on
1:53:36
already beating humans on most of these tasks and finally it's
1:53:39
most of these tasks and finally it's
1:53:39
most of these tasks and finally it's also
1:53:40
also
1:53:40
also uh not very friendly to the environment
1:53:42
uh not very friendly to the environment
1:53:42
uh not very friendly to the environment as it was mentioned
1:53:43
as it was mentioned
1:53:43
as it was mentioned in the talk by uh josh earlier um
1:53:46
in the talk by uh josh earlier um
1:53:46
in the talk by uh josh earlier um so last year there was a paper that
1:53:49
so last year there was a paper that
1:53:49
so last year there was a paper that showed that
1:53:49
showed that
1:53:49
showed that um that showed the estimated the co2
1:53:52
um that showed the estimated the co2
1:53:52
um that showed the estimated the co2 emissions from
1:53:54
emissions from
1:53:54
emissions from training common nlp models and compared
1:53:56
training common nlp models and compared
1:53:56
training common nlp models and compared it with
1:53:57
it with
1:53:57
it with familiar consumption and this was even
1:53:59
familiar consumption and this was even
1:53:59
familiar consumption and this was even before the
1:54:00
before the
1:54:00
before the very large models but it was already
1:54:02
very large models but it was already
1:54:02
very large models but it was already painting a pretty concerning picture
1:54:06
so these models are huge and they're
1:54:08
so these models are huge and they're
1:54:08
so these models are huge and they're computationally expensive to train and
1:54:10
computationally expensive to train and
1:54:10
computationally expensive to train and use
1:54:11
use
1:54:11
use and on that note i would i would add
1:54:12
and on that note i would i would add
1:54:12
and on that note i would i would add that um i am co-organizing a workshop
1:54:15
that um i am co-organizing a workshop
1:54:15
that um i am co-organizing a workshop called sustainability
1:54:17
called sustainability
1:54:17
called sustainability uh which will be held next month
1:54:19
uh which will be held next month
1:54:19
uh which will be held next month virtually at emlp
1:54:21
virtually at emlp
1:54:21
virtually at emlp where we encourage building simpler and
1:54:24
where we encourage building simpler and
1:54:24
where we encourage building simpler and more efficient nlp models so
1:54:26
more efficient nlp models so
1:54:26
more efficient nlp models so please participate
1:54:30
okay moving on to the different type of
1:54:32
okay moving on to the different type of
1:54:32
okay moving on to the different type of tasks which is the more
1:54:34
tasks which is the more
1:54:34
tasks which is the more complex tasks that require deeper
1:54:36
complex tasks that require deeper
1:54:36
complex tasks that require deeper understanding
1:54:39
understanding
1:54:39
understanding so i want to show an example of a real
1:54:41
so i want to show an example of a real
1:54:41
so i want to show an example of a real world question answering
1:54:42
world question answering
1:54:42
world question answering system this is a demo that someone
1:54:45
system this is a demo that someone
1:54:45
system this is a demo that someone shared with me and
1:54:46
shared with me and
1:54:46
shared with me and i don't mean to say anything but
1:54:47
i don't mean to say anything but
1:54:47
i don't mean to say anything but specifically about this system i just
1:54:49
specifically about this system i just
1:54:49
specifically about this system i just want to show
1:54:49
want to show
1:54:50
want to show how incredibly difficult this task is
1:54:53
how incredibly difficult this task is
1:54:53
how incredibly difficult this task is so this is a demo meant to answer
1:54:55
so this is a demo meant to answer
1:54:55
so this is a demo meant to answer questions about kovi 19
1:54:57
questions about kovi 19
1:54:57
questions about kovi 19 and i asked it about the risk of kobe 19
1:54:59
and i asked it about the risk of kobe 19
1:54:59
and i asked it about the risk of kobe 19 infection from touching public surfaces
1:55:02
infection from touching public surfaces
1:55:02
infection from touching public surfaces and it's likely that it doesn't actually
1:55:04
and it's likely that it doesn't actually
1:55:04
and it's likely that it doesn't actually contain the answer for that
1:55:05
contain the answer for that
1:55:06
contain the answer for that but what i got in return were actually
1:55:08
but what i got in return were actually
1:55:08
but what i got in return were actually answers for different questions
1:55:10
answers for different questions
1:55:10
answers for different questions so uh for example uh what is the
1:55:12
so uh for example uh what is the
1:55:12
so uh for example uh what is the probability of transmission of the virus
1:55:14
probability of transmission of the virus
1:55:14
probability of transmission of the virus from wuhan to other cities in china
1:55:16
from wuhan to other cities in china
1:55:16
from wuhan to other cities in china before the quarantine
1:55:17
before the quarantine
1:55:18
before the quarantine what is the risk of getting infected by
1:55:19
what is the risk of getting infected by
1:55:19
what is the risk of getting infected by children
1:55:21
children
1:55:21
children what is the ability to detect imported
1:55:23
what is the ability to detect imported
1:55:23
what is the ability to detect imported cases among high surveillance locations
1:55:25
cases among high surveillance locations
1:55:25
cases among high surveillance locations and finally the only paragraph that
1:55:27
and finally the only paragraph that
1:55:27
and finally the only paragraph that seems somewhat relevant is the last one
1:55:30
seems somewhat relevant is the last one
1:55:30
seems somewhat relevant is the last one but it still doesn't answer the question
1:55:34
so this is uh to illustrate the
1:55:36
so this is uh to illustrate the
1:55:36
so this is uh to illustrate the difference between solving a task
1:55:38
difference between solving a task
1:55:38
difference between solving a task like question answering which is uh
1:55:40
like question answering which is uh
1:55:40
like question answering which is uh incredibly complex
1:55:42
incredibly complex
1:55:42
incredibly complex uh to solving a data set which is
1:55:43
uh to solving a data set which is
1:55:43
uh to solving a data set which is something we do pretty well today
1:55:47
something we do pretty well today
1:55:47
something we do pretty well today and now let me show you an example of
1:55:49
and now let me show you an example of
1:55:49
and now let me show you an example of solving a data set
1:55:50
solving a data set
1:55:50
solving a data set this is uh for a different task called
1:55:52
this is uh for a different task called
1:55:52
this is uh for a different task called natural language
1:55:53
natural language
1:55:53
natural language inference or nli so in
1:55:57
inference or nli so in
1:55:57
inference or nli so in this task the model is given two
1:55:58
this task the model is given two
1:55:58
this task the model is given two sentences a premise and a hypothesis the
1:56:01
sentences a premise and a hypothesis the
1:56:01
sentences a premise and a hypothesis the premise is known to have happened
1:56:03
premise is known to have happened
1:56:03
premise is known to have happened and the the goal is to to determine
1:56:07
and the the goal is to to determine
1:56:07
and the the goal is to to determine the truth value of the hypothesis so for
1:56:10
the truth value of the hypothesis so for
1:56:10
the truth value of the hypothesis so for example
1:56:11
example
1:56:11
example for the premise straight performer is
1:56:13
for the premise straight performer is
1:56:13
for the premise straight performer is doing is his act for kids
1:56:16
doing is his act for kids
1:56:16
doing is his act for kids we can have the hypothesis a person
1:56:18
we can have the hypothesis a person
1:56:18
we can have the hypothesis a person performing for
1:56:19
performing for
1:56:19
performing for children on the street which is
1:56:21
children on the street which is
1:56:21
children on the street which is considered entailed because
1:56:23
considered entailed because
1:56:23
considered entailed because it's not adding any new information this
1:56:25
it's not adding any new information this
1:56:25
it's not adding any new information this is pretty much repeating the premise but
1:56:27
is pretty much repeating the premise but
1:56:27
is pretty much repeating the premise but making it a bit more general
1:56:29
making it a bit more general
1:56:29
making it a bit more general uh a juggler entertaining a group of
1:56:31
uh a juggler entertaining a group of
1:56:31
uh a juggler entertaining a group of children on the street is neutral
1:56:32
children on the street is neutral
1:56:32
children on the street is neutral because
1:56:33
because
1:56:33
because it may be true but we don't know for
1:56:35
it may be true but we don't know for
1:56:35
it may be true but we don't know for sure that the straight performer is a
1:56:37
sure that the straight performer is a
1:56:37
sure that the straight performer is a juggler so it's adding information
1:56:40
juggler so it's adding information
1:56:40
juggler so it's adding information and finally a magician performing for an
1:56:42
and finally a magician performing for an
1:56:42
and finally a magician performing for an audience in a nightclub
1:56:43
audience in a nightclub
1:56:43
audience in a nightclub is considered contradicting because
1:56:46
is considered contradicting because
1:56:46
is considered contradicting because assuming we discuss the same events
1:56:48
assuming we discuss the same events
1:56:48
assuming we discuss the same events um the one of them happens on the street
1:56:51
um the one of them happens on the street
1:56:51
um the one of them happens on the street while the other one happens in a
1:56:52
while the other one happens in a
1:56:52
while the other one happens in a nightclub
1:56:53
nightclub
1:56:54
nightclub if you didn't understand the task that's
1:56:55
if you didn't understand the task that's
1:56:55
if you didn't understand the task that's perfectly fine um
1:56:57
perfectly fine um
1:56:57
perfectly fine um this is a pretty complicated task that
1:56:59
this is a pretty complicated task that
1:56:59
this is a pretty complicated task that requires knowledge about word meanings
1:57:02
requires knowledge about word meanings
1:57:02
requires knowledge about word meanings understanding of syntax ability to
1:57:04
understanding of syntax ability to
1:57:04
understanding of syntax ability to resolve references to entities
1:57:06
resolve references to entities
1:57:06
resolve references to entities word knowledge and common sense
1:57:07
word knowledge and common sense
1:57:07
word knowledge and common sense reasoning um
1:57:09
reasoning um
1:57:09
reasoning um yet uh the best neural models they
1:57:11
yet uh the best neural models they
1:57:11
yet uh the best neural models they already uh
1:57:12
already uh
1:57:12
already uh beat humans or are are at human level on
1:57:16
beat humans or are are at human level on
1:57:16
beat humans or are are at human level on these
1:57:16
these
1:57:16
these uh on the standard data sets for the
1:57:18
uh on the standard data sets for the
1:57:18
uh on the standard data sets for the task
1:57:20
task
1:57:20
task um so why this why is this happening is
1:57:22
um so why this why is this happening is
1:57:22
um so why this why is this happening is it because they actually solve the nli
1:57:24
it because they actually solve the nli
1:57:24
it because they actually solve the nli task
1:57:25
task
1:57:25
task um not so much it's because they solve
1:57:27
um not so much it's because they solve
1:57:27
um not so much it's because they solve the data
1:57:28
the data
1:57:28
the data sets and they do that by
1:57:31
sets and they do that by
1:57:31
sets and they do that by finding uh these shallow
1:57:34
finding uh these shallow
1:57:34
finding uh these shallow patterns of that that correlate between
1:57:37
patterns of that that correlate between
1:57:38
patterns of that that correlate between the inputs and the outputs in the data
1:57:39
the inputs and the outputs in the data
1:57:39
the inputs and the outputs in the data set which are not always indicative of
1:57:41
set which are not always indicative of
1:57:41
set which are not always indicative of the task
1:57:42
the task
1:57:42
the task and one um one pretty neat way to show
1:57:45
and one um one pretty neat way to show
1:57:45
and one um one pretty neat way to show that uh it has been shown
1:57:47
that uh it has been shown
1:57:47
that uh it has been shown uh two years ago by two papers
1:57:50
uh two years ago by two papers
1:57:50
uh two years ago by two papers um is that if you provide a model with
1:57:53
um is that if you provide a model with
1:57:53
um is that if you provide a model with just a part of the input
1:57:54
just a part of the input
1:57:54
just a part of the input just the hypothesis it's still doing
1:57:57
just the hypothesis it's still doing
1:57:57
just the hypothesis it's still doing better much better than
1:57:58
better much better than
1:57:58
better much better than chance uh where whereas the human uh
1:58:01
chance uh where whereas the human uh
1:58:01
chance uh where whereas the human uh provider just the hypothesis would not
1:58:03
provider just the hypothesis would not
1:58:03
provider just the hypothesis would not be able to solve the
1:58:04
be able to solve the
1:58:04
be able to solve the the instance so what it learns are
1:58:07
the instance so what it learns are
1:58:07
the instance so what it learns are things like
1:58:08
things like
1:58:08
things like that if the hypothesis contains a word
1:58:10
that if the hypothesis contains a word
1:58:10
that if the hypothesis contains a word like outside which is pretty general
1:58:13
like outside which is pretty general
1:58:13
like outside which is pretty general then it's likely entailing while uh
1:58:16
then it's likely entailing while uh
1:58:16
then it's likely entailing while uh longer hypothesis indicate neutral
1:58:18
longer hypothesis indicate neutral
1:58:18
longer hypothesis indicate neutral because usually you would add
1:58:19
because usually you would add
1:58:19
because usually you would add information
1:58:20
information
1:58:20
information uh or this one is my absolute favorite
1:58:23
uh or this one is my absolute favorite
1:58:23
uh or this one is my absolute favorite uh
1:58:23
uh
1:58:23
uh the existence of the word cat in the
1:58:25
the existence of the word cat in the
1:58:25
the existence of the word cat in the hypothesis
1:58:27
hypothesis
1:58:27
hypothesis is correlates with contradiction because
1:58:30
is correlates with contradiction because
1:58:30
is correlates with contradiction because the premises in the data set were taken
1:58:32
the premises in the data set were taken
1:58:32
the premises in the data set were taken from image captions
1:58:34
from image captions
1:58:34
from image captions and they contain a lot of dogs and as
1:58:36
and they contain a lot of dogs and as
1:58:36
and they contain a lot of dogs and as you all know
1:58:37
you all know
1:58:38
you all know cat is the opposite of dom
1:58:41
so models rely on shallow patterns
1:58:44
so models rely on shallow patterns
1:58:44
so models rely on shallow patterns or spurious correlations and they fail
1:58:47
or spurious correlations and they fail
1:58:47
or spurious correlations and they fail to generalize outside our training
1:58:49
to generalize outside our training
1:58:49
to generalize outside our training domain
1:58:50
domain
1:58:50
domain these patterns are often not very
1:58:52
these patterns are often not very
1:58:52
these patterns are often not very indicative of the task itself they're
1:58:54
indicative of the task itself they're
1:58:54
indicative of the task itself they're just
1:58:54
just
1:58:54
just indicative of a data set specifically on
1:58:58
indicative of a data set specifically on
1:58:58
indicative of a data set specifically on the nli
1:58:59
the nli
1:58:59
the nli example the reason that this happens is
1:59:01
example the reason that this happens is
1:59:01
example the reason that this happens is because the data set is somewhat
1:59:02
because the data set is somewhat
1:59:02
because the data set is somewhat artificial
1:59:03
artificial
1:59:04
artificial so the way it was built is that the um
1:59:07
so the way it was built is that the um
1:59:07
so the way it was built is that the um the premises were taken from image
1:59:09
the premises were taken from image
1:59:09
the premises were taken from image captions but then
1:59:10
captions but then
1:59:10
captions but then uh crowdsourcing workers were asked to
1:59:12
uh crowdsourcing workers were asked to
1:59:12
uh crowdsourcing workers were asked to generate the
1:59:13
generate the
1:59:13
generate the entailing neutral and contradicting
1:59:15
entailing neutral and contradicting
1:59:15
entailing neutral and contradicting hypothesis
1:59:17
hypothesis
1:59:17
hypothesis and what they likely did which is what
1:59:20
and what they likely did which is what
1:59:20
and what they likely did which is what everyone would do if they had to
1:59:21
everyone would do if they had to
1:59:21
everyone would do if they had to answer a lot of tasks and they get paid
1:59:23
answer a lot of tasks and they get paid
1:59:24
answer a lot of tasks and they get paid by task
1:59:25
by task
1:59:25
by task is to come up with these cognitive
1:59:26
is to come up with these cognitive
1:59:26
is to come up with these cognitive shortcuts or strategies to answer as
1:59:29
shortcuts or strategies to answer as
1:59:29
shortcuts or strategies to answer as many questions
1:59:30
many questions
1:59:30
many questions um as fast as possible so for example
1:59:34
um as fast as possible so for example
1:59:34
um as fast as possible so for example replacing
1:59:34
replacing
1:59:34
replacing dogs by cats or any negation negation to
1:59:37
dogs by cats or any negation negation to
1:59:38
dogs by cats or any negation negation to um to contradict the sentence and these
1:59:41
um to contradict the sentence and these
1:59:41
um to contradict the sentence and these things are
1:59:41
things are
1:59:41
things are incredibly easy for neural model to pick
1:59:43
incredibly easy for neural model to pick
1:59:43
incredibly easy for neural model to pick up on
1:59:47
another problem is that it's hard to
1:59:48
another problem is that it's hard to
1:59:48
another problem is that it's hard to tell whether a model learned valid or
1:59:50
tell whether a model learned valid or
1:59:50
tell whether a model learned valid or spurious patterns for a specific task
1:59:52
spurious patterns for a specific task
1:59:52
spurious patterns for a specific task because
1:59:53
because
1:59:53
because neural models are just not interpretable
1:59:56
neural models are just not interpretable
1:59:56
neural models are just not interpretable and this leads to a different problem
1:59:58
and this leads to a different problem
1:59:58
and this leads to a different problem which is even worse
1:59:59
which is even worse
1:59:59
which is even worse which is uh fairness problems
2:00:02
which is uh fairness problems
2:00:02
which is uh fairness problems uh where our machine learning models
2:00:05
uh where our machine learning models
2:00:05
uh where our machine learning models they learn
2:00:06
they learn
2:00:06
they learn uh our bad habits and our uh
2:00:09
uh our bad habits and our uh
2:00:09
uh our bad habits and our uh biases from us and it's very hard to to
2:00:12
biases from us and it's very hard to to
2:00:12
biases from us and it's very hard to to find it
2:00:14
find it
2:00:14
find it so here's just a handful of examples
2:00:16
so here's just a handful of examples
2:00:16
so here's just a handful of examples from many examples over the last few
2:00:19
from many examples over the last few
2:00:19
from many examples over the last few years
2:00:20
years
2:00:20
years so for example microsoft chatbot that
2:00:23
so for example microsoft chatbot that
2:00:23
so for example microsoft chatbot that became racist in one day
2:00:25
became racist in one day
2:00:25
became racist in one day or lots of gender bias and racial bias
2:00:28
or lots of gender bias and racial bias
2:00:28
or lots of gender bias and racial bias in various applications
2:00:30
in various applications
2:00:30
in various applications there's also the example of amazon that
2:00:33
there's also the example of amazon that
2:00:33
there's also the example of amazon that built
2:00:33
built
2:00:33
built a tool meant to filter out cvs for
2:00:37
a tool meant to filter out cvs for
2:00:37
a tool meant to filter out cvs for recruiting purposes but it couldn't
2:00:39
recruiting purposes but it couldn't
2:00:39
recruiting purposes but it couldn't actually use it in in practice because
2:00:41
actually use it in in practice because
2:00:41
actually use it in in practice because uh they found it was discriminating
2:00:43
uh they found it was discriminating
2:00:43
uh they found it was discriminating against women
2:00:46
against women
2:00:46
against women so neural models perpetuate social
2:00:48
so neural models perpetuate social
2:00:48
so neural models perpetuate social biases that are found in their training
2:00:50
biases that are found in their training
2:00:50
biases that are found in their training data
2:00:52
data
2:00:52
data let's look at the amazon cv filtering
2:00:55
let's look at the amazon cv filtering
2:00:55
let's look at the amazon cv filtering example and try to figure out what could
2:00:56
example and try to figure out what could
2:00:56
example and try to figure out what could go wrong
2:00:58
go wrong
2:00:58
go wrong so if you had to build a cv filtering
2:01:01
so if you had to build a cv filtering
2:01:01
so if you had to build a cv filtering system what you would
2:01:02
system what you would
2:01:02
system what you would uh usually do is you would build a data
2:01:05
uh usually do is you would build a data
2:01:05
uh usually do is you would build a data set
2:01:06
set
2:01:06
set that's based on past decisions made by
2:01:08
that's based on past decisions made by
2:01:08
that's based on past decisions made by people
2:01:09
people
2:01:09
people so you would take the uh cvs that were
2:01:11
so you would take the uh cvs that were
2:01:11
so you would take the uh cvs that were um
2:01:12
um
2:01:12
um sent for for applications and then the
2:01:15
sent for for applications and then the
2:01:15
sent for for applications and then the decisions that the
2:01:16
decisions that the
2:01:16
decisions that the the responsible person decided whether
2:01:18
the responsible person decided whether
2:01:18
the responsible person decided whether to uh go forward with with them or
2:01:21
to uh go forward with with them or
2:01:21
to uh go forward with with them or or not the problem is that the the
2:01:23
or not the problem is that the the
2:01:23
or not the problem is that the the people that provide these supervisions
2:01:25
people that provide these supervisions
2:01:25
people that provide these supervisions are likely biased
2:01:27
are likely biased
2:01:27
are likely biased uh so it may be for example uh that in
2:01:30
uh so it may be for example uh that in
2:01:30
uh so it may be for example uh that in this specific case
2:01:31
this specific case
2:01:31
this specific case um someone thought that men are better
2:01:33
um someone thought that men are better
2:01:33
um someone thought that men are better qualified for some job than women
2:01:36
qualified for some job than women
2:01:36
qualified for some job than women it's also likely that they were not
2:01:38
it's also likely that they were not
2:01:38
it's also likely that they were not wanting were not enough women in the
2:01:40
wanting were not enough women in the
2:01:40
wanting were not enough women in the training data
2:01:42
training data
2:01:42
training data the second problem is that as i've
2:01:44
the second problem is that as i've
2:01:44
the second problem is that as i've mentioned we use pre-trained
2:01:46
mentioned we use pre-trained
2:01:46
mentioned we use pre-trained representations
2:01:47
representations
2:01:47
representations so these represent representations are
2:01:50
so these represent representations are
2:01:50
so these represent representations are also biased
2:01:51
also biased
2:01:51
also biased uh and so for example it has been shown
2:01:54
uh and so for example it has been shown
2:01:54
uh and so for example it has been shown that
2:01:54
that
2:01:54
that uh for uh word representations they are
2:01:57
uh for uh word representations they are
2:01:57
uh for uh word representations they are they
2:01:58
they
2:01:58
they contain um racial and gender uh biases
2:02:02
contain um racial and gender uh biases
2:02:02
contain um racial and gender uh biases so for example um a programmer would be
2:02:05
so for example um a programmer would be
2:02:05
so for example um a programmer would be more
2:02:06
more
2:02:06
more considered more semantically similar to
2:02:08
considered more semantically similar to
2:02:08
considered more semantically similar to men than to women
2:02:09
men than to women
2:02:09
men than to women which is more similar to homemaker um
2:02:12
which is more similar to homemaker um
2:02:12
which is more similar to homemaker um and
2:02:13
and
2:02:13
and once we use these representations in our
2:02:15
once we use these representations in our
2:02:15
once we use these representations in our systems it just amplifies the biases
2:02:19
systems it just amplifies the biases
2:02:19
systems it just amplifies the biases and finally as i mentioned since uh
2:02:21
and finally as i mentioned since uh
2:02:21
and finally as i mentioned since uh these neural models are not
2:02:22
these neural models are not
2:02:22
these neural models are not interpretable it's pretty hard to
2:02:24
interpretable it's pretty hard to
2:02:24
interpretable it's pretty hard to actually
2:02:25
actually
2:02:25
actually figure out that you have this system
2:02:27
figure out that you have this system
2:02:27
figure out that you have this system that amplifies biases
2:02:29
that amplifies biases
2:02:29
that amplifies biases unless you specifically probe it by for
2:02:31
unless you specifically probe it by for
2:02:31
unless you specifically probe it by for example
2:02:32
example
2:02:32
example providing it with two identical cvs that
2:02:35
providing it with two identical cvs that
2:02:35
providing it with two identical cvs that would only be
2:02:35
would only be
2:02:35
would only be different in the gender of the applicant
2:02:38
different in the gender of the applicant
2:02:38
different in the gender of the applicant and then seeing whether the predictions
2:02:42
and then seeing whether the predictions
2:02:42
and then seeing whether the predictions are consistent or not
2:02:45
are consistent or not
2:02:46
are consistent or not uh moving on to a different topic this
2:02:47
uh moving on to a different topic this
2:02:47
uh moving on to a different topic this is the positive one uh dynamic versus
2:02:50
is the positive one uh dynamic versus
2:02:50
is the positive one uh dynamic versus static word vectors i think this was all
2:02:51
static word vectors i think this was all
2:02:52
static word vectors i think this was all also mentioned earlier today
2:02:53
also mentioned earlier today
2:02:53
also mentioned earlier today um so the way that we obtain word
2:02:56
um so the way that we obtain word
2:02:56
um so the way that we obtain word represent so
2:02:57
represent so
2:02:57
represent so just to mention first that obviously
2:02:58
just to mention first that obviously
2:02:58
just to mention first that obviously words have multiple meanings
2:03:01
words have multiple meanings
2:03:01
words have multiple meanings and the way that we obtain word
2:03:03
and the way that we obtain word
2:03:03
and the way that we obtain word representations today
2:03:06
representations today
2:03:06
representations today reflects that so we use language models
2:03:09
reflects that so we use language models
2:03:09
reflects that so we use language models which
2:03:09
which
2:03:09
which take into account the entire context
2:03:11
take into account the entire context
2:03:11
take into account the entire context that the word is found in
2:03:14
that the word is found in
2:03:14
that the word is found in so for example in a sentence like i'm
2:03:16
so for example in a sentence like i'm
2:03:16
so for example in a sentence like i'm using both the keyboard and the mouse
2:03:18
using both the keyboard and the mouse
2:03:18
using both the keyboard and the mouse the representation would be dynamically
2:03:21
the representation would be dynamically
2:03:21
the representation would be dynamically built for the sentence
2:03:22
built for the sentence
2:03:22
built for the sentence and it would reflect the computer device
2:03:25
and it would reflect the computer device
2:03:25
and it would reflect the computer device sense of mouse while in a sentence like
2:03:29
sense of mouse while in a sentence like
2:03:29
sense of mouse while in a sentence like the chase the cat chased away the mouse
2:03:31
the chase the cat chased away the mouse
2:03:31
the chase the cat chased away the mouse um
2:03:32
um
2:03:32
um it's going to tend more towards the
2:03:35
it's going to tend more towards the
2:03:35
it's going to tend more towards the animal sense
2:03:37
animal sense
2:03:37
animal sense uh this is different from only about two
2:03:39
uh this is different from only about two
2:03:39
uh this is different from only about two years ago where it was pretty customary
2:03:41
years ago where it was pretty customary
2:03:41
years ago where it was pretty customary to use static embeddings
2:03:43
to use static embeddings
2:03:43
to use static embeddings so you would just retrieve the single
2:03:45
so you would just retrieve the single
2:03:45
so you would just retrieve the single emitting
2:03:46
emitting
2:03:46
emitting uh for mouse which would in a weird way
2:03:51
consensus and then it would be up to
2:03:54
consensus and then it would be up to
2:03:54
consensus and then it would be up to your
2:03:55
your
2:03:55
your downstream application to take care of
2:03:57
downstream application to take care of
2:03:57
downstream application to take care of distinguishing these two senses
2:03:59
distinguishing these two senses
2:03:59
distinguishing these two senses if you do that at all so uh finally most
2:04:02
if you do that at all so uh finally most
2:04:02
if you do that at all so uh finally most of nlp research is context sensitive
2:04:04
of nlp research is context sensitive
2:04:04
of nlp research is context sensitive this is really great news
2:04:08
um this is using the transformer
2:04:10
um this is using the transformer
2:04:10
um this is using the transformer architecture
2:04:11
architecture
2:04:11
architecture in most models today but it's not the
2:04:13
in most models today but it's not the
2:04:13
in most models today but it's not the only context-sensitive representation
2:04:15
only context-sensitive representation
2:04:15
only context-sensitive representation but it's the dominant one right now
2:04:17
but it's the dominant one right now
2:04:17
but it's the dominant one right now uh and the way that it works is that i
2:04:19
uh and the way that it works is that i
2:04:19
uh and the way that it works is that i mean largely speaking
2:04:21
mean largely speaking
2:04:21
mean largely speaking uh every item in a sequence is
2:04:23
uh every item in a sequence is
2:04:24
uh every item in a sequence is represented as
2:04:25
represented as
2:04:25
represented as the um deep the average of the deep
2:04:27
the um deep the average of the deep
2:04:27
the um deep the average of the deep representation of
2:04:28
representation of
2:04:28
representation of all the other items in the sequence
2:04:33
uh but modeling context is hardly enough
2:04:35
uh but modeling context is hardly enough
2:04:35
uh but modeling context is hardly enough and there is so much more than
2:04:37
and there is so much more than
2:04:37
and there is so much more than that these text representations don't
2:04:38
that these text representations don't
2:04:38
that these text representations don't really capture today
2:04:40
really capture today
2:04:40
really capture today so um for example things like implicit
2:04:43
so um for example things like implicit
2:04:43
so um for example things like implicit meaning
2:04:44
meaning
2:04:44
meaning if i say that i didn't eat anything
2:04:46
if i say that i didn't eat anything
2:04:46
if i say that i didn't eat anything since the morning then
2:04:47
since the morning then
2:04:47
since the morning then i'm likely meaning i likely mean that
2:04:49
i'm likely meaning i likely mean that
2:04:49
i'm likely meaning i likely mean that i'm hungry
2:04:51
i'm hungry
2:04:51
i'm hungry or broader context like an utterance
2:04:54
or broader context like an utterance
2:04:54
or broader context like an utterance like no
2:04:54
like no
2:04:54
like no thanks on its own it's not very
2:04:56
thanks on its own it's not very
2:04:56
thanks on its own it's not very meaningful but
2:04:58
meaningful but
2:04:58
meaningful but in that context it can mean something
2:04:59
in that context it can mean something
2:04:59
in that context it can mean something like i refuse to offer
2:05:01
like i refuse to offer
2:05:01
like i refuse to offer food or non-literal meaning
2:05:04
food or non-literal meaning
2:05:04
food or non-literal meaning so i could eat a horse doesn't actually
2:05:07
so i could eat a horse doesn't actually
2:05:07
so i could eat a horse doesn't actually mean that i would eat a horse if
2:05:09
mean that i would eat a horse if
2:05:09
mean that i would eat a horse if if i had the opportunity it just means
2:05:11
if i had the opportunity it just means
2:05:11
if i had the opportunity it just means that i'm really hungry
2:05:14
that i'm really hungry
2:05:14
that i'm really hungry programmatics um so things like do you
2:05:16
programmatics um so things like do you
2:05:16
programmatics um so things like do you have some food here is just a polite way
2:05:18
have some food here is just a polite way
2:05:18
have some food here is just a polite way to ask for some
2:05:20
to ask for some
2:05:20
to ask for some and a common background so if i tell you
2:05:23
and a common background so if i tell you
2:05:23
and a common background so if i tell you there's a new restaurant in the city
2:05:24
there's a new restaurant in the city
2:05:24
there's a new restaurant in the city i likely refer to the city we're in now
2:05:28
i likely refer to the city we're in now
2:05:28
i likely refer to the city we're in now and i keep using this example in virtual
2:05:30
and i keep using this example in virtual
2:05:30
and i keep using this example in virtual conferences i realize it's a bit strange
2:05:32
conferences i realize it's a bit strange
2:05:32
conferences i realize it's a bit strange but
2:05:33
but
2:05:33
but yeah uh yeah this gets even more
2:05:35
yeah uh yeah this gets even more
2:05:35
yeah uh yeah this gets even more complicated because
2:05:36
complicated because
2:05:36
complicated because in the us depending on where you are or
2:05:38
in the us depending on where you are or
2:05:38
in the us depending on where you are or where you're from
2:05:39
where you're from
2:05:39
where you're from if you say the city there is a whole
2:05:41
if you say the city there is a whole
2:05:41
if you say the city there is a whole convention of
2:05:42
convention of
2:05:42
convention of uh which city you refer to it doesn't
2:05:45
uh which city you refer to it doesn't
2:05:45
uh which city you refer to it doesn't even even have to be the city you're in
2:05:47
even even have to be the city you're in
2:05:47
even even have to be the city you're in now
2:05:48
now
2:05:48
now and all these complicated things are not
2:05:51
and all these complicated things are not
2:05:51
and all these complicated things are not really represented very well on
2:05:53
really represented very well on
2:05:53
really represented very well on in these representations which hardly
2:05:55
in these representations which hardly
2:05:55
in these representations which hardly capture
2:05:56
capture
2:05:56
capture any meaning under the surface
2:06:00
do these text representations have
2:06:01
do these text representations have
2:06:01
do these text representations have common sense knowledge
2:06:03
common sense knowledge
2:06:03
common sense knowledge so the answer here is mixed so
2:06:06
so the answer here is mixed so
2:06:06
so the answer here is mixed so on the one hand it's possible to extract
2:06:08
on the one hand it's possible to extract
2:06:08
on the one hand it's possible to extract knowledge from language models
2:06:10
knowledge from language models
2:06:10
knowledge from language models so um for example if you provide it with
2:06:14
so um for example if you provide it with
2:06:14
so um for example if you provide it with facts or candidate facts you can uh
2:06:18
facts or candidate facts you can uh
2:06:18
facts or candidate facts you can uh score them to see which ones of them are
2:06:20
score them to see which ones of them are
2:06:20
score them to see which ones of them are more plausible
2:06:21
more plausible
2:06:21
more plausible uh or more more likely to be correct
2:06:23
uh or more more likely to be correct
2:06:23
uh or more more likely to be correct than others
2:06:25
than others
2:06:25
than others or um they're they're also capable of
2:06:28
or um they're they're also capable of
2:06:28
or um they're they're also capable of associating
2:06:29
associating
2:06:29
associating concepts with their prominent properties
2:06:31
concepts with their prominent properties
2:06:31
concepts with their prominent properties so for example a lion has fur is big and
2:06:33
so for example a lion has fur is big and
2:06:33
so for example a lion has fur is big and has
2:06:34
has
2:06:34
has has claws and um they're
2:06:37
has claws and um they're
2:06:37
has claws and um they're pretty good at generating definitions
2:06:40
pretty good at generating definitions
2:06:40
pretty good at generating definitions and properties of concepts
2:06:45
and properties of concepts
2:06:45
and properties of concepts on the other hand the knowledge that
2:06:47
on the other hand the knowledge that
2:06:47
on the other hand the knowledge that they
2:06:48
they
2:06:48
they that they generate is not very reliable
2:06:50
that they generate is not very reliable
2:06:50
that they generate is not very reliable it's it's not very accurate
2:06:53
it's it's not very accurate
2:06:53
it's it's not very accurate and there's a whole line of work right
2:06:55
and there's a whole line of work right
2:06:55
and there's a whole line of work right now showing all the limitations of these
2:06:56
now showing all the limitations of these
2:06:56
now showing all the limitations of these models
2:06:57
models
2:06:57
models so for example they are not very
2:07:00
so for example they are not very
2:07:00
so for example they are not very sensitive to negations so they can
2:07:04
sensitive to negations so they can
2:07:04
sensitive to negations so they can score a fact like birds can fly and
2:07:06
score a fact like birds can fly and
2:07:06
score a fact like birds can fly and birds can't fly
2:07:07
birds can't fly
2:07:07
birds can't fly almost equally likely they also have
2:07:11
almost equally likely they also have
2:07:11
almost equally likely they also have these really weird generalizations so
2:07:13
these really weird generalizations so
2:07:13
these really weird generalizations so for example
2:07:14
for example
2:07:14
for example they uh can generate something like
2:07:16
they uh can generate something like
2:07:16
they uh can generate something like barack's wife hillary because
2:07:18
barack's wife hillary because
2:07:18
barack's wife hillary because they just know that these two names are
2:07:20
they just know that these two names are
2:07:20
they just know that these two names are somehow
2:07:21
somehow
2:07:21
somehow semantically similar but they don't
2:07:23
semantically similar but they don't
2:07:23
semantically similar but they don't really know how to express their
2:07:25
really know how to express their
2:07:25
really know how to express their semantic relation between them and this
2:07:28
semantic relation between them and this
2:07:28
semantic relation between them and this is a similar phenomenon so
2:07:30
is a similar phenomenon so
2:07:30
is a similar phenomenon so in certain constructs they know that
2:07:33
in certain constructs they know that
2:07:33
in certain constructs they know that they should predict the color for
2:07:35
they should predict the color for
2:07:35
they should predict the color for example so that something banana is
2:07:37
example so that something banana is
2:07:37
example so that something banana is tasty
2:07:38
tasty
2:07:38
tasty but they don't necessarily know what's
2:07:40
but they don't necessarily know what's
2:07:40
but they don't necessarily know what's the color of a banana so they just
2:07:42
the color of a banana so they just
2:07:42
the color of a banana so they just output different colors
2:07:45
output different colors
2:07:45
output different colors and uh they also they are also sensitive
2:07:47
and uh they also they are also sensitive
2:07:48
and uh they also they are also sensitive to the phrasing so they would provide
2:07:49
to the phrasing so they would provide
2:07:49
to the phrasing so they would provide different answers if you
2:07:51
different answers if you
2:07:51
different answers if you try to complete a sentence like the sky
2:07:54
try to complete a sentence like the sky
2:07:54
try to complete a sentence like the sky is something today
2:07:56
is something today
2:07:56
is something today as opposed to today the sky is something
2:08:00
so the representations lack sufficient
2:08:02
so the representations lack sufficient
2:08:02
so the representations lack sufficient common sense
2:08:05
common sense
2:08:05
common sense and the root of the problem is in
2:08:07
and the root of the problem is in
2:08:07
and the root of the problem is in reporting bias which
2:08:08
reporting bias which
2:08:08
reporting bias which is or at least some some of the uh
2:08:11
is or at least some some of the uh
2:08:11
is or at least some some of the uh root of the problem of the problem is in
2:08:13
root of the problem of the problem is in
2:08:13
root of the problem of the problem is in reporting math
2:08:15
reporting math
2:08:15
reporting math uh where um the problem is that the
2:08:18
uh where um the problem is that the
2:08:18
uh where um the problem is that the we tend to talk more about the
2:08:20
we tend to talk more about the
2:08:20
we tend to talk more about the sensational and noteworthy things than
2:08:22
sensational and noteworthy things than
2:08:22
sensational and noteworthy things than the trivial things
2:08:23
the trivial things
2:08:23
the trivial things so um text corpora hardly um
2:08:27
so um text corpora hardly um
2:08:27
so um text corpora hardly um contain any very trivial things and then
2:08:31
contain any very trivial things and then
2:08:31
contain any very trivial things and then it has been shown like a few years back
2:08:33
it has been shown like a few years back
2:08:33
it has been shown like a few years back that if you look at the frequency of
2:08:35
that if you look at the frequency of
2:08:35
that if you look at the frequency of people's actions in text
2:08:37
people's actions in text
2:08:37
people's actions in text you might conclude that people murder
2:08:39
you might conclude that people murder
2:08:39
you might conclude that people murder more than they breathe
2:08:41
more than they breathe
2:08:41
more than they breathe the reason that this happens is because
2:08:43
the reason that this happens is because
2:08:43
the reason that this happens is because you wouldn't find reports
2:08:45
you wouldn't find reports
2:08:45
you wouldn't find reports or news reports or any discussion on the
2:08:49
or news reports or any discussion on the
2:08:49
or news reports or any discussion on the internet on
2:08:50
internet on
2:08:50
internet on people breathing and
2:08:53
people breathing and
2:08:53
people breathing and language models um they don't really
2:08:56
language models um they don't really
2:08:56
language models um they don't really solve this problem completely and in
2:08:58
solve this problem completely and in
2:08:58
solve this problem completely and in some ways they even amplify it
2:09:02
so the way forward might be to learn
2:09:03
so the way forward might be to learn
2:09:03
so the way forward might be to learn from additional modalities not just
2:09:05
from additional modalities not just
2:09:05
from additional modalities not just from text but also from vision or from
2:09:08
from text but also from vision or from
2:09:08
from text but also from vision or from images or from videos
2:09:10
images or from videos
2:09:10
images or from videos so for example aggregating across
2:09:13
so for example aggregating across
2:09:13
so for example aggregating across similar
2:09:14
similar
2:09:14
similar images you might learn that in a class
2:09:16
images you might learn that in a class
2:09:16
images you might learn that in a class photo if someone is in the front
2:09:18
photo if someone is in the front
2:09:18
photo if someone is in the front row then they are likely seated maybe
2:09:21
row then they are likely seated maybe
2:09:21
row then they are likely seated maybe cross-legged
2:09:23
cross-legged
2:09:23
cross-legged while if they are on the last row then
2:09:25
while if they are on the last row then
2:09:25
while if they are on the last row then they are likely standing otherwise they
2:09:27
they are likely standing otherwise they
2:09:27
they are likely standing otherwise they wouldn't be seen
2:09:30
so let's to conclude let's go over the
2:09:32
so let's to conclude let's go over the
2:09:32
so let's to conclude let's go over the checklist of what we want our models to
2:09:34
checklist of what we want our models to
2:09:34
checklist of what we want our models to do
2:09:34
do
2:09:34
do so we want them to seamlessly learn
2:09:36
so we want them to seamlessly learn
2:09:36
so we want them to seamlessly learn input output mappings
2:09:38
input output mappings
2:09:38
input output mappings to uh we don't want to train them from
2:09:39
to uh we don't want to train them from
2:09:40
to uh we don't want to train them from scratch every time and we want them to
2:09:41
scratch every time and we want them to
2:09:41
scratch every time and we want them to be context sensitive
2:09:44
be context sensitive
2:09:44
be context sensitive this is pretty much solved but we also
2:09:46
this is pretty much solved but we also
2:09:46
this is pretty much solved but we also want them to use
2:09:47
want them to use
2:09:47
want them to use fewer training examples to generalize
2:09:50
fewer training examples to generalize
2:09:50
fewer training examples to generalize outside our training domain
2:09:52
outside our training domain
2:09:52
outside our training domain to be smaller to solve the task instead
2:09:54
to be smaller to solve the task instead
2:09:54
to be smaller to solve the task instead of learning data specific shortcuts
2:09:56
of learning data specific shortcuts
2:09:56
of learning data specific shortcuts to be interpretable not to perpetuate
2:10:00
to be interpretable not to perpetuate
2:10:00
to be interpretable not to perpetuate social biases to capture meaning under
2:10:03
social biases to capture meaning under
2:10:03
social biases to capture meaning under the surface
2:10:04
the surface
2:10:04
the surface to capture accurate common sense
2:10:06
to capture accurate common sense
2:10:06
to capture accurate common sense knowledge yeah that's it and
2:10:08
knowledge yeah that's it and
2:10:08
knowledge yeah that's it and um this is pretty much all uh still work
2:10:11
um this is pretty much all uh still work
2:10:11
um this is pretty much all uh still work in progress
2:10:12
in progress
2:10:12
in progress thank you oh wow
2:10:16
thank you oh wow
2:10:16
thank you oh wow that was amazing varied um and also that
2:10:19
that was amazing varied um and also that
2:10:19
that was amazing varied um and also that sort of end
2:10:20
sort of end
2:10:20
sort of end checklist looks a bit like my general
2:10:23
checklist looks a bit like my general
2:10:23
checklist looks a bit like my general to-do list like it always feels like
2:10:24
to-do list like it always feels like
2:10:24
to-do list like it always feels like you're just doing the top few and then
2:10:26
you're just doing the top few and then
2:10:26
you're just doing the top few and then you're like oh there's so much
2:10:27
you're like oh there's so much
2:10:28
you're like oh there's so much more we need to do in this space um so
2:10:30
more we need to do in this space um so
2:10:30
more we need to do in this space um so thanks
2:10:31
thanks
2:10:31
thanks thank you so much for sharing that um we
2:10:34
thank you so much for sharing that um we
2:10:34
thank you so much for sharing that um we we actually have some really great
2:10:36
we actually have some really great
2:10:36
we actually have some really great questions in the chat but
2:10:37
questions in the chat but
2:10:38
questions in the chat but um please hold those as a thought and
2:10:40
um please hold those as a thought and
2:10:40
um please hold those as a thought and what we're going to do is we're going to
2:10:43
what we're going to do is we're going to
2:10:43
what we're going to do is we're going to go through our last and final talk and
2:10:45
go through our last and final talk and
2:10:45
go through our last and final talk and then we'll bring back
2:10:46
then we'll bring back
2:10:46
then we'll bring back varied along with our next speaker and
2:10:48
varied along with our next speaker and
2:10:48
varied along with our next speaker and we'll have a really really good
2:10:49
we'll have a really really good
2:10:50
we'll have a really really good conversation
2:10:50
conversation
2:10:50
conversation and hopefully get all those questions
2:10:52
and hopefully get all those questions
2:10:52
and hopefully get all those questions answered so thank you so much foreign if
2:10:54
answered so thank you so much foreign if
2:10:54
answered so thank you so much foreign if you can hang around for a little bit
2:10:56
you can hang around for a little bit
2:10:56
you can hang around for a little bit longer with us
2:10:57
longer with us
2:10:57
longer with us um we're very excited to speak to you
2:10:59
um we're very excited to speak to you
2:11:00
um we're very excited to speak to you again
2:11:01
again
2:11:01
again thanks amazing so next
2:11:04
thanks amazing so next
2:11:04
thanks amazing so next person up is uh inez montani
2:11:07
person up is uh inez montani
2:11:07
person up is uh inez montani some of you may know her as the founder
2:11:09
some of you may know her as the founder
2:11:09
some of you may know her as the founder of explosion and also one of the core
2:11:11
of explosion and also one of the core
2:11:11
of explosion and also one of the core contributors to spacey
2:11:13
contributors to spacey
2:11:13
contributors to spacey and which is an incredibly popular
2:11:16
and which is an incredibly popular
2:11:16
and which is an incredibly popular package to use
2:11:17
package to use
2:11:17
package to use when looking at natural language
2:11:18
when looking at natural language
2:11:18
when looking at natural language processing so
2:11:20
processing so
2:11:20
processing so uh welcome vinas hey
2:11:25
uh welcome vinas hey
2:11:25
uh welcome vinas hey can you hear us okay i think i think i
2:11:26
can you hear us okay i think i think i
2:11:26
can you hear us okay i think i think i was muted but i think i can
2:11:28
was muted but i think i can
2:11:28
was muted but i think i can um now we're getting really loud and
2:11:30
um now we're getting really loud and
2:11:30
um now we're getting really loud and clear i was gonna say
2:11:32
clear i was gonna say
2:11:32
clear i was gonna say it is not a conference call or a online
2:11:35
it is not a conference call or a online
2:11:35
it is not a conference call or a online event unless we go you're muted
2:11:36
event unless we go you're muted
2:11:36
event unless we go you're muted can you on me no also people always
2:11:39
can you on me no also people always
2:11:39
can you on me no also people always assume that because i live in a big city
2:11:41
assume that because i live in a big city
2:11:41
assume that because i live in a big city i have a great internet connection which
2:11:43
i have a great internet connection which
2:11:43
i have a great internet connection which is
2:11:43
is
2:11:43
is absolutely not the case in germany
2:11:46
absolutely not the case in germany
2:11:46
absolutely not the case in germany it's it's so hit and miss isn't it and
2:11:48
it's it's so hit and miss isn't it and
2:11:48
it's it's so hit and miss isn't it and also i find
2:11:49
also i find
2:11:49
also i find times of the day someone in the comments
2:11:52
times of the day someone in the comments
2:11:52
times of the day someone in the comments let me know if this is actually true of
2:11:54
let me know if this is actually true of
2:11:54
let me know if this is actually true of my wi-fi but
2:11:55
my wi-fi but
2:11:55
my wi-fi but i feel like the minute people like stop
2:11:58
i feel like the minute people like stop
2:11:58
i feel like the minute people like stop work which tends to be i
2:11:59
work which tends to be i
2:11:59
work which tends to be i work a lot with the us and being from
2:12:01
work a lot with the us and being from
2:12:01
work a lot with the us and being from the uk that means my evenings are quite
2:12:03
the uk that means my evenings are quite
2:12:03
the uk that means my evenings are quite full
2:12:03
full
2:12:03
full uh i'm exactly the same yeah it's so
2:12:07
uh i'm exactly the same yeah it's so
2:12:07
uh i'm exactly the same yeah it's so bad isn't it netflix time right now so
2:12:10
bad isn't it netflix time right now so
2:12:10
bad isn't it netflix time right now so everyone is consuming the network
2:12:14
everyone is consuming the network
2:12:14
everyone is consuming the network i honestly think that's what it is yeah
2:12:17
i honestly think that's what it is yeah
2:12:17
i honestly think that's what it is yeah fantastic well inez can you
2:12:19
fantastic well inez can you
2:12:19
fantastic well inez can you introduce yourself i don't think i did
2:12:20
introduce yourself i don't think i did
2:12:20
introduce yourself i don't think i did you justice so if you can just tell us a
2:12:22
you justice so if you can just tell us a
2:12:22
you justice so if you can just tell us a little bit about yourself
2:12:24
little bit about yourself
2:12:24
little bit about yourself oh i mean i think you did i really like
2:12:26
oh i mean i think you did i really like
2:12:26
oh i mean i think you did i really like the introduction because i usually start
2:12:27
the introduction because i usually start
2:12:27
the introduction because i usually start my introduction with like hi
2:12:29
my introduction with like hi
2:12:29
my introduction with like hi i mean as some of you might know me from
2:12:31
i mean as some of you might know me from
2:12:31
i mean as some of you might know me from my work on spacey
2:12:34
my work on spacey
2:12:34
my work on spacey so that was actually that's exactly what
2:12:35
so that was actually that's exactly what
2:12:35
so that was actually that's exactly what i would have said so uh but yeah i'm
2:12:37
i would have said so uh but yeah i'm
2:12:37
i would have said so uh but yeah i'm i'm the co-founder of a company called
2:12:39
i'm the co-founder of a company called
2:12:39
i'm the co-founder of a company called explosion and uh we specialize in
2:12:41
explosion and uh we specialize in
2:12:41
explosion and uh we specialize in developer tools for
2:12:42
developer tools for
2:12:42
developer tools for machine learning and nlp and yeah spacey
2:12:45
machine learning and nlp and yeah spacey
2:12:45
machine learning and nlp and yeah spacey is probably our most
2:12:47
is probably our most
2:12:47
is probably our most popular open source library but we also
2:12:49
popular open source library but we also
2:12:49
popular open source library but we also do a bunch of other stuff we publish
2:12:51
do a bunch of other stuff we publish
2:12:51
do a bunch of other stuff we publish um a data annotation tool called prodigy
2:12:54
um a data annotation tool called prodigy
2:12:54
um a data annotation tool called prodigy which helps users really create their
2:12:55
which helps users really create their
2:12:55
which helps users really create their own data sets and also you know
2:12:57
own data sets and also you know
2:12:57
own data sets and also you know work against some of the problems that
2:12:59
work against some of the problems that
2:12:59
work against some of the problems that we also just heard about
2:13:01
we also just heard about
2:13:01
we also just heard about in the previous talk um so yeah that's
2:13:03
in the previous talk um so yeah that's
2:13:03
in the previous talk um so yeah that's what we do and we just released a new
2:13:05
what we do and we just released a new
2:13:05
what we do and we just released a new version of
2:13:05
version of
2:13:05
version of spacey well the pre-release um a nightly
2:13:07
spacey well the pre-release um a nightly
2:13:08
spacey well the pre-release um a nightly version people can try out so that's
2:13:09
version people can try out so that's
2:13:09
version people can try out so that's also what i'm going to be
2:13:10
also what i'm going to be
2:13:10
also what i'm going to be talking about today um and it's like the
2:13:13
talking about today um and it's like the
2:13:13
talking about today um and it's like the first time i'm talking about
2:13:14
first time i'm talking about
2:13:14
first time i'm talking about this at any conference so this is quite
2:13:16
this at any conference so this is quite
2:13:16
this at any conference so this is quite special
2:13:17
special
2:13:18
special oh my goodness we have an explanation
2:13:20
oh my goodness we have an explanation
2:13:20
oh my goodness we have an explanation [Laughter]
2:13:22
[Laughter]
2:13:22
[Laughter] we need to be fair with the speakers
2:13:25
we need to be fair with the speakers
2:13:25
we need to be fair with the speakers with the speakers with
2:13:26
with the speakers with
2:13:26
with the speakers with the viewers in as her session is
2:13:29
the viewers in as her session is
2:13:29
the viewers in as her session is pre-recorded because of the network that
2:13:31
pre-recorded because of the network that
2:13:31
pre-recorded because of the network that is not always so stable
2:13:33
is not always so stable
2:13:33
is not always so stable um so let's uh i think let's have a look
2:13:36
um so let's uh i think let's have a look
2:13:36
um so let's uh i think let's have a look at it
2:13:36
at it
2:13:36
at it right hi i'm ines
2:13:40
right hi i'm ines
2:13:40
right hi i'm ines i'm the co-founder of explosion and a
2:13:41
i'm the co-founder of explosion and a
2:13:41
i'm the co-founder of explosion and a co-developer of spacey a popular open
2:13:43
co-developer of spacey a popular open
2:13:43
co-developer of spacey a popular open source library for natural language
2:13:45
source library for natural language
2:13:45
source library for natural language processing in python
2:13:46
processing in python
2:13:46
processing in python spacey helps you build advanced natural
2:13:48
spacey helps you build advanced natural
2:13:48
spacey helps you build advanced natural language understanding pipelines using
2:13:50
language understanding pipelines using
2:13:50
language understanding pipelines using the latest state-of-the-art machine
2:13:51
the latest state-of-the-art machine
2:13:51
the latest state-of-the-art machine learning techniques
2:13:52
learning techniques
2:13:52
learning techniques as well as fully custom and rule-based
2:13:54
as well as fully custom and rule-based
2:13:54
as well as fully custom and rule-based approaches for uofa specific use case
2:13:58
approaches for uofa specific use case
2:13:58
approaches for uofa specific use case spacey has come a long way since its
2:13:59
spacey has come a long way since its
2:13:59
spacey has come a long way since its first release in 2015.
2:14:01
first release in 2015.
2:14:02
first release in 2015. it's been downloaded over 20 million
2:14:03
it's been downloaded over 20 million
2:14:03
it's been downloaded over 20 million times with a large community and
2:14:05
times with a large community and
2:14:05
times with a large community and ecosystem
2:14:06
ecosystem
2:14:06
ecosystem what makes spacey special is that it was
2:14:08
what makes spacey special is that it was
2:14:08
what makes spacey special is that it was designed for production use from day one
2:14:11
designed for production use from day one
2:14:11
designed for production use from day one this also means that we think about a
2:14:13
this also means that we think about a
2:14:13
this also means that we think about a lot of things that aren't really
2:14:14
lot of things that aren't really
2:14:14
lot of things that aren't really relevant to experiment code
2:14:16
relevant to experiment code
2:14:16
relevant to experiment code like we make sure the library is easy to
2:14:19
like we make sure the library is easy to
2:14:19
like we make sure the library is easy to deploy and practical to run on lots of
2:14:21
deploy and practical to run on lots of
2:14:21
deploy and practical to run on lots of text
2:14:21
text
2:14:21
text and we also focus on the developer
2:14:23
and we also focus on the developer
2:14:23
and we also focus on the developer experience like getting the right
2:14:25
experience like getting the right
2:14:25
experience like getting the right conceptual model behind the api
2:14:27
conceptual model behind the api
2:14:27
conceptual model behind the api consistent naming good error handling
2:14:29
consistent naming good error handling
2:14:29
consistent naming good error handling and great documentation
2:14:33
version 3 of spacey brings a whole new
2:14:35
version 3 of spacey brings a whole new
2:14:35
version 3 of spacey brings a whole new level of this
2:14:36
level of this
2:14:36
level of this new transformer-based pipelines that get
2:14:38
new transformer-based pipelines that get
2:14:38
new transformer-based pipelines that get spaces accuracy right up to the current
2:14:40
spaces accuracy right up to the current
2:14:40
spaces accuracy right up to the current state of the art
2:14:41
state of the art
2:14:41
state of the art training is now fully configurable from
2:14:43
training is now fully configurable from
2:14:43
training is now fully configurable from start to finish
2:14:44
start to finish
2:14:44
start to finish and the new spacey projects let you
2:14:46
and the new spacey projects let you
2:14:46
and the new spacey projects let you manage end-to-end workflows from
2:14:47
manage end-to-end workflows from
2:14:47
manage end-to-end workflows from prototype to production
2:14:52
transformer models are a family of
2:14:54
transformer models are a family of
2:14:54
transformer models are a family of neural network architectures
2:14:56
neural network architectures
2:14:56
neural network architectures that have proven very successful for nlp
2:14:58
that have proven very successful for nlp
2:14:58
that have proven very successful for nlp especially when used with
2:15:00
especially when used with
2:15:00
especially when used with language model pre-training the most
2:15:02
language model pre-training the most
2:15:02
language model pre-training the most important advantage of transformer
2:15:03
important advantage of transformer
2:15:03
important advantage of transformer models is that they scale up better
2:15:06
models is that they scale up better
2:15:06
models is that they scale up better as you add more parameters transformers
2:15:09
as you add more parameters transformers
2:15:09
as you add more parameters transformers keep improving more steadily than
2:15:11
keep improving more steadily than
2:15:11
keep improving more steadily than earlier alternatives like the cnn or
2:15:13
earlier alternatives like the cnn or
2:15:13
earlier alternatives like the cnn or lscm
2:15:14
lscm
2:15:14
lscm transformers also use gpu architectures
2:15:16
transformers also use gpu architectures
2:15:16
transformers also use gpu architectures more efficiently and gpus have
2:15:19
more efficiently and gpus have
2:15:19
more efficiently and gpus have been getting more powerful while cpus
2:15:21
been getting more powerful while cpus
2:15:21
been getting more powerful while cpus have been kind of stagnant
2:15:23
have been kind of stagnant
2:15:23
have been kind of stagnant so while we keep getting new boundary
2:15:26
so while we keep getting new boundary
2:15:26
so while we keep getting new boundary pushing transformer models that are a
2:15:28
pushing transformer models that are a
2:15:28
pushing transformer models that are a bit too big for many practical purposes
2:15:30
bit too big for many practical purposes
2:15:30
bit too big for many practical purposes like nvidia's megatron openai's gpt3
2:15:34
like nvidia's megatron openai's gpt3
2:15:34
like nvidia's megatron openai's gpt3 models like bird base that felt quite
2:15:37
models like bird base that felt quite
2:15:37
models like bird base that felt quite extravagant
2:15:38
extravagant
2:15:38
extravagant a few years ago now feel pretty
2:15:41
a few years ago now feel pretty
2:15:41
a few years ago now feel pretty practical given how gpu prices have been
2:15:43
practical given how gpu prices have been
2:15:43
practical given how gpu prices have been falling
2:15:45
falling
2:15:45
falling so with all that context in mind we're
2:15:48
so with all that context in mind we're
2:15:48
so with all that context in mind we're pleased to introduce a new family of
2:15:50
pleased to introduce a new family of
2:15:50
pleased to introduce a new family of trained pipelines based on transformer
2:15:52
trained pipelines based on transformer
2:15:52
trained pipelines based on transformer architectures
2:15:54
architectures
2:15:54
architectures the pipelines run best on gpu and you'll
2:15:56
the pipelines run best on gpu and you'll
2:15:56
the pipelines run best on gpu and you'll definitely want a
2:15:57
definitely want a
2:15:57
definitely want a gpu for training but you can also deploy
2:16:00
gpu for training but you can also deploy
2:16:00
gpu for training but you can also deploy them on cpu
2:16:02
them on cpu
2:16:02
them on cpu under the hood the transformer-based
2:16:03
under the hood the transformer-based
2:16:03
under the hood the transformer-based pipelines use the hugging phase
2:16:04
pipelines use the hugging phase
2:16:04
pipelines use the hugging phase transformers library
2:16:06
transformers library
2:16:06
transformers library and pytorch so you can use any pie
2:16:08
and pytorch so you can use any pie
2:16:08
and pytorch so you can use any pie touch-based model
2:16:09
touch-based model
2:16:09
touch-based model that having face published very easily
2:16:11
that having face published very easily
2:16:11
that having face published very easily and out of the box
2:16:17
for the english transformer pipeline we
2:16:19
for the english transformer pipeline we
2:16:19
for the english transformer pipeline we used the roberta based model that was
2:16:20
used the roberta based model that was
2:16:20
used the roberta based model that was published by researchers at facebook
2:16:22
published by researchers at facebook
2:16:22
published by researchers at facebook shortly after the seminal bird paper by
2:16:24
shortly after the seminal bird paper by
2:16:24
shortly after the seminal bird paper by devlin at all which showed the success
2:16:27
devlin at all which showed the success
2:16:27
devlin at all which showed the success of language model pre-training for large
2:16:28
of language model pre-training for large
2:16:28
of language model pre-training for large transformers
2:16:30
transformers
2:16:30
transformers we fine-tuned the roberto-based
2:16:31
we fine-tuned the roberto-based
2:16:31
we fine-tuned the roberto-based transformer on
2:16:33
transformer on
2:16:33
transformer on for all of the tasks in the pipeline
2:16:35
for all of the tasks in the pipeline
2:16:35
for all of the tasks in the pipeline like that includes dependency parsing
2:16:37
like that includes dependency parsing
2:16:37
like that includes dependency parsing part of speech tagging and named entity
2:16:39
part of speech tagging and named entity
2:16:39
part of speech tagging and named entity recognition
2:16:42
we're really happy with the accuracies
2:16:44
we're really happy with the accuracies
2:16:44
we're really happy with the accuracies we've been able to achieve so far with
2:16:45
we've been able to achieve so far with
2:16:45
we've been able to achieve so far with the transformer based pipelines
2:16:47
the transformer based pipelines
2:16:47
the transformer based pipelines using this approach spacey's ner
2:16:49
using this approach spacey's ner
2:16:49
using this approach spacey's ner accuracy is now right up there with the
2:16:51
accuracy is now right up there with the
2:16:51
accuracy is now right up there with the current
2:16:52
current
2:16:52
current state of the art and the other
2:16:54
state of the art and the other
2:16:54
state of the art and the other components also get accuracies that are
2:16:56
components also get accuracies that are
2:16:56
components also get accuracies that are up to date
2:16:56
up to date
2:16:56
up to date with modern systems as well
2:17:01
with modern systems as well
2:17:01
with modern systems as well there are definitely trade-offs with the
2:17:03
there are definitely trade-offs with the
2:17:03
there are definitely trade-offs with the transformer-based pipeline store
2:17:05
transformer-based pipeline store
2:17:05
transformer-based pipeline store there are more dependencies gpus are
2:17:08
there are more dependencies gpus are
2:17:08
there are more dependencies gpus are more expensive
2:17:09
more expensive
2:17:09
more expensive and less reliable and latency can also
2:17:11
and less reliable and latency can also
2:17:11
and less reliable and latency can also be a problem on the other hand
2:17:13
be a problem on the other hand
2:17:13
be a problem on the other hand um the system does make 30 fewer errors
2:17:16
um the system does make 30 fewer errors
2:17:16
um the system does make 30 fewer errors across these tasks which can definitely
2:17:17
across these tasks which can definitely
2:17:17
across these tasks which can definitely be important
2:17:19
be important
2:17:19
be important so if you find yourself um working on
2:17:21
so if you find yourself um working on
2:17:21
so if you find yourself um working on something and accuracy really is the
2:17:23
something and accuracy really is the
2:17:23
something and accuracy really is the main blocker for you
2:17:24
main blocker for you
2:17:24
main blocker for you you should definitely consider trying
2:17:25
you should definitely consider trying
2:17:25
you should definitely consider trying out the transformer
2:17:31
the transformer support in space e3 is
2:17:33
the transformer support in space e3 is
2:17:33
the transformer support in space e3 is also very flexible so you can build your
2:17:35
also very flexible so you can build your
2:17:35
also very flexible so you can build your own solutions
2:17:36
own solutions
2:17:36
own solutions you're not limited to just the pipelines
2:17:38
you're not limited to just the pipelines
2:17:38
you're not limited to just the pipelines that we've trained and distributed
2:17:40
that we've trained and distributed
2:17:40
that we've trained and distributed and probably the feature that's most
2:17:42
and probably the feature that's most
2:17:42
and probably the feature that's most important is the ability to put the
2:17:44
important is the ability to put the
2:17:44
important is the ability to put the transformer weights into their own
2:17:46
transformer weights into their own
2:17:46
transformer weights into their own component
2:17:47
component
2:17:47
component which other components in your pipeline
2:17:49
which other components in your pipeline
2:17:49
which other components in your pipeline will then be able to connect to
2:17:52
will then be able to connect to
2:17:52
will then be able to connect to this makes it really easy to do
2:17:53
this makes it really easy to do
2:17:53
this makes it really easy to do multitask learning and it's how we have
2:17:55
multitask learning and it's how we have
2:17:55
multitask learning and it's how we have one transformer powering the tagger
2:17:57
one transformer powering the tagger
2:17:57
one transformer powering the tagger parser
2:17:57
parser
2:17:58
parser and ner all at once
2:18:01
and ner all at once
2:18:01
and ner all at once it makes a big difference for your
2:18:02
it makes a big difference for your
2:18:02
it makes a big difference for your performance because the transformer only
2:18:04
performance because the transformer only
2:18:04
performance because the transformer only has to run once
2:18:05
has to run once
2:18:05
has to run once even if your pipeline consists of a lot
2:18:07
even if your pipeline consists of a lot
2:18:07
even if your pipeline consists of a lot of separate components
2:18:11
of separate components
2:18:11
of separate components we've also made sure that the
2:18:12
we've also made sure that the
2:18:12
we've also made sure that the transformer-based pipelines behave
2:18:14
transformer-based pipelines behave
2:18:14
transformer-based pipelines behave just the way you'd expect other spacey
2:18:15
just the way you'd expect other spacey
2:18:16
just the way you'd expect other spacey pipelines to work when it comes to
2:18:17
pipelines to work when it comes to
2:18:17
pipelines to work when it comes to things like saving
2:18:18
things like saving
2:18:18
things like saving loading and installing the train
2:18:20
loading and installing the train
2:18:20
loading and installing the train pipeline for example
2:18:21
pipeline for example
2:18:21
pipeline for example the transformer-based pipelines saves
2:18:23
the transformer-based pipelines saves
2:18:23
the transformer-based pipelines saves out the fine-tuned weights
2:18:25
out the fine-tuned weights
2:18:25
out the fine-tuned weights into the directory for you so the
2:18:27
into the directory for you so the
2:18:27
into the directory for you so the directory is fully self-contained
2:18:29
directory is fully self-contained
2:18:29
directory is fully self-contained you can test your artifacts and when
2:18:31
you can test your artifacts and when
2:18:31
you can test your artifacts and when you're deploying the model you don't
2:18:32
you're deploying the model you don't
2:18:32
you're deploying the model you don't have to worry about your workers
2:18:33
have to worry about your workers
2:18:33
have to worry about your workers downloading things at runtime
2:18:35
downloading things at runtime
2:18:35
downloading things at runtime and if you you package the model using
2:18:37
and if you you package the model using
2:18:37
and if you you package the model using the spacey package command
2:18:39
the spacey package command
2:18:39
the spacey package command it can be installed via pip and the
2:18:41
it can be installed via pip and the
2:18:41
it can be installed via pip and the requirements will all be specified for
2:18:43
requirements will all be specified for
2:18:43
requirements will all be specified for you
2:18:43
you
2:18:43
you making installation really really easy
2:18:50
transformer based pipelines are really
2:18:52
transformer based pipelines are really
2:18:52
transformer based pipelines are really an example of a relatively complex way
2:18:54
an example of a relatively complex way
2:18:54
an example of a relatively complex way to arrange your pipeline
2:18:56
to arrange your pipeline
2:18:56
to arrange your pipeline the transformer itself introduces a
2:18:59
the transformer itself introduces a
2:18:59
the transformer itself introduces a number of options and then you have the
2:19:01
number of options and then you have the
2:19:01
number of options and then you have the other components that need to connect to
2:19:03
other components that need to connect to
2:19:03
other components that need to connect to the transformer
2:19:04
the transformer
2:19:04
the transformer and set up their other options and
2:19:07
and set up their other options and
2:19:07
and set up their other options and there are many ways for this
2:19:08
there are many ways for this
2:19:08
there are many ways for this configuration to be wrong resulting in a
2:19:10
configuration to be wrong resulting in a
2:19:10
configuration to be wrong resulting in a pipeline that can't run
2:19:12
pipeline that can't run
2:19:12
pipeline that can't run or won't learn anything useful and
2:19:15
or won't learn anything useful and
2:19:15
or won't learn anything useful and the fact is this is already standard for
2:19:17
the fact is this is already standard for
2:19:17
the fact is this is already standard for machine learning of course spacey is
2:19:18
machine learning of course spacey is
2:19:18
machine learning of course spacey is definitely not the only
2:19:19
definitely not the only
2:19:20
definitely not the only library that has to think about this
2:19:21
library that has to think about this
2:19:21
library that has to think about this every machine learning system is sitting
2:19:23
every machine learning system is sitting
2:19:23
every machine learning system is sitting on a big pile of configuration
2:19:26
on a big pile of configuration
2:19:26
on a big pile of configuration always one flag away from blowing up and
2:19:29
always one flag away from blowing up and
2:19:29
always one flag away from blowing up and ruining everything
2:19:30
ruining everything
2:19:30
ruining everything and the problem is pretty fundamental
2:19:33
and the problem is pretty fundamental
2:19:33
and the problem is pretty fundamental it's what deep learning is if you think
2:19:35
it's what deep learning is if you think
2:19:35
it's what deep learning is if you think about it
2:19:35
about it
2:19:36
about it the whole premise or the whole promise
2:19:38
the whole premise or the whole promise
2:19:38
the whole premise or the whole promise of deep learning is that we can
2:19:40
of deep learning is that we can
2:19:40
of deep learning is that we can stack a relatively small number of
2:19:42
stack a relatively small number of
2:19:42
stack a relatively small number of layers
2:19:43
layers
2:19:43
layers of layer types together to create
2:19:46
of layer types together to create
2:19:46
of layer types together to create complex systems
2:19:47
complex systems
2:19:47
complex systems and then the optimization process
2:19:49
and then the optimization process
2:19:49
and then the optimization process introduces even more settings and we're
2:19:52
introduces even more settings and we're
2:19:52
introduces even more settings and we're constantly tweaking the dynamics to try
2:19:53
constantly tweaking the dynamics to try
2:19:53
constantly tweaking the dynamics to try and guide the model
2:19:54
and guide the model
2:19:54
and guide the model to better solutions so settings
2:19:57
to better solutions so settings
2:19:57
to better solutions so settings everywhere
2:19:59
everywhere
2:19:59
everywhere and we've definitely felt the pain of
2:20:01
and we've definitely felt the pain of
2:20:02
and we've definitely felt the pain of letting these settings
2:20:03
letting these settings
2:20:03
letting these settings get out of control and we think it's
2:20:05
get out of control and we think it's
2:20:05
get out of control and we think it's really an important problem
2:20:07
really an important problem
2:20:07
really an important problem so we've also put a lot of work into
2:20:09
so we've also put a lot of work into
2:20:09
so we've also put a lot of work into getting spacey
2:20:11
getting spacey
2:20:11
getting spacey up set up to really do this right
2:20:14
up set up to really do this right
2:20:14
up set up to really do this right so here's an example of a common problem
2:20:16
so here's an example of a common problem
2:20:16
so here's an example of a common problem that arises from
2:20:17
that arises from
2:20:17
that arises from configuration settings it's really
2:20:20
configuration settings it's really
2:20:20
configuration settings it's really common to pass in configuration that
2:20:21
common to pass in configuration that
2:20:22
common to pass in configuration that gets passed from one function into
2:20:23
gets passed from one function into
2:20:23
gets passed from one function into another
2:20:24
another
2:20:24
another of course each of those functions um
2:20:27
of course each of those functions um
2:20:27
of course each of those functions um have to have a usable api so they're
2:20:29
have to have a usable api so they're
2:20:29
have to have a usable api so they're likely going to have default values and
2:20:31
likely going to have default values and
2:20:32
likely going to have default values and passing all those defaults along is
2:20:33
passing all those defaults along is
2:20:33
passing all those defaults along is really really error prone
2:20:35
really really error prone
2:20:35
really really error prone it's hard to figure out where some value
2:20:36
it's hard to figure out where some value
2:20:36
it's hard to figure out where some value is getting injected and it's hard to
2:20:38
is getting injected and it's hard to
2:20:38
is getting injected and it's hard to keep
2:20:39
keep
2:20:39
keep all the defaults aligned and
2:20:42
all the defaults aligned and
2:20:42
all the defaults aligned and you're also exposing a lot of details of
2:20:44
you're also exposing a lot of details of
2:20:44
you're also exposing a lot of details of the functions you're calling into
2:20:47
the functions you're calling into
2:20:47
the functions you're calling into um the outer api which really ties
2:20:50
um the outer api which really ties
2:20:50
um the outer api which really ties um into that one type of object it's
2:20:53
um into that one type of object it's
2:20:53
um into that one type of object it's constructing
2:20:54
constructing
2:20:54
constructing so if you later add the option to create
2:20:57
so if you later add the option to create
2:20:57
so if you later add the option to create a
2:20:57
a
2:20:58
a cnn or transformer here instead of an
2:21:00
cnn or transformer here instead of an
2:21:00
cnn or transformer here instead of an lstm you'll have a bunch of options
2:21:02
lstm you'll have a bunch of options
2:21:02
lstm you'll have a bunch of options they're only valid in some situations
2:21:04
they're only valid in some situations
2:21:04
they're only valid in some situations and that's when signatures really get
2:21:06
and that's when signatures really get
2:21:06
and that's when signatures really get out of control and your editor starts
2:21:08
out of control and your editor starts
2:21:08
out of control and your editor starts giving you those
2:21:09
giving you those
2:21:09
giving you those whole screen tool tips that nobody
2:21:11
whole screen tool tips that nobody
2:21:11
whole screen tool tips that nobody really wants to see
2:21:13
really wants to see
2:21:13
really wants to see so here's an example snippet of spaces
2:21:16
so here's an example snippet of spaces
2:21:16
so here's an example snippet of spaces configuration
2:21:17
configuration
2:21:17
configuration it's based on python's built-in config
2:21:19
it's based on python's built-in config
2:21:19
it's based on python's built-in config parser module with a few extensions that
2:21:22
parser module with a few extensions that
2:21:22
parser module with a few extensions that we
2:21:22
we
2:21:22
we wrote specifically for it one is that we
2:21:25
wrote specifically for it one is that we
2:21:25
wrote specifically for it one is that we use json to pass the values so you can
2:21:27
use json to pass the values so you can
2:21:27
use json to pass the values so you can easily make the values lists or mappings
2:21:29
easily make the values lists or mappings
2:21:29
easily make the values lists or mappings using
2:21:30
using
2:21:30
using a simple and also familiar syntax um
2:21:33
a simple and also familiar syntax um
2:21:33
a simple and also familiar syntax um that you probably use in other places as
2:21:35
that you probably use in other places as
2:21:35
that you probably use in other places as well and second
2:21:37
well and second
2:21:37
well and second we follow thomas convention of using a
2:21:40
we follow thomas convention of using a
2:21:40
we follow thomas convention of using a dot notation
2:21:41
dot notation
2:21:41
dot notation in the section names to denote nested
2:21:42
in the section names to denote nested
2:21:42
in the section names to denote nested section and this keeps each block simple
2:21:45
section and this keeps each block simple
2:21:45
section and this keeps each block simple and readable and avoids
2:21:47
and readable and avoids
2:21:47
and readable and avoids the indentation problems that can arise
2:21:49
the indentation problems that can arise
2:21:49
the indentation problems that can arise in very long config files
2:21:51
in very long config files
2:21:51
in very long config files and we're really happy with these
2:21:52
and we're really happy with these
2:21:52
and we're really happy with these syntactic conventions and we think the
2:21:54
syntactic conventions and we think the
2:21:54
syntactic conventions and we think the resulting conflict stays
2:21:55
resulting conflict stays
2:21:55
resulting conflict stays pretty neat and easy to read
2:21:59
pretty neat and easy to read
2:21:59
pretty neat and easy to read but the thing that we're really proud of
2:22:01
but the thing that we're really proud of
2:22:02
but the thing that we're really proud of in the config system
2:22:04
in the config system
2:22:04
in the config system is the whole system itself and the whole
2:22:08
is the whole system itself and the whole
2:22:08
is the whole system itself and the whole you know way it works um and what it
2:22:11
you know way it works um and what it
2:22:11
you know way it works um and what it does
2:22:11
does
2:22:11
does because um what really sets it apart i
2:22:14
because um what really sets it apart i
2:22:14
because um what really sets it apart i think is the registered functions
2:22:15
think is the registered functions
2:22:15
think is the registered functions integration
2:22:16
integration
2:22:16
integration your config file can describe a function
2:22:19
your config file can describe a function
2:22:19
your config file can describe a function or call a function call or
2:22:21
or call a function call or
2:22:21
or call a function call or object creation by referring to a
2:22:23
object creation by referring to a
2:22:23
object creation by referring to a function that will then be retrieved
2:22:25
function that will then be retrieved
2:22:25
function that will then be retrieved from an extensible table that we term
2:22:27
from an extensible table that we term
2:22:27
from an extensible table that we term the registry
2:22:29
the registry
2:22:29
the registry so in this example the registry will
2:22:31
so in this example the registry will
2:22:31
so in this example the registry will look in its optimizers table
2:22:33
look in its optimizers table
2:22:33
look in its optimizers table for a function named according to the
2:22:35
for a function named according to the
2:22:35
for a function named according to the specified key adam v1
2:22:38
specified key adam v1
2:22:38
specified key adam v1 this function will then be called with
2:22:39
this function will then be called with
2:22:39
this function will then be called with the other elements of the block passed
2:22:41
the other elements of the block passed
2:22:41
the other elements of the block passed in as arguments
2:22:42
in as arguments
2:22:42
in as arguments in this example the other argument is
2:22:44
in this example the other argument is
2:22:44
in this example the other argument is the learning rate
2:22:45
the learning rate
2:22:46
the learning rate which is also created via a registered
2:22:48
which is also created via a registered
2:22:48
which is also created via a registered function
2:22:49
function
2:22:49
function the configuration is resolved bottom up
2:22:51
the configuration is resolved bottom up
2:22:51
the configuration is resolved bottom up so
2:22:52
so
2:22:52
so the result of that function is computed
2:22:54
the result of that function is computed
2:22:54
the result of that function is computed and the resulting object is passed into
2:22:57
and the resulting object is passed into
2:22:57
and the resulting object is passed into adam
2:22:58
adam
2:22:58
adam in this example the learning rate here
2:23:00
in this example the learning rate here
2:23:00
in this example the learning rate here will come from the schedules table
2:23:02
will come from the schedules table
2:23:02
will come from the schedules table and it will actually be a generator so
2:23:05
and it will actually be a generator so
2:23:05
and it will actually be a generator so i know that's a lot to take in but i
2:23:07
i know that's a lot to take in but i
2:23:07
i know that's a lot to take in but i really think it's worth taking a second
2:23:09
really think it's worth taking a second
2:23:09
really think it's worth taking a second here
2:23:11
here
2:23:11
here what we're doing here is letting the
2:23:12
what we're doing here is letting the
2:23:12
what we're doing here is letting the tree of objects get
2:23:14
tree of objects get
2:23:14
tree of objects get built bottom up instead of top down
2:23:17
built bottom up instead of top down
2:23:17
built bottom up instead of top down in a normal config system you'd pass the
2:23:19
in a normal config system you'd pass the
2:23:19
in a normal config system you'd pass the optimizer settings for the learning rate
2:23:21
optimizer settings for the learning rate
2:23:21
optimizer settings for the learning rate and
2:23:22
and
2:23:22
and the optimizer would then go and
2:23:23
the optimizer would then go and
2:23:23
the optimizer would then go and construct the object so all the config
2:23:26
construct the object so all the config
2:23:26
construct the object so all the config goes into the object at the top of the
2:23:28
goes into the object at the top of the
2:23:28
goes into the object at the top of the tree and travels downwards getting
2:23:29
tree and travels downwards getting
2:23:29
tree and travels downwards getting passed along the call chain
2:23:32
passed along the call chain
2:23:32
passed along the call chain instead here each object only receives
2:23:35
instead here each object only receives
2:23:35
instead here each object only receives the configuration it needs itself
2:23:37
the configuration it needs itself
2:23:37
the configuration it needs itself because it doesn't need to create any
2:23:38
because it doesn't need to create any
2:23:38
because it doesn't need to create any objects it just gets the object
2:23:40
objects it just gets the object
2:23:40
objects it just gets the object instances
2:23:41
instances
2:23:41
instances and this is achieved by the simple
2:23:43
and this is achieved by the simple
2:23:43
and this is achieved by the simple mechanism of having a registry system so
2:23:45
mechanism of having a registry system so
2:23:45
mechanism of having a registry system so you can refer
2:23:45
you can refer
2:23:46
you can refer to functions by name of course the
2:23:49
to functions by name of course the
2:23:49
to functions by name of course the registry system wouldn't be much fun if
2:23:50
registry system wouldn't be much fun if
2:23:50
registry system wouldn't be much fun if you couldn't
2:23:51
you couldn't
2:23:51
you couldn't also add things to it yourself so you're
2:23:53
also add things to it yourself so you're
2:23:53
also add things to it yourself so you're not limited to just the functions
2:23:55
not limited to just the functions
2:23:55
not limited to just the functions rewrite and register for you it's really
2:23:57
rewrite and register for you it's really
2:23:57
rewrite and register for you it's really easy to register your own functions
2:23:59
easy to register your own functions
2:23:59
easy to register your own functions using the decorator behind the scenes
2:24:01
using the decorator behind the scenes
2:24:01
using the decorator behind the scenes the config system does a lot of cool
2:24:03
the config system does a lot of cool
2:24:03
the config system does a lot of cool stuff for you it looks at your function
2:24:05
stuff for you it looks at your function
2:24:05
stuff for you it looks at your function signature
2:24:05
signature
2:24:05
signature and uses it to build a pedentic data
2:24:07
and uses it to build a pedentic data
2:24:07
and uses it to build a pedentic data model that can be used to validate your
2:24:09
model that can be used to validate your
2:24:09
model that can be used to validate your config
2:24:10
config
2:24:10
config so if your config specifies arguments
2:24:12
so if your config specifies arguments
2:24:12
so if your config specifies arguments that will result in an error
2:24:14
that will result in an error
2:24:14
that will result in an error like wrong names missing required values
2:24:16
like wrong names missing required values
2:24:16
like wrong names missing required values even type errors like passing a string
2:24:19
even type errors like passing a string
2:24:19
even type errors like passing a string instead of an end we can raise an error
2:24:21
instead of an end we can raise an error
2:24:21
instead of an end we can raise an error message for you right away
2:24:22
message for you right away
2:24:22
message for you right away so we don't need to run everything and
2:24:25
so we don't need to run everything and
2:24:25
so we don't need to run everything and let you see some weird trace back
2:24:26
let you see some weird trace back
2:24:26
let you see some weird trace back five minutes later just because you made
2:24:28
five minutes later just because you made
2:24:28
five minutes later just because you made some typo
2:24:31
the cli also adds a few cool features
2:24:33
the cli also adds a few cool features
2:24:33
the cli also adds a few cool features that really improve
2:24:34
that really improve
2:24:34
that really improve the developer experience around this i
2:24:36
the developer experience around this i
2:24:36
the developer experience around this i think
2:24:37
think
2:24:37
think you can quickly set up a new config file
2:24:40
you can quickly set up a new config file
2:24:40
you can quickly set up a new config file using the recommended defaults
2:24:42
using the recommended defaults
2:24:42
using the recommended defaults given the pipeline you're trying to
2:24:43
given the pipeline you're trying to
2:24:43
given the pipeline you're trying to train and you can even tell it about the
2:24:45
train and you can even tell it about the
2:24:45
train and you can even tell it about the trade-offs that
2:24:46
trade-offs that
2:24:46
trade-offs that are relevant to you so it can recommend
2:24:48
are relevant to you so it can recommend
2:24:48
are relevant to you so it can recommend you a config that's tuned for accuracy
2:24:50
you a config that's tuned for accuracy
2:24:50
you a config that's tuned for accuracy rather than just
2:24:51
rather than just
2:24:51
rather than just efficiency and speed or vice versa and
2:24:54
efficiency and speed or vice versa and
2:24:54
efficiency and speed or vice versa and once your config is ready you can then
2:24:55
once your config is ready you can then
2:24:55
once your config is ready you can then pass it
2:24:56
pass it
2:24:56
pass it as an argument to the spacey train
2:24:57
as an argument to the spacey train
2:24:58
as an argument to the spacey train command and the config system is another
2:25:00
command and the config system is another
2:25:00
command and the config system is another step in the cli experience that we've
2:25:02
step in the cli experience that we've
2:25:02
step in the cli experience that we've been building for spacey for quite some
2:25:04
been building for spacey for quite some
2:25:04
been building for spacey for quite some time now
2:25:05
time now
2:25:05
time now and you can define your pipeline in the
2:25:07
and you can define your pipeline in the
2:25:07
and you can define your pipeline in the config system and then
2:25:08
config system and then
2:25:08
config system and then execute the training with a simple and
2:25:10
execute the training with a simple and
2:25:10
execute the training with a simple and consistent command
2:25:12
consistent command
2:25:12
consistent command while still being able to change things
2:25:14
while still being able to change things
2:25:14
while still being able to change things on the command line
2:25:15
on the command line
2:25:15
on the command line if your workflow requires it and once
2:25:18
if your workflow requires it and once
2:25:18
if your workflow requires it and once the model is trained it can be converted
2:25:20
the model is trained it can be converted
2:25:20
the model is trained it can be converted into a python package
2:25:21
into a python package
2:25:21
into a python package which gives you a really easy way to
2:25:23
which gives you a really easy way to
2:25:23
which gives you a really easy way to ship it and
2:25:25
ship it and
2:25:25
ship it and deploy it into production
2:25:30
and as you can see here machine learning
2:25:31
and as you can see here machine learning
2:25:32
and as you can see here machine learning projects inherently consist
2:25:33
projects inherently consist
2:25:34
projects inherently consist of several steps even in the simplest
2:25:36
of several steps even in the simplest
2:25:36
of several steps even in the simplest case
2:25:37
case
2:25:37
case you're going to run some sort of
2:25:38
you're going to run some sort of
2:25:38
you're going to run some sort of training process save out a trade model
2:25:40
training process save out a trade model
2:25:40
training process save out a trade model and then you'll be loading it back in to
2:25:43
and then you'll be loading it back in to
2:25:43
and then you'll be loading it back in to predict with it
2:25:43
predict with it
2:25:44
predict with it and in practice they're usually many
2:25:45
and in practice they're usually many
2:25:45
and in practice they're usually many more steps in the process
2:25:47
more steps in the process
2:25:47
more steps in the process like you might need to go over the
2:25:49
like you might need to go over the
2:25:49
like you might need to go over the corpus before you create your models
2:25:51
corpus before you create your models
2:25:51
corpus before you create your models in order to build a vocabulary list or
2:25:53
in order to build a vocabulary list or
2:25:53
in order to build a vocabulary list or you might need to run some sort of data
2:25:55
you might need to run some sort of data
2:25:55
you might need to run some sort of data conversion pre-processing or
2:25:57
conversion pre-processing or
2:25:57
conversion pre-processing or augmentation step and your project might
2:25:59
augmentation step and your project might
2:25:59
augmentation step and your project might not consist of just one model
2:26:01
not consist of just one model
2:26:01
not consist of just one model maybe you're doing something like
2:26:03
maybe you're doing something like
2:26:03
maybe you're doing something like aspect-oriented sentiment analysis
2:26:05
aspect-oriented sentiment analysis
2:26:05
aspect-oriented sentiment analysis and only 0.5 percent of your texts are
2:26:07
and only 0.5 percent of your texts are
2:26:08
and only 0.5 percent of your texts are even relevant
2:26:09
even relevant
2:26:09
even relevant so you need to train and run a text
2:26:10
so you need to train and run a text
2:26:10
so you need to train and run a text classifier first to find the stuff
2:26:12
classifier first to find the stuff
2:26:12
classifier first to find the stuff that's on topic
2:26:13
that's on topic
2:26:13
that's on topic or maybe you're doing a dialogue task
2:26:15
or maybe you're doing a dialogue task
2:26:15
or maybe you're doing a dialogue task and you need many layers of analysis
2:26:19
and you need many layers of analysis
2:26:19
and you need many layers of analysis the new spacey project system helps you
2:26:21
the new spacey project system helps you
2:26:21
the new spacey project system helps you keep those multi-stage projects
2:26:22
keep those multi-stage projects
2:26:22
keep those multi-stage projects organized
2:26:23
organized
2:26:23
organized and we're especially excited about how
2:26:25
and we're especially excited about how
2:26:25
and we're especially excited about how much easier this makes it to share and
2:26:27
much easier this makes it to share and
2:26:27
much easier this makes it to share and reuse work
2:26:28
reuse work
2:26:28
reuse work instead of everyone having to write
2:26:30
instead of everyone having to write
2:26:30
instead of everyone having to write their own scripts and instructions
2:26:32
their own scripts and instructions
2:26:32
their own scripts and instructions to run the various steps in their repo
2:26:35
to run the various steps in their repo
2:26:35
to run the various steps in their repo we can just
2:26:35
we can just
2:26:36
we can just all use a common standard and for the
2:26:38
all use a common standard and for the
2:26:38
all use a common standard and for the most common workflows like training and
2:26:40
most common workflows like training and
2:26:40
most common workflows like training and entity recognizer you'll be able to
2:26:42
entity recognizer you'll be able to
2:26:42
entity recognizer you'll be able to clone our project templates
2:26:43
clone our project templates
2:26:43
clone our project templates modify them for your data and
2:26:46
modify them for your data and
2:26:46
modify them for your data and requirements
2:26:47
requirements
2:26:47
requirements and then just run the commands and then
2:26:50
and then just run the commands and then
2:26:50
and then just run the commands and then when it's time to ship your work into
2:26:51
when it's time to ship your work into
2:26:51
when it's time to ship your work into production
2:26:52
production
2:26:52
production you'll have everything in a format
2:26:53
you'll have everything in a format
2:26:53
you'll have everything in a format that's super easy to automate and
2:26:56
that's super easy to automate and
2:26:56
that's super easy to automate and just run and deploy so here's what we
2:26:58
just run and deploy so here's what we
2:26:58
just run and deploy so here's what we expect
2:26:59
expect
2:26:59
expect everyday usage to look like once you've
2:27:02
everyday usage to look like once you've
2:27:02
everyday usage to look like once you've found a project template you want to
2:27:03
found a project template you want to
2:27:03
found a project template you want to work from you you can use the spacey
2:27:05
work from you you can use the spacey
2:27:05
work from you you can use the spacey project clone
2:27:06
project clone
2:27:06
project clone command to create a local copy just like
2:27:09
command to create a local copy just like
2:27:09
command to create a local copy just like git clone
2:27:10
git clone
2:27:10
git clone at this point you might want to edit the
2:27:12
at this point you might want to edit the
2:27:12
at this point you might want to edit the project template to point to your own
2:27:13
project template to point to your own
2:27:13
project template to point to your own data files or customize the template
2:27:16
data files or customize the template
2:27:16
data files or customize the template to me to fit better fit to your
2:27:18
to me to fit better fit to your
2:27:18
to me to fit better fit to your requirements in some other way
2:27:20
requirements in some other way
2:27:20
requirements in some other way and next you'll run the spacey project
2:27:22
and next you'll run the spacey project
2:27:22
and next you'll run the spacey project assets command to download or verify the
2:27:24
assets command to download or verify the
2:27:24
assets command to download or verify the data required by the project
2:27:27
data required by the project
2:27:27
data required by the project project templates can point to assets
2:27:29
project templates can point to assets
2:27:29
project templates can point to assets hosted in lots of different ways
2:27:31
hosted in lots of different ways
2:27:31
hosted in lots of different ways um they can be um
2:27:34
um they can be um
2:27:34
um they can be um [Music]
2:27:35
[Music]
2:27:35
[Music] behind the url they can be in a git repo
2:27:38
behind the url they can be in a git repo
2:27:38
behind the url they can be in a git repo or you can also describe assets without
2:27:40
or you can also describe assets without
2:27:40
or you can also describe assets without a source url
2:27:41
a source url
2:27:41
a source url you'll still need to get those into the
2:27:42
you'll still need to get those into the
2:27:42
you'll still need to get those into the right place yourself but
2:27:44
right place yourself but
2:27:44
right place yourself but at least you'll know when they're
2:27:45
at least you'll know when they're
2:27:45
at least you'll know when they're missing and if the project template
2:27:46
missing and if the project template
2:27:46
missing and if the project template specifies a checksum
2:27:48
specifies a checksum
2:27:48
specifies a checksum it can also tell you whether you're
2:27:49
it can also tell you whether you're
2:27:49
it can also tell you whether you're working from the right version
2:27:51
working from the right version
2:27:51
working from the right version and once you have the assets you'll
2:27:52
and once you have the assets you'll
2:27:52
and once you have the assets you'll usually use the spacey project run
2:27:54
usually use the spacey project run
2:27:54
usually use the spacey project run command to execute a command or workflow
2:27:57
command to execute a command or workflow
2:27:57
command to execute a command or workflow where a workflow is just a list of
2:27:59
where a workflow is just a list of
2:27:59
where a workflow is just a list of separate commands
2:28:01
separate commands
2:28:01
separate commands as the steps are completed the checksums
2:28:03
as the steps are completed the checksums
2:28:03
as the steps are completed the checksums of the outputs are saved out for you
2:28:05
of the outputs are saved out for you
2:28:05
of the outputs are saved out for you you to a little metadata file and this
2:28:07
you to a little metadata file and this
2:28:08
you to a little metadata file and this helps you
2:28:08
helps you
2:28:08
helps you replicate your work later and also makes
2:28:10
replicate your work later and also makes
2:28:10
replicate your work later and also makes it easy for spacey to check
2:28:11
it easy for spacey to check
2:28:12
it easy for spacey to check whether some step can be skipped
2:28:13
whether some step can be skipped
2:28:13
whether some step can be skipped probably the biggest inspiration for the
2:28:15
probably the biggest inspiration for the
2:28:15
probably the biggest inspiration for the spacey project system was dvc
2:28:18
spacey project system was dvc
2:28:18
spacey project system was dvc data version control there's even an
2:28:20
data version control there's even an
2:28:20
data version control there's even an integration for dvc users to make it
2:28:23
integration for dvc users to make it
2:28:23
integration for dvc users to make it easy to use the two together
2:28:26
easy to use the two together
2:28:26
easy to use the two together integrations work really well within the
2:28:27
integrations work really well within the
2:28:27
integrations work really well within the spacey project system because many tools
2:28:29
spacey project system because many tools
2:28:29
spacey project system because many tools in the ecosystem
2:28:31
in the ecosystem
2:28:31
in the ecosystem operate outside of your local python
2:28:32
operate outside of your local python
2:28:32
operate outside of your local python installation they're often services that
2:28:35
installation they're often services that
2:28:35
installation they're often services that help you fill in
2:28:36
help you fill in
2:28:36
help you fill in various parts of your workflow for
2:28:38
various parts of your workflow for
2:28:38
various parts of your workflow for instance when you're training a model
2:28:39
instance when you're training a model
2:28:40
instance when you're training a model and running different experiments you
2:28:41
and running different experiments you
2:28:42
and running different experiments you typically want to track the results
2:28:43
typically want to track the results
2:28:44
typically want to track the results weights and biases is a popular tool for
2:28:45
weights and biases is a popular tool for
2:28:46
weights and biases is a popular tool for this and spacey integrates with it out
2:28:47
this and spacey integrates with it out
2:28:47
this and spacey integrates with it out of the box using a logger that you can
2:28:49
of the box using a logger that you can
2:28:49
of the box using a logger that you can add
2:28:49
add
2:28:49
add to your training config it will lock the
2:28:52
to your training config it will lock the
2:28:52
to your training config it will lock the entire config
2:28:53
entire config
2:28:53
entire config including all settings hyper parameters
2:28:55
including all settings hyper parameters
2:28:55
including all settings hyper parameters and even registered functions
2:28:57
and even registered functions
2:28:57
and even registered functions so you always have a record of how your
2:28:59
so you always have a record of how your
2:28:59
so you always have a record of how your model was trained and how the different
2:29:01
model was trained and how the different
2:29:01
model was trained and how the different settings
2:29:02
settings
2:29:02
settings impact the results
2:29:06
once you've finished training a model
2:29:08
once you've finished training a model
2:29:08
once you've finished training a model you can also spin up an interactive
2:29:10
you can also spin up an interactive
2:29:10
you can also spin up an interactive streamlit app to explore its predictions
2:29:12
streamlit app to explore its predictions
2:29:12
streamlit app to explore its predictions or compare it to other pipelines our
2:29:15
or compare it to other pipelines our
2:29:15
or compare it to other pipelines our spacey streamlight package provides
2:29:16
spacey streamlight package provides
2:29:16
spacey streamlight package provides building blocks for visualizing
2:29:18
building blocks for visualizing
2:29:18
building blocks for visualizing different components
2:29:19
different components
2:29:19
different components you can also use it as a step in your
2:29:21
you can also use it as a step in your
2:29:21
you can also use it as a step in your project so you can
2:29:22
project so you can
2:29:22
project so you can run spacey project run visualize to spin
2:29:25
run spacey project run visualize to spin
2:29:25
run spacey project run visualize to spin up an app
2:29:25
up an app
2:29:25
up an app with a pipeline trained in a previous
2:29:27
with a pipeline trained in a previous
2:29:27
with a pipeline trained in a previous step
2:29:31
the project unconfig systems work
2:29:33
the project unconfig systems work
2:29:33
the project unconfig systems work together to let you set up and train
2:29:36
together to let you set up and train
2:29:36
together to let you set up and train powerful language understanding systems
2:29:38
powerful language understanding systems
2:29:38
powerful language understanding systems and here a pipeline
2:29:39
and here a pipeline
2:29:39
and here a pipeline is a sequence of components and it's you
2:29:42
is a sequence of components and it's you
2:29:42
is a sequence of components and it's you can think of it like an assembly line
2:29:44
can think of it like an assembly line
2:29:44
can think of it like an assembly line the doc object travels along it and each
2:29:46
the doc object travels along it and each
2:29:46
the doc object travels along it and each component does
2:29:47
component does
2:29:47
component does its little bit of work updating the dock
2:29:49
its little bit of work updating the dock
2:29:49
its little bit of work updating the dock with new annotations
2:29:51
with new annotations
2:29:51
with new annotations and at the end the dock api stores the
2:29:53
and at the end the dock api stores the
2:29:53
and at the end the dock api stores the annotations efficiently
2:29:55
annotations efficiently
2:29:55
annotations efficiently and makes it easy to use the different
2:29:56
and makes it easy to use the different
2:29:56
and makes it easy to use the different annotations together
2:29:58
annotations together
2:29:58
annotations together here's part of a config file that
2:30:00
here's part of a config file that
2:30:00
here's part of a config file that describes some pipeline components
2:30:02
describes some pipeline components
2:30:02
describes some pipeline components the nlp object will look up a factory
2:30:04
the nlp object will look up a factory
2:30:04
the nlp object will look up a factory function
2:30:05
function
2:30:05
function using this string name and create it
2:30:07
using this string name and create it
2:30:07
using this string name and create it with the arguments defined in the config
2:30:10
with the arguments defined in the config
2:30:10
with the arguments defined in the config a subsection like model is just another
2:30:12
a subsection like model is just another
2:30:12
a subsection like model is just another argument
2:30:13
argument
2:30:13
argument and here it uses a model architecture
2:30:16
and here it uses a model architecture
2:30:16
and here it uses a model architecture created via the function registry system
2:30:19
created via the function registry system
2:30:19
created via the function registry system that i talked about earlier
2:30:23
this example creates a transition based
2:30:25
this example creates a transition based
2:30:25
this example creates a transition based named entity recognizer
2:30:27
named entity recognizer
2:30:27
named entity recognizer and its model's token to vector
2:30:28
and its model's token to vector
2:30:28
and its model's token to vector embedding layer is a transformer
2:30:30
embedding layer is a transformer
2:30:30
embedding layer is a transformer listener
2:30:31
listener
2:30:31
listener which connects to the transformer
2:30:33
which connects to the transformer
2:30:33
which connects to the transformer component in the pipeline and allows
2:30:35
component in the pipeline and allows
2:30:35
component in the pipeline and allows multiple components to share the same
2:30:37
multiple components to share the same
2:30:37
multiple components to share the same transformer embeddings
2:30:39
transformer embeddings
2:30:39
transformer embeddings the doctorate layer also takes a pooling
2:30:41
the doctorate layer also takes a pooling
2:30:41
the doctorate layer also takes a pooling layer as an argument
2:30:42
layer as an argument
2:30:42
layer as an argument in this case reduce mean for mean
2:30:44
in this case reduce mean for mean
2:30:44
in this case reduce mean for mean pooling all of these are things you can
2:30:46
pooling all of these are things you can
2:30:46
pooling all of these are things you can customize
2:30:47
customize
2:30:47
customize you could for instance change the
2:30:49
you could for instance change the
2:30:49
you could for instance change the component to use a buyer listing
2:30:51
component to use a buyer listing
2:30:51
component to use a buyer listing instead of a transformer or you could
2:30:53
instead of a transformer or you could
2:30:53
instead of a transformer or you could change the final pooling strategy
2:30:55
change the final pooling strategy
2:30:55
change the final pooling strategy so that only the first workpiece token
2:30:57
so that only the first workpiece token
2:30:57
so that only the first workpiece token is used rather than the average
2:30:59
is used rather than the average
2:30:59
is used rather than the average you can also implement um entirely
2:31:02
you can also implement um entirely
2:31:02
you can also implement um entirely custom
2:31:03
custom
2:31:03
custom models for instance you might make a
2:31:05
models for instance you might make a
2:31:05
models for instance you might make a sparse linear model like we had in
2:31:07
sparse linear model like we had in
2:31:07
sparse linear model like we had in spacey version one
2:31:08
spacey version one
2:31:08
spacey version one or you can register an entirely new
2:31:11
or you can register an entirely new
2:31:11
or you can register an entirely new component for instance maybe you want a
2:31:13
component for instance maybe you want a
2:31:13
component for instance maybe you want a crf based named entity recognizer rather
2:31:15
crf based named entity recognizer rather
2:31:15
crf based named entity recognizer rather than the transition based approach
2:31:17
than the transition based approach
2:31:17
than the transition based approach just as with normal code it's extensible
2:31:20
just as with normal code it's extensible
2:31:20
just as with normal code it's extensible in
2:31:21
in
2:31:21
in several places and you can choose the
2:31:23
several places and you can choose the
2:31:23
several places and you can choose the abstraction
2:31:24
abstraction
2:31:24
abstraction at which you want to get involved you
2:31:27
at which you want to get involved you
2:31:27
at which you want to get involved you can also reuse the weights
2:31:28
can also reuse the weights
2:31:28
can also reuse the weights or configuration of a component from
2:31:31
or configuration of a component from
2:31:31
or configuration of a component from another pipeline as a starting point
2:31:33
another pipeline as a starting point
2:31:33
another pipeline as a starting point this can be useful to kind of import
2:31:35
this can be useful to kind of import
2:31:35
this can be useful to kind of import pipeline components into a new pipeline
2:31:37
pipeline components into a new pipeline
2:31:37
pipeline components into a new pipeline or to do things like domain adaptation
2:31:39
or to do things like domain adaptation
2:31:39
or to do things like domain adaptation or self training
2:31:41
or self training
2:31:41
or self training if you just want to import a component
2:31:43
if you just want to import a component
2:31:43
if you just want to import a component for instance to reuse an aimed entity
2:31:44
for instance to reuse an aimed entity
2:31:44
for instance to reuse an aimed entity recognizer in a new text classification
2:31:46
recognizer in a new text classification
2:31:46
recognizer in a new text classification project
2:31:47
project
2:31:47
project you can set the component weights to
2:31:49
you can set the component weights to
2:31:49
you can set the component weights to frozen this tells
2:31:51
frozen this tells
2:31:51
frozen this tells tell spacey not to treat the components
2:31:53
tell spacey not to treat the components
2:31:53
tell spacey not to treat the components as usual and
2:31:54
as usual and
2:31:54
as usual and to try to update it um and this way we
2:31:57
to try to update it um and this way we
2:31:57
to try to update it um and this way we can make sure
2:31:59
can make sure
2:31:59
can make sure that you can use the spacey train
2:32:00
that you can use the spacey train
2:32:00
that you can use the spacey train command in
2:32:02
command in
2:32:02
command in really most situations uh there's also
2:32:05
really most situations uh there's also
2:32:05
really most situations uh there's also callbacks for different points in the
2:32:06
callbacks for different points in the
2:32:06
callbacks for different points in the life cycle
2:32:08
life cycle
2:32:08
life cycle if you really need to train from code
2:32:10
if you really need to train from code
2:32:10
if you really need to train from code and want to you can definitely do that
2:32:12
and want to you can definitely do that
2:32:12
and want to you can definitely do that um and the abstractions here are easy to
2:32:14
um and the abstractions here are easy to
2:32:14
um and the abstractions here are easy to use too but generally the spacey train
2:32:16
use too but generally the spacey train
2:32:16
use too but generally the spacey train command should really have you covered
2:32:19
command should really have you covered
2:32:19
command should really have you covered for train components the most important
2:32:21
for train components the most important
2:32:21
for train components the most important type of customization you want to do
2:32:24
type of customization you want to do
2:32:24
type of customization you want to do is of course the model spacey can
2:32:27
is of course the model spacey can
2:32:27
is of course the model spacey can communicate with models defined in major
2:32:29
communicate with models defined in major
2:32:29
communicate with models defined in major frameworks like pytorch tensorflow and
2:32:31
frameworks like pytorch tensorflow and
2:32:31
frameworks like pytorch tensorflow and mxnet
2:32:32
mxnet
2:32:32
mxnet via our machine learning library think
2:32:34
via our machine learning library think
2:32:34
via our machine learning library think this was a key design goal we had for
2:32:36
this was a key design goal we had for
2:32:36
this was a key design goal we had for think
2:32:37
think
2:32:37
think it works as an interface layer between
2:32:39
it works as an interface layer between
2:32:39
it works as an interface layer between spacey and other machine learning
2:32:41
spacey and other machine learning
2:32:41
spacey and other machine learning libraries by wrapping their models and
2:32:43
libraries by wrapping their models and
2:32:43
libraries by wrapping their models and acting as a shame
2:32:44
acting as a shame
2:32:44
acting as a shame for example thinks pytorch wrapper layer
2:32:47
for example thinks pytorch wrapper layer
2:32:47
for example thinks pytorch wrapper layer takes a python model and returns a think
2:32:49
takes a python model and returns a think
2:32:49
takes a python model and returns a think model instance
2:32:50
model instance
2:32:50
model instance you could then use it inside your
2:32:52
you could then use it inside your
2:32:52
you could then use it inside your registered functions to build up the
2:32:53
registered functions to build up the
2:32:53
registered functions to build up the think model
2:32:55
think model
2:32:55
think model and then use that to power your spacey
2:32:57
and then use that to power your spacey
2:32:57
and then use that to power your spacey components
2:32:58
components
2:32:58
components so yeah that's what's in space e3 and
2:33:00
so yeah that's what's in space e3 and
2:33:00
so yeah that's what's in space e3 and why it's there and we've been
2:33:02
why it's there and we've been
2:33:02
why it's there and we've been learning a lot from helping spacey grow
2:33:04
learning a lot from helping spacey grow
2:33:04
learning a lot from helping spacey grow and evolve the last five years and
2:33:06
and evolve the last five years and
2:33:06
and evolve the last five years and we've actually really been looking
2:33:07
we've actually really been looking
2:33:07
we've actually really been looking forward to sharing this all with you
2:33:10
forward to sharing this all with you
2:33:10
forward to sharing this all with you and one of the aspects of spacey that
2:33:13
and one of the aspects of spacey that
2:33:13
and one of the aspects of spacey that we're most proud of
2:33:14
we're most proud of
2:33:14
we're most proud of is the really awesome ecosystem of
2:33:16
is the really awesome ecosystem of
2:33:16
is the really awesome ecosystem of plugins
2:33:17
plugins
2:33:17
plugins and extensions that's grown up around
2:33:19
and extensions that's grown up around
2:33:19
and extensions that's grown up around the library
2:33:20
the library
2:33:20
the library with so many awesome people developing
2:33:23
with so many awesome people developing
2:33:23
with so many awesome people developing these things
2:33:24
these things
2:33:24
these things and publishing them and helping other
2:33:27
and publishing them and helping other
2:33:27
and publishing them and helping other users
2:33:27
users
2:33:27
users in their nlp projects and now with all
2:33:30
in their nlp projects and now with all
2:33:30
in their nlp projects and now with all the new features
2:33:30
the new features
2:33:30
the new features and configurability we're really looking
2:33:33
and configurability we're really looking
2:33:33
and configurability we're really looking forward to seeing the ecosystem
2:33:35
forward to seeing the ecosystem
2:33:35
forward to seeing the ecosystem evolve even further so i'll help you try
2:33:38
evolve even further so i'll help you try
2:33:38
evolve even further so i'll help you try it out and
2:33:39
it out and
2:33:39
it out and definitely let us know what you built
2:33:46
amazing we're really looking for i'm
2:33:49
amazing we're really looking for i'm
2:33:49
amazing we're really looking for i'm sure
2:33:49
sure
2:33:49
sure many people are looking forward to try
2:33:52
many people are looking forward to try
2:33:52
many people are looking forward to try out this a new version of spacey
2:33:55
out this a new version of spacey
2:33:55
out this a new version of spacey and as you might see in his connection
2:33:58
and as you might see in his connection
2:33:58
and as you might see in his connection goes
2:33:58
goes
2:33:58
goes up and on and off so
2:34:02
up and on and off so
2:34:02
up and on and off so yeah actually i'm so glad i pre-recorded
2:34:04
yeah actually i'm so glad i pre-recorded
2:34:04
yeah actually i'm so glad i pre-recorded this because literally like 10 minutes
2:34:06
this because literally like 10 minutes
2:34:06
this because literally like 10 minutes in
2:34:06
in
2:34:06
in my internet connection dropped out and
2:34:08
my internet connection dropped out and
2:34:08
my internet connection dropped out and imagine this that happened live
2:34:09
imagine this that happened live
2:34:10
imagine this that happened live no i mean for me it's more like you know
2:34:11
no i mean for me it's more like you know
2:34:11
no i mean for me it's more like you know if i do something i want to do it
2:34:12
if i do something i want to do it
2:34:12
if i do something i want to do it properly like i don't think
2:34:14
properly like i don't think
2:34:14
properly like i don't think i either do it properly or not at all
2:34:16
i either do it properly or not at all
2:34:16
i either do it properly or not at all and i think this was the best value
2:34:18
and i think this was the best value
2:34:18
and i think this was the best value um for whoever is currently watching
2:34:20
um for whoever is currently watching
2:34:20
um for whoever is currently watching this
2:34:21
this
2:34:21
this we are looking forward to try it out
2:34:23
we are looking forward to try it out
2:34:23
we are looking forward to try it out with all the new features that have been
2:34:25
with all the new features that have been
2:34:25
with all the new features that have been added to
2:34:26
added to
2:34:26
added to the library we had some questions
2:34:30
the library we had some questions
2:34:30
the library we had some questions from the public and one is uh are you
2:34:33
from the public and one is uh are you
2:34:33
from the public and one is uh are you working on
2:34:34
working on
2:34:34
working on making your spacey like explainable
2:34:38
making your spacey like explainable
2:34:38
making your spacey like explainable um you mean as in adding features
2:34:42
um you mean as in adding features
2:34:42
um you mean as in adding features that help people yeah why does it make
2:34:45
that help people yeah why does it make
2:34:45
that help people yeah why does it make that
2:34:46
that
2:34:46
that decision and we actually um so
2:34:49
decision and we actually um so
2:34:49
decision and we actually um so i think that actually could be a really
2:34:50
i think that actually could be a really
2:34:50
i think that actually could be a really great extension now with
2:34:52
great extension now with
2:34:52
great extension now with um you know the sort of new way
2:34:54
um you know the sort of new way
2:34:54
um you know the sort of new way extensible config system and structure
2:34:56
extensible config system and structure
2:34:56
extensible config system and structure that we have because it makes it much
2:34:57
that we have because it makes it much
2:34:57
that we have because it makes it much much easier to slot
2:34:58
much easier to slot
2:34:58
much easier to slot something into every um you know point
2:35:02
something into every um you know point
2:35:02
something into every um you know point of the system of the model um so
2:35:04
of the system of the model um so
2:35:04
of the system of the model um so actually that's something i'm really i'm
2:35:05
actually that's something i'm really i'm
2:35:05
actually that's something i'm really i'm looking forward to exploring how
2:35:08
looking forward to exploring how
2:35:08
looking forward to exploring how we can integrate the models and the
2:35:10
we can integrate the models and the
2:35:10
we can integrate the models and the config with like a lot of the cool tools
2:35:12
config with like a lot of the cool tools
2:35:12
config with like a lot of the cool tools that already exist
2:35:13
that already exist
2:35:13
that already exist in the ecosystem so um yeah i think that
2:35:16
in the ecosystem so um yeah i think that
2:35:16
in the ecosystem so um yeah i think that should definitely be
2:35:17
should definitely be
2:35:17
should definitely be possible and much easier now okay cool
2:35:21
possible and much easier now okay cool
2:35:21
possible and much easier now okay cool another question was concerning
2:35:25
another question was concerning
2:35:25
another question was concerning named entity recognition um is it
2:35:28
named entity recognition um is it
2:35:28
named entity recognition um is it possible to use
2:35:29
possible to use
2:35:29
possible to use pre-trained transform models for tasks
2:35:32
pre-trained transform models for tasks
2:35:32
pre-trained transform models for tasks just as a
2:35:33
just as a
2:35:33
just as a and then named entity recognition
2:35:36
and then named entity recognition
2:35:36
and then named entity recognition um well yeah i mean the models uh the
2:35:39
um well yeah i mean the models uh the
2:35:39
um well yeah i mean the models uh the examples i showed there
2:35:40
examples i showed there
2:35:40
examples i showed there um um i think i even showed like a
2:35:42
um um i think i even showed like a
2:35:42
um um i think i even showed like a config for
2:35:43
config for
2:35:43
config for a named entity recognition component
2:35:45
a named entity recognition component
2:35:45
a named entity recognition component that uses pre-trained transformer
2:35:47
that uses pre-trained transformer
2:35:47
that uses pre-trained transformer weights so yes that's possible we
2:35:49
weights so yes that's possible we
2:35:49
weights so yes that's possible we we've um pre-trained a few um pipelines
2:35:51
we've um pre-trained a few um pipelines
2:35:52
we've um pre-trained a few um pipelines that you can download and test
2:35:53
that you can download and test
2:35:53
that you can download and test on some corpora including uh named
2:35:56
on some corpora including uh named
2:35:56
on some corpora including uh named entity recognition
2:35:57
entity recognition
2:35:57
entity recognition corpora that are available um so yes
2:35:59
corpora that are available um so yes
2:36:00
corpora that are available um so yes pretty much
2:36:00
pretty much
2:36:00
pretty much any model definition that you can write
2:36:02
any model definition that you can write
2:36:02
any model definition that you can write that predicts something
2:36:03
that predicts something
2:36:03
that predicts something that's useful in that context you can
2:36:05
that's useful in that context you can
2:36:05
that's useful in that context you can also use transform ambitions for
2:36:08
also use transform ambitions for
2:36:08
also use transform ambitions for yeah very cool um oh i have a quick
2:36:11
yeah very cool um oh i have a quick
2:36:11
yeah very cool um oh i have a quick question and as
2:36:12
question and as
2:36:12
question and as so this is me being selfish again and
2:36:14
so this is me being selfish again and
2:36:14
so this is me being selfish again and like wanting to ask all my own questions
2:36:16
like wanting to ask all my own questions
2:36:16
like wanting to ask all my own questions um so one of the things that really came
2:36:18
um so one of the things that really came
2:36:18
um so one of the things that really came to mind uh from watching your talk was
2:36:21
to mind uh from watching your talk was
2:36:21
to mind uh from watching your talk was there's quite actually a lot of
2:36:22
there's quite actually a lot of
2:36:22
there's quite actually a lot of developer considerations so almost
2:36:24
developer considerations so almost
2:36:24
developer considerations so almost thinking more about the end to end and
2:36:27
thinking more about the end to end and
2:36:27
thinking more about the end to end and uh we spoke about this with josh one of
2:36:29
uh we spoke about this with josh one of
2:36:29
uh we spoke about this with josh one of our previous guests where
2:36:31
our previous guests where
2:36:31
our previous guests where he was kind of saying unfortunately like
2:36:32
he was kind of saying unfortunately like
2:36:32
he was kind of saying unfortunately like the exciting experiment
2:36:34
the exciting experiment
2:36:34
the exciting experiment like using different models and doing a
2:36:37
like using different models and doing a
2:36:37
like using different models and doing a b testing tends to be like the smallest
2:36:39
b testing tends to be like the smallest
2:36:39
b testing tends to be like the smallest part of the project
2:36:40
part of the project
2:36:40
part of the project and it feels like you've thought a lot
2:36:42
and it feels like you've thought a lot
2:36:42
and it feels like you've thought a lot about kind of how do we
2:36:44
about kind of how do we
2:36:44
about kind of how do we um reuse different assets and stuff like
2:36:47
um reuse different assets and stuff like
2:36:47
um reuse different assets and stuff like that so is that an
2:36:47
that so is that an
2:36:48
that so is that an ethos that spacey is going to have
2:36:50
ethos that spacey is going to have
2:36:50
ethos that spacey is going to have moving forwards
2:36:51
moving forwards
2:36:51
moving forwards is everything kind of we'll give you
2:36:53
is everything kind of we'll give you
2:36:53
is everything kind of we'll give you some out of the box but actually you can
2:36:54
some out of the box but actually you can
2:36:54
some out of the box but actually you can customize so much
2:36:56
customize so much
2:36:56
customize so much yeah exactly i mean i think i feel like
2:36:58
yeah exactly i mean i think i feel like
2:36:58
yeah exactly i mean i think i feel like this has always been something that we
2:36:59
this has always been something that we
2:36:59
this has always been something that we wanted um and it's been like a process
2:37:01
wanted um and it's been like a process
2:37:01
wanted um and it's been like a process um
2:37:02
um
2:37:02
um and i think it also kind of reflects uh
2:37:04
and i think it also kind of reflects uh
2:37:04
and i think it also kind of reflects uh the ecosystem and the developments in
2:37:05
the ecosystem and the developments in
2:37:05
the ecosystem and the developments in the ecosystem and in the broader
2:37:07
the ecosystem and in the broader
2:37:07
the ecosystem and in the broader developer community a lot
2:37:08
developer community a lot
2:37:08
developer community a lot because when we started with spacey in
2:37:10
because when we started with spacey in
2:37:10
because when we started with spacey in 2015 people
2:37:11
2015 people
2:37:11
2015 people really just wanted to have some named
2:37:13
really just wanted to have some named
2:37:13
really just wanted to have some named entity recognizer that was kind of you
2:37:15
entity recognizer that was kind of you
2:37:15
entity recognizer that was kind of you know what you used at the time and just
2:37:17
know what you used at the time and just
2:37:17
know what you used at the time and just wanted to run it maybe train it that's
2:37:19
wanted to run it maybe train it that's
2:37:19
wanted to run it maybe train it that's it
2:37:19
it
2:37:19
it now especially with the transformer
2:37:21
now especially with the transformer
2:37:21
now especially with the transformer embeddings there's some people who still
2:37:22
embeddings there's some people who still
2:37:22
embeddings there's some people who still just want to have
2:37:23
just want to have
2:37:23
just want to have great defaults something that just works
2:37:25
great defaults something that just works
2:37:25
great defaults something that just works and of course we want to be able to
2:37:26
and of course we want to be able to
2:37:26
and of course we want to be able to provide that i think that's super
2:37:27
provide that i think that's super
2:37:28
provide that i think that's super important
2:37:28
important
2:37:28
important but especially with the transformers
2:37:31
but especially with the transformers
2:37:31
but especially with the transformers they're more like you you can think of
2:37:32
they're more like you you can think of
2:37:32
they're more like you you can think of it more of like a sub
2:37:33
it more of like a sub
2:37:33
it more of like a sub network of like your larger network and
2:37:35
network of like your larger network and
2:37:35
network of like your larger network and so that also means that people
2:37:37
so that also means that people
2:37:37
so that also means that people you do want to work at a much more
2:37:38
you do want to work at a much more
2:37:38
you do want to work at a much more fine-grained level you want to be
2:37:39
fine-grained level you want to be
2:37:39
fine-grained level you want to be working at
2:37:40
working at
2:37:40
working at the tensor level and do you know
2:37:43
the tensor level and do you know
2:37:43
the tensor level and do you know use different prediction heads whatever
2:37:44
use different prediction heads whatever
2:37:44
use different prediction heads whatever you want to do and those are both
2:37:46
you want to do and those are both
2:37:46
you want to do and those are both workflows that are totally valid
2:37:48
workflows that are totally valid
2:37:48
workflows that are totally valid and that we want to support and i think
2:37:49
and that we want to support and i think
2:37:49
and that we want to support and i think that that was definitely something that
2:37:51
that that was definitely something that
2:37:51
that that was definitely something that really influenced
2:37:52
really influenced
2:37:52
really influenced that design um that we went for for
2:37:54
that design um that we went for for
2:37:54
that design um that we went for for space e3
2:37:56
space e3
2:37:56
space e3 well i think it's a time also to to get
2:37:59
well i think it's a time also to to get
2:37:59
well i think it's a time also to to get our
2:38:00
our
2:38:00
our third guest back into the in the video
2:38:04
third guest back into the in the video
2:38:04
third guest back into the in the video um let's invite verit again to the
2:38:06
um let's invite verit again to the
2:38:06
um let's invite verit again to the screen
2:38:07
screen
2:38:07
screen hi veretz thanks for joining us again
2:38:10
hi veretz thanks for joining us again
2:38:10
hi veretz thanks for joining us again um you probably saw the the section of
2:38:13
um you probably saw the the section of
2:38:13
um you probably saw the the section of ines
2:38:14
ines
2:38:14
ines just now what do you think about spaceyv
2:38:17
just now what do you think about spaceyv
2:38:17
just now what do you think about spaceyv 3.0
2:38:20
3.0
2:38:20
3.0 it's crazy i really like using it i
2:38:21
it's crazy i really like using it i
2:38:21
it's crazy i really like using it i think that among the
2:38:23
think that among the
2:38:23
think that among the different packages python packages that
2:38:26
different packages python packages that
2:38:26
different packages python packages that i'm using for my research
2:38:28
i'm using for my research
2:38:28
i'm using for my research space is i think the most reliable and
2:38:31
space is i think the most reliable and
2:38:31
space is i think the most reliable and and
2:38:31
and
2:38:31
and consistent package where you can just
2:38:34
consistent package where you can just
2:38:34
consistent package where you can just keep using the same code it doesn't
2:38:35
keep using the same code it doesn't
2:38:35
keep using the same code it doesn't change
2:38:36
change
2:38:36
change and they keep adding oh thanks no this
2:38:38
and they keep adding oh thanks no this
2:38:38
and they keep adding oh thanks no this is super nice to hear because that's
2:38:39
is super nice to hear because that's
2:38:40
is super nice to hear because that's very that's definitely something that's
2:38:41
very that's definitely something that's
2:38:41
very that's definitely something that's also important to us
2:38:42
also important to us
2:38:42
also important to us um yeah and that's something we're going
2:38:43
um yeah and that's something we're going
2:38:44
um yeah and that's something we're going for so it's great um you know that it
2:38:45
for so it's great um you know that it
2:38:45
for so it's great um you know that it actually works like that
2:38:47
actually works like that
2:38:47
actually works like that yeah i think it's true that um
2:38:50
yeah i think it's true that um
2:38:50
yeah i think it's true that um when packages are being developed by
2:38:52
when packages are being developed by
2:38:52
when packages are being developed by people who
2:38:53
people who
2:38:53
people who uh maybe emphasize their research more
2:38:56
uh maybe emphasize their research more
2:38:56
uh maybe emphasize their research more than they emphasize the
2:38:57
than they emphasize the
2:38:57
than they emphasize the ease of use then um normally you would
2:39:00
ease of use then um normally you would
2:39:00
ease of use then um normally you would get packages that
2:39:01
get packages that
2:39:01
get packages that um are in in some way hard to work with
2:39:04
um are in in some way hard to work with
2:39:04
um are in in some way hard to work with because you have to always
2:39:06
because you have to always
2:39:06
because you have to always like if you want to use your code from
2:39:08
like if you want to use your code from
2:39:08
like if you want to use your code from two years ago it wouldn't work anymore
2:39:10
two years ago it wouldn't work anymore
2:39:10
two years ago it wouldn't work anymore but
2:39:11
but
2:39:11
but this is really not a problem in space
2:39:12
this is really not a problem in space
2:39:12
this is really not a problem in space it's it seems like it
2:39:14
it's it seems like it
2:39:14
it's it seems like it it's very well thought in that aspect
2:39:17
it's very well thought in that aspect
2:39:17
it's very well thought in that aspect thanks but yeah i think also the
2:39:19
thanks but yeah i think also the
2:39:19
thanks but yeah i think also the production news aspect is definitely
2:39:21
production news aspect is definitely
2:39:21
production news aspect is definitely some you know what makes a difference
2:39:22
some you know what makes a difference
2:39:22
some you know what makes a difference and also i wouldn't i would never
2:39:23
and also i wouldn't i would never
2:39:23
and also i wouldn't i would never on research code because
2:39:25
on research code because
2:39:25
on research code because um you know this research code has a
2:39:27
um you know this research code has a
2:39:27
um you know this research code has a place and i also feel like it's not a
2:39:28
place and i also feel like it's not a
2:39:28
place and i also feel like it's not a researcher's responsibility to make sure
2:39:30
researcher's responsibility to make sure
2:39:30
researcher's responsibility to make sure that like
2:39:31
that like
2:39:31
that like um i don't know their code runs on like
2:39:35
um i don't know their code runs on like
2:39:35
um i don't know their code runs on like um you know production context and it's
2:39:36
um you know production context and it's
2:39:36
um you know production context and it's fast and runs on whatever machine or
2:39:38
fast and runs on whatever machine or
2:39:38
fast and runs on whatever machine or even runs on cpu
2:39:39
even runs on cpu
2:39:40
even runs on cpu that's like remember seeing some like
2:39:41
that's like remember seeing some like
2:39:41
that's like remember seeing some like twitter thread where i think matt said
2:39:43
twitter thread where i think matt said
2:39:43
twitter thread where i think matt said something like well look
2:39:44
something like well look
2:39:44
something like well look um asking whether it runs on cpu is a
2:39:46
um asking whether it runs on cpu is a
2:39:46
um asking whether it runs on cpu is a question like
2:39:47
question like
2:39:47
question like does it run on a toaster i don't know
2:39:49
does it run on a toaster i don't know
2:39:49
does it run on a toaster i don't know does not matter like that's not what
2:39:50
does not matter like that's not what
2:39:50
does not matter like that's not what we're trying to do here so i always
2:39:52
we're trying to do here so i always
2:39:52
we're trying to do here so i always think it's important
2:39:53
think it's important
2:39:53
think it's important to keep that distinction and it's not
2:39:55
to keep that distinction and it's not
2:39:55
to keep that distinction and it's not the research as well
2:39:57
the research as well
2:39:58
the research as well yeah and actually i find it quite
2:40:00
yeah and actually i find it quite
2:40:00
yeah and actually i find it quite interesting
2:40:01
interesting
2:40:01
interesting i love ferret that you use spacey also
2:40:03
i love ferret that you use spacey also
2:40:03
i love ferret that you use spacey also our previous guest elrond
2:40:05
our previous guest elrond
2:40:05
our previous guest elrond his uh his chat bot software also used
2:40:08
his uh his chat bot software also used
2:40:08
his uh his chat bot software also used spacey so yeah everyone's everyone's
2:40:10
spacey so yeah everyone's everyone's
2:40:10
spacey so yeah everyone's everyone's using spacey which is a treat
2:40:12
using spacey which is a treat
2:40:12
using spacey which is a treat um but one of the things i kind of
2:40:14
um but one of the things i kind of
2:40:14
um but one of the things i kind of wanted to say was
2:40:15
wanted to say was
2:40:15
wanted to say was it's it's nice to hear very that you're
2:40:17
it's it's nice to hear very that you're
2:40:17
it's it's nice to hear very that you're using it
2:40:18
using it
2:40:18
using it do and work a lot with research do you
2:40:21
do and work a lot with research do you
2:40:21
do and work a lot with research do you do you chat with people like berad and
2:40:23
do you chat with people like berad and
2:40:23
do you chat with people like berad and kind of to hear what's going on in the
2:40:25
kind of to hear what's going on in the
2:40:25
kind of to hear what's going on in the space and kind of work that into your
2:40:27
space and kind of work that into your
2:40:27
space and kind of work that into your strategy
2:40:28
strategy
2:40:28
strategy yeah of course like it's always the idea
2:40:31
yeah of course like it's always the idea
2:40:31
yeah of course like it's always the idea has always been well we see
2:40:32
has always been well we see
2:40:32
has always been well we see what's what people are working on what
2:40:34
what's what people are working on what
2:40:34
what's what people are working on what is working and what also can generalize
2:40:36
is working and what also can generalize
2:40:36
is working and what also can generalize to a broader context
2:40:37
to a broader context
2:40:37
to a broader context and what we can actually make use for
2:40:39
and what we can actually make use for
2:40:39
and what we can actually make use for people for people so i think you know at
2:40:41
people for people so i think you know at
2:40:41
people for people so i think you know at this point with like for example the
2:40:42
this point with like for example the
2:40:42
this point with like for example the transformers model
2:40:44
transformers model
2:40:44
transformers model we came up with an idea and with an
2:40:46
we came up with an idea and with an
2:40:46
we came up with an idea and with an architecture that actually makes it work
2:40:48
architecture that actually makes it work
2:40:48
architecture that actually makes it work for example
2:40:49
for example
2:40:49
for example for the multitask learning you can share
2:40:51
for the multitask learning you can share
2:40:51
for the multitask learning you can share component with other compo
2:40:53
component with other compo
2:40:53
component with other compo one transformer with other components
2:40:54
one transformer with other components
2:40:54
one transformer with other components like all of these ideas actually make it
2:40:56
like all of these ideas actually make it
2:40:56
like all of these ideas actually make it like
2:40:57
like
2:40:57
like work well and make it useful um so but
2:41:00
work well and make it useful um so but
2:41:00
work well and make it useful um so but there's always like a point where you
2:41:01
there's always like a point where you
2:41:01
there's always like a point where you have a lot
2:41:01
have a lot
2:41:02
have a lot of fake cutting-edge stuff that we of
2:41:03
of fake cutting-edge stuff that we of
2:41:03
of fake cutting-edge stuff that we of course follow but you you know
2:41:05
course follow but you you know
2:41:05
course follow but you you know you always want to take some time and
2:41:06
you always want to take some time and
2:41:06
you always want to take some time and some distance to see what works what
2:41:08
some distance to see what works what
2:41:08
some distance to see what works what makes sense and
2:41:09
makes sense and
2:41:09
makes sense and what do we actually want to tell people
2:41:10
what do we actually want to tell people
2:41:10
what do we actually want to tell people to use but the idea has always been yeah
2:41:12
to use but the idea has always been yeah
2:41:12
to use but the idea has always been yeah we see what works in research
2:41:14
we see what works in research
2:41:14
we see what works in research and how can we actually make that bring
2:41:16
and how can we actually make that bring
2:41:16
and how can we actually make that bring that to developers with
2:41:18
that to developers with
2:41:18
that to developers with a good development experience we
2:41:21
a good development experience we
2:41:21
a good development experience we uh ever very talked a lot about
2:41:25
uh ever very talked a lot about
2:41:25
uh ever very talked a lot about bias also within data how how is that
2:41:28
bias also within data how how is that
2:41:28
bias also within data how how is that within spacey how do you handle
2:41:29
within spacey how do you handle
2:41:30
within spacey how do you handle that i mean i think it's difficult to
2:41:33
that i mean i think it's difficult to
2:41:33
that i mean i think it's difficult to you know phrase it like that in terms of
2:41:35
you know phrase it like that in terms of
2:41:35
you know phrase it like that in terms of oh bias is it good or bad like in
2:41:36
oh bias is it good or bad like in
2:41:36
oh bias is it good or bad like in general
2:41:37
general
2:41:37
general well sure you you have data and you
2:41:39
well sure you you have data and you
2:41:39
well sure you you have data and you train a model on it and that model will
2:41:41
train a model on it and that model will
2:41:41
train a model on it and that model will reflect
2:41:41
reflect
2:41:42
reflect whatever is um in the data and that
2:41:44
whatever is um in the data and that
2:41:44
whatever is um in the data and that could be
2:41:45
could be
2:41:45
could be good or bad depending on what you want
2:41:48
good or bad depending on what you want
2:41:48
good or bad depending on what you want to do
2:41:48
to do
2:41:48
to do like for example you know if you want to
2:41:51
like for example you know if you want to
2:41:51
like for example you know if you want to analyze
2:41:51
analyze
2:41:51
analyze um comments on reddit um that people
2:41:54
um comments on reddit um that people
2:41:54
um comments on reddit um that people have posted um yeah you can train a
2:41:55
have posted um yeah you can train a
2:41:55
have posted um yeah you can train a model on that and the model will be able
2:41:57
model on that and the model will be able
2:41:57
model on that and the model will be able to reflect the sort of language people
2:41:58
to reflect the sort of language people
2:41:58
to reflect the sort of language people use and what topics people talk about
2:42:00
use and what topics people talk about
2:42:00
use and what topics people talk about and that's very interesting
2:42:02
and that's very interesting
2:42:02
and that's very interesting and then but then if you take that model
2:42:03
and then but then if you take that model
2:42:04
and then but then if you take that model and you deploy it into production or use
2:42:05
and you deploy it into production or use
2:42:05
and you deploy it into production or use it in your chat bot to talk to people
2:42:08
it in your chat bot to talk to people
2:42:08
it in your chat bot to talk to people that's pretty that could be potentially
2:42:09
that's pretty that could be potentially
2:42:09
that's pretty that could be potentially terrible because your model might go and
2:42:11
terrible because your model might go and
2:42:11
terrible because your model might go and like just i don't know
2:42:12
like just i don't know
2:42:12
like just i don't know um use um very terrible language that it
2:42:15
um use um very terrible language that it
2:42:15
um use um very terrible language that it picked up from
2:42:16
picked up from
2:42:16
picked up from from reddit if you make a generate
2:42:17
from reddit if you make a generate
2:42:17
from reddit if you make a generate language or whatever so i think it's or
2:42:19
language or whatever so i think it's or
2:42:19
language or whatever so i think it's or if you analyze
2:42:20
if you analyze
2:42:20
if you analyze text about people back in a day in
2:42:22
text about people back in a day in
2:42:22
text about people back in a day in science yeah that will have an
2:42:23
science yeah that will have an
2:42:23
science yeah that will have an incredible gender bias because most of
2:42:25
incredible gender bias because most of
2:42:25
incredible gender bias because most of the people that i talked about are men
2:42:27
the people that i talked about are men
2:42:27
the people that i talked about are men um not that women didn't exist but women
2:42:29
um not that women didn't exist but women
2:42:29
um not that women didn't exist but women weren't written about so
2:42:31
weren't written about so
2:42:31
weren't written about so yeah that's just that's a fact and
2:42:33
yeah that's just that's a fact and
2:42:33
yeah that's just that's a fact and that's not something we need to
2:42:35
that's not something we need to
2:42:35
that's not something we need to fix the problem comes when you know
2:42:37
fix the problem comes when you know
2:42:37
fix the problem comes when you know you're taking a model and you're using
2:42:39
you're taking a model and you're using
2:42:39
you're taking a model and you're using it and what you're actually trying to do
2:42:41
it and what you're actually trying to do
2:42:41
it and what you're actually trying to do with it and i think that's
2:42:42
with it and i think that's
2:42:42
with it and i think that's so it's i think it comes down a lot more
2:42:43
so it's i think it comes down a lot more
2:42:43
so it's i think it comes down a lot more to the that sort of awareness rather
2:42:46
to the that sort of awareness rather
2:42:46
to the that sort of awareness rather than you know
2:42:47
than you know
2:42:47
than you know kind of library fixed bias no it's like
2:42:49
kind of library fixed bias no it's like
2:42:49
kind of library fixed bias no it's like bias is not
2:42:50
bias is not
2:42:50
bias is not inherently bad it's just some you know
2:42:52
inherently bad it's just some you know
2:42:52
inherently bad it's just some you know something that can have a really bad
2:42:53
something that can have a really bad
2:42:53
something that can have a really bad impact
2:42:54
impact
2:42:54
impact that's what you put what you what goes
2:42:57
that's what you put what you what goes
2:42:57
that's what you put what you what goes in goes out
2:42:58
in goes out
2:42:58
in goes out yeah and i actually i'm quite critical
2:43:00
yeah and i actually i'm quite critical
2:43:00
yeah and i actually i'm quite critical of like the this
2:43:01
of like the this
2:43:02
of like the this whole like um you know anti-bias as a
2:43:06
whole like um you know anti-bias as a
2:43:06
whole like um you know anti-bias as a service
2:43:06
service
2:43:06
service um thing that like comes up now that
2:43:09
um thing that like comes up now that
2:43:09
um thing that like comes up now that it's like oh
2:43:10
it's like oh
2:43:10
it's like oh it's trendy to talk about and you know
2:43:11
it's trendy to talk about and you know
2:43:11
it's trendy to talk about and you know now you have companies who are like oh
2:43:13
now you have companies who are like oh
2:43:13
now you have companies who are like oh we promised you a model that's like
2:43:15
we promised you a model that's like
2:43:15
we promised you a model that's like buyers free and it's like and people are
2:43:17
buyers free and it's like and people are
2:43:17
buyers free and it's like and people are like oh great if i could just
2:43:18
like oh great if i could just
2:43:18
like oh great if i could just pay you a few million and you give me
2:43:20
pay you a few million and you give me
2:43:20
pay you a few million and you give me you know certificate there's no bias in
2:43:22
you know certificate there's no bias in
2:43:22
you know certificate there's no bias in my model great
2:43:23
my model great
2:43:23
my model great um so i think you know
2:43:29
oh have we sammy are you there we might
2:43:32
oh have we sammy are you there we might
2:43:32
oh have we sammy are you there we might have lost something little i'm here i'm
2:43:34
have lost something little i'm here i'm
2:43:34
have lost something little i'm here i'm sorry
2:43:39
um actually i have a quick question
2:43:40
um actually i have a quick question
2:43:40
um actually i have a quick question following on from that and this one's
2:43:42
following on from that and this one's
2:43:42
following on from that and this one's probably for
2:43:42
probably for
2:43:42
probably for um varied a little bit more how you
2:43:45
um varied a little bit more how you
2:43:45
um varied a little bit more how you spoke
2:43:46
spoke
2:43:46
spoke a little bit about these um the problems
2:43:49
a little bit about these um the problems
2:43:49
a little bit about these um the problems we see specifically with nlp when it
2:43:51
we see specifically with nlp when it
2:43:51
we see specifically with nlp when it comes to ethics
2:43:52
comes to ethics
2:43:52
comes to ethics and so one of the interesting pieces i
2:43:55
and so one of the interesting pieces i
2:43:55
and so one of the interesting pieces i was thinking about was i was like
2:43:56
was thinking about was i was like
2:43:56
was thinking about was i was like how do you when you're doing your
2:43:59
how do you when you're doing your
2:43:59
how do you when you're doing your research
2:44:00
research
2:44:00
research kind of either comment or make reference
2:44:03
kind of either comment or make reference
2:44:03
kind of either comment or make reference to whether there would be bias because
2:44:05
to whether there would be bias because
2:44:05
to whether there would be bias because as you said like
2:44:06
as you said like
2:44:06
as you said like we know that a lot of maybe the um
2:44:09
we know that a lot of maybe the um
2:44:09
we know that a lot of maybe the um data sets that are put together uh that
2:44:12
data sets that are put together uh that
2:44:12
data sets that are put together uh that research users tend to have some biases
2:44:14
research users tend to have some biases
2:44:14
research users tend to have some biases in them
2:44:15
in them
2:44:15
in them and how do you kind of work around that
2:44:17
and how do you kind of work around that
2:44:17
and how do you kind of work around that or work with it
2:44:19
or work with it
2:44:19
or work with it um so i i agree with the ines that it's
2:44:22
um so i i agree with the ines that it's
2:44:22
um so i i agree with the ines that it's mostly about the awareness right now
2:44:24
mostly about the awareness right now
2:44:24
mostly about the awareness right now um so just so it is at
2:44:29
um so just so it is at
2:44:29
um so just so it is at various different levels so first of all
2:44:31
various different levels so first of all
2:44:31
various different levels so first of all um
2:44:32
um
2:44:32
um maybe first deciding what kind of models
2:44:36
maybe first deciding what kind of models
2:44:36
maybe first deciding what kind of models to train versus what models
2:44:38
to train versus what models
2:44:38
to train versus what models are not a good idea to train like maybe
2:44:41
are not a good idea to train like maybe
2:44:41
are not a good idea to train like maybe i i wouldn't train a comment automatic
2:44:45
i i wouldn't train a comment automatic
2:44:45
i i wouldn't train a comment automatic comment generation on on reddit data
2:44:47
comment generation on on reddit data
2:44:47
comment generation on on reddit data because i know it's
2:44:48
because i know it's
2:44:48
because i know it's full of uh hate speech um
2:44:51
full of uh hate speech um
2:44:51
full of uh hate speech um but uh of course i mean we don't have
2:44:54
but uh of course i mean we don't have
2:44:54
but uh of course i mean we don't have control
2:44:55
control
2:44:55
control on like every model we train uh would
2:44:58
on like every model we train uh would
2:44:58
on like every model we train uh would have
2:44:59
have
2:44:59
have uh various types of biases that are
2:45:01
uh various types of biases that are
2:45:01
uh various types of biases that are found in the
2:45:02
found in the
2:45:02
found in the training data um so we should mostly at
2:45:05
training data um so we should mostly at
2:45:05
training data um so we should mostly at this point be aware of that and
2:45:07
this point be aware of that and
2:45:07
this point be aware of that and um there are researchers focusing on
2:45:11
um there are researchers focusing on
2:45:11
um there are researchers focusing on on these de-biasing algorithms but they
2:45:13
on these de-biasing algorithms but they
2:45:13
on these de-biasing algorithms but they are obviously far from perfect
2:45:15
are obviously far from perfect
2:45:15
are obviously far from perfect uh among several issues because they
2:45:19
uh among several issues because they
2:45:19
uh among several issues because they it's pretty hard to to find these
2:45:21
it's pretty hard to to find these
2:45:21
it's pretty hard to to find these boundaries between
2:45:22
boundaries between
2:45:22
boundaries between what is a valid statistical um
2:45:25
what is a valid statistical um
2:45:25
what is a valid statistical um fact versus what is um something that is
2:45:28
fact versus what is um something that is
2:45:28
fact versus what is um something that is harmful or might amplify biases
2:45:31
harmful or might amplify biases
2:45:31
harmful or might amplify biases um from things that were maybe true in
2:45:33
um from things that were maybe true in
2:45:33
um from things that were maybe true in the past but we're trying to make them
2:45:35
the past but we're trying to make them
2:45:35
the past but we're trying to make them better in the future like
2:45:36
better in the future like
2:45:36
better in the future like uh gender imbalance in various fields
2:45:40
uh gender imbalance in various fields
2:45:40
uh gender imbalance in various fields um and so and and even when they
2:45:43
um and so and and even when they
2:45:43
um and so and and even when they when we maybe some people can take the
2:45:47
when we maybe some people can take the
2:45:47
when we maybe some people can take the um decision on what should be valid and
2:45:49
um decision on what should be valid and
2:45:49
um decision on what should be valid and what not
2:45:50
what not
2:45:50
what not then you still have to specifically
2:45:51
then you still have to specifically
2:45:51
then you still have to specifically define okay now we're trying to
2:45:53
define okay now we're trying to
2:45:54
define okay now we're trying to target gender bias which we can expect
2:45:56
target gender bias which we can expect
2:45:56
target gender bias which we can expect to find
2:45:57
to find
2:45:57
to find in i don't know maybe texts related to
2:46:00
in i don't know maybe texts related to
2:46:00
in i don't know maybe texts related to occupations or
2:46:01
occupations or
2:46:01
occupations or um i don't know but you have to
2:46:05
um i don't know but you have to
2:46:05
um i don't know but you have to specifically define and you there's no
2:46:08
specifically define and you there's no
2:46:08
specifically define and you there's no method that finds all the biases and
2:46:10
method that finds all the biases and
2:46:10
method that finds all the biases and fixes them
2:46:12
fixes them
2:46:12
fixes them and i i don't think there will be
2:46:13
and i i don't think there will be
2:46:13
and i i don't think there will be because it's just it's something that we
2:46:15
because it's just it's something that we
2:46:15
because it's just it's something that we as
2:46:16
as
2:46:16
as humans decide on yeah i was gonna say no
2:46:19
humans decide on yeah i was gonna say no
2:46:19
humans decide on yeah i was gonna say no it's good to point it out as well that
2:46:21
it's good to point it out as well that
2:46:21
it's good to point it out as well that is such a early area as well isn't it
2:46:24
is such a early area as well isn't it
2:46:24
is such a early area as well isn't it there are like lots of people working in
2:46:26
there are like lots of people working in
2:46:26
there are like lots of people working in that space and hopefully we can
2:46:28
that space and hopefully we can
2:46:28
that space and hopefully we can find things but that can support um
2:46:32
find things but that can support um
2:46:32
find things but that can support um maybe data scientists to know a little
2:46:33
maybe data scientists to know a little
2:46:33
maybe data scientists to know a little bit more about
2:46:35
bit more about
2:46:35
bit more about um before they send in things to
2:46:37
um before they send in things to
2:46:37
um before they send in things to production whether you know
2:46:38
production whether you know
2:46:38
production whether you know certain flags that might be able to help
2:46:40
certain flags that might be able to help
2:46:40
certain flags that might be able to help them and stuff like that so thank you
2:46:41
them and stuff like that so thank you
2:46:41
them and stuff like that so thank you for
2:46:42
for
2:46:42
for um yeah explaining that so eloquently
2:46:44
um yeah explaining that so eloquently
2:46:44
um yeah explaining that so eloquently probably actually just one thing to add
2:46:46
probably actually just one thing to add
2:46:46
probably actually just one thing to add i think also in general the
2:46:47
i think also in general the
2:46:47
i think also in general the developer culture around it is has an
2:46:50
developer culture around it is has an
2:46:50
developer culture around it is has an impact as well it's um
2:46:51
impact as well it's um
2:46:51
impact as well it's um you know it's not just a tooling if you
2:46:53
you know it's not just a tooling if you
2:46:53
you know it's not just a tooling if you have um you know if your
2:46:55
have um you know if your
2:46:55
have um you know if your um team working on things isn't very
2:46:58
um team working on things isn't very
2:46:58
um team working on things isn't very diverse at all then well you might run
2:46:59
diverse at all then well you might run
2:47:00
diverse at all then well you might run into
2:47:00
into
2:47:00
into um problems that are caused by that
2:47:02
um problems that are caused by that
2:47:02
um problems that are caused by that because nobody thinks of these issues
2:47:04
because nobody thinks of these issues
2:47:04
because nobody thinks of these issues and i think
2:47:04
and i think
2:47:04
and i think as you have more diverse teams with
2:47:06
as you have more diverse teams with
2:47:06
as you have more diverse teams with different backgrounds working on these
2:47:07
different backgrounds working on these
2:47:08
different backgrounds working on these things we also have a higher chance of
2:47:09
things we also have a higher chance of
2:47:10
things we also have a higher chance of um actually you know detecting more
2:47:12
um actually you know detecting more
2:47:12
um actually you know detecting more problems and making maybe even more
2:47:14
problems and making maybe even more
2:47:14
problems and making maybe even more reasonable decisions in the first place
2:47:17
reasonable decisions in the first place
2:47:17
reasonable decisions in the first place yeah
2:47:18
yeah
2:47:18
yeah yes if you would compare spacey compared
2:47:22
yes if you would compare spacey compared
2:47:22
yes if you would compare spacey compared with
2:47:22
with
2:47:22
with nlp tools that are api services from
2:47:26
nlp tools that are api services from
2:47:26
nlp tools that are api services from from the big cloud providers how would
2:47:28
from the big cloud providers how would
2:47:28
from the big cloud providers how would you compare it with each other
2:47:31
you compare it with each other
2:47:31
you compare it with each other um i mean i find it's kind of hard to
2:47:32
um i mean i find it's kind of hard to
2:47:32
um i mean i find it's kind of hard to compare because it's two very
2:47:33
compare because it's two very
2:47:33
compare because it's two very fundamentally
2:47:34
fundamentally
2:47:34
fundamentally um you know different things um the big
2:47:37
um you know different things um the big
2:47:37
um you know different things um the big appeal of spacey
2:47:38
appeal of spacey
2:47:38
appeal of spacey or any library is that well you can
2:47:40
or any library is that well you can
2:47:40
or any library is that well you can build stuff with it you can run it
2:47:41
build stuff with it you can run it
2:47:41
build stuff with it you can run it yourself and you own your data and
2:47:43
yourself and you own your data and
2:47:43
yourself and you own your data and models that's also what's incredibly
2:47:44
models that's also what's incredibly
2:47:44
models that's also what's incredibly important to
2:47:45
important to
2:47:45
important to i would say most of our users people
2:47:47
i would say most of our users people
2:47:47
i would say most of our users people develop their nlp
2:47:49
develop their nlp
2:47:49
develop their nlp technologies in-house that you don't
2:47:50
technologies in-house that you don't
2:47:50
technologies in-house that you don't want to be sending all your data to
2:47:53
want to be sending all your data to
2:47:53
want to be sending all your data to um insert large tech company here um you
2:47:55
um insert large tech company here um you
2:47:55
um insert large tech company here um you wanna
2:47:56
wanna
2:47:56
wanna uh you know own your day to train your
2:47:58
uh you know own your day to train your
2:47:58
uh you know own your day to train your own models and that's what a library
2:48:00
own models and that's what a library
2:48:00
own models and that's what a library lets you do
2:48:00
lets you do
2:48:00
lets you do that said sure there are lots of there
2:48:02
that said sure there are lots of there
2:48:02
that said sure there are lots of there are also lots of api services that
2:48:04
are also lots of api services that
2:48:04
are also lots of api services that kind of run a custom spacing model
2:48:06
kind of run a custom spacing model
2:48:06
kind of run a custom spacing model behind the scenes
2:48:07
behind the scenes
2:48:07
behind the scenes even from larger tech companies um
2:48:10
even from larger tech companies um
2:48:10
even from larger tech companies um but uh you know i'm not saying oh it's
2:48:12
but uh you know i'm not saying oh it's
2:48:12
but uh you know i'm not saying oh it's bad you shouldn't
2:48:13
bad you shouldn't
2:48:13
bad you shouldn't you shouldn't be using these i think for
2:48:15
you shouldn't be using these i think for
2:48:15
you shouldn't be using these i think for like there's a lot of small
2:48:16
like there's a lot of small
2:48:16
like there's a lot of small functionality
2:48:17
functionality
2:48:17
functionality where you know if that's nlp is not your
2:48:18
where you know if that's nlp is not your
2:48:18
where you know if that's nlp is not your mail main goal could be pretty useful to
2:48:20
mail main goal could be pretty useful to
2:48:20
mail main goal could be pretty useful to just plug something in
2:48:22
just plug something in
2:48:22
just plug something in for some basic translation why not
2:48:25
for some basic translation why not
2:48:25
for some basic translation why not but um often you know if you're really
2:48:27
but um often you know if you're really
2:48:27
but um often you know if you're really serious about what you're doing
2:48:28
serious about what you're doing
2:48:28
serious about what you're doing the most interesting problems and the
2:48:31
the most interesting problems and the
2:48:31
the most interesting problems and the most interesting insights you're trying
2:48:32
most interesting insights you're trying
2:48:32
most interesting insights you're trying to gain by
2:48:33
to gain by
2:48:33
to gain by doing points up for example information
2:48:35
doing points up for example information
2:48:35
doing points up for example information extraction with mlp
2:48:37
extraction with mlp
2:48:37
extraction with mlp are the things that are incredibly
2:48:38
are the things that are incredibly
2:48:38
are the things that are incredibly specific to your problem and what you're
2:48:41
specific to your problem and what you're
2:48:41
specific to your problem and what you're thinking about
2:48:42
thinking about
2:48:42
thinking about and that's the stuff that doesn't just
2:48:43
and that's the stuff that doesn't just
2:48:43
and that's the stuff that doesn't just generalize you if you could download it
2:48:45
generalize you if you could download it
2:48:45
generalize you if you could download it from the internet it wouldn't be very
2:48:46
from the internet it wouldn't be very
2:48:46
from the internet it wouldn't be very valuable
2:48:47
valuable
2:48:47
valuable uh for your company so that's where you
2:48:50
uh for your company so that's where you
2:48:50
uh for your company so that's where you know people
2:48:51
know people
2:48:51
know people train their own models and that's where
2:48:53
train their own models and that's where
2:48:53
train their own models and that's where library comes in
2:48:54
library comes in
2:48:54
library comes in yeah true i think it's a good comparison
2:48:57
yeah true i think it's a good comparison
2:48:57
yeah true i think it's a good comparison with what josh
2:48:58
with what josh
2:48:58
with what josh was telling from vmware they were using
2:49:00
was telling from vmware they were using
2:49:00
was telling from vmware they were using the birth model but of course they had
2:49:02
the birth model but of course they had
2:49:02
the birth model but of course they had to
2:49:02
to
2:49:02
to add a lot of own data to it to be able
2:49:05
add a lot of own data to it to be able
2:49:05
add a lot of own data to it to be able to recognize their own
2:49:07
to recognize their own
2:49:07
to recognize their own working vocabulary because it's not some
2:49:10
working vocabulary because it's not some
2:49:10
working vocabulary because it's not some basic language that they're using and
2:49:11
basic language that they're using and
2:49:12
basic language that they're using and the example he was giving about the
2:49:13
the example he was giving about the
2:49:13
the example he was giving about the switch
2:49:14
switch
2:49:14
switch yes which could be for uh for switching
2:49:16
yes which could be for uh for switching
2:49:16
yes which could be for uh for switching on the lights
2:49:17
on the lights
2:49:17
on the lights it could be a network switch it can mean
2:49:19
it could be a network switch it can mean
2:49:19
it could be a network switch it can mean so many different things and again that
2:49:21
so many different things and again that
2:49:21
so many different things and again that changes from
2:49:22
changes from
2:49:22
changes from one from one
2:49:25
one from one
2:49:25
one from one technology or industry to another
2:49:27
technology or industry to another
2:49:27
technology or industry to another industry
2:49:29
industry
2:49:29
industry to go to something totally different um
2:49:32
to go to something totally different um
2:49:32
to go to something totally different um how do you feel about accessibility of
2:49:34
how do you feel about accessibility of
2:49:34
how do you feel about accessibility of specific models that have been built
2:49:38
specific models that have been built
2:49:38
specific models that have been built for example the gbt3 that got uh
2:49:41
for example the gbt3 that got uh
2:49:42
for example the gbt3 that got uh released they they built it they had a
2:49:44
released they they built it they had a
2:49:44
released they they built it they had a previous version where they said it was
2:49:46
previous version where they said it was
2:49:46
previous version where they said it was not so
2:49:47
not so
2:49:47
not so it was bits well they were not sure
2:49:49
it was bits well they were not sure
2:49:49
it was bits well they were not sure about releasing it
2:49:51
about releasing it
2:49:51
about releasing it but is this thing that needs to should
2:49:53
but is this thing that needs to should
2:49:53
but is this thing that needs to should be released and open source available
2:49:55
be released and open source available
2:49:55
be released and open source available for everyone
2:49:56
for everyone
2:49:56
for everyone or you think it's better that big
2:49:58
or you think it's better that big
2:49:58
or you think it's better that big providers keep track of those things
2:50:06
um
2:50:16
i don't think that we need to keep
2:50:18
i don't think that we need to keep
2:50:18
i don't think that we need to keep training huge models
2:50:19
training huge models
2:50:19
training huge models uh at all but but if we do um
2:50:23
uh at all but but if we do um
2:50:23
uh at all but but if we do um i i do think that uh i mean i do uh
2:50:26
i i do think that uh i mean i do uh
2:50:26
i i do think that uh i mean i do uh support releasing everything and making
2:50:28
support releasing everything and making
2:50:28
support releasing everything and making everything publicly available
2:50:30
everything publicly available
2:50:30
everything publicly available and um i i would give openai the benefit
2:50:34
and um i i would give openai the benefit
2:50:34
and um i i would give openai the benefit of the doubt that they really didn't
2:50:36
of the doubt that they really didn't
2:50:36
of the doubt that they really didn't release their model because they were
2:50:37
release their model because they were
2:50:37
release their model because they were concerned about
2:50:38
concerned about
2:50:38
concerned about uh how it can be used for malicious
2:50:41
uh how it can be used for malicious
2:50:41
uh how it can be used for malicious purposes versus
2:50:42
purposes versus
2:50:42
purposes versus um because they wanted to keep them it
2:50:45
um because they wanted to keep them it
2:50:45
um because they wanted to keep them it to themselves
2:50:47
to themselves
2:50:47
to themselves but it doesn't really matter because
2:50:49
but it doesn't really matter because
2:50:49
but it doesn't really matter because eventually
2:50:50
eventually
2:50:50
eventually uh when one of these models are um
2:50:54
uh when one of these models are um
2:50:54
uh when one of these models are um are published even if they're not
2:50:55
are published even if they're not
2:50:56
are published even if they're not released it's really
2:50:57
released it's really
2:50:57
released it's really it's relatively easy for some other
2:51:00
it's relatively easy for some other
2:51:00
it's relatively easy for some other research groups to
2:51:01
research groups to
2:51:01
research groups to to to just train something similar maybe
2:51:05
to to just train something similar maybe
2:51:05
to to just train something similar maybe not in the same scale but it's not i
2:51:08
not in the same scale but it's not i
2:51:08
not in the same scale but it's not i mean they don't have like
2:51:09
mean they don't have like
2:51:09
mean they don't have like um they don't have some special
2:51:12
um they don't have some special
2:51:12
um they don't have some special uh architecture or something that wasn't
2:51:14
uh architecture or something that wasn't
2:51:14
uh architecture or something that wasn't known before they just trained it
2:51:17
known before they just trained it
2:51:17
known before they just trained it bigger and on more data um
2:51:20
bigger and on more data um
2:51:20
bigger and on more data um so if there is a risk to that it's
2:51:22
so if there is a risk to that it's
2:51:22
so if there is a risk to that it's already out there and
2:51:23
already out there and
2:51:24
already out there and i don't think there's a reason for one
2:51:26
i don't think there's a reason for one
2:51:26
i don't think there's a reason for one company or another to
2:51:27
company or another to
2:51:27
company or another to um to keep their models private
2:51:30
um to keep their models private
2:51:30
um to keep their models private not for that reason okay
2:51:34
not for that reason okay
2:51:34
not for that reason okay yeah yeah i mean i do think yeah i mean
2:51:36
yeah yeah i mean i do think yeah i mean
2:51:36
yeah yeah i mean i do think yeah i mean in general yes i
2:51:37
in general yes i
2:51:37
in general yes i um i agree um with those points and
2:51:41
um i agree um with those points and
2:51:41
um i agree um with those points and um i also think well it's that you know
2:51:43
um i also think well it's that you know
2:51:43
um i also think well it's that you know the algorithms out there
2:51:45
the algorithms out there
2:51:45
the algorithms out there that's not what you should protect um
2:51:47
that's not what you should protect um
2:51:47
that's not what you should protect um although if you look in a completely
2:51:48
although if you look in a completely
2:51:48
although if you look in a completely different direction
2:51:49
different direction
2:51:49
different direction of data privacy more general yes i do
2:51:51
of data privacy more general yes i do
2:51:52
of data privacy more general yes i do think it's good to
2:51:52
think it's good to
2:51:52
think it's good to for companies or whatever you're doing
2:51:55
for companies or whatever you're doing
2:51:55
for companies or whatever you're doing to be aware that like no you shouldn't
2:51:56
to be aware that like no you shouldn't
2:51:56
to be aware that like no you shouldn't just be sending all your data to like
2:51:58
just be sending all your data to like
2:51:58
just be sending all your data to like some random startup or some random
2:52:00
some random startup or some random
2:52:00
some random startup or some random some large tech company um and you know
2:52:03
some large tech company um and you know
2:52:03
some large tech company um and you know just give it to them
2:52:04
just give it to them
2:52:04
just give it to them like it's important i think it's good
2:52:06
like it's important i think it's good
2:52:06
like it's important i think it's good that more and more people
2:52:08
that more and more people
2:52:08
that more and more people entering the field are skilled in like
2:52:10
entering the field are skilled in like
2:52:10
entering the field are skilled in like the domain and that companies are able
2:52:11
the domain and that companies are able
2:52:11
the domain and that companies are able to develop
2:52:12
to develop
2:52:12
to develop things in-house and own what they train
2:52:15
things in-house and own what they train
2:52:15
things in-house and own what they train and own
2:52:16
and own
2:52:16
and own their data and don't um you know make
2:52:18
their data and don't um you know make
2:52:18
their data and don't um you know make themselves dependent on like one or two
2:52:19
themselves dependent on like one or two
2:52:20
themselves dependent on like one or two large companies
2:52:21
large companies
2:52:21
large companies um that then claim almost ownership over
2:52:23
um that then claim almost ownership over
2:52:23
um that then claim almost ownership over um
2:52:24
um
2:52:24
um you know everything you're doing yeah
2:52:26
you know everything you're doing yeah
2:52:26
you know everything you're doing yeah okay
2:52:28
okay
2:52:28
okay now general ai it's something that we
2:52:31
now general ai it's something that we
2:52:31
now general ai it's something that we hear
2:52:31
hear
2:52:32
hear a lot of uh people some we have a lot of
2:52:34
a lot of uh people some we have a lot of
2:52:34
a lot of uh people some we have a lot of people who do not know anything about ai
2:52:37
people who do not know anything about ai
2:52:37
people who do not know anything about ai and they think ai can fix everything it
2:52:39
and they think ai can fix everything it
2:52:39
and they think ai can fix everything it can fix the world
2:52:41
can fix the world
2:52:41
can fix the world but we know it's we're far from there
2:52:43
but we know it's we're far from there
2:52:43
but we know it's we're far from there but do you think on nlp that we will
2:52:45
but do you think on nlp that we will
2:52:45
but do you think on nlp that we will ever
2:52:46
ever
2:52:46
ever gonna be able to say uh these are nlp
2:52:49
gonna be able to say uh these are nlp
2:52:49
gonna be able to say uh these are nlp models that actually can understand
2:52:51
models that actually can understand
2:52:51
models that actually can understand everything no
2:52:54
everything no
2:52:54
everything no i mean it's
2:53:07
define human level performance um
2:53:11
define human level performance um
2:53:11
define human level performance um like even you know even the things we're
2:53:12
like even you know even the things we're
2:53:12
like even you know even the things we're looking at now where um okay you have
2:53:14
looking at now where um okay you have
2:53:14
looking at now where um okay you have one system that on a specific task
2:53:16
one system that on a specific task
2:53:16
one system that on a specific task achieves above human performance
2:53:18
achieves above human performance
2:53:18
achieves above human performance that's objective you know that's
2:53:19
that's objective you know that's
2:53:20
that's objective you know that's objectively impressive but it also
2:53:21
objectively impressive but it also
2:53:22
objectively impressive but it also um you know what it means is well on
2:53:25
um you know what it means is well on
2:53:25
um you know what it means is well on average compared to humans that have
2:53:28
average compared to humans that have
2:53:28
average compared to humans that have been presented
2:53:29
been presented
2:53:29
been presented this task um and that has done it under
2:53:31
this task um and that has done it under
2:53:31
this task um and that has done it under certain conditions and so on
2:53:33
certain conditions and so on
2:53:33
certain conditions and so on not necessarily the machine is better
2:53:34
not necessarily the machine is better
2:53:34
not necessarily the machine is better than a human because it's like
2:53:36
than a human because it's like
2:53:36
than a human because it's like not okay i think it's also it's the
2:53:40
not okay i think it's also it's the
2:53:40
not okay i think it's also it's the wrong question to ask like i always felt
2:53:41
wrong question to ask like i always felt
2:53:41
wrong question to ask like i always felt like when people ask about like agi it's
2:53:43
like when people ask about like agi it's
2:53:43
like when people ask about like agi it's like oh
2:53:44
like oh
2:53:44
like oh oh oh the machine's gonna kill us it's
2:53:46
oh oh the machine's gonna kill us it's
2:53:46
oh oh the machine's gonna kill us it's like oh if we're at that point you have
2:53:47
like oh if we're at that point you have
2:53:47
like oh if we're at that point you have like you have other problems like it's
2:53:49
like you have other problems like it's
2:53:49
like you have other problems like it's it's not it's not
2:53:51
it's not it's not
2:53:51
it's not it's not worth it to like i don't know engage on
2:53:53
worth it to like i don't know engage on
2:53:53
worth it to like i don't know engage on that sort of level because i think it's
2:53:55
that sort of level because i think it's
2:53:55
that sort of level because i think it's um to the point there's so many other
2:53:57
um to the point there's so many other
2:53:57
um to the point there's so many other like more you know pressing
2:53:59
like more you know pressing
2:53:59
like more you know pressing questions and answers and problems um
2:54:01
questions and answers and problems um
2:54:01
questions and answers and problems um well
2:54:02
well
2:54:02
well yeah very pointed out many many problems
2:54:04
yeah very pointed out many many problems
2:54:04
yeah very pointed out many many problems still that won't be getting us to that
2:54:06
still that won't be getting us to that
2:54:06
still that won't be getting us to that point right
2:54:07
point right
2:54:07
point right so um no that's it it's so so good
2:54:10
so um no that's it it's so so good
2:54:10
so um no that's it it's so so good i guess maybe something slightly closer
2:54:11
i guess maybe something slightly closer
2:54:11
i guess maybe something slightly closer to home then for you
2:54:13
to home then for you
2:54:13
to home then for you is um the direction we're going down at
2:54:16
is um the direction we're going down at
2:54:16
is um the direction we're going down at the moment with nlp
2:54:18
the moment with nlp
2:54:18
the moment with nlp is that other other approaches happening
2:54:20
is that other other approaches happening
2:54:20
is that other other approaches happening as well that are a little bit different
2:54:21
as well that are a little bit different
2:54:21
as well that are a little bit different and can you tell us a bit about them
2:54:24
and can you tell us a bit about them
2:54:24
and can you tell us a bit about them um so i think that um maybe just to add
2:54:27
um so i think that um maybe just to add
2:54:27
um so i think that um maybe just to add first to the uh previous question i
2:54:29
first to the uh previous question i
2:54:29
first to the uh previous question i think that uh agi is
2:54:31
think that uh agi is
2:54:31
think that uh agi is is also the wrong question because
2:54:34
is also the wrong question because
2:54:34
is also the wrong question because currently we don't train any model
2:54:35
currently we don't train any model
2:54:35
currently we don't train any model that's good at
2:54:36
that's good at
2:54:36
that's good at everything we train specific models that
2:54:39
everything we train specific models that
2:54:39
everything we train specific models that are good at specific tasks on specific
2:54:41
are good at specific tasks on specific
2:54:41
are good at specific tasks on specific data sets
2:54:42
data sets
2:54:42
data sets um we're lucky enough to have these
2:54:44
um we're lucky enough to have these
2:54:44
um we're lucky enough to have these pre-trained models that have at least
2:54:46
pre-trained models that have at least
2:54:46
pre-trained models that have at least a certain level of being able to
2:54:49
a certain level of being able to
2:54:49
a certain level of being able to represent
2:54:50
represent
2:54:50
represent uh text in in natural language but um
2:54:53
uh text in in natural language but um
2:54:53
uh text in in natural language but um but we are building very specific models
2:54:56
but we are building very specific models
2:54:56
but we are building very specific models um
2:54:56
um
2:54:56
um and um specifically for
2:55:00
and um specifically for
2:55:00
and um specifically for nlp i i don't know to say that
2:55:03
nlp i i don't know to say that
2:55:03
nlp i i don't know to say that we're not going to have a model that
2:55:05
we're not going to have a model that
2:55:05
we're not going to have a model that understands language but
2:55:07
understands language but
2:55:07
understands language but it doesn't seem like we're on the way
2:55:08
it doesn't seem like we're on the way
2:55:08
it doesn't seem like we're on the way there now and i think the vast majority
2:55:11
there now and i think the vast majority
2:55:11
there now and i think the vast majority of work on
2:55:11
of work on
2:55:12
of work on nlp currently is just based on deep
2:55:14
nlp currently is just based on deep
2:55:14
nlp currently is just based on deep learning uh
2:55:16
learning uh
2:55:16
learning uh i would even say that it's it's quite
2:55:18
i would even say that it's it's quite
2:55:18
i would even say that it's it's quite hard i think to get a paper accepted to
2:55:20
hard i think to get a paper accepted to
2:55:20
hard i think to get a paper accepted to a conference
2:55:21
a conference
2:55:21
a conference an lb conference if you're not working
2:55:23
an lb conference if you're not working
2:55:23
an lb conference if you're not working on uh deep learning
2:55:25
on uh deep learning
2:55:25
on uh deep learning uh and i don't think that the um
2:55:28
uh and i don't think that the um
2:55:28
uh and i don't think that the um the model that understands language
2:55:29
the model that understands language
2:55:29
the model that understands language would come from just training
2:55:31
would come from just training
2:55:31
would come from just training larger models with i don't know more
2:55:33
larger models with i don't know more
2:55:33
larger models with i don't know more layers or more data
2:55:35
layers or more data
2:55:35
layers or more data uh it definitely helps for some things
2:55:37
uh it definitely helps for some things
2:55:37
uh it definitely helps for some things but uh we're gonna have to have
2:55:39
but uh we're gonna have to have
2:55:39
but uh we're gonna have to have some combination with a different
2:55:42
some combination with a different
2:55:42
some combination with a different different approach
2:55:43
different approach
2:55:43
different approach which i don't know yeah yeah i think
2:55:45
which i don't know yeah yeah i think
2:55:45
which i don't know yeah yeah i think this is interesting you say that because
2:55:47
this is interesting you say that because
2:55:47
this is interesting you say that because i feel like this also lines up quite
2:55:48
i feel like this also lines up quite
2:55:48
i feel like this also lines up quite well with what we're seeing more from
2:55:49
well with what we're seeing more from
2:55:49
well with what we're seeing more from like you know industry use cases
2:55:51
like you know industry use cases
2:55:51
like you know industry use cases and it's like yeah it's cool that you
2:55:53
and it's like yeah it's cool that you
2:55:53
and it's like yeah it's cool that you know you have these language models you
2:55:54
know you have these language models you
2:55:54
know you have these language models you can do a lot more
2:55:55
can do a lot more
2:55:55
can do a lot more that previously wasn't possible but
2:55:57
that previously wasn't possible but
2:55:57
that previously wasn't possible but there's always
2:55:58
there's always
2:55:58
there's always an additional need for stuff that maybe
2:56:00
an additional need for stuff that maybe
2:56:00
an additional need for stuff that maybe you know isn't an end-to-end
2:56:02
you know isn't an end-to-end
2:56:02
you know isn't an end-to-end um prediction task or something you know
2:56:04
um prediction task or something you know
2:56:04
um prediction task or something you know where
2:56:05
where
2:56:05
where you train a model to do exactly the
2:56:08
you train a model to do exactly the
2:56:08
you train a model to do exactly the business
2:56:08
business
2:56:08
business purpose you know that exactly solves the
2:56:11
purpose you know that exactly solves the
2:56:11
purpose you know that exactly solves the business um
2:56:12
business um
2:56:12
business um problem that you have like that's also
2:56:14
problem that you have like that's also
2:56:14
problem that you have like that's also quite unrealistic like many
2:56:16
quite unrealistic like many
2:56:16
quite unrealistic like many of our users surely they might want to
2:56:18
of our users surely they might want to
2:56:18
of our users surely they might want to use like the latest state-of-the-art
2:56:20
use like the latest state-of-the-art
2:56:20
use like the latest state-of-the-art cutting-edge transformer system but then
2:56:23
cutting-edge transformer system but then
2:56:23
cutting-edge transformer system but then you know they'd add you know you want to
2:56:24
you know they'd add you know you want to
2:56:24
you know they'd add you know you want to add a component on top that makes sure
2:56:26
add a component on top that makes sure
2:56:26
add a component on top that makes sure that it
2:56:27
that it
2:56:27
that it definitely always gets your company name
2:56:29
definitely always gets your company name
2:56:29
definitely always gets your company name correctly recognized and you do that by
2:56:31
correctly recognized and you do that by
2:56:31
correctly recognized and you do that by writing a regular expression
2:56:33
writing a regular expression
2:56:33
writing a regular expression or you know you want to augment your
2:56:35
or you know you want to augment your
2:56:35
or you know you want to augment your system with stuff that you can look up
2:56:37
system with stuff that you can look up
2:56:37
system with stuff that you can look up somewhere on the internet and so on like
2:56:39
somewhere on the internet and so on like
2:56:39
somewhere on the internet and so on like there's a real-life solution often
2:56:41
there's a real-life solution often
2:56:41
there's a real-life solution often consists of a lot more parts
2:56:43
consists of a lot more parts
2:56:43
consists of a lot more parts um yes
2:56:47
um yes
2:56:47
um yes yeah i know i i think we see that a lot
2:56:49
yeah i know i i think we see that a lot
2:56:49
yeah i know i i think we see that a lot as well like i work a lot with um
2:56:52
as well like i work a lot with um
2:56:52
as well like i work a lot with um cloud architectures and so that is like
2:56:54
cloud architectures and so that is like
2:56:54
cloud architectures and so that is like a huge point that we sometimes
2:56:56
a huge point that we sometimes
2:56:56
a huge point that we sometimes i sometimes like get across where i'm
2:56:57
i sometimes like get across where i'm
2:56:58
i sometimes like get across where i'm like often a lot of these solutions are
2:57:00
like often a lot of these solutions are
2:57:00
like often a lot of these solutions are like
2:57:00
like
2:57:00
like multiple pieces and you pick the right
2:57:03
multiple pieces and you pick the right
2:57:03
multiple pieces and you pick the right stuff
2:57:04
stuff
2:57:04
stuff um elrone was talking a little bit about
2:57:06
um elrone was talking a little bit about
2:57:06
um elrone was talking a little bit about ensemble models similar kind of idea
2:57:08
ensemble models similar kind of idea
2:57:08
ensemble models similar kind of idea right using your strengths when you need
2:57:10
right using your strengths when you need
2:57:10
right using your strengths when you need them and then
2:57:11
them and then
2:57:11
them and then kind of going down the right route
2:57:12
kind of going down the right route
2:57:12
kind of going down the right route hopefully to
2:57:14
hopefully to
2:57:14
hopefully to make something seem intelligent and we
2:57:17
make something seem intelligent and we
2:57:17
make something seem intelligent and we also had a really
2:57:18
also had a really
2:57:18
also had a really amazing talk on kind of the history of
2:57:20
amazing talk on kind of the history of
2:57:20
amazing talk on kind of the history of ai in our first session at the start of
2:57:22
ai in our first session at the start of
2:57:22
ai in our first session at the start of october
2:57:23
october
2:57:23
october and it's interesting varied what you say
2:57:25
and it's interesting varied what you say
2:57:25
and it's interesting varied what you say and and as a kind of about
2:57:27
and and as a kind of about
2:57:27
and and as a kind of about okay we're running down this neural
2:57:29
okay we're running down this neural
2:57:29
okay we're running down this neural network group which
2:57:30
network group which
2:57:30
network group which was like the opposite in history like we
2:57:33
was like the opposite in history like we
2:57:33
was like the opposite in history like we were running down like svns and like
2:57:35
were running down like svns and like
2:57:35
were running down like svns and like more
2:57:35
more
2:57:36
more kind of traditional um machine learning
2:57:38
kind of traditional um machine learning
2:57:38
kind of traditional um machine learning routes and then
2:57:39
routes and then
2:57:39
routes and then all of a sudden connectionism and that
2:57:41
all of a sudden connectionism and that
2:57:41
all of a sudden connectionism and that became like the big thing
2:57:42
became like the big thing
2:57:42
became like the big thing and it like broke the glass ceilings
2:57:44
and it like broke the glass ceilings
2:57:44
and it like broke the glass ceilings that we'd seen before
2:57:46
that we'd seen before
2:57:46
that we'd seen before and so yeah it's fascinating to think
2:57:47
and so yeah it's fascinating to think
2:57:47
and so yeah it's fascinating to think like okay like we're working on this
2:57:49
like okay like we're working on this
2:57:49
like okay like we're working on this route now and this is this is good but i
2:57:51
route now and this is this is good but i
2:57:51
route now and this is this is good but i wonder what's coming next like yeah that
2:57:53
wonder what's coming next like yeah that
2:57:53
wonder what's coming next like yeah that always boggles my mind
2:57:56
always boggles my mind
2:57:56
always boggles my mind yeah so i think we kind of see people
2:57:58
yeah so i think we kind of see people
2:57:58
yeah so i think we kind of see people going a bit back to like you know the
2:58:00
going a bit back to like you know the
2:58:00
going a bit back to like you know the language itself and more you know the
2:58:02
language itself and more you know the
2:58:02
language itself and more you know the computational linguistic side of things
2:58:04
computational linguistic side of things
2:58:04
computational linguistic side of things so at least that's kind of you know what
2:58:05
so at least that's kind of you know what
2:58:05
so at least that's kind of you know what i'm
2:58:05
i'm
2:58:05
i'm hoping um uh you know we because it
2:58:09
hoping um uh you know we because it
2:58:09
hoping um uh you know we because it that stuff is relevant if you want to
2:58:11
that stuff is relevant if you want to
2:58:11
that stuff is relevant if you want to reason about which technology to choose
2:58:13
reason about which technology to choose
2:58:13
reason about which technology to choose and i think that's what it comes down to
2:58:14
and i think that's what it comes down to
2:58:14
and i think that's what it comes down to we have so many tools available
2:58:15
we have so many tools available
2:58:15
we have so many tools available but you still need to pick the right
2:58:17
but you still need to pick the right
2:58:17
but you still need to pick the right tool out of your toolbox like oh you
2:58:19
tool out of your toolbox like oh you
2:58:19
tool out of your toolbox like oh you want to you know you want to put a
2:58:20
want to you know you want to put a
2:58:20
want to you know you want to put a picture on the wall what you have all
2:58:21
picture on the wall what you have all
2:58:21
picture on the wall what you have all these tools do you use a hammer do you
2:58:23
these tools do you use a hammer do you
2:58:23
these tools do you use a hammer do you use like
2:58:24
use like
2:58:24
use like you know the heavy um i don't know drill
2:58:27
you know the heavy um i don't know drill
2:58:27
you know the heavy um i don't know drill what do you what do you do and that's a
2:58:28
what do you what do you do and that's a
2:58:28
what do you what do you do and that's a decision you still need to make and
2:58:30
decision you still need to make and
2:58:30
decision you still need to make and often that does come down to well
2:58:31
often that does come down to well
2:58:31
often that does come down to well understanding the technology but also
2:58:33
understanding the technology but also
2:58:33
understanding the technology but also understanding how does language work
2:58:34
understanding how does language work
2:58:34
understanding how does language work how does language work across different
2:58:36
how does language work across different
2:58:36
how does language work across different languages what are um the challenges and
2:58:38
languages what are um the challenges and
2:58:38
languages what are um the challenges and how
2:58:39
how
2:58:39
how do you actually solve um the problem
2:58:42
do you actually solve um the problem
2:58:42
do you actually solve um the problem with all the tools we have available
2:58:44
with all the tools we have available
2:58:44
with all the tools we have available somehow brings me a bit to the oh okay
2:58:46
somehow brings me a bit to the oh okay
2:58:46
somehow brings me a bit to the oh okay go ahead harry sorry oh no
2:58:48
go ahead harry sorry oh no
2:58:48
go ahead harry sorry oh no i just want to say i think it's a good
2:58:49
i just want to say i think it's a good
2:58:49
i just want to say i think it's a good approach to think of deep learning as a
2:58:51
approach to think of deep learning as a
2:58:51
approach to think of deep learning as a tool
2:58:51
tool
2:58:51
tool out of many possible tools that we can
2:58:54
out of many possible tools that we can
2:58:54
out of many possible tools that we can use
2:58:56
use
2:58:56
use true so it brings me a bit to the more
2:58:59
true so it brings me a bit to the more
2:58:59
true so it brings me a bit to the more to the question
2:59:00
to the question
2:59:00
to the question um if we look to the future where will
2:59:02
um if we look to the future where will
2:59:02
um if we look to the future where will nlp bring us
2:59:03
nlp bring us
2:59:03
nlp bring us what's the what's new what's going to
2:59:06
what's the what's new what's going to
2:59:06
what's the what's new what's going to change in
2:59:07
change in
2:59:07
change in nlp that we don't have now but that we
2:59:09
nlp that we don't have now but that we
2:59:09
nlp that we don't have now but that we might have in
2:59:11
might have in
2:59:11
might have in let's say five years
2:59:14
let's say five years
2:59:14
let's say five years um i don't know about applications but i
2:59:17
um i don't know about applications but i
2:59:17
um i don't know about applications but i think maybe
2:59:18
think maybe
2:59:18
think maybe uh more collaboration with other
2:59:21
uh more collaboration with other
2:59:21
uh more collaboration with other fields like uh maybe i mentioned in the
2:59:24
fields like uh maybe i mentioned in the
2:59:24
fields like uh maybe i mentioned in the end of my presentation maybe
2:59:26
end of my presentation maybe
2:59:26
end of my presentation maybe combining language and vision there's
2:59:28
combining language and vision there's
2:59:28
combining language and vision there's already some work
2:59:30
already some work
2:59:30
already some work on that but um i think maybe more of
2:59:34
on that but um i think maybe more of
2:59:34
on that but um i think maybe more of that because
2:59:35
that because
2:59:35
that because we're starting to talk like in the nlp
2:59:38
we're starting to talk like in the nlp
2:59:38
we're starting to talk like in the nlp community
2:59:38
community
2:59:38
community we're starting to talk about the
2:59:39
we're starting to talk about the
2:59:39
we're starting to talk about the limitations of learning just from text
2:59:43
limitations of learning just from text
2:59:43
limitations of learning just from text because humans don't learn about the
2:59:45
because humans don't learn about the
2:59:45
because humans don't learn about the world just from text so you can't really
2:59:47
world just from text so you can't really
2:59:47
world just from text so you can't really expect a model to do that
2:59:49
expect a model to do that
2:59:49
expect a model to do that we also see things and we hear things
2:59:51
we also see things and we hear things
2:59:51
we also see things and we hear things and
2:59:52
and
2:59:52
and um use all our senses so um
2:59:55
um use all our senses so um
2:59:56
um use all our senses so um yeah some something like that maybe a
2:59:58
yeah some something like that maybe a
2:59:58
yeah some something like that maybe a combination with vision or with robotics
3:00:02
combination with vision or with robotics
3:00:02
combination with vision or with robotics yeah i think from my perspective um one
3:00:05
yeah i think from my perspective um one
3:00:05
yeah i think from my perspective um one thing definitely we're already seeing
3:00:06
thing definitely we're already seeing
3:00:06
thing definitely we're already seeing and that we're
3:00:07
and that we're
3:00:07
and that we're hopefully going to be seeing even more
3:00:09
hopefully going to be seeing even more
3:00:09
hopefully going to be seeing even more of is more people from all kinds of
3:00:10
of is more people from all kinds of
3:00:10
of is more people from all kinds of different
3:00:11
different
3:00:11
different areas and domains and research fields um
3:00:14
areas and domains and research fields um
3:00:14
areas and domains and research fields um using nlp um in their work like we
3:00:17
using nlp um in their work like we
3:00:17
using nlp um in their work like we already have see a lot in like digital
3:00:19
already have see a lot in like digital
3:00:19
already have see a lot in like digital humanities
3:00:19
humanities
3:00:20
humanities social sciences but there's so many
3:00:21
social sciences but there's so many
3:00:21
social sciences but there's so many other fields
3:00:23
other fields
3:00:23
other fields and with the technology actually
3:00:24
and with the technology actually
3:00:24
and with the technology actually becoming you know more usable and more
3:00:26
becoming you know more usable and more
3:00:26
becoming you know more usable and more people just you know picking up
3:00:27
people just you know picking up
3:00:27
people just you know picking up um from programming um because you know
3:00:30
um from programming um because you know
3:00:30
um from programming um because you know it works and
3:00:31
it works and
3:00:31
it works and bringing in the domain expertise so i
3:00:33
bringing in the domain expertise so i
3:00:33
bringing in the domain expertise so i think that's that's gonna be really
3:00:34
think that's that's gonna be really
3:00:34
think that's that's gonna be really interesting
3:00:35
interesting
3:00:35
interesting um otherwise also that's more kind of
3:00:38
um otherwise also that's more kind of
3:00:38
um otherwise also that's more kind of from a developer experience perspective
3:00:39
from a developer experience perspective
3:00:40
from a developer experience perspective like a lot of
3:00:41
like a lot of
3:00:41
like a lot of python is probably one of the main
3:00:42
python is probably one of the main
3:00:42
python is probably one of the main languages people use and a lot of
3:00:44
languages people use and a lot of
3:00:44
languages people use and a lot of exciting developments
3:00:45
exciting developments
3:00:45
exciting developments in that ecosystem which also um you know
3:00:48
in that ecosystem which also um you know
3:00:48
in that ecosystem which also um you know briefly mentioned in my talk um
3:00:50
briefly mentioned in my talk um
3:00:50
briefly mentioned in my talk um and which we're also you know focusing
3:00:52
and which we're also you know focusing
3:00:52
and which we're also you know focusing on a lot within some of the stuff we're
3:00:54
on a lot within some of the stuff we're
3:00:54
on a lot within some of the stuff we're doing like type hints
3:00:55
doing like type hints
3:00:55
doing like type hints um you know some of the new um you know
3:00:58
um you know some of the new um you know
3:00:58
um you know some of the new um you know new stuff you could do
3:00:59
new stuff you could do
3:00:59
new stuff you could do um uh yeah and also all of that kind of
3:01:03
um uh yeah and also all of that kind of
3:01:03
um uh yeah and also all of that kind of in
3:01:03
in
3:01:04
in also enables um a much um
3:01:08
also enables um a much um
3:01:08
also enables um a much um a closer uh cycle between
3:01:11
a closer uh cycle between
3:01:11
a closer uh cycle between um research and production or it brings
3:01:13
um research and production or it brings
3:01:13
um research and production or it brings basically brings research and production
3:01:15
basically brings research and production
3:01:15
basically brings research and production closer
3:01:15
closer
3:01:16
closer um together a prototype and production
3:01:18
um together a prototype and production
3:01:18
um together a prototype and production that sort i mean actually
3:01:19
that sort i mean actually
3:01:19
that sort i mean actually you know research maybe um you know
3:01:22
you know research maybe um you know
3:01:22
you know research maybe um you know should maybe also
3:01:23
should maybe also
3:01:23
should maybe also always be kind of separate but like the
3:01:25
always be kind of separate but like the
3:01:25
always be kind of separate but like the idea of okay you're trying something out
3:01:26
idea of okay you're trying something out
3:01:26
idea of okay you're trying something out um you're prototyping something you're
3:01:28
um you're prototyping something you're
3:01:28
um you're prototyping something you're actually shipping it um
3:01:31
actually shipping it um
3:01:31
actually shipping it um that sort of um loop um if we close that
3:01:34
that sort of um loop um if we close that
3:01:34
that sort of um loop um if we close that gap
3:01:35
gap
3:01:35
gap i think um it will also be much easier
3:01:37
i think um it will also be much easier
3:01:37
i think um it will also be much easier to build systems that work
3:01:38
to build systems that work
3:01:38
to build systems that work um yeah yeah oh that that's so true
3:01:41
um yeah yeah oh that that's so true
3:01:41
um yeah yeah oh that that's so true isn't it and i think now i don't know
3:01:42
isn't it and i think now i don't know
3:01:42
isn't it and i think now i don't know about
3:01:43
about
3:01:43
about all of you i feel like there's um maybe
3:01:45
all of you i feel like there's um maybe
3:01:46
all of you i feel like there's um maybe like more of a demand than ever for
3:01:48
like more of a demand than ever for
3:01:48
like more of a demand than ever for someone in the data science space to
3:01:49
someone in the data science space to
3:01:49
someone in the data science space to have to have a really large set of
3:01:51
have to have a really large set of
3:01:51
have to have a really large set of skills
3:01:52
skills
3:01:52
skills like you're not now just even focusing
3:01:54
like you're not now just even focusing
3:01:54
like you're not now just even focusing on the model elements
3:01:55
on the model elements
3:01:55
on the model elements you're now focusing on like how do i
3:01:57
you're now focusing on like how do i
3:01:57
you're now focusing on like how do i make sure this is production ready code
3:01:59
make sure this is production ready code
3:01:59
make sure this is production ready code how does this
3:02:00
how does this
3:02:00
how does this uh root into config files and stuff like
3:02:02
uh root into config files and stuff like
3:02:02
uh root into config files and stuff like that
3:02:03
that
3:02:03
that interestingly in us someone um wrote us
3:02:06
interestingly in us someone um wrote us
3:02:06
interestingly in us someone um wrote us in the audience
3:02:07
in the audience
3:02:07
in the audience said um spacey v3 is super cool
3:02:10
said um spacey v3 is super cool
3:02:10
said um spacey v3 is super cool i wanna throw that one in there straight
3:02:12
i wanna throw that one in there straight
3:02:12
i wanna throw that one in there straight away for you but they also depart and
3:02:14
away for you but they also depart and
3:02:14
away for you but they also depart and i'm just starting to understand a lot of
3:02:16
i'm just starting to understand a lot of
3:02:16
i'm just starting to understand a lot of the new elements we just realized that
3:02:18
the new elements we just realized that
3:02:18
the new elements we just realized that the config system
3:02:19
the config system
3:02:20
the config system fits perfectly for non-machine learning
3:02:22
fits perfectly for non-machine learning
3:02:22
fits perfectly for non-machine learning projects
3:02:23
projects
3:02:23
projects um they said did you do did you consider
3:02:26
um they said did you do did you consider
3:02:26
um they said did you do did you consider having the config system as its own
3:02:28
having the config system as its own
3:02:28
having the config system as its own package
3:02:28
package
3:02:28
package instead of as a part of think yeah
3:02:31
instead of as a part of think yeah
3:02:31
instead of as a part of think yeah that's a really interesting question um
3:02:33
that's a really interesting question um
3:02:33
that's a really interesting question um i mean it's it's a good question because
3:02:35
i mean it's it's a good question because
3:02:35
i mean it's it's a good question because it's something we've been discussing
3:02:36
it's something we've been discussing
3:02:36
it's something we've been discussing internally
3:02:37
internally
3:02:37
internally um so advice that someone else had this
3:02:39
um so advice that someone else had this
3:02:39
um so advice that someone else had this idea
3:02:40
idea
3:02:40
idea because we were like i'm not sure if
3:02:41
because we were like i'm not sure if
3:02:41
because we were like i'm not sure if that's something people want but like
3:02:43
that's something people want but like
3:02:43
that's something people want but like yes while we were building this we were
3:02:44
yes while we were building this we were
3:02:44
yes while we were building this we were like oh even you know i wasn't even
3:02:46
like oh even you know i wasn't even
3:02:46
like oh even you know i wasn't even thinking about like non-machine learning
3:02:47
thinking about like non-machine learning
3:02:48
thinking about like non-machine learning projects but i was
3:02:48
projects but i was
3:02:48
projects but i was like well it should just be its own
3:02:50
like well it should just be its own
3:02:50
like well it should just be its own library um stand-alone so
3:02:53
library um stand-alone so
3:02:53
library um stand-alone so yeah it's something you've been thinking
3:02:54
yeah it's something you've been thinking
3:02:54
yeah it's something you've been thinking about good stuff whoever that
3:02:57
about good stuff whoever that
3:02:57
about good stuff whoever that question was from uh hit enter you need
3:02:59
question was from uh hit enter you need
3:03:00
question was from uh hit enter you need a job
3:03:02
a job
3:03:02
a job and then there was just one more from
3:03:03
and then there was just one more from
3:03:04
and then there was just one more from the audience as well just to make sure
3:03:05
the audience as well just to make sure
3:03:05
the audience as well just to make sure our audience that
3:03:06
our audience that
3:03:06
our audience that gets all of their questions answered by
3:03:08
gets all of their questions answered by
3:03:08
gets all of their questions answered by our amazing speakers so um
3:03:10
our amazing speakers so um
3:03:10
our amazing speakers so um how do the transformer models in spacey
3:03:12
how do the transformer models in spacey
3:03:12
how do the transformer models in spacey uh
3:03:13
uh
3:03:13
uh v3 integrate with prodigy
3:03:16
v3 integrate with prodigy
3:03:16
v3 integrate with prodigy um that's also a good question so
3:03:19
um that's also a good question so
3:03:19
um that's also a good question so prodigy is our
3:03:20
prodigy is our
3:03:20
prodigy is our annotation tool and it also it's a
3:03:21
annotation tool and it also it's a
3:03:22
annotation tool and it also it's a developer tool um you run it locally it
3:03:24
developer tool um you run it locally it
3:03:24
developer tool um you run it locally it integrates
3:03:24
integrates
3:03:24
integrates very well with spacey obviously because
3:03:26
very well with spacey obviously because
3:03:26
very well with spacey obviously because that's also a library we develop
3:03:28
that's also a library we develop
3:03:28
that's also a library we develop um and yes of course you'll be able to
3:03:31
um and yes of course you'll be able to
3:03:31
um and yes of course you'll be able to use these
3:03:32
use these
3:03:32
use these new transformer models in prodigy for
3:03:34
new transformer models in prodigy for
3:03:34
new transformer models in prodigy for example to help you
3:03:35
example to help you
3:03:35
example to help you label your data to suggest annotations
3:03:38
label your data to suggest annotations
3:03:38
label your data to suggest annotations for you that you can then um you know
3:03:40
for you that you can then um you know
3:03:40
for you that you can then um you know just to be more efficient instead of
3:03:42
just to be more efficient instead of
3:03:42
just to be more efficient instead of just doing everything from scratch to
3:03:43
just doing everything from scratch to
3:03:43
just doing everything from scratch to correct the models
3:03:44
correct the models
3:03:44
correct the models um suggestions but we still need a bit
3:03:47
um suggestions but we still need a bit
3:03:47
um suggestions but we still need a bit more experimentation here because
3:03:49
more experimentation here because
3:03:49
more experimentation here because the reality is these models are a lot
3:03:51
the reality is these models are a lot
3:03:51
the reality is these models are a lot larger and
3:03:52
larger and
3:03:52
larger and um a bit slower so um you know really
3:03:55
um a bit slower so um you know really
3:03:55
um a bit slower so um you know really having them having them in a loop is
3:03:56
having them having them in a loop is
3:03:56
having them having them in a loop is just a bit more
3:03:57
just a bit more
3:03:57
just a bit more um um you know more challenging um but
3:04:00
um um you know more challenging um but
3:04:00
um um you know more challenging um but yeah like prodigy will just use spacey
3:04:03
yeah like prodigy will just use spacey
3:04:03
yeah like prodigy will just use spacey three out of the box soon and um then
3:04:06
three out of the box soon and um then
3:04:06
three out of the box soon and um then you know you'll just be able to use it
3:04:08
you know you'll just be able to use it
3:04:08
you know you'll just be able to use it like you currently use it fbls
3:04:11
like you currently use it fbls
3:04:11
like you currently use it fbls always good i'm sure we have a lot of
3:04:14
always good i'm sure we have a lot of
3:04:14
always good i'm sure we have a lot of viewers who who might be interested in
3:04:17
viewers who who might be interested in
3:04:17
viewers who who might be interested in llp but maybe never really started in it
3:04:20
llp but maybe never really started in it
3:04:20
llp but maybe never really started in it how do you get started in yeah in a
3:04:23
how do you get started in yeah in a
3:04:23
how do you get started in yeah in a project
3:04:24
project
3:04:24
project an nlp project how do you start learning
3:04:26
an nlp project how do you start learning
3:04:26
an nlp project how do you start learning and the skills for it
3:04:30
any any resources you'd recommend that
3:04:32
any any resources you'd recommend that
3:04:32
any any resources you'd recommend that you
3:04:33
you
3:04:33
you i think people do well with online um
3:04:36
i think people do well with online um
3:04:36
i think people do well with online um i'm always a big coursera fan
3:04:39
i'm always a big coursera fan
3:04:39
i'm always a big coursera fan i mean their machine learning course is
3:04:40
i mean their machine learning course is
3:04:40
i mean their machine learning course is unreal so
3:04:45
i was like oh maybe you're the better
3:04:47
i was like oh maybe you're the better
3:04:48
i was like oh maybe you're the better person to answer this because you're a
3:04:49
person to answer this because you're a
3:04:49
person to answer this because you're a much more classic nlp person
3:04:51
much more classic nlp person
3:04:51
much more classic nlp person yeah i'm gonna say it's a bit hard for
3:04:53
yeah i'm gonna say it's a bit hard for
3:04:53
yeah i'm gonna say it's a bit hard for me to answer that because i
3:04:54
me to answer that because i
3:04:54
me to answer that because i i learned it in the university so um
3:04:58
i learned it in the university so um
3:04:58
i learned it in the university so um it's hard to recommend an online course
3:05:00
it's hard to recommend an online course
3:05:00
it's hard to recommend an online course that i know that is good
3:05:02
that i know that is good
3:05:02
that i know that is good uh but i know i mean there are many
3:05:04
uh but i know i mean there are many
3:05:04
uh but i know i mean there are many courses um
3:05:06
courses um
3:05:06
courses um available i would look maybe for um
3:05:09
available i would look maybe for um
3:05:09
available i would look maybe for um maybe from one of the big universities
3:05:11
maybe from one of the big universities
3:05:11
maybe from one of the big universities sometimes they uh
3:05:12
sometimes they uh
3:05:12
sometimes they uh they upload their introductory nlp
3:05:16
they upload their introductory nlp
3:05:16
they upload their introductory nlp course videos
3:05:17
course videos
3:05:17
course videos i think maybe that that could be a
3:05:21
i think maybe that that could be a
3:05:21
i think maybe that that could be a good resource but also depending on the
3:05:24
good resource but also depending on the
3:05:24
good resource but also depending on the need if you
3:05:24
need if you
3:05:24
need if you just want to quickly learn how to
3:05:27
just want to quickly learn how to
3:05:27
just want to quickly learn how to develop a
3:05:28
develop a
3:05:28
develop a simple tool then maybe just look for a
3:05:30
simple tool then maybe just look for a
3:05:30
simple tool then maybe just look for a tutorial for specifically what you need
3:05:32
tutorial for specifically what you need
3:05:32
tutorial for specifically what you need as opposed to if you just want to learn
3:05:34
as opposed to if you just want to learn
3:05:34
as opposed to if you just want to learn about the field
3:05:37
about the field
3:05:37
about the field i guess if you're interested in spacey
3:05:39
i guess if you're interested in spacey
3:05:40
i guess if you're interested in spacey more specifically um we we do have a
3:05:42
more specifically um we we do have a
3:05:42
more specifically um we we do have a free online course that i um built which
3:05:44
free online course that i um built which
3:05:44
free online course that i um built which is kind of interactive you can do like
3:05:45
is kind of interactive you can do like
3:05:45
is kind of interactive you can do like code exercises and it also
3:05:47
code exercises and it also
3:05:47
code exercises and it also sort of works i think as kind of a nice
3:05:49
sort of works i think as kind of a nice
3:05:49
sort of works i think as kind of a nice intro to nlp
3:05:51
intro to nlp
3:05:51
intro to nlp and you learn um spacey along the way
3:05:53
and you learn um spacey along the way
3:05:53
and you learn um spacey along the way and it's course.spacey.io you can you
3:05:55
and it's course.spacey.io you can you
3:05:55
and it's course.spacey.io you can you can find it online
3:05:56
can find it online
3:05:56
can find it online um and you can just do it in your
3:05:57
um and you can just do it in your
3:05:57
um and you can just do it in your browser so that might be cool like it's
3:05:59
browser so that might be cool like it's
3:05:59
browser so that might be cool like it's not going to teach you all about nlp but
3:06:00
not going to teach you all about nlp but
3:06:00
not going to teach you all about nlp but like it's
3:06:01
like it's
3:06:01
like it's i mean i think it serves that one
3:06:03
i mean i think it serves that one
3:06:03
i mean i think it serves that one purpose and i guess from my experience
3:06:06
purpose and i guess from my experience
3:06:06
purpose and i guess from my experience i personally i always need like a
3:06:07
i personally i always need like a
3:06:07
i personally i always need like a project to work on like i'm not i can't
3:06:09
project to work on like i'm not i can't
3:06:09
project to work on like i'm not i can't sit down and just learn a thing
3:06:11
sit down and just learn a thing
3:06:11
sit down and just learn a thing um i'm like okay often i'm really bad at
3:06:13
um i'm like okay often i'm really bad at
3:06:13
um i'm like okay often i'm really bad at giving advice i usually avoid giving
3:06:15
giving advice i usually avoid giving
3:06:15
giving advice i usually avoid giving people advice
3:06:15
people advice
3:06:16
people advice on these things but if if there's one
3:06:17
on these things but if if there's one
3:06:17
on these things but if if there's one piece of advice it'd probably be like
3:06:19
piece of advice it'd probably be like
3:06:19
piece of advice it'd probably be like well
3:06:19
well
3:06:19
well i don't know find something else you're
3:06:20
i don't know find something else you're
3:06:20
i don't know find something else you're passionate about like maybe you're
3:06:22
passionate about like maybe you're
3:06:22
passionate about like maybe you're passionate about
3:06:23
passionate about
3:06:23
passionate about football or you're passionate about food
3:06:26
football or you're passionate about food
3:06:26
football or you're passionate about food and then
3:06:26
and then
3:06:26
and then you can somehow combine that with like
3:06:29
you can somehow combine that with like
3:06:29
you can somehow combine that with like some interesting problem or insight you
3:06:31
some interesting problem or insight you
3:06:31
some interesting problem or insight you want to get like oh maybe you want to
3:06:32
want to get like oh maybe you want to
3:06:32
want to get like oh maybe you want to analyze a bunch of text and find
3:06:34
analyze a bunch of text and find
3:06:34
analyze a bunch of text and find something out about it
3:06:35
something out about it
3:06:35
something out about it um and then you have this sort of extra
3:06:37
um and then you have this sort of extra
3:06:37
um and then you have this sort of extra motivation from
3:06:39
motivation from
3:06:39
motivation from a field or from an area that you know
3:06:41
a field or from an area that you know
3:06:41
a field or from an area that you know something about and that you're also
3:06:42
something about and that you're also
3:06:42
something about and that you're also interested in
3:06:43
interested in
3:06:43
interested in and then um you know you built some app
3:06:45
and then um you know you built some app
3:06:45
and then um you know you built some app built like a little website
3:06:47
built like a little website
3:06:47
built like a little website write a blog post um it shows what you
3:06:50
write a blog post um it shows what you
3:06:50
write a blog post um it shows what you did so i think
3:06:51
did so i think
3:06:51
did so i think for people who are like that and really
3:06:52
for people who are like that and really
3:06:52
for people who are like that and really need a hands-on thing i think that could
3:06:54
need a hands-on thing i think that could
3:06:54
need a hands-on thing i think that could be a good
3:06:55
be a good
3:06:55
be a good way to get into it great thank you very
3:06:58
way to get into it great thank you very
3:06:58
way to get into it great thank you very much i think that's a nice
3:06:59
much i think that's a nice
3:07:00
much i think that's a nice closure for tonight or for today
3:07:02
closure for tonight or for today
3:07:02
closure for tonight or for today depending from where you're watching
3:07:04
depending from where you're watching
3:07:04
depending from where you're watching of course um ines and veretz we thank
3:07:07
of course um ines and veretz we thank
3:07:07
of course um ines and veretz we thank you very much for your time
3:07:08
you very much for your time
3:07:08
you very much for your time for the very interesting sessions you
3:07:10
for the very interesting sessions you
3:07:10
for the very interesting sessions you shared with us today we're looking
3:07:12
shared with us today we're looking
3:07:12
shared with us today we're looking forward to
3:07:12
forward to
3:07:12
forward to see more uh about all the things you're
3:07:15
see more uh about all the things you're
3:07:15
see more uh about all the things you're doing
3:07:16
doing
3:07:16
doing around nlp and thank you very much for
3:07:19
around nlp and thank you very much for
3:07:19
around nlp and thank you very much for joining us today
3:07:21
joining us today
3:07:21
joining us today yeah thank you thanks so much we'll see
3:07:24
yeah thank you thanks so much we'll see
3:07:24
yeah thank you thanks so much we'll see you soon
3:07:25
you soon
3:07:25
you soon yeah so with this we're uh closing
3:07:29
yeah so with this we're uh closing
3:07:29
yeah so with this we're uh closing uh our third episode of the
3:07:32
uh our third episode of the
3:07:32
uh our third episode of the if you look into another
3:07:33
if you look into another
3:07:33
if you look into another [Laughter]
3:07:35
[Laughter]
3:07:36
[Laughter] so we're closing our third episode of
3:07:38
so we're closing our third episode of
3:07:38
so we're closing our third episode of the
3:07:39
the
3:07:39
the global ai october sessions um
3:07:42
global ai october sessions um
3:07:42
global ai october sessions um don't forget send it send something
3:07:45
don't forget send it send something
3:07:45
don't forget send it send something global ai community on twitter
3:07:47
global ai community on twitter
3:07:47
global ai community on twitter we will pick one of the of the tweets if
3:07:51
we will pick one of the of the tweets if
3:07:51
we will pick one of the of the tweets if you also add them
3:07:52
you also add them
3:07:52
you also add them into our form uh so that's global ai dot
3:07:57
into our form uh so that's global ai dot
3:07:57
into our form uh so that's global ai dot live slash win dash action just send us
3:08:00
live slash win dash action just send us
3:08:00
live slash win dash action just send us a link to your tweet we will randomly
3:08:02
a link to your tweet we will randomly
3:08:02
a link to your tweet we will randomly pick someone
3:08:03
pick someone
3:08:03
pick someone and they have the chance to win a 50
3:08:06
and they have the chance to win a 50
3:08:06
and they have the chance to win a 50 amazon
3:08:07
amazon
3:08:07
amazon voucher so who knows maybe it's you
3:08:10
voucher so who knows maybe it's you
3:08:10
voucher so who knows maybe it's you amy thank you very much for joining me
3:08:13
amy thank you very much for joining me
3:08:13
amy thank you very much for joining me today and hosting this i think it was a
3:08:15
today and hosting this i think it was a
3:08:16
today and hosting this i think it was a very nice
3:08:17
very nice
3:08:17
very nice episode today i personally learned a lot
3:08:20
episode today i personally learned a lot
3:08:20
episode today i personally learned a lot what about you
3:08:21
what about you
3:08:21
what about you yeah absolutely all of our guests were
3:08:23
yeah absolutely all of our guests were
3:08:24
yeah absolutely all of our guests were spectacular
3:08:24
spectacular
3:08:24
spectacular like they always are um it's gone so
3:08:27
like they always are um it's gone so
3:08:27
like they always are um it's gone so fast i can't believe it's been three
3:08:28
fast i can't believe it's been three
3:08:28
fast i can't believe it's been three hours already
3:08:29
hours already
3:08:29
hours already um but yeah just like huge thank you to
3:08:32
um but yeah just like huge thank you to
3:08:32
um but yeah just like huge thank you to all the attendees as well um for
3:08:34
all the attendees as well um for
3:08:34
all the attendees as well um for submitting questions because it really
3:08:36
submitting questions because it really
3:08:36
submitting questions because it really does like so your questions are
3:08:37
does like so your questions are
3:08:37
does like so your questions are incredible
3:08:38
incredible
3:08:38
incredible um so yeah do continue to spend time
3:08:40
um so yeah do continue to spend time
3:08:40
um so yeah do continue to spend time with us as globally our community and
3:08:42
with us as globally our community and
3:08:42
with us as globally our community and make sure you get your questions
3:08:43
make sure you get your questions
3:08:43
make sure you get your questions answered by these amazing speakers
3:08:45
answered by these amazing speakers
3:08:45
answered by these amazing speakers yeah so we had an introduction to
3:08:49
yeah so we had an introduction to
3:08:49
yeah so we had an introduction to nai we talked about computer vision we
3:08:52
nai we talked about computer vision we
3:08:52
nai we talked about computer vision we talked today about
3:08:53
talked today about
3:08:53
talked today about natural language processing and next
3:08:55
natural language processing and next
3:08:55
natural language processing and next week we're back and
3:08:57
week we're back and
3:08:57
week we're back and we'll be talking more about advanced ai
3:08:59
we'll be talking more about advanced ai
3:08:59
we'll be talking more about advanced ai so
3:09:00
so
3:09:00
so go to our website register if you
3:09:02
go to our website register if you
3:09:02
go to our website register if you haven't registered yet
3:09:03
haven't registered yet
3:09:03
haven't registered yet and we hope to see you all back next
3:09:06
and we hope to see you all back next
3:09:06
and we hope to see you all back next week thank you very much for joining
3:09:09
week thank you very much for joining
3:09:09
week thank you very much for joining see you later
3:09:17
see you later
3:09:17
see you later [Music]
3:09:22
[Music]
3:09:22
[Music] you