videocam_off
This livestream is currently offline
Check back later when the stream goes live
Up next in 10
How to ingest unstructured PDF files from AWS S3 into Salesforce Data 360?
Jan 12, 2026
Show More Show Less
View Video Transcript
0:06
Hello everyone. In this video, we are
0:09
going to see how to ingest unstructured
0:13
PDF files from Amazon AWS S3 bucket into
0:19
Salesforce data 360 previously called as
0:23
data cloud.
0:25
If you have a requirement where you
0:28
wanted to
0:30
keep data cloud or data 360 updated
0:35
with the unstructured files that are
0:38
stored in S3 bucket then you can make
0:40
use of this video as a reference.
0:43
My use case was whenever a file is
0:46
created, updated or deleted,
0:49
those files should be
0:53
available in data cloud uh in Salesforce
0:59
in UDLO
1:02
unstructured data lake object and then
1:04
it should be available in UDMO
1:07
unstructured data
1:10
model object and then with the help of
1:13
search index and custom retriever we can
1:16
make use of it in agent force or in
1:19
prompt builder
1:21
let's see how to do this setup
1:28
uh please check the video description in
1:30
the video description I have shared my
1:32
blog post from the blog post you should
1:34
be able to get all the steps sample
1:37
commands and also sample um
1:41
configuration file for your reference.
1:45
In this blog post, I have given
1:50
detailed steps and also highle steps. In
1:53
this video, let's go over the highle
1:56
steps. If you are pretty new to Amazon
2:00
AWS and Salesforce data 360, then you
2:03
can make use of the highle uh steps.
2:05
Sorry. Uh the detailed steps if you have
2:08
good understanding of Amazon AWS and
2:11
then Salesforce data 360 then you can
2:14
make use of the highle steps. If you
2:17
scroll down to this blog post
2:21
you should see highle steps.
2:26
The first step is we have to create a S3
2:29
bucket. So I created a S3 bucket with
2:32
the name enterprise level storage. and
2:35
it is in US East Wagayo. So the region
2:39
is US - East - 2.
2:43
Next,
2:45
this bucket should be created in your
2:48
Amazon
2:49
uh account region. So if your Amazon
2:52
account region was in west, then you
2:54
should create the bucket in west region.
2:57
If your Amazon buck um region account
3:00
region is in east then you should create
3:02
the bucket in east uh region. If you
3:05
create in two different uh regions then
3:08
you will get redirect errors and then um
3:11
there are so many complex uh uh
3:13
configurations to uh to achieve it. So
3:15
in order to make it simple try to make
3:17
sure the bucket is in the same location
3:20
of your uh Amazon AWS account. Next go
3:25
to IM in U
3:28
Amazon AWS console and then create an IM
3:32
user. When you create this AM user,
3:36
assign Amazon S3 full access permission
3:38
policy to this user.
3:41
If you want to restrict this access and
3:44
then if you wanted to have a better uh
3:47
um security around providing this access
3:51
then make use of the technical
3:53
recommendations and best practices uh
3:56
section in which I have covered
4:00
how to provide
4:03
um
4:05
secured access to this integration user.
4:12
So this is uh one. So check the least
4:15
privilege principle in this technical
4:17
recommendations and best practices.
4:20
Okay.
4:21
The next step is
4:24
um we have to create AWS S3 connector in
4:28
Salesforce.
4:30
So go to data cloud setup. So we have to
4:33
go to data cloud setup.
4:37
Under external integrations, select
4:40
other connectors. Click new
4:44
and for the source select Amazon S3
4:50
and then you have to enter the access
4:53
key and the secret key which you got
4:55
while creating the user in IM in AWS.
4:59
You will see a button called test
5:01
connection. So click that test
5:03
connection button to test the
5:05
connectivity. You should get a success
5:08
prompt so that the connectivity was uh
5:10
successfully done and also the AWS
5:14
access key and the secret key was
5:15
accurate. Click the save button. Now the
5:19
AWS connector setup is done. The next uh
5:22
thing is we have to create UDL in
5:25
Salesforce data 360. So go open data
5:30
cloud application. Go to data lake
5:32
objects tab. Click the new button.
5:36
Select from external files. Click next.
5:39
Select Amazon S3. You will be able to
5:42
see this option only when this
5:45
particular connector setup was
5:47
successfully done. And make sure this is
5:49
also active.
5:52
Select it. And then you should be able
5:54
to
5:56
configure it. Here
6:00
you have to select the connection
6:04
when you go. Yeah, here you have to
6:05
select the connection which you created
6:07
in this connector. So the connector name
6:09
is AWS S3. So here I'm able to add
6:13
select AWS S3.
6:17
Next in the directory uh field you if
6:20
let's say inside the S3 bucket you have
6:23
folders then you can mention the folder
6:26
names here you can add up to four or
6:28
five um folders I guess it also looks
6:32
into the data um of subdirectory uh
6:35
let's say you have a parent directory
6:37
and then you have subdirectories inside
6:39
that uh um parent directory then it will
6:42
also try to pull all the information
6:43
from uh subdirectory here in the file
6:47
name pattern use star dot PDF because
6:52
the use case for me was to get all the
6:54
PDF files.
6:59
Yeah, you have to just type this
7:02
information without double quotes. Okay.
7:06
And then you have to give a name for uh
7:09
UDLO and as well as for UDMO. Once that
7:12
is done um when you click next button
7:16
you should see uh an option called
7:18
enable semantic search with system
7:21
defaults. You can make use of this to
7:24
create a search index uh in data 360
7:27
this search index uh uh uh is required
7:31
to create a custom retriever. So make
7:33
sure you are enabling this option. Once
7:36
this is done, click save and make sure
7:40
the status of uh
7:43
the data lake is active. When you create
7:46
this UDL,
7:48
Salesforce will create three um UDLs.
7:52
One for the file names, other one is for
7:56
chunk and the other one is for vector
7:58
embedding which is index.
8:03
Okay. Next is so the connectivity is
8:07
done uh in Salesforce uh to connect
8:11
Amazon S3 uh bucket. So that is done and
8:16
you have created UDL and then it it
8:18
would have automatically created UDMO
8:20
since we gave the object API name and
8:22
also the name for the UDMO and the
8:24
mapping would have been done all
8:26
automatically by uh Salesforce. Now
8:31
we have to push the files whenever it is
8:34
created, created, updated or deleted in
8:39
S3 bucket. In order to do that, we have
8:41
to set up file notification pipeline. In
8:45
order to achieve it, we are going to
8:47
create a lambda function which will uh u
8:50
make use of uh um the updated file
8:54
notifications or created or deleted file
8:57
notification and then it will push it to
8:59
Salesforce data 360. Uh in order to uh
9:03
do that install AWS CLA, install jq and
9:08
also open SSL, install all these three
9:11
applications.
9:13
The next step is we have to create
9:15
private public key pair and certificate.
9:19
In order to do that make use of open SSL
9:24
command. So this will create a keypad.pm
9:28
file. So I created a folder called uh
9:31
AWS
9:35
not AWS forert I created a file uh a
9:39
folder called uert
9:45
inside this particular folder using my
9:49
terminal I was able to run this command
9:52
and then it created a PEM file. Next it
9:55
created a CRT file which is assert and
9:59
then it created a private key. So make
10:02
sure you are creating a folder and then
10:05
inside and in the terminal
10:08
uh make sure the path of the terminal is
10:11
uh this particular folder. Run these
10:14
three commands so that it will create PM
10:17
filesert file and a private key file.
10:21
Okay. Once this is done, open these two
10:24
GitHub link.
10:27
This is the installer script. We are
10:29
going to execute it so that it will
10:31
create a lambda function role u S3
10:35
bucket um and also event notification in
10:39
AWS so that whenever a file is created,
10:42
updated or deleted, it will send those
10:44
information to Salesforce data 360.
10:48
this particular AWS lambda function
10:51
contains the lambda function. So if you
10:54
if you are an AWS expert then you can
10:57
look into the code that is uh uh
11:00
developed for this lambda function and
11:02
then you can also do some modifications
11:04
to it. Okay, the next step is we have to
11:08
create a external client app in
11:10
Salesforce. Previously uh we were making
11:13
use of connected app in Salesforce and
11:15
now we have to make use of um external
11:18
client credential app. So here create a
11:20
external client credential app in the
11:22
call back URL make use of your
11:24
Salesforce my domain URL
11:27
in in order to authenticate the
11:29
connectivity between Salesforce uh uh
11:32
data 360 and uh um Amazon S3 we are
11:37
going to make use of this ooth uh in
11:40
order once the oath authorization is
11:42
done we are going to redirect the user
11:46
to a particular URL. So that is the
11:49
redirect URI in uh oath. So that is the
11:53
call back URL. You have to set it up
11:55
here. Make sure you are selecting these
11:57
three oath scopes. One is to manage the
12:00
user data. One is to perform uh uh
12:03
request for refresh token and offline
12:05
access and the other one is data cloud
12:07
injection API data so that it can ingest
12:10
the data from uh um lambda function in
12:14
Amazon S3 into Salesforce data 360. Make
12:19
sure to select JWT bearer uh flow option
12:24
in the certificate. Select the
12:26
certificate which we created. So make
12:28
use of the dosert file which we created
12:31
using the open SSL command.
12:34
Okay. Once that is also done make a note
12:37
of consumer key and secret in Salesforce
12:40
setup. Search for who a open id connect
12:45
settings.
12:49
If you go to that setup,
12:54
you should see
12:57
all oath username password flows.
13:04
You can toggle it on.
13:13
Once that is toggled on,
13:16
now make use of your Salesforce my
13:18
domain URL/services/2
13:21
/ authorize. Um, and we are passing
13:24
multiple parameters here. Response type
13:26
equal to code. Client ID equal to your
13:29
Salesforce consumer key from the
13:31
external client application. scope API
13:34
refresh token and CPI CDP_gest
13:38
API and redirect URI equal to the call
13:40
back URL which we used it
13:45
in the connect in the external client
13:47
credential app
13:50
and then
13:52
um code_ch challenges sa 256. So this is
13:56
the algorithm. Once this is done, you
13:58
will see a prompt with uh three scopes.
14:04
So these three scopes, you should see it
14:07
on the prompt. Make sure you are getting
14:09
only those three pro um scopes not uh
14:12
additional and also make sure you are
14:14
not adding any additional scopes here
14:16
which are not needed. I have seen uh
14:18
people adding full access. Please do not
14:20
do that. It will expose so much of your
14:23
Salesforce information. Um if if someone
14:26
hacks a consumer key and consumer secret
14:29
so make sure you are not doing that. Um
14:34
once it is done once you click allow it
14:37
will redirect to the call back URL that
14:40
was configured and used it in the URL
14:43
also. So it will redirect
14:46
to the call back URL and then it will it
14:48
will authorize. Once that is done
14:52
unzip the S3 file notification
14:54
installation script. So this is the
14:56
installation script. Um you would have
14:59
downloaded it from GitHub.
15:02
Unzip it. Move it to a folder. So I
15:05
created a folder called AWS. And uh if
15:09
you go to documents, if I go to AWS, I
15:12
should see um so these are all generated
15:16
ones. So you should see three files
15:18
input parameters, setup
15:22
s3 and setup s3. So these three files
15:25
you should see. Now you have to update
15:27
this input parameters
15:30
s3 confile. This is a very crucial step.
15:34
Um so here
15:38
I am I made use of visual studio code uh
15:41
to update uh this
15:42
input_parameters_s3.com
15:45
file. Here enter use your username
15:49
and make use of your Salesforce my
15:50
domain URL. Since I have um uh directly
15:55
connected it with my developer edition
15:57
or I'm making use of my
15:59
login.salforce.com URL here. Use your
16:02
12digit AWS account ID. Make sure the
16:05
region is accurate. Uh make use of the
16:10
uh S3 bucket API name here. Um let's say
16:15
you have a folder inside your uh S3
16:18
bucket, then you should make use of that
16:20
particular folder. Uh I don't have any
16:22
folder inside it, so I left it blank. So
16:25
it makes use of the entire S3 bucket
16:28
access.
16:30
Um next
16:32
uh here I I created I used this um uh
16:35
name like AWS - S3 - SF.Lambda. So
16:40
Salesforce will create a um S3 bucket
16:44
with this name uh in your uh Amazon uh
16:47
uh console if this particular uh bucket
16:50
is not already available and then it
16:52
will put the GitHub
16:56
lambda function.zip file which we
16:58
downloaded. So the same file will be
17:00
uploaded in uh uh this particular S3
17:03
bucket. Um
17:07
you can leave this uh um uh key as
17:11
blank. It is not u required. Next here
17:15
you have to mention where is the source
17:18
for the
17:19
um AWS lambda_unction.zip
17:23
file. Uh this is the file we would have
17:25
downloaded it uh for the file
17:27
notification pipeline. Um so this is the
17:29
role name I have defined and this is the
17:32
lambda function. Um so when the lambda
17:35
function is created the name of the
17:36
lambda function will be this one. If you
17:38
go to lambda in uh Amazon console if you
17:41
search for this lambda function you
17:43
should see it once the script executed
17:45
successfully. Uh next uh this is the key
17:50
that will be used in the lambda function
17:52
uh to store your consumer key of AWS
17:55
sorry Salesforce and this is the
17:57
consumer key uh Salesforce consumer key
18:00
for the from the external client
18:02
credential app. Uh next in order to
18:05
store the private key uh this is the um
18:08
private key uh name uh I have set it up.
18:12
These are all something um uh the naming
18:15
convention I have used. So Salesforce
18:18
AWS S3 and iPhone RSA private key. Uh so
18:22
I made use of the example from
18:24
Salesforce and then I created it. Uh and
18:27
for the PM file you have to mention the
18:30
exact location where the keeper.pm
18:34
file um is stored. Uh this file was
18:37
generated with the help of open SSL
18:40
command.
18:41
Okay. So, make sure this
18:44
input_parameters_s3.com
18:47
file is configured without any errors.
18:50
If you configure with any errors, then
18:53
uh the installation script will fail and
18:56
then the lambda function creation and uh
18:59
the policy creation, role creation,
19:00
everything will uh fail. So, make sure
19:02
it is um accurate. Okay. Once uh that is
19:07
done now you have to in in your terminal
19:11
you have to change the path to the
19:13
directory where those three files are.
19:16
So this is the folder where um I have
19:19
stored all the three uh files. So in uh
19:23
make sure you are in this particular
19:26
folder in your terminal using the cd
19:28
command change directory command. Once
19:30
that is done, make use of this command.
19:33
Populate with your account access key,
19:36
secret key, your uh session token and
19:38
also your AWS region. Run this command
19:41
in your terminal so that your uh AWS uh
19:46
uh environment variables are set. Once
19:48
that is done, you have to run this
19:50
command.
19:52
This command will take few minutes to
19:54
complete. Make sure all the steps are
19:57
completed. Once it is completed then you
20:01
should see a uh S3 bucket with the name
20:05
that was mentioned here.
20:09
Okay. No no not here.
20:16
Yeah. So this is the bucket name where
20:20
the files are stored.
20:23
You should see a lambda function with
20:24
this name.
20:26
And also you should see
20:29
this particular Oh, okay. So this is the
20:32
bucket name I have used. So this is the
20:36
bucket name. So this bucket you should
20:39
see it and inside that you should see
20:43
AWS lambda_unction.zip.
20:46
So this is the file uh which contains
20:48
the lambda function code. So Salesforce
20:50
would have created this for you. Okay.
20:53
And the next thing is
20:58
you have to create a uh retriever. So go
21:01
to Einstein studio tab, go to retriever,
21:05
click new and then you should be able to
21:08
create a retriever.
21:10
Make use of the data model object and
21:14
also the search index which we
21:16
configured. Once that is done, you can
21:19
create a simple prompt template. In
21:23
order to verify it, I created a flex
21:25
prompt template and then I was able to
21:27
verify it.
21:34
So here
21:37
you have to use
21:51
Yeah. So I'm making use of the input uh
21:54
string against the retriever to respond
21:58
to the customer.
22:00
So in the input
22:09
Yeah. So in the input the information
22:12
what I passed was change the
22:14
advertisement plan. So this is the
22:17
information that is passed to search
22:19
against this particular retriever. So
22:21
the retriever was able to fetch all this
22:23
information from the file and then it
22:25
was able to generate a nice response
22:27
back to the user.
22:35
I have shared the
22:36
input_parameters_s3.com
22:40
file for your reference. Um I had hard
22:43
time in configuring this because I'm not
22:45
an AWS expert. Um so if you run into any
22:49
issues or if you wanted to recreate
22:52
things, go to the S3 bucket.
22:59
You should see an event notification
23:02
created for the S3 bucket.
23:05
So if you go to the properties,
23:11
you should see an event notification.
23:13
You can delete it and then you can rerun
23:16
this particular uh
23:19
command again and again.
23:21
So once this is uh uh done successfully,
23:26
if you create a file, if you update a
23:29
file and also if you delete a file, the
23:32
chunking whatever happened in Salesforce
23:35
in data 360 will be up to date. I um I
23:40
tested both uh
23:43
file creation or file upload and also
23:46
file uh deletion. it was able to sync it
23:49
properly with the data 360 and then I
23:52
was able to create a uh prompt template
23:55
and then I was able to verify it.
23:58
If you go to lambda in Salesforce, you
24:01
should see a lambda with the name that
24:04
was mentioned
24:06
here. So this is a lambda function name.
24:09
If you are a uh AWS expert then you
24:12
should you can make use of uh the
24:15
monitor tab. You can click view cloud
24:17
watch logs and then you should be able
24:19
to troubleshoot it. I had some errors
24:23
with the help of
24:26
um with the help of uh failed invocation
24:30
execution error. I was able to debug it
24:33
and then I was able to fix it.
24:36
So if you upload or uh if you delete a
24:38
file and then if the data is not updated
24:41
uh in Salesforce data 360, please make
24:44
use of
24:46
view uh cloud watch logs which will be
24:48
very very helpful.
24:53
I hope it was helpful.
25:00
Thank you for watching.
#Internet Software