Create AWS S3 Bucket and Read Parquet File from Fabric Notebook

Hello everyone and compliments of the season

In this video, I'm going to show you how to read a Leak House Backup file into Amazon

S3 Web Service using the Fabric Notebook. This is going to be a comprehensive end-to-end project because I'm going to walk you through

what is Amazon S3, how you can create buckets, create access keys, secret keys and of course

how you can read the data from Fabric Notebook into the S3 Web Service and of course how

you can query the data in the S3 query window. So let's get started

Now what is Amazon S3? Now the S3 simply means Simple Storage Service and it is a scalable object storage service

that is offered as part of the Amazon Web Services and it allows you to store and retrieve

any amount of data at any time from any location on the web

The S3 is designed to be a highly durable, available and scalable service to store data

or form data backup, archive your data, distribute content and of course it serves as a data lake

Using the Amazon Web Service, we're going to create what's called buckets. Now buckets are basically containers for storing any kind of objects like the PDF

like the parquet file, CSV, PNG or whatever file you have. Enough of talking, let's get started

I am currently in the console.aws.amazon.com and of course we can click on the services

to see all the services in the AWS or I can even search the S3 in the search bar and then

click on this S3 service. For the first time, we're going to see this Amazon S3 store and retrieve any amount of

data from anywhere because I do not have any container buckets created

So I'm going to click on this create buckets that can store different kind of objects and

of course we can specify the AWS region in the general configuration

I'm going to scroll down to, there we go, EU01 and of course we can choose the bucket

type, we can choose the general purpose or directory. I'm going to go with the general purpose which is the recommended and I'm going to scroll down

Now for the bucket name, you can see this kind of example, we can't use uppercase as

a bucket name. It's not going to allow that. So I'm going to type in sales data 1234 just to make it more unique and of course you can

even select a bucket in S3 if we have any existing bucket but I do not have any bucket

so I'm not going to choose this and let's scroll down. For the object ownership, now we can specify the access control levels, okay so this is

going to be disabled, ACL and let's scroll down. Now for the block public access settings for this bucket, we're going to block all the

public access just for now. For the bucket version, I'm going to click on enable and this allows me to get any objects

that has been deleted in a bucket so we can see version is a means of keeping multiple

variants of an object in the same bucket. So enabled and of course we can optionally specify tags and for the default encryption

we want to use the default server-side encryption with Amazon S3 managed keys and let's go down

and of course for the advanced settings, we don't need to do anything here for now, let's

just go down and click on create bucket, okay I'm going to go up, okay I can see that bucket

with the same name already exists. Now I do not have any bucket with the same name but I don't know why

I'm just going to type in 12 and then sales data 12, oh okay I'm still seeing the same

error or let's use 007, let me click out and this is accepted sales data 007 and then

let's click on create bucket, successfully created bucket sales data 007 and that is

super amazing. So we can see the name of the bucket sales data 007, we can see the AWS region in EU

01 covering Stockholm and some other places. So for the access, we can see bucket and objects not public and that is fine for now

I can click on the bucket name and then we can see different kind of parameters like

the object, the properties, the permissions, the metrics, management and the access points

Now what I'm going to do is to go ahead and create what's called access key and secret

key in the identity and access management service. So I'm going to come here, I'm going to type in AIM and then I'm going to click on this

Now in the identity and access management AIM service, we can see the access management

and we can see different kind of the dashboard rather, the AIM dashboard, we can see the

security recommendations and so on. Now I want to click on users, I want to create a new user and click on create user

Now I'm going to call it my name, Abiola Dede, so this is going to be the username

Now I'm not going to provide user access to the AWS management consult

So I'm going to go ahead and click on next. And then for the permissions options, under the set permissions, now we can add to a group

we can copy permission from a user to this particular new user

We can even attach individual direct policies. So I'm going to choose this attach policies directly and then I'm going to apply these policies

I'm going to click on this administrative access Amplify. And I think let me just grant, okay, device setup

I think that is fine for this user. I'm going to scroll down, then click on next

And then we can see the review and create. So this is the username, the console password type, none, require password reset, none

And of course we can see the permission summary. So scroll down and click on create user

And there we go, Abiola Dede user created with a few errors. That is fine

There's no problem. Now I'm going to click on the user we just created

And then in the user, we're going to create what's called the access key

So I'm going to click on this create access key. And of course we can specify the access key

We can choose the use case for command line interface CLI against the local code application

running on AWS compute service, third party service. We can even click on or choose application running outside AWS

I'm going to choose this third party service. We plan to use this access key to enable access for a third party application or service that

monitors or manage your AWS resources. So I'm going to scroll down and I'm going to click on this

I understand above recommendation. I want to proceed to create an access key

So click on next. And then for the description tag, let's just call it, you know, access, go ahead and create

access key. Okay. So you can see this is the only time that the secret access can be viewed or downloaded

So what I'm going to do is I'm going to click on this, the secret code. So I'm not going to click on it anyway, but this is the access key

So what I'm going to do is click on this download.csv. So I'm going to download to my personal PC

There we go. AbuelaDavidAccessKey.csv. That's fine. And go ahead and click done. Okay

So you can see the access key one created, and of course it is now active

What I'm going to do finally for now, I'm going to click on sales data to investigate

Now you can see in this case, we do not have any object

So let's head over to the Fabric Notebook. Now basically in this Fabric Notebook workspace, we can see we have this AWS S3 leak house

I'm going to click on it. And of course we can see we have this file uploaded sales data.csv

I can click on it to investigate. So we can see the sales data

And of course, I'm going to click on this file. So what I did just to click on these three ellipses, and of course just load to the existing

table, this particular sales data. Okay, that's fine. I'm going to open a notebook

So click on new notebook. Now we need to install two libraries

The first one is the Boto3, which allows us to interact with AWS services and also S3FS

to interact specifically with the S3 service in AWS. So let's do pip install Boto3 and control enter to run the cell

While that is doing its job, I'm going to click to add a new cell

Now I'm going to install pip install. Now I'm going to install the S3FS

And let's wait for this to finish its job. Okay, there we go

So it has been installed. And I'm going to come to the pip install

Okay, so you can see command executed in few seconds. So I'm going to scroll down

First we need to import the Boto3 and of course we want to import pandas and then SPD

So import Boto3. Okay, and then we'll import pandas as SPD

So let's run this cell and let's see. There we go. So command executed in 312 milliseconds and that is amazing

So let's click on a new cell. Now we'll initialize the instance of the Boto3 and then we'll see the resource

So we'll specify the service name, the region name, the AWS access key ID, AWS secret access key

So I'm going to just give the variable AWS S3. Okay, S3

Okay, and then we'll initialize the Boto3.resource and then open the brackets

Now we'll specify the name of the service. So let's call this one service underscore name

And that's going to be S3, right? Amazon S3 and then put in a comma

And then we'll provide the region, region underscore name. And that has to be, let me just double check, that should be EU01

So EU iPhone North, iPhone 1 and then put in a comma

Now we need to provide the AWS underscore access, access key ID

So, and there we go. So we can see the AWS access key ID

Now specify the AWS underscore secret underscore access key. So AWS underscore secret underscore access key

And that has to be inside a single code. Okay, so there we go

Now there we go. So we can see the access service name, the region, the access key ID

and of course the secret access key ID. So let's go ahead and control enter to run the cell

There we go. So command executed successfully. Now, the next thing we need to do is to read the parquet file

into pd.region underscore parquet. So I'm going to come to the sales data, okay

And click on this horizontal ellipses. And I want to copy the path

So copy that and let's add a new cell. Now I'm just going to do df equals to pd.read underscore parquet

And of course, inside double quote, I'm going to control V extension

and then click enter. Now we need to go ahead and store the data frame into parquet extension

So I'm going to do df.to underscore parquet. And then inside open and close brackets, now single quote

I'm going to call this one sales data dot parquet. Okay, so let's check it out

So df.to underscore parquet. Okay, this is fine. Control enter. And let's see

There we go. So command executed in just two seconds and 490 milliseconds

Now I'm going to scroll down. So the final thing we need to do is to go ahead and upload this particular

sales data dot parquet into the AWS. So I'm going to call the AWS S3 that we defined

And if we want to access the bucket, so inside open and close brackets

I'm going to specify the name of the bucket, which is sales 007

So sales data, sales data 007, 007. Okay. And then we'll use the dot upload underscore file

And of course, we'll actually specify the file name. So file name, that's most equal to this sales data dot parquet

Let me just copy it. And inside single quotes, control V. And of course, we want to specify the key

So this is key. And inside that has to equal to inside single quote

Let's just call it sales data. So let's go through it again

So first we specify the bucket. And then we specify the name of the bucket

And then we use the dot upload file. And of course, we specify the file name, which is sales data dot parquet

And then the key, which is sales data. So this is going to be what's going to be seen as the key in the S3

So let's go ahead and control enter to run the cell. There we go

So command executed in 879 milliseconds by Abiola Abiola. And then we can go to the S3

Now, I'm just going to go ahead and refresh the page. Now, this is the moment of truth

I'm going to click on the sales data 007 bucket. And there we go

Sales data last modified December 21st, 2023 by 1838. And that's exactly the time we are

And that is super cool. And of course, you can see the size for 81.7 kilobytes

And this is super amazing. And of course, you can see the storage class is standard

I'm going to click on the object, the sales data. And of course, you can see the properties, the permission, the version

So you can see the object overview, owner, AWS region, the last modified date, size

We can see the S3 URI and, of course, the object URL

And, of course, you can see the key. So this is exactly what we specified, the sales data

If you don't forget, they've got this particular key. So you can see the key here

This is super amazing. And finally, I'm going to click on this object actions

And I'm going to click on this query with S3 select. So for the input settings, now we can see the path and, of course, the SAS

Now I'm going to choose, because this is actually a package, I'm going to choose a patch package

And for the compression, this is not supported. That's fine. I'm going to scroll down

And then I can select the output settings. So for the output, you can choose the CSV or JSON

I can even choose the CSV delimiter. So that is fine. Just go ahead and scroll down

Now this is going to be select star from S3 object. And this is S as an alias

I'm going to see the limits or the first 10 records. I'm going to scroll down

I can see in the results query there's nothing to display. I'm going to click on run SQL query

Let's see. Successfully returned 10 records in 2045 milliseconds. I'm going to scroll down

I can see the raw data. I'm going to choose the formatted. And there we go

So we can see the data. So this is super amazing. So this is the end-to-end project on how to create S3 buckets

how we can create access key, secret key. We can read data from Fabric Boot Booth

And of course, how we can query the data in the S3 query window

I trust you enjoyed this video. If you do, like, share, comment, and see you in the new year

Thank you and bye for now. Cheers