Friendship of Data with Cloud: Close friends or Best friends? by Dhruv Aggarwal

We'll be discussing about friendship of data with cloud, close friends or best friends, like, I think many of us have been using cloud

Like we must be beginners, we must be professionals, like, so we don't know the relationship between data and cloud

don't know whether they are best friends or whether they are close friends

So we'll be getting to know in further slides like what's the relationship between them

whether they are best friends or close friends. So I'm working as a software engineer at Excel Technologies

I have worked with ytics with their geeks for geeks, project pros, CFT in data science, big data

technologies as an intern, two plus years of experience as an intern

Obviously, remote working was there. We were not allowed to work in office

Also, I'm currently a AWS community builder for machine learning for a second consecutive year

Cagle 3X expert. So like in cloud technologies, we are using AWS, Azure, GCP, Red Hat also

So starting of it, the agenda would be the issues and challenges

involving on-prem system with data effectively informed decisions. Next we'll be covering major tools and technologies

in cloud that are solving the problems and why we should shift to cloud

rather than on-prem systems. Then thirdly we'll be deriving the relationship with the data and cloud technology

like whether they are close friends or whether they are best friends. Right

So starting of it, anyone of you can explain what is cloud computing

like it's obviously not computers can fly it's obviously not so do you know what is cloud

computing yeah that's a more more competitive way right so I was hoping for a slightly

more detailed explanation of how cloud computing works it's it's a magic so yeah and

it's not feels like we're a masman which i'm going and then there's an internet on the

cloud forming server it's not like that it's basically basically like and it's also not like the we are renting a car right we are obviously renting

a data center of some company we are using the virtual machines we are using using more

technologies out of it so please get this out of your mind like we are renting a car it's not

like that okay so uh yeah something winifun now it's like we are using infrastructure as a service

we are using platform as a service we are using software as a service so you must have out

heard the terms like IWS, PWS, and SaaS, software as a service

Like, software as a service involves, you know, these application security databases

Platform as a service involves application and infrastructure as a service, application security databases

Like most of the companies work directly on SaaS. Some companies work on PAS

There are only few companies which work directly on three of these, like, yeah, three of these

terminologies, right? And then we have enterprise IT where they work on all these technologies

like data center, networking, storage, server, virtualization operating system databases. Right. Then we are going to look on issues and challenges involving on-prem systems

and why companies need to move to cloud technologies. So you must have heard of like they save

a lot of money out of it, right? Security is, you know, a conflict. They can be secure

They cannot be secure. It's a 50-50 ratio because last six months

we have heard that there is a cloud breach in the security part. So yeah, then we have on-prem system that can't adjust

to the growing company needs. That's, yeah, one of the main issues is that many companies

have to scale effectively. Like, for example, we have hot star, right

You must have been watching cricket matches, like India versus Pakistan is there

So recently I was watching a YouTube video where they have explained that the viewership count was approximately 1 and 30 lakhs So why they on a single day like so they were using AWAS EC2 instance

EC2 is basically elastic compute. So the number one reason for them is that their EC2 instance have to scale

means they have to scale their compute resources. So that's a major term for our company to choose the cloud over their online

cloud over their on-prem system. Next is the cost perspective. Next is the cost perspective

Like on the server part, if we are, you know, installing server, keeping networks in our offices

like it involves a huge cost. Like we have to maintain IT team for that

keeping a check on the security part also. So that part, we just move to the AWS or GCP

and other cloud services. and other cloud services parts. So the cost gets highly reduced

Next is a huge capital expense compared to the affordable cloud services

Yeah, that's the same. Like high cause and capital expenses get reduced

The IT team we have to keep, that gets all erased. Then we have a risk of losing critical information

Yeah, so that's a major concern. If you take an example of Hot Star part

like there, the data is currently stored on the, their servers only. They're not keeping on that their on AWS servers. So this point is

being widely discussed in many conferences, especially like the security part. So it's a 50-50

rate ratio like many trust the cloud services, many not trust the cloud services. Okay

Then maintaining on-to-m system is a costly affair and time consuming for companies as it can be

monthly quarterly. So if you choose a yearly plan, or yearly service for our cloud services

we get a lot of discount on the cost perspective, right? So that's another point

Yeah, the skillability part is that the cloud is amazing. So skill we're so cloud

So that's the main point. Now we talk, now we'll be talking about security part

So data security is a major concern in cloud. that's a myth

you use cloud computing your work must be so secure yeah that security part

they give us the in return that your data would be secured

but many companies don't trust them right so now major tools and technologies

provided in cloud that are solving problem and using emerging technologies so here I have taken an example

of Google compute engine computer engine basically involved virtual machines disk and network. So anyone knows what is virtual machine

Okay, so anyone know containers, what are containers? and what's the difference between containers and virtual machines like, yeah

Do you know that? Yeah, storage and operating system also can also be there, right

Correct, correct, correct, correct, correct. Yeah. So this part, if we take on the server, we buy a server we keep in our office

So the cost perspective is a lot more, yeah, in lags and crores

So rather than buying, maintaining an IT team, which is using the Google Cloud or AW, that's your choice

Azure also offers the same service with a different name. So this service we can use

Right. Then we have a microservices part where the container part is also there

Next is that, yeah, if we need a Docker or a Kubernetes service, right

So in the Hot Star example, so Google Cloud Platform supporting Docker and Kubernetes like Boss

yeah, it can be GCP, it can be Azure and it can be AWS, anyone like that

So in the Hot Star example that I have shared, there they were using Kubernetes

Yeah, so Kubernetes as instances, containers, et cetera, et cetera. So their scalability powers was like they were able to scale much accessly Now you must have heard serverless in cloud computing right So in serverless like it not that we don have a server It not like that it just a name that have been given to serverless technology

So in that, you keep using that word, yeah. Now, config management

Do you know about conflict management? Anyone knows about config management? Some word of that, yeah

Yeah, hardware and system can be there. Yeah, hardware and system can be there

Correct, correct, correct. So yeah, serverless technologies, many have a myth like that

If you don't need config management, if you use serverless, that's completely not 100% correct

We need config management and if you are using serverless, some of it can be deduced or cannot be done, but we need that

So this is an example of N2L machine learning using cloud services

Okay, so in this architecture we have used in EC2 instance. EC2 instance is basically of AWS part

So using cloud services we can build an N2N machine learning model

we can collect the data, basically Twitter data. We have tweets, okay

We can collect the real-time data. And then we can put into the NIFI, Kafka

and we can process the data using Spark. So Spark, NIFI, Kafka are all big data technologies

that can be used. And MongoDB, this is the storage part, MongoDB, that is used as a no-SQL database

Okay? And then we can plot the data using Dash or Plotly server

So the cloud services and we are using easy to instance there

So this configuration, this architecture has a reduced cost that can scale effectively and for a team that can, if a miss happening happens, right

If a mis happening is there, so this part can be corrected easily if you put that architecture on the cloud

rather than being on a server of a company that is purchased. right so end-to-end website monitoring in dual-time on cloud yeah this service

website monitoring can also be done using only the cloud services only like

we can not we are not using any third-party application here so what is so so so

what we are doing is that we are using Amazon EC2 instance for the compute part

we are getting the website that's a website data we are using Kinesis for the

data streams and then the lambda for the serverless technologies here and then we are using

Aurora and SNS service yeah so don't we don't need any third-party service with the third-party

applications third-party software here everything can be done using cloud and the data

has a and the data has a real-time access here right okay so here okay so here

what we have done is that we have an EC, so we created an EC2 instance, AWS EC2 instance here

right? We are getting the data, that's the data source, okay? So we have, so we are getting

the data from the Twitter API, we have linked that, okay? And then here we are using

NYFA and we are using Kafka, Kafka for the collection, and I have for the ingestion of the data

Okay, and then when the data is, okay, yeah, so when the data is, is

pulled into the event stream, the machine learning model is already running there in the

backend to process the data. Then the, yeah, so the real-time data that is cutting, that is coming from the Twitter that

has a large volume. Okay, so that's why we are using Spark here

That's your choice if your data is large, you can use Spark or anything else

And that, after processing Spark Emly is a machine learning library of the Spark

We are reading the data, we are yzing data, and then after getting that, it's getting stored

into the database that can we access later on. And then we need to visualize the data It your choice if you need to visualize if you need to get the ysis of the data So we visualize the data using plotly or dash

And that's completely linked to the EC2 instance. For the EC2 instance, we are using the command line here. Okay

Does anyone have a doubt here? No. Yes, correct, correct, correct, that's the main point

We only pay for what we use. We don't have to purchase a license for a one year or a two year term, right

So that's the benefit of using APIs, that's the benefit of using Spark getting linked to the

EC2 instance. That's yeah. Okay so yeah we can also do end-to-end

website monitoring rather than we use third-party applications. We can directly use the various services of the cloud technologies. Here it's being

used only of AWS you can use you can try with a GCP or Azure. So now we

talk about friendship of data with the cloud. So it's not a close friendship like the

the Munna, Mbbs, the circuit and the Sanjad. Or the best friends like the Shole, I think

I don't remember the name. Yeah, JNVU, correct, correct, correct. So yeah, so it can be rather than a close phase or best friends, right

So, yeah, data and cloud can be close friends, and data can rely on cloud services and trust in

So in the last slide that we have seen the into website monitoring, the data is already trusting the cloud services, right

That's why the person who is doing that, framing that architecture, has already used the services

So the close friends that the data can deliver in cloud services and trust its various technologies

for its process, yeah, so data can be processed and pipelines can be built easily because data

so the data trust the cloud services, right? But for the best friends we already know

that they share everything but they don't trust. Right? So Dead and Cloud can be best friends only when they share everything between them

and specific to industry sectors. Like if we talk about ed tech sector

for the tech sector they can be best friends and close friends both

because they need to trust the services because there is a live interaction between the mentor and a student

Okay, one of, so they can't afford the breakdown of the services

services. So if there is a breakdown, the doubt so the students cannot be, so cannot be dissolved

on time. So yeah, this is for the, so that this is for the funny part, right? So we conclude

that data and cloud majorly are close friends because we have seen that in the industry

segments they both trust the services but they can't share everything. You can't build an application or a software just on the cloud services

You need third party applications. You need servers on your own. You need data on your servers as a private security

So now is the situation that they are close friends. But in the future, if the cloud is scaled, more services are introduced

They can be best friends also and you don't need a server room in your office or college or anything else. Right

So yeah. If you need the slides, you can scan the code for the future reference

Or if you want the memes, you can take that. You can copy there

So yeah, thank you. You can connect with me on LinkedIn, on Twitter

If you have any doubts related to cloud, so if you're not, then we'll be live interaction. Yeah. Yeah

No, no, close friends, like. Not best friend, we'll be close. Right

So, now underground on the same