RDD in Spark

In this video we are discussing RDD in Spark

RD stands for resilient distributed data set. It is one data structure with the help of which all the data sets which will be

deciding in this RDD will be distributed onto multiple different servers and nodes

So whenever computations are required, the competitions can be done simultaneously in parallel

on multiple different servers and nodes where these data sets are available

So, let us go for more discussion on this RDD in Spark

So, what is RDD in Spark? RDD stands for resilient distributed dataset

It is the basic data structure of Apache Spark. And each and every data set in RDD is logically partitioned across different servers

And using this partition's different nodes and clusters can do the competition in parallel

at the same time. And that's why RDD is faster in our operations. So, RDD is an immutable

collection of objects So it is immutable collection of objects So RDD are resilient so they are fault tolerant So as a result of that it uses the DAG and using DAG it is able to recompute missing partitions if there is any and it is

distributed because data can be stored on multiple different nodes or servers and finally RDD is a data set so the user can load data set externally

using the CSV files or JSON files according to the recommend availability now consider this very diagram so here actually we are getting two

main operations one is the transformations another one is the actions to two main

operations are there one is the transformations and another one is the actions

now what is the basic difference between these transformations and actions in

case of transformations will be getting another new odd dd from the previous one

but in case of actions no separate RDD will be created from the previous one. So, that is the main difference between the transformations and actions

So at first we are going for these two different operations into details

So what are the operations of SparkRDD The operations of SparkRDD are of two types So the first one is the transformation and the next one is the action So transformation The transformations are like a function that takes RDD as input and returns

one or more RDD as output. Here you can have multiple different methods we can execute here

So it creates a new RDD by applying tasks like our map method, filter method, reduce by key method, etc

it will be creating multiple new oddidies from the parent one the transformations are lazy

operations that means it will not get pre-computed and it creates some odddids but they are not executed

until one action is performed so until the actions are getting performed no new art ds are going

to get created so that's why it is known as lazy operation so there are two types of transformations

are possible and they are the narrow transformation and the white transformation

So that is about the required discussion on transformation. So let us go for the actions

Now we are discussing action here. The final result of RDT competition is the action And it uses the DAG to execute the tasks and at first it loads the data into the original RDD and then performs all intermediate transformations jobs and finally returns to the driver program So it is not going to create any kind of new RDD at the end of the actions Now what are the different actions we can carry out Some actions in Spark are first method or say take method

reduce method, collect method and count method, etc. Using transformation, we can create

RDD from another existing RDD. In the previous diagram also we have shown that using

transformations, the new RDD or multiple new RDs will be created from the mother one

So using transformations, we can create RDD from another existing RDD to work with the actual

data set, we should use the actions. And during the execution of actions, it does not create any new RDD, so its outcome can be stored onto the

save, onto certain files or display that one on the console. So that is our RDD in Spark. So we have

discussed this one with some discussions and with one proper diagram. Thanks for watching this video