Features of RDDs

In this video we are discussing features of RDT

We shall be discussing multiple different features of RDD for your better conception

So let us start with the discussion here. RDT features. Here we are discussing 10 different RDD features and they are location stickiness, fault

tolerance, parallel, in-memory computation, no limitation, course gained operation, partitioning, immutability, lazy evaluation and persistence

So we shall go one by one. At first we are starting with the location stickiness

To compute partitions, RDTs are capable of defining placement preference. And moreover, placement preference refers to information about the location of RDD

So although the DAG scheduler places the partitions in such a way that task is close to the data as much as possible

So if the data on which the tax will be operating, if it is placed as much as much

as possible close to that, then obviously it will speed up the competition on time

Next we are going to discuss that is the fault tolerance If any worker note fails by using lineage of operations we can recompute the lost partition of RDD from the original one So it is so it is possible to recover

lost data very easily. Now let us discuss on parallel. What is parallel? So while we talk

about parallel processing, RDD processes the data parallelly over the cluster. So as a result of that, the competitional time will be quite low

Next point we are discussing that is in-memory competition. Basically, while storing data in RDD, data is stored in memory for as long as we want to

store it. So it improves the performance by an order of magnitudes by keeping the data in the

memory. So it reduces disk red-write operations to manifold and 100-tampers. faster operation will be observed in memory computation next we are going for no

limitation so there are no limitations to use the number of spark RDD and we can use as any as any number of RDs and basically the limit depends on the size of the disk and the memory So that why there is no limitations that for how many number of Spark IDDs will be working with

So depending upon the workload, depending upon the size of the disk and the memory, the number of RDDDs can be selected

Next we are discussing course grand operation. So generally we apply course grand transformations to

SparkRDD it means the operation applies to the whole dataset not on the single

element in the data set of RDD in Spark next we are discussing partitioning

basically RD partition the records logically and also distributes the data across various nodes in the cluster and moreover the logical divisions are only

for processing and internally it has no division So hence it provides parallelism. Immutability

So imitability is the next feature of RDD we are going to discuss

And immutability means once we create an RDD we cannot manipulate it Moreover we can create a new RDD by performing any transformation Also we achieve consistency through the immutability

Next, a very interesting feature is the lazy evaluation. Spark lazy evaluation means the data

inside RDDs are not evaluated on the go. Basically, only after an action triggers all the changes

or the competition will be performed. And therefore, it limits that how much, you can't

how much work it has to do at that particular instant of time

As the last feature, we want to discuss persistence. In in-memory, we can store the frequently used RDD and also we can retrieve them directly

from memory without going to disk. It results in the speed of the execution

Moreover, we can perform multiple operations on the same time, on the same data

and it is also possible by storing the data explicitly in memory by calling parsist or cache functions

So in this way, in this particular video, we have discussed RDD features and different kinds of features we have discussed into details

Thanks for watching this video