Limitations of Apache Spark

In this video we are discussing limitations of Apache Spark

We know that Apache Spark has got certain advantages but some limitations are also there

So let us discuss them one by one. So here we have mentioned five different limitations

The first one is the problem with small files. We know that each in every file, each in every file will be represented as a small partition

into our Apache Spark. And in case of large file, it will be

be divided into multiple smaller partitions. And obviously, problem with the small files

should be there and these particular partitions will be known as the small partitions

Next, we're having the no file management system. In Apache Spark, that is no inbuilt

file management system for it. It depends on Hadoop. We're having the latency. Apache Spark

latency is higher compared to the Apache Flink. So that's why latency is one of the problems

We're having the manual optimization. So here, no automatic optimization is there

The manual optimization synchronizations are to be done. So that is one of the limitations of Apache Spark

We're having the expensive. We know that Apache Spark supports in-memory computation

That means the data will be available in the memory. During competition, when the intermediate results will be produced and they will be kept

in the memory at the same time. We know that memory is a very costly thing

that is a primary memory or RAM is a very costly thing. So, that's why putting huge data into the memory means we require huge amount of memory. So that is a very expensive affairs

So let us discuss all of them into some more details So there are some limitations of Apache Spark and some of them will be like this So problems with small files In the SparkRDD each file is a small partition and for a large

file there will be a large number of small partitions will be there. To perform tasks in

efficient way, we need to repartition them into manageable format. So this particular repartition

will be a time consuming one. Next one is no file management system. So Spark has got no file

management system in depends on some other platform like our Hadoop, etc. So spark itself is having

no file management system. So it depends on other file management systems we can consider here

as an Hadoop as one of the examples. Next one is the expensive. The spark will be very expensive

when we want to do cost-efficient processing of big data. And keeping data in the memory is very expensive, and we need lots of RAM to do such

work smoothly. Because in-memory computation means the huge data will be loaded onto the memory requires

also the huge primary memory or RAM. Manual optimization. So that is a very headacheful thing

So the Spark job is needed to be manually optimized to specific data sets

and to partition and cache in Spark is also to be correct

So, these are the manual optimization will be a very troublesome for the users

We are having the latency. That is the spark has higher latency compared to the Apache Flink

So, these are the different limitations of Apache Spark. We have discussed that one with some diagram and detailing

Thanks for watching this video