MapReduce and Design Patterns - Generating Data Pattern Overview

In this video, we are discussing generating data pattern overview

So, in this case, the data will not be read from the HDFS

It will be generated by some random functions and as a result of that, we'll be getting

the virtual data set, virtual input splits and so on. Sometimes this particular design pattern can also be used

When we are having a small data set, we can put some newly generated data to make it a huge

data set. So, inclusion of this dummy records within that small data set will become a huge

data set for the operation. So, let us discuss more on this design pattern. So what is generating

data pattern? So this pattern does not take the input from SDFS. It creates data from the

scratch. So it is not reading its data from the SDFS and it is creating data from the very

scratch using some random function generations. So, generating random data is used in different

cases and one of the use of is that to build some sort of representative data set to convert

the actual small data set to a large one. So where we are having a few number of records

in a small data set we can put such new data sets from the scratch using some random functions and with the help of which we are going to get a healthy long data set So the task is not so easy to implement in the Hadoop MapReduce framework

and as there are no input split or records so we need to fool the framework to think

there are splits are there and so many records are there. So actually we are creating a

virtual data set virtual input splits on which the MapReduce task can easily work

So, here this is the diagram with the help of which you can discuss that is generating data pattern

So, here we are having the virtual input splits. That means, so here the data have got formed from the very scratch, there was no physically

existing or pre-existing data sets. So that data set is a huge one, so we are having multiple input splits

So virtual input splits were having multiple such. we are having the record reader which will be reading this records from the virtual input

splits then we're having the identity mapper and using this identity mapper you can put

some business logic in the mapper and the mapper will work accordingly and it will produce

the respective output parts so you can find that this is the generic diagram with help of which

we have discussed the concept of generating data pattern overview in the next videos we'll be

discussing lots on these particular concepts please watch all of them and thanks for watching this