MapReduce and Design Patterns - Filtering Pattern Overview

In this video we are discussing filtering pattern overview

From the very name filtering it is quite obvious that this particular pattern will be

required to select some set of records depending upon some filtering criteria and that's

why the name has got given as filtering pattern overview. So let us go for some more discussion on this topic

So what is filtering pattern? So as the name suggests this type of pattern is used

to filter records based on some conditions or criteria. So, using different conditions, it checks whether a record should stay or it should be dismissed

This pattern is used to find a subset from a huge data set with certain conditions

So we are going to have our huge data set in about big data, and depending upon the given

condition or criteria, we are supposed to decide whether a record should stay or should

get dismissed. So actually, it is calculating. or it is accumulating some subset of our given records

So as an example, if we want all records related to the user ID, let it be x, y, z

then we have to use the filtering pattern to do that one

So using this particular pattern we can do the filtering on our dataset So what is filtering pattern So the filtering pattern structure is very simple So map key value

So map is the, this is having the key value pair, so this is the key and this is the

respective value. So if we want to keep that record, return the key value of it

That means it will return the key value of the respective record

So you can find here we are having the input splits. So, here the input will be a big one, huge amount of data sets and records will be there

So, input splits will be obtained and then they will be assigned to the filter mapper

So, here we are having the mapper tasks. So this input splits will be taken as input by this filter mapper and then the output

splits will be obtained. And there is a basic organization of this filtering design pattern

this section we will see mainly four types of filtering pattern so they are simple random

sampling next we shall go for bloom filtering top 10 and distinct but we will see one

sample and very simple filtering pattern example before going for this all detailing

about this four different categories so let us go for the next videos for the implementation

of this simple filtering pattern thanks for watching this video