MapReduce and Design Patterns - Bloom Filtering Pattern Overview

In this video we are discussing bloom filtering pattern overview

Now what is the bloom filtering pattern? So in this case we'll be having some predefined set of values and these values will be known

as the hot values. From the records, from the input records will be extracting some features and those features

will be searched on these hot values. And if the value has been found, if the feature has been found there then the record

will be kept otherwise the record will be discarded and dismissed. So, there is the main concept

in our bloom filtering pattern overview. So now let us go for some more discussion on this

topic. What is bloom filtering pattern? This is also a basic filtering pattern, but it has

unique evaluation function for each record. To extract the feature of each and every record

we are supposed to have some unique evaluation function. In the first, we are supposed to have some unique evaluation function. In this pattern we can filter those records which are present in some predefined set of

values and these predefined set of values will be known as hot values So we be having some predefined set of values and that set will be known as a hot value set So for each record it finds the feature of that record If it is present in the hot value set then the record will be kept

Otherwise, the record will be discarded and it will be ignored. So, in this way, the filtering will take place

So, Bloom filtering criteria. So let us go for a summarization. So there are some criteria for Bloom filtering

So, what are the criteria? Here you have mentioned four such criteria. Let me discuss one

by one. So, first one is that data can be separated into different records. So, a data

can be separated into multiple different records according to our requirement. An extracted feature from the records that could be in a set of hot values. So some evaluation

function will be there with the help of which this feature value will be extracted from each

every records and that will be searched onto some set of hot values

There is a set of predefined hot values. So we'll be having some predefined hot values

So there this searching will be done. Sometimes some unwanted data can be there after filtering also and it is also acceptable So if there is some unwanted data is there after filtering then also the data will be accepted so these are the today bloom filtering criteria

bloom filtering structure to perform the bloom filtering job at first we need to train our

system from the data set so at first the model that the model will be trained with some data set

so when the training will be done then we require here number of datasets so that my model can be trained in a proper way

So, after completing the training, it generates a data and it will be stored onto the SDFS

So, after completion of the training, the model will generate some data

And those data will be stored onto our HDFS. So, this is our phase number one

In the second phase, when the actual MR task is performed, it uses the intermediate

data which was stored in, in our phase number one. in HDFS then that particular data set and those values which is saved which are saved

on this SDFS will be accessed now here we are having one diagram for the better

understanding so we had two phases phase one and phase two so now phase one is actually the step one so step one means the filter training We having the input split we having a huge data set so the data set will be

split into multiple input splits. So that input split will be available to the Bloom filter training

So this input split will make this Bloom filter model to be trained and it will produce

some outputs and the output will be saved on the output file on the HDFS

Now, we are having the phase number 2 that is our step 2. So step 2, bloom filtering via map reduce

Now the MapReduce tasks will be coming in the scene. So input split, Bloom filter mapper and this Bloom filter mapper will refer load filter

from distributed cache and then the Bloom filter test will take place

Now the test will have two outputs. One is the maybe and the one is the no

So no means the respective record will be discarded, maybe means it will be stored. Maybe it will be stored under the output files

We are having multiple mappers that is a bloom filter mapper so they'll be working in

parallel simultaneously on multiple input splits. So that is the basic concepts of bloom filtering criteria in our map reviews

So thanks for watching this video