MapReduce and Design Patterns - Structured to Hierarchical Pattern Overview

Here in this video, we are going to discuss structure to hierarchical pattern overview

So, structured data means the data will be in the form of RDMS, that means it will

be coming in form of rows, and this hierarchical means the data will be converted in the

Hadoop format that is our JSON format or XML format. So that is our hierarchical format

So here these patterns are falling under the data organization pattern. So as a result of that, here will be having one interface

of the data and that will be transferred to the another interface of data

So this transforming will be done here in this case from the structured data to the hierarchical data

So let us go for more discussion on this topic. What is structured to hierarchical pattern

So this pattern creates new records from a totally different structure. So new records will be in the hierarchical format but from a different structure that is our structured

So, using this pattern, we can convert row-based data into hierarchical data like your JSON

or XML. Because you know that in case of RDBMS data, there is a structure data, so data will be

represented in multiple rows. So we convert the data from the RDBMS to the Hadoop system and we need to use these

type of design patterns. Structure to hierarchical pattern structure. So to add multiple data sources into one hierarchical data, we can use the Hadoop class called

multiple inputs. It may happen that we will be dealing with multiple input sets, input data sets

In those cases, we require one Hadoop class. The Hadoop class is multiple inputs which is available in this respective package The Mapper class takes the data and parsed the records into one cohesive format So mapper will take the structure data and it will parse that one and it will be converted

to the respective one cohesive format which will be helping the reducer to do the needful next

The reducer will take data from the different sources and using the keys and it will create

XML or JSON from the data using the selected keys. So, this particular reducer will take the output of the mapper and now it will be converting

it to the respective hierarchical format that is either in the form of JSON or XML

So now let us discuss this with one diagram. You can find that here we are having the dataset A, we are having the dataset B

So depending upon the size of these datasets, the number of input splits will be decided

So, here we are having three input splits and that's why we require dataset A mapper and

three such mappers. And for these input splits we require dataset B mappers and two such mappers

So now its output will be like this. There is a post ID comma post data

It might be say parent ID comma child data. So this particular mapper will produce output in the form of key value pairs

Now the shuffle and short, that is the softling and shorting will take place

And now we are having this hierarchy reducer because we are supposed to convert this one in the

respective hierarchical format. So that's why we are having this hierarchical reducer and then the output part will be obtained

as the final output. So in this particular discussion we have discussed only the overview

But in the next video we'll be discussing how to implement it. So please watch the next video for the implementation process

Thanks for watching this