Spark Transformation Functions

in this video we are discussing spark

transformation functions so there are so

many functions are there and you'll be

concentrating only those functions which

are very much regularly used and in our

spark related applications so here we

are having one table the first column is

containing the function name and the

sinem column is containing the purpose

and description of the function so first

function is map so returns new or DT

formed by passing elements from the

source through the map so in the sub map

the user can put their respective

business logic and here it will take the

same partitions same number of

partitions as as input and same number

of partitions will be produced as output

so that is a main concept are in this

map and like all transformation

functions it will take our duty as input

and produces our DD as output next one

is the filter returns new or DD by

selecting those element who where filter

returns true so filtered will have some

criteria so depending upon the criteria

those records will be selected from the

input to the output which are matching

with the criteria next one is a flat map

so here each input item can be mapped to

0 or more output items so here in case

of flat map it will have say in number

of partitions as input and it can

produce 0 1 or more than 1 partitions as

output so that is a basic concept of

this flat map method and here each input

item can be mapped to 0 or more than

output items and more output items and

that is known as the flat map method we

are having a separate video on this map

and flatmap method you can also watch

that one next one is the map partitions

this particular method so same as map

but runs map tasks for each partitions

of an RDD next one is Union so returns

the new oddity of the Performing Union

operation on the data so it will return

that separate another ID or Duty there

is a

or did he and which is performing the

union of our data next we are going for

group by key method so this particular

function is that a data set with key

value pair will be converted to key and

then iterable values because it is a

group by key so he will be there for the

same key that it a table on the values

will be obtained in case of group by key

method next one is the shot by key

function and shorts the data on the key

of key value pairs so it will shot it

depending upon the key value pairs so

depending upon the key in the key value

pairs next one is the joint function and

when called on kV and kW the key value

pairs

it returns K V Kamata blue and that is

known as the joint operation next one is

the Kardashian so fine Cartesian product

of two data sets that means all possible

combinations of two data sets next one

is the repartition and suffle the

content of an RDD randomly and it may

create more fewer more or fewer data

sets and that is known as the real

partition so these are the different

functions which are available in our

transformation on our didi thanks for

watching this video