Machine Learning - Preprocessing Structured Data - Imputers

In this video we are going to discuss inputters

So inputting refers to using a model to replace missing values. So that means whenever we're having some missing values in the data set

how to replace the missing value with some other value using some logic

So that will be handled by this imputing process. So there are many options we could consider when replacing a missing value

We're having some examples here. So example here is a cost. constant value that has meaning with the domain such as zero distinct from all other values

So depending upon the domain, obviously, domain means from the, that is a collection of data

from where the data value came. So that's why we can select in case of a numeric domain, we can select zero that is a distinct

from all other values. A value from another randomly selected record. So we can select the respective attribute value which is missing

We can select any other record randomly. and you can pick up the respective attribute value to replace this missing value

A mean, median or mode value for the column. A value estimated by another predictive model

So there are multiple different ways in which we can fill up these missing values

We can handle with the missing values. Any imputing performed on the training dataset will have to be performed on the new data

in future when predictions are needed from the finalized model. As example, if you choose to impute with mean column values, these mean column values will need to be stored to a file for the later use on the new data that has missing values

So if I found, if I decide that we shall be filling up those or replacing those missing values by the column mean in that case, those mean values, respective columns are to be saved in a certain file so that when the new tuples

will be coming with some missing values in some attributes, then from the file we shall

pick up the respective missing value, which was nothing but the mean of the column in

the previous instance of the dataset to replace that one So that has to be maintained Using Python imputors missing values can be replaced by the mean the median or the most

frequent value using the strategy hyper parameter. So there are multiple different ways with help of which we can deal with the missing values

So let us go for one practical example for the better understanding on this topic

So here is the example for you. Let us discuss SK learn� pre-processing imputer class

So how to use it in our Python code to handle with the missing values

So at first we are going for preliminaries. So we are importing punders, import punders as PD, import Numpi as NP, and also from SKlearn

Pre-Processing import importer. So input it is nothing but a class under which will be defining objects passing some hyperparameters

So at first we are creating one data. So one data frame we are going to create

So we are creating the empty data set at first. So that is an empty data frame we are creating the data frame object is DF

And then it is going to have two columns. One is the X0 and the one is the X1

So two column heads are there. Here we're having the set of values. Here we're having the set of values

You can find that this particular value has been initialized with the respective none

That is our NP dot NN. What is the NP dot NN? that is a num pi not a number so here we're having one missing value there and here

we're printing the respective data frame here so let me execute my code so here you

can find that my data set has got my data frame has got printed and here we're

having one n a n what is the n that is the not a number so now we are going for fit

imputed so create an imputer object that looks for nn values then replaces them

with the mean value of the of the feature by columns axis is equal to zero what is the

meaning of axis is equal to zero that means along the column that means not along

the row along the column wise will be doing the respective replacement so mean

imputed we are defining one object under the class imputed passing parameters that is a missing values is equal to n say we are having say a missing value means those particular cells having got the value zero so we are passing the respective value

here strategy is equal to mean so here we are going to replace it by the mean values and then

axis is equal to zero that is column wise so what are the strategies are there let me go for some more

detailing about this imputer class so here i'm having this help imputer so let me execute it

it will give me the idea so help on class imputed in module escalon

pre-processing dot imputation so here we're having the class imputed which is

having some set of parameters so we're having the missing values is equal to n strategy is equal to mean axis is equal to zero Barbus is equal to zero copy

is equal to true so here we can find that these are the default values are there

so now for the missing values the integer we can put some integer or

or n n option optional default will be n and so on and for the strategy the default

is mean other strategies are there median and most frequent so if mean then replace

missing values using the mean along the axis and if median then replace the

missing values with the median along the axis and if most frequent then replace

the missing using the most frequent value along the axis so here the axis will

be by default 0. So, axis is equal to 0 means along the columns. Axis is equal to one means

impute along the rows. We're having the verbose, that is the integer optional default is zero

controls the verbosity of the imputed. We're having the copy. There is a bullion, optional

and here the default value is true. If copy, a copy of X will be created. If false

the imputation will be done in place whenever possible. Note that in the following

cases a new copy will always be made even if the copy is equal to false so for

this particular cases the new column will be created always whenever also the

copy is equal to false is there so this is a respective health we are availing against the glass imputed so you can go through it later on for the detailing and this is a code which has to be executed to get the same okay now let me come

to my point so missing values is equal to n n so we are considering that

n n will be considered as a missing value strategy is equal to mean and axis

is equal to 0 that is a column wise and train the imputer on the DF

data set so now we're going for the training this imputer on the Df data

set so that is a mean underscore imputed so whatever the object we define

under the class imputed so mean underscore imputed dot df so the data frame has

been given so the respective columns the n n will be replaced by the mean of

the column here also you can make median if I if I like to do so so apply the imputer

to the data set so now input df is equal to mean imputer

dot transform df values so here we're having the respective transformation will take place and then we are going for

the view of the data so now let me execute this one and then we are going for

this now viewing this one you can find that initially this one is having the

n-n in the column number X1 attribute name X1 so here you can see that the

attribute name X1 here it has been replaced by point 4927 double three double three so notice that point four nine two seven double three

is the imputed value replacing the np.n.n. value. So in this way, in this particular program

we have given you the demonstration. I'm just putting all the codes in front of you so that you can

see. You can also type the same. You can do the experiments at your end. Here also you can go for

the median to get it. Let me do it that one for you. So we are going for this median. I'm not

changing this object names. So let it be there. So now if you print it, you can find that

the nn value has been replaced by the respective median one so i think you have got the idea that how to use this

escalan dot pre-processing imputer class or to do the imputation on our data set on the null values

thanks for watching this video