Machine Learning - Preprocessing Structured Data - Imputers

In this video, we are going to discuss inputters

So, inputting refers to using a model to replace missing values. So that means whenever we're having some missing values in the data set, how to replace

the missing value with some other value using some logic. So that will be handled by this imputing process

So there are many options we could consider when replacing a missing value

We're having some examples here. So example here is a cost. constant value that has meaning with the domain such as zero distinct from all other values

So depending upon the domain, obviously, domain means from the, that is a collection of data

from where the data value came. So that's why we can select in case of, say, numeric domain, we can select zero that is

a distinct from all other values. A value from another randomly selected record

So we can select the respective attribute value which is missing. We can select any other record randomly

and you can pick up the respective attribute value to replace this missing value

A mean, median or mode value for the column. A value estimated by another predictive model

So there are multiple different ways in which we can fill up these missing values

We can handle with the missing values. Any imputing performed on the training dataset will have to be performed on the new data

in future when predictions are needed from the finalized model. As example, if you choose to impute with mean column values, these mean column values will need to be stored to a file for the later use on the new data that has missing values

So if I found, if I decide that we shall be filling up those or replacing those missing values by the column mean in that case, those mean values, respective columns are to be saved in a certain file so that when the new tuples

will be coming with some missing values in some attributes, then from the file we shall

pick up the respective missing value, which was nothing but the mean of the column in

the previous instance of the dataset to replace that one So that has to be maintained Using Python imputors missing values can be replaced by the mean the median or the most

frequent value using the strategy hyper parameter. So there are multiple different ways with help of which we can deal with the missing values

So let us go for one practical example for the better understanding on this topic

So here is the example for you. Let us discuss SK learn

Pre-Processing Imputer class. So how to use it in our Python code

to handle with the missing values? So at first we are going for preliminaries

So we are importing punders, import punders as PD, import Numpai as NP, and also from SK learn

Pre-processing import inputter. So inputted is nothing but a class under which

will be defining objects passing some hyper parameters. So at first we are creating one data

So one data frame we are going to create so we are creating the empty data set at

first so that is a empty data frame we are creating the data frame object is DF and

then it is going to have two columns one is the x0 and the one is the X1 so two

column heads are there here we're having the set of values here we're having

the set of values you can find that this particular value has been

initialized with the respective none that is our NP dot NN what is the NP

dot NN that is a NN pi not a number So here we're having one missing value there

And here we're printing the respective data frame here. So let me execute my code

So here you can find that my data set has got my data frame has got printed and here

we're having one NAN. What is the NN? That is the not a number. So now we are going for fit imputer

So create an imputer object that looks for NN values, then replaces them with the

mean value of the of the feature by columns. axis is equal to 0 what is the meaning of axis is equal to 0 that means along

the column that means not along the row along the column wise will be doing the

respective replacement so mean imputed we are defining one object under the class

imputed passing parameters that is a missing values is equal to n say we are having say a missing value means those particular cells having got the value zero so we are passing the respective value here strategy is

equal to mean so here we are going to replace it by the mean values and then

axis is equal to zero that is column wise so what are the strategies are there let me go

for some more detailing about this imputer class so here I'm having this

help imputer so let me execute it it will give me the I

idea so help on class imputed in module escalon dot pre-processing dot imputation

so here we're having the class imputed which is having some set of parameters

so we're having the missing values is equal to n strategy is equal to mean axis is

equal to zero Barbus is equal to zero copy is equal to true so here we can find

that these are the default values are there so now for the missing values the

integer we can put some integer or Nn option of default will be n and so on and for the strategy the default is mean other

strategies that they are median and most frequent so if mean then replace

missing values using the mean along the axis and if median then replace the

missing values with the median along the axis and if most frequent then

replace the missing using the most frequent value along the axis so here the

axis will be by default zero so axis is equal to zero means

along the columns axis is equal to one means impute along the rows we're having the

verbose that is an integer optional default is zero controls the verbosity of the imputed

we're having the copy that is a bullion optional and here the default value is true

if copy a copy of x will be created if false the imputation would be done in place

whenever possible note that in the following cases a new copy will will always be made

made even if the copy is equal to false. So for this particular cases, the new column will be created

always whenever also the copy is equal to false is there. So this is the respective health

we are availing against the glass imputed So you can go through it later on for the detailing and this is a code which has to be executed to get the same okay now let me come

to my point so missing values is equal to n n so we are considering that

n n will be considered as a missing value strategy is equal to mean and axis

is equal to zero that is a column wise and train the imputer on the DF

dataset so now we're going for the training this imputer on the Df

set so that is a mean underscore imputed so whatever the object we define

under the class imputed so mean underscore imputed dot df so the data frame has

been given so the respective columns the n n will be replaced by the mean of

the column here also you can make median if I if I like to do so so apply the imputer

to the data set so now impute df is equal to mean imputer

dot transform df values so here we're having the respective transformation will take place and then we are going for

the view of the data so now let me execute this one and then we are going for

this now viewing this one you can find that initially this one is having the

n-n in the column number X1 attribute name X1 so here you can see that the

attribute name X1 here the it has been replaced by point 4927

double three double three so notice that point four nine two seven double three

is the imputed value replacing the np. np.n value. So in this way in this particular

program we have given you the demonstration. I'm just putting all the codes in front of

you so that you can see you can also type the same you can do the experiments at your

end here also you can go for the median to get it let me do it that one for you

so we are going for this median I'm not changing this object names so let it be

there so now if you print it you can find that the nn value has been replaced by the respective median one so i think you have got the idea that how to use this

escalon dot pre-processing imputer class or to do the imputation on our data set on the null values

thanks for watching this video