Machine Learning - Preprocessing Structured Data - Detecting Outliers

In this video we are discussing detecting outlayers so we shall give you one demonstration on this very topic

So here is the demonstration. Let us discuss outlet detection and outlet handling

So before going for that, let me discuss what are the types of outliers

The first type is known as the global outlier. So what is the global outlayer

So here the object significantly deviates from the rest of the dataset

next one is a contextual outlayer what is a contextual outlayer object deviates

significantly based on a selected context for example say 28 degrees centigrade is an outlayer of a Moscow winter but it is not an outlayer in

another context that is say 28 degrees centigrade is not an outlayer for Moscow

summer so that is a contextual outlier last one is a collective outlayer

so a subset of data objects collectively deviate significantly from the whole data set and even if the individual data objects

may not be the outlayers for example say a large set of transactions of the same

stock among a small party in a short period of time can be considered as an

evidence of market manipulation so in this way we're having three different types

of outlayers global outliers contextual outliers and collective outliers so let us go

for the coding now so here we have imported the respective models and here we have

created one data frame with three attributes price bedrooms and size the data

frame name is flat that is a house or flat I'm considering so here we are

going for the respective printing so I'm executing this so here is my data frame

consider that last record here the prices are all of five digit numbers but

it is of they are of six digit numbers and it is of seven digit number so here the bedrooms all of them are lesser than 10 or you can consider 20 also but it is it is 125 I think it is a very big compared to the rest of this So there is a global outlayers we can consider

Next one here all the sizes are four digits but here the size is of five digits. So here we're having some outlayer. So one record at least I'm finding that it is containing some outlier values. So now how to detect it. So the method we are showing here is the eleventh

envelope assumes that the data is normally distributed and based on the

assumption draws an ellipse around the data and classifying any observation inside the ellipse as an in layer labeled as one and any observation outside

the ellipse will be will be labeled as outlayers and labeled as minus one a

major limitation of this approach is the need to specify a contamination parameter

which is the proportion of observations that are outlayers and the value we don't know here we're

considering that this contamination parameter here we are considering as that 10% data will be

outlayers but here it is very difficult to guess it if we know that 10% data is outlayered so

I think we are having a good concept on the data set but that might not be always

available so this elliptic envelope this particular method is having some limitations

for that so here we're defining one out layered underscore detector object under the

class elliptic envelope contamination parameter has passed as 10% that is a

point one fit detector so executing the fit method and also the predict method

you can find that here it is being labeled as one one one so these four

records it has been considered as non outlayers that is the in layers and the

last record has been considered and labeled as minus one it has been treated as outliers so next one is that how to deal with them so at first we can go for the

drop so here here we are using one formula one condition that is if the bedroom size is less than 20 then it is it is okay other than it is the record has to be dropped so if we execute

the same you can find that the last record the last row has been dropped

because its size was 125 which is greater than 20 you can easily find it out here

it is 125.0 okay next one the mark that means I do not want

to delete but I want to mark it with 0 and 1 so here you can go for that is

that when the flat bedroom is less than 20 mark it as 0 otherwise mark it is

as 1 and I'm creating another column there is a flat out layer so now you can

find that this column has got filled up with the values zeros and ones and

zeros have been put against those rules where the flat size is here you can find

that the bedroom size is less than our 20 but whenever the bedroom

room size is greater than 20 I'm finding this one has been labeled as one

according to the given code next one is a rescaling so here we're creating one

new column head that is a log of square feet so the log of square feet is there

so here is a we are taking the x we are taking the x log of the x and then

x is the respective flat size so on the flat size that means on this very

column you have done this log and you have done the scaling to some extent so now

you have if you print the respective data frame it is getting printed one

extra column that is a log of square feet so now let me go for the that is the I

QA that is in that is an intermediate quartile range so here you can find that

flat is equal to DF so DF I'm just keeping I kept this DF earlier also

so that I want to work on the original data frame so DF is this one so now I've kept

this original value there in DF so now here we are having this inter quartile range so I QR inter quartile range so the the flat has been initialized with the old values so those two extra columns have got deleted now and then we are

calculating the quartile one and quartile three we know that quartel one means 25% data will be

below that value and quartile two means the median that is a 50% data will be below that value

and quartile three that is a 75% data will be below that value so flat dot quater

quantile that is a point two five so we are calculating the quartile one and he

are calculating the quartile two the method name is quantile and that is your

point two five and point seven five I have passed so inter quartile range so

here you can find this one as Q3 minus Q1 there is the IQ word print IQR so

I'm printing this value here you can find that if you print this I keyword

I'm finding that the IQ values for price bedroom and size so

they are this so now when the when Q1 minus 1.5 into IQR and Q1 plus 1.5 into Iqr so any flat

whose whose corresponding value is less than this or greater than this will be

considered as the respective outlayer so it has been treated as any axis is equal to

one so this condition will be applied column wise and then meant out will be

the new data frame you can find that the last record has got deleted so here you can find that we have we are putting this one as not

that means those records which will not be satisfying both these two conditions will be will be

persisting and those records which will be satisfying any one of the conditions will be deleted

and so the third row has got deleted from here so this is my total code here

you can also type the code you can do the respective uh experiments on it

to have the better understanding that how to detect and how to deal with the outlayers and what are the different types of outlayers

We have discussed that one into details. Thanks for watching this