0:00
Hello everyone, welcome back to Fat
0:01
Trader uh where we talk all things
0:03
trading, investing, and building wealth.
0:04
Today's video is something that's been
0:06
in popular demand from many of you in
0:07
the community, especially those of you
0:09
who just starting in algo trading or
0:10
learning Python. Um, one of the first
0:12
questions that I get asked is how do I
0:14
download NC historical data
0:15
automatically, both EOD as well as
0:17
intraday. Uh, by intraday, I mean the 1
0:19
minute, 5 minute, 15, 30, 60 minutes uh
0:21
type of data uh for stocks, futures and
0:24
option symbols. For most uh experienced
0:26
algo traders, they get this data
0:27
directly from their broker APIs.
0:28
However, for people who are starting new
0:30
and don't have a broker API setup, they
0:32
need this type of uh utility. Well, uh
0:34
the good news is per your request, I've
0:35
created a simple Python utility uh that
0:37
can download the NSE historical data. So
0:39
whether you're back testing strategies
0:40
or just want to analyze price action,
0:42
this tool gives you clean, reliable
0:43
market data straight from the official
0:45
NSE website, and it's uh completely free
0:47
as well. In this video, I'll walk you
0:49
through the utility, its code, and I'll
0:50
show you also how to use it step by
0:52
step. You can copy this code for free
0:53
and use it as you please. So stick
0:55
around and smash that like button if you
0:57
haven't done so already.
0:59
If this is your first time here,
1:00
welcome. My name is Vive and I'm a
1:02
financially independent algo trader.
1:03
This channel is all about building a
1:04
community of algo traders. We discuss
1:06
everything about a trading using Python,
1:07
building and practicing trading
1:08
strategies, market updates, much more.
1:10
Please do visit our community website
1:12
fabtrader.in. Also, do check out my
1:13
other YouTube channel Fabe where I talk
1:15
about my own financial independence
1:16
journey and share tools, methods, and
1:18
strategies that help me achieve my
1:20
financial freedom. Thank you.
1:26
A few days back I'd uh written this blog
1:28
article. I also did a video on this
1:29
where I had created an NS utility uh in
1:32
Python which using which I think you can
1:33
do a bunch of activities like for
1:35
example tracking live price you can
1:36
download B copies uh you can extract
1:38
corporate auctions corporate
1:40
announcements um you know a whole lot of
1:42
things uh that you can basically fetch
1:43
directly from the NSE website and this
1:45
is that video uh if you haven't seen
1:46
this video uh or read this article I'll
1:48
provide the link in the description so
1:50
take a look at it. Um the video and the
1:52
blog article was received quite well. Uh
1:54
a lot of people told me that they are
1:56
now starting to use this on daily basis
1:58
and they found it very useful. And as
2:00
part of that video comments uh and
2:01
people who pinged me told me that you
2:03
know if uh I can do something on the
2:05
historic data as well because most
2:07
people are looking to download NSE
2:08
historic data both the EOD data as well
2:10
as the intraday data which is the 1
2:12
minute 5 minute 15-minut type data. Um
2:15
not just for the stock data they also
2:16
wanted uh you know stock options uh and
2:19
then uh futures data as well. So they
2:21
had asked me if I can create a similar
2:23
utility uh where I can download uh this
2:24
data from the nse. Uh it'll be very
2:26
useful to them. So based on popular
2:28
demand, I've created a a simple tool
2:30
that will do just that. So the details
2:32
that I'm going to be talking through the
2:33
in this video, all that information is
2:34
available in this blog article and I'll
2:36
post the link uh in the description so
2:38
you can read through this. Uh and then
2:40
what this tool uh would do is basically
2:42
uh just as you guys asked uh it'll get
2:44
historical data not just for the the uh
2:47
the underlying and the the dys but also
2:49
for the futures and options data for
2:51
those stocks and uh uh indices. The the
2:53
article gives you a very detailed
2:55
step-by-step process on how to install
2:56
this and how to use this. And this is
2:58
the entire code. Uh you can just simply
3:00
copy this and then uh you know run it as
3:02
you wish. You don't really need anything
3:03
apart from just pandas and uh python
3:05
installed on your machine. Before I walk
3:07
you through the Python code and the the
3:09
actual Python implementation, uh let
3:10
let's just start from the fundamentals,
3:12
right? What exactly are we trying to do
3:13
here and how are we getting this data.
3:15
So I'm sure you are aware of this
3:16
website which is charting.nse.com which
3:18
is the official nse website. Uh and then
3:20
this basically has charts for all the
3:22
instruments uh that that are in
3:23
question. Uh all that the Python program
3:25
that we trying to do is trying to
3:27
extract data from this chart and that's
3:29
where you're getting your EOD data and
3:31
all the intraday data. Right? At at a
3:32
very high level this is exactly what
3:34
that Python uh code is doing. Um for
3:37
people who are slightly interested in uh
3:38
you know technically understanding how
3:39
this is done I know a couple of
3:40
subscribers had asked me that you
3:42
provide code which is all good but try
3:44
and do uh you know try and explain how
3:45
this code works also so that we can
3:47
learn a thing or two. So for people who
3:48
are slightly more interested technically
3:50
on how this works um what we're trying
3:52
to do is that if any website if you go
3:54
uh it basically works uh as a series of
3:56
requests and response right um so if you
3:59
press F12 from the from any web screen
4:00
that you're currently in uh you would
4:02
you would see something like this and
4:03
then when you go into network uh this is
4:05
where you see how this particular client
4:07
which is your browser is talking to the
4:09
the server which in this case is the NSE
4:11
server right how how these two uh items
4:14
basically talk to each other uh can be
4:15
seen from this window So at a high level
4:18
how this works is uh you know the the
4:20
the browser or the client basically
4:21
sends a web request to the server in a
4:23
specific format that the server can
4:24
understand and then then the server gets
4:26
that request it basically responds back
4:28
with the actual data. So in this case if
4:30
you really see these are the server
4:32
requests that are going out to the
4:33
server from this particular web browser
4:35
and it is sending the data in this
4:36
particular format where it is telling uh
4:38
the server as to which specific uh you
4:40
know um uh script code or the the script
4:42
code in this case nifty uh 26,000 as as
4:44
everybody knows this is a script code
4:45
for nifty and then what time frame the
4:47
data is is is required in this case the
4:50
interval is 1 minute as you can see here
4:52
the time interval is one so it basically
4:53
sends this information in a JSON format
4:56
that the server can understand and then
4:57
server basically uh serves that page
4:59
with the with the data that is required.
5:00
Right? So in in the Python program, what
5:02
we're trying to do is just emulate as if
5:04
a real user is trying to, you know,
5:06
request NSE for data. Right? In this
5:08
case, there's no real web browser or
5:09
anything involved. But we're using
5:11
Python program to basically emulate as
5:13
if a user is trying to get this uh
5:14
request done through a web browser. And
5:16
that's exactly how we are trying to send
5:17
that request and we get that data back
5:19
from the server and we we trap that data
5:21
and that's how we get our historical
5:23
data. If you haven't joined our Telegram
5:25
group, please do so. I share market
5:27
insights, algo trading tips and new
5:28
video notifications and this way you can
5:29
stay up to date with our community news
5:32
events. Now with that background now
5:34
let's look at the Python implementation.
5:36
U maybe if you look at the code it'll
5:37
it'll make a lot more sense. Now so this
5:39
is the code uh right now from a intraday
5:42
perspective these are all the timelines
5:43
uh the time frames that it's you know
5:45
kind of supports which is 1 minute 3
5:46
minute 5 minute 10 minutes 15 30 minutes
5:48
and 1 hour which is 60 minutes the daily
5:50
1 week and 1 month. So this is all the
5:52
data that you can basically extract for
5:54
any stock any indices uh including you
5:56
know futures or option symbols even
5:58
stock options including right uh so
6:00
that's what this does um the way it is
6:02
implemented is we basically have created
6:03
a class where the the the key
6:05
functionality uh is basically already
6:06
coded in this is the class uh for people
6:09
who are not familiar with class class is
6:10
one of the most powerful data structures
6:11
within Python uh if you're familiar with
6:13
object-oriented programming um I'm sure
6:15
you've heard of class um if I want to
6:18
give you a very cr example on what this
6:20
really means assume that you you're
6:21
building plastic dolls or silicone
6:23
dolls, right? Um to to build a doll like
6:25
that, what you first do is you create a
6:27
mold first, right? The mold is basically
6:29
the class year and then once the mold is
6:31
ready, you can create as many dolls as
6:33
you want using that mold and and and
6:35
these dolls are basically the objects
6:37
that you're creating based out of the
6:38
the class that you're creating, right?
6:39
So the the class becomes a mold and your
6:41
dolls becomes the object. In this case,
6:43
we we're creating a a class called the
6:44
NC master data. And uh like I mentioned
6:47
what this is trying to do is that it's
6:49
it's basically trying to access these
6:51
URLs and then emulating as if a user is
6:54
sending the request to the NCS web
6:55
servers and then the server responds
6:57
with the actual data that we want which
6:59
is this data and then once the data
7:01
comes through uh we we're basically
7:02
extracting the data that we want convert
7:04
that into a pandas data frame and uh we
7:06
we download it and use it the way we
7:07
want it. Hope that kind of makes sense
7:09
and I hope I'm not confusing you. Uh but
7:11
but let's just move on to how this code
7:13
really works. So at the start uh I told
7:15
you about the nse master data being the
7:16
class and then you create an instant
7:18
called nse. This is this is where this
7:19
this becomes the mold and nse becomes
7:21
the doll that you're creating. So the
7:22
nse is the the object or the instance
7:24
that you're creating based on the mold
7:25
which is the class. Right? So in this
7:27
case nse comes to life here as an
7:28
object. Uh first thing is before we even
7:30
download anything we'll have to download
7:32
the entire nse symbol master which is
7:33
both including the nse as well as the
7:35
nfo. This step is very important. You'll
7:37
have to run it every time. uh this all
7:39
it does is basically downloads the
7:40
entire symbol master for NSE and NFO and
7:42
creates a cross reference file. Um and
7:44
then for downloading any data of course
7:46
you need the the start date and end
7:47
date. In this case I'm just taking just
7:49
for testing purposes uh 6 days. So uh
7:51
end date is today and my start date is 6
7:53
days prior to the the end date which is
7:55
basically the 6 day window is what I'm
7:56
considering. So feel free to change this
7:58
uh number to whatever uh time frame that
8:00
you want and accordingly the data will
8:01
be downloaded. Um before we download the
8:04
historic data, there is one additional u
8:06
utility that I've created here which can
8:07
be of a bit of use. Um not all of us
8:10
know the entire uh you know the symbol
8:11
list and its namesh by heart. So this is
8:14
basically a simple search facility. So
8:16
you can you can give even the entire
8:18
full uh name of this uh the symbol or
8:20
partially whatever you know and then you
8:22
can uh tell which exchange that you're
8:24
basically looking at and then when you
8:25
try to run this it'll give you the
8:26
entire list of the symbol code from nse.
8:29
In this case, I've given a nifty bank
8:31
and uh there is only one uh symbol there
8:33
within nse which is this nifty bank
8:34
which is again very clear very familiar
8:36
2609 is the bank nifty's uh symbol. So
8:39
it basically searches and gives you the
8:40
the details along with the script code.
8:42
Similarly you can do a search on nfo uh
8:44
the fo symbols as well. So in this case
8:46
I've given the partial bank nifty 25th
8:48
April but I want to extract all symbols
8:50
that kind of roughly match uh this wild
8:52
card. So when I run this, this gives me
8:54
the entire list of all the you know the
8:56
futures as well as the the option
8:58
symbols for bank nifty. So using this
9:00
you can basically zero in copy and then
9:01
run uh historic download on any specific
9:06
want. Now that you understand how the
9:08
the symbol search utility works uh let's
9:10
get to the the meat of the video which
9:11
is how to actually download the historic
9:13
data. Right? Uh so I've given sample
9:15
snippet for every type here and then I'
9:18
I've just printed the first two rows so
9:19
that you know you can you get to see the
9:21
actual output also. I've run this
9:22
already. Uh so let's quickly take a look
9:24
at you know what are the things that we
9:25
can do here. In this case I'm basically
9:26
downloading the EOD data for Nifty. Uh
9:29
here I've given interval as 1 day. Uh
9:31
that's why you getting the the EOD data
9:33
for Nifty which is here. Right. So you
9:34
typically get the open high low close
9:36
and volume for Nifty50 in this case. U
9:39
moving on. Um this is index data but
9:42
this is for a lower time frame time
9:43
frame which is 1 minute data. So I'm
9:44
basically having Bank Nifty data for 1
9:46
minute data uh which is again the
9:48
underlying data index for it. So this is
9:50
how it basically prints for every minute
9:52
it gives you the um the data for 1
9:55
minute and then similarly you can do it
9:57
for uh in this case I'm doing it for the
9:59
TCS which is again the the underlying
10:00
stock alone for 10 minutes so this is
10:02
where the TCS 10 minutes data is printed
10:05
um you can again extend this to um you
10:07
know index futures in this case nifty 50
10:10
futures is what I'm taking the April
10:11
futures so that also is downloaded I've
10:13
given a 1 hour data here so that is
10:14
downloaded um and then if you want the
10:17
stock futures again you can give the
10:18
stock name and then the the future
10:20
symbol that you want and in this case
10:22
Reliance data is printed. I've asked for
10:24
1 hour of data and 1 hour data is what
10:28
Um similarly if you go for index options
10:31
uh given bank nifty index options uh you
10:33
know the 50,000 PE and that's downloaded
10:35
for 5 minutes and then uh if you also
10:38
want the stock options that is also
10:39
available in this case the TCS stock
10:40
options were 3,000 CE and that is also
10:42
available here. So I' I've given a
10:44
flavor of each of the functionalities or
10:46
the symbol types that you can download
10:48
so that you know you you can run it and
10:49
then uh you know try it out for
10:51
yourself. There are a couple of items
10:53
that I would like to remind you. Uh
10:54
number one is uh nse the data is not
10:58
very clean. I mean in the sense that you
10:59
know for example if uh sometimes what
11:01
happens is instead of 9:15 uh it gives
11:03
you dates timestamps as 94 59 seconds
11:06
right? Uh I I've done my best to clean
11:09
up all of that to to ensure that you
11:10
know you get a proper time stamp just
11:12
the way you usually get it on uh you
11:13
know trading view. So I've done most of
11:15
the cleanup uh you know using uh I've
11:17
tampered with the code a little bit and
11:18
then cleaned up all of that issues for
11:19
you. Um but I would still like to warn
11:22
you that for some of the instruments
11:23
especially you know stock options where
11:25
there's not lot of you know volume and
11:26
all that you sometimes might get some
11:28
weird time stamps here. Uh so take a a
11:30
closer look at it and then fix it you
11:32
know accordingly. Uh the second aspect
11:34
is uh for majority of the the stock
11:36
options you know when you run it it'll
11:37
give you an empty one. So if there is
11:39
not enough activity for a specific
11:40
expiry or a strike you know you'll get
11:42
an empty uh data frame. So please make
11:44
sure so in case there is no data coming
11:46
from some of your options instruments.
11:48
So don't blame me. That's how NYSE is
11:49
right because u I've seen some of the
11:52
comments saying that your code is
11:53
and it's not working and all
11:54
that. So uh so I try to do my best here
11:57
uh to give you a cleaner code but
11:59
typically the data is something that's
12:00
coming from NC and I can't do much about
12:02
it. So uh so don't blame me,
12:04
right? Hope you like this tool. Um a lot
12:07
of effort has gone into doing this. So I
12:09
mean I would really appreciate if you
12:10
can like and subscribe uh because almost
12:12
85% of the viewers don't subscribe to
12:14
the channel. So if you can really do
12:15
that, that'll encourage me to do more
12:16
and more uh of these type of tools that
12:18
will be useful to you. So do try this
12:20
out and let me know how this goes and if
12:22
there are any issues uh just put those
12:23
uh you know in the comment section. I'll
12:25
definitely get to it. Until then um
12:27
thanks for watching and I'll see you on
12:28
another video. Bye-bye. If you genuinely
12:31
found this video useful, please consider
12:32
subscribing and liking the video. And I
12:34
will see you soon in another video. And
12:35
until then, take care and happy trading.
