Build a Node.js Express Google Search Console API With OAuth2 to Fetch Website SEO Data in Browser

uh Hello friends welcome to this video so in this video we will look at the Google search console API and we will

interact with it inside nodejs and express and we will actually fetch the data which you see in the search console

so you can see we have this website listed out there and we have this data right here so we need to fetch this data

inside our own application so we have this authentication built in login with Google so if you click this button you

will be redirected to the login page where you need to choose your Google account in which once you select the

Google account you need to allow the permission so you need to click continue and then you will see the list of

websites which your Gmail account has so we have all these websites listed and I

can select any of these website and then I can select the data from the start

date and the end date so let's suppose I want all the data to be coming from 1 January 2024 to the end date and here I

can select today's date which is 27th of November so we have this date picker

available and now if I click the button fetch data you will see it will fetch all the data the clicks The Impressions

CTR position that you see in the Google search console so this is a individual page right here this is a search query

and this is a number of clicks Impressions CTR and the position so you will see we will fetch the top 50 pages

that has the most website traffic and you can just match right here if I open

this in the pages tab right here in the so all this data that you see we are

fetching it and displaying it in the browser so you can

change the using this date picker you can change the start date if I pick

Let's suppose this 2022 so now the date I will change you

will see this is actually the top 50 pages from this time period and it is displaying all this the page actual page

the actual cery the clicks impression CTR position so we will be using this

and displaying this and this will be a complete example uh the full source code is given in the description so this is

we are using ejs nodejs Express and let me show you the modules

that we are using so we are using as I already told you Google search console

API for interacting with it we are using this module which is Google API so it's

a Google API client specifically for nodejs so we are using this module it

has got over 1 million weekly download so this is a simple command so we are using this and if you just search for

this API which is Google search console API this is actually the

thing we using this so you can read more about it by going to this so we using it

inside nodejs and express and apart from that we are also

using ejs which is actually a template engine where we actually

render JavaScript templates inside HTML so it's a template engine inside Express

so this is a command and last but not least we are using the express server in nodejs which is used

to make the rest API so these are the three technologies that we using to build this application so I

will now see you in the next section where we will start developing this application uh now to get started guys

we will start building the application so I will be there in my project

directory so I will simply open this the command line so I will make a new folder

I will simply say here make a new folder which I will call this Google search

console clone I will CD into

this and open inside vs code and we will simply make a package.json file to

initialize a nodejs project npm in-

y so this will actually create the package.json file with the default

options so now we need to install the modules so

npmi Google apis Express which is will be the web server

and EGS template engine so these are the three modules which are required you simply install

them so using this command so it will install

this it will create this node modules folder and in the dependency sections

you will find out these three modules listed out we are using the

latest version and for starting your application we using this tool which is

node one which automatically starts your application once you make any sort of changes so install them globally now we

can start just make a index.js file which will be the starting point of the

application so first of all we can simply make a simple Express app

requiring Express and just starting this

application on Local Host

5,000 so if you now want to run this application you will simply say node

mode and then index rojs the file and it will start this

and you will see you will see this console lock message that is app is listening on Port

5000 and now we need to Simply set the view engine as well which we installed

which is EGS in this way we can set the view engine by using the set method and

here we can simply make a simple file when we go to the homepage we need to

load a template so we use this render function and inside this we mention the file name so in this case we are loading

the index template we don't write the extension so for actually storing the

templates we do need to make a view directory and here you need to store the

templates so we have naming this index. ejs which is actually the extension

name so this will be a simple HTML I will simply say Google search console

we simply have a simple heading so if you go to

the Local Host 5,000 you will see now this is loading that

template so instead of this now we need to design the actual page where we will

be accepting first of all we doing the authentication for doing the

authentication we need to basically import that that module from Google aps

so we are actually in including this module Google apis which will do the

authentication and now for actually doing the authentication we do need to go to Google Cloud

console and get our client ID and client secret and for using any Google API you

do need to create account in here Google Cloud console I've already created one

after that you need to enable the apis by going to the section enabled apis and services and you need to enable the

Google search console API so just search for this API Google search

console and the very first result which comes this is the API that you need to

enable simply click the enable button I've already done that so then you need

to create the credentials by clicking this button manage

and you need to click this create credentials and you need to select the

second option here o or client ID and from the application type this will be

building a web application simply select that and here you need to paste your

homepage URL so I am running this on Local Host

5,000 so this will be the authorized JavaScript origin and then for the

redirect URI we will actually be having this URI we will make this route

which will be the call back Ur So just write o or2 call back this can be

anything but I'm just calling this as o or2 call back you can call this

anything and then after you do this simply click the create

button so this will create your client ID so now you can download the Json file

by click this button so now this will download everything inside this Json

file so you can open this file using vs code

and it will give you this information that you see and I will simply copy

this and create a file here which I will call this as

credentials. Json and I will paste all this information that we copied from Google Cloud console simply transfer

that file inside your project you'll be using this simply close that after we

get get this file now we can use this file by loading it for loading this we

do need to have the file system module in nodejs we will include this module

and uh then we need to load these credentials load the Au Au credentials

using using this file so we can simply load this credentials by first of all converting this to Json

so json. pass and then we'll be reading this using this function in file system

module read file sync and we will be providing the path here which is

credentials. Json and then here in the second argument it takes

actually the encoding type so here it will be tf8 so it will read everything and store

them inside this variable credentials so after that we will basically make the

O client O2 client for making this we'll create an actual variable and we'll be

instantiating Google and it contains this function Au and it contains this

O2 and it actually takes three options the client ID client secret and the redirect

URI so we can load everything inside web. client ID so if you open

this credentials Json file you will see there is a web property it's a web object inside this we have the client ID

redirect URI and the client secret as well so these three things we are loading right

here we simply here we need to Lo load the

client secret and thirdly we need to load the

redirect URI so we loading these three things to

create this o to client now we can actually use this to create the

URL so this contains a function right here generate Au URL and here we can

pass two things which access type we need to set this as offline the second option is scope scope is nothing but

what information that you are accessing from the Google so here you need to provide exactly the information that we

want to accept googleapis.com web Masters read only scope that we want

from the Google search console API so here we are providing the scope so then we'll be declaring a

tokens variable which will be null so here we'll be storing the access token which will be coming after you do after

user do the authentication and now in this get

route we will actually pass some information to this template here is a second argument which will be an object

so we'll be passing this Au URL and then the error variable we'll

set it to null so we are passing these two things to this temp template index

template this authentication URL that we constructed right here so now going to

this template index. J ejs we now we'll be having a simple

button which will be responsible for redirecting the user so here we'll be

using some bootstrap so just include the CDN right here after the title so we using bootstrap and then we

will be using a simple container class

bootstrap we having a simple heading which is Google search cons so mini

clone so here we'll be embedding some EGS so the

square if any sort of error is there in that case this is just the syntax of EGS

so essentially what we are doing we are checking for if any sort of error is there we displaying the error

message and just below that we will have a simple button in the anchor

tag so here we are embedding that variable that we got Au URL that we passed like this we can embed this using

EGS giving a bootstrap class to make it a as a button simply say login with

Google so here this is actually these two variables that we are embedding

first is the error and this is the Au URL so we are passing these two things

to this template if you see so now if you refresh you will see a button will

be present login with Google if you press that button you will be redirected to

your account you need to select the account and then you need to Grant the

permission click on continue and you will see it will show

you this error message that this authentication call back URL that you mentioned is not defined we haven't

Define this URL so we do need to Define this URL and exchange this authorization code

that you see in the URL with the access token so now we need just need to Define

this redirect URL so what I will do I will

simply Define this just below this you can Define the redirect

URL this will be a get request so

app.get so this needs to match in the Google Cloud console whatever that you

provide you need to provide the same here both things things should match and

now inside this we will write the code in try catch block so if any sort of error take place we can console log the

error message and now in the tri block we will basically extract authorization

code from the URL so we can do this request or

query we can extract the authorization code after doing that we need to

exchange this for Access token you can easily do this using this

Library you can use a weight here so o client it contains a function to get the access

token we have this function of get token so this will return your access

token and here you just need to pass the code authorization code as an argument

so this will return your access token so now we can store this access token in the variable that we had earlier so you

can say access tokens so you can see earlier we declared a global variable which which is tokens right here so we

are just using this variable to store the access token so after we actually set the access token

we just need to set the credentials and pass this access

token so after we do the authentication now we can easily call our search console API to fetch the website data so

first of all we do need to get the list of websites that goog uh user has for

automatically populating those websites we can use a function right here which is google.

webms and here you need to specify two things first is the which version of the

API that you're using so I'm using the datest version which is version three the second option that we need to tell

is the O client so we are need to pass the O client that we are using so

o we simply passing the client so after setting this now you can call the

API so first of all we will fetch the list of websites that the user has so

for fetching that we have a

function we can use this object it contains a function which is do sites do

list so it will list all the websites that the user has

and uh we can store this inside this variable site list

response dot data and inside that we have site

entry if no sites are returned then we would be equal to empty array so if user has

no sites then empt array will be assigned so now I can just console log

this information just to show you these information has been returned let me also console log that

just you can see in the browser that the information is

returning so this is your access

token so now if you refresh if I go to Local Host

5,000 if I repeat the same process again clicking the button again you need to Grant you just

need to do this for the very first time after that it will store that access toen in the you don't need to do this

again and again so once you grant the permission

it will give you their access token

so now you can see that guys if I click this button you will see in the console we

will return the list of websites that the user own you will see all these websites will be returned to me in the

Json response so now we can Loop through all these websites and display it in the

browser in the select Fe field so that the user can select which website to select so right here

uh inside this redirect URL we need to actually show a

template let me call this as dashboard so once the user is authenticated we need to redirect the

user so we can use the redirect function and redirect to the/ dashboard route I'm

just calling it/ dashboard and now we just need to make this route SL

dashboard and inside this we will be say async

function and here you'll be checking that if the access token is not defined then we just need to return to the

homepage essentially the user is not authen authenticated if the access token is not

present so now assuming that you have the access token then we can

fetch actually the information that we want to display so we can

again use the same thing again pass the version that we are using version three

the second one is that your authentication client so o or

client so then we can repeat the same process to fetch the

sites so in the TR catch blog we can write the code

so you'll set return the sites and you make a variable selected site

and this site will be pre selected when you load the page so the very first site

so site entry. site URL

so then we need to load a template so we'll call this template as results or I

can call this as dashboard you can call this anything extension is not written and here we'll

be passing this information first is the array of sites response. data. site

entry the second is your selected site the third information is your

performance data which will be by now it will be null error will also be null so

we are passing these four things to this template now we need to create this

template in the views folder so you can call this as

dashboard so this is your dashboard page

so right here we again need to use bootstrap so just include the CDM after

the title and right here uh we will use the container class of

bootstrap and simply say here search

console analytics if you just refresh your

browser just go to click on this button we already will

be authenticated and redirected to the dashboard page if you see now this is your dashboard page now we need to show

the list of websites so we will have the form right here and the method here will

be post and we will be making a simple post request to/

analytics so we will submit the information and make the post request to get the data

and here we will have some bootstrap classes

so here you'll be having a simple label where we allow the user to Simply have

the select list so we will simply say uh this will be for your site URL I'll be

giving a bootstrap class of form label and here will be simply saying that

select site and then we'll be having a select tag we giving an

name parameter which is required we give it site

URL and we giving a bootstrap class of form select and it should be

required and inside the select list we will be dynamically be looping through all the

sites so you'll be using some ejs right here so here we can use JavaScript code

and we have this array we can Loop through we can use the for each Loop and

for each site we can simply you need to wrap everything in

these TXS this is uh EGS embedded JavaScript

template so in between these we can create your options for the select

Tex and for the value here we need to embed the dynamic variable which will be side site

URL

and all the source code that I'm using here is given in the description so you'll have the set of

options so here if you refresh now you will see the

set of websites prepopulated you will see the website owners that owns

so this will be different for you if you have some websites it will be populated here so in this way you can show the

series of websites set you own so after this select

list we will have two date Pickers where we allow the user to pick the from date

and to date so here it'll be having two date Pickers so I will simply paste the code for

that so you will see if I refresh now we have we have two date Pickers where we

allow the starting date and the ending date [Music]

so if you see we just have given these name parameters start date and this is

for the end date so name parameters important you

need to give them in order to get access to the values in

Express and lastly we will have a simple button

a bootstrap button let me just format so we have the button to fetch

the data so now this is your dashboard page completed so once you select the dates

and submit the form it should display the data but if you do this now it is telling that it cannot find the route so

we do need to make this route in the index.js file so just make this route

app. poost SL analytics and it will be Asing

function so here we do need to get the things that we submitted through this

form so first of all we will again check if no access token is there in that case

we just need to return to the

homepage so then we'll be extracting all the things which will be starting

date and the end date and the actual site

URL and you'll be extracting it using the request or body headers you will see

request. body and then after we get all this

information uh let me just console log it just to check if it is submitting or not

so if you select your website and then you provide the starting

date and the ending date click on fetch data and if you see the

console it is studing cannot D structure property uh here you will get error

because I think we haven't set the body head uh body passer middleware so for doing this we do need

to set our body passs and middleware so we need to include a line

here app. use Express dot URL encoded extended to

false and express. Json so these two middle lines we do need to add because

we are transferring information from the HTML file to the Json file so we do need

to give these name parameters as I already mentioned to each input field so

that we can get their value so we have given these name

paramet this is for the site URL so if you try this now hopefully it should

work

so now you will see all these three things have been captured and it has been transferred to the server side

Express Now using these three things we can actually fetch our

data so just include these two middleware lines after we get this we

can make a simple rest API call to the Google search console

API so we can simply here say con

const Google dot again we need to mention the which API that we are calling so

right here we need to mention the version number and the pass also pass

the o or client after doing this we will make a simple

TR catch block so right here uh we will make the

call to the API we say search console dot search analytics. query this is the

method that we have and here we need to pass an object first it takes the site URL that we are targeting secondly it

takes the request body object here we need to pass the starting date the end

date and here it takes the dimensions it will be an array so Dimensions refers to

which properties you are targeting so we need to get get the latest the top pages

of the website so provide the first property as page the second is the top queries top search queries that we need

these two information the page and query and lastly it has the property of row

limit so how many pages that you want to show I need to find out the top 50 pages of my website which has the maximum

amount of traffic so it is exactly meant by this and after

this we get the row back so response. dat.

rows so we can just console log

this console log the response so

after that we need to construct using these rows variable this would be an array here we

can map each row and right here we can

simply inside this we will have different properties first is the page number so row. Keys

zero and actually the query which will be equal to row. keys this will be the

second property and number of clicks we can get this using row.

clicks if nothing is there then we put zero and we get our impressions as

well so this will be row.

Impressions and then we click through rate so we can say row Dot

CTR so you will basically convert into a whole number we can use this tary

operator we will divide multiplied with by 100 to get the click through rate in

a decimal number to convert the percentage and then we lastly we get the

position of the website so we can have the property row position again will

be converting this to fixed value so two fixed

one so now we constructed this uh whole thing which is your performance data

which will be an object which have these six properties

the page number query clicks impression CTR position so we have these six

properties now we need to rent ENT the the template once again which will be

our dashboard template and here we need to pass all

these things so we need pass the sites so be basically making a function

right here which I will call this as get

sites which will actually get our sites the total SES so

this will be a simple Asing function I will just Define here so the sole

process of this function is to return the number of sites that user has so we

are just calling this function right here the second property that we need to

pass right here is the selected site is our site URL the third thing is

our performance data

this will be we are directly passing it so we can

directly write like this and error will be

null so if any sort of error take place in that case also we can rerender the template

so now we need to go to the template which is dashboard EGS and we need to Loop through the

object that we are passing to this template which is your performance

data so just go to this file right

here and just after the form is ending right here after the form

tag so once again we'll be checking first of all for the error if any error

has there in that case we can display simple alert

message so after this we will actually have to display the information in the

tblo structure so we will have a simple table but before that we will check this

in the if condition that if the performance data is available

and if the length is greater than zero in that case only we need to display

this

information so just make sure that you close all these brackets this is valid ejs code so after you do this we have

the table tag where we display the information so we'll have the bootstrap classes attached to it so we give it a

table class table striped

then we will have the table head tag so then we will have these six

columns first is your page number second column will be for the

cury third column is number of clicks fourth one is for the

Impressions then we have the CTR click through date and lastly we have the

position so now this T after table head tag we have the table body

tag so inside this we will write the EGS and we will looping through this

performance data variable and we'll use the for each Loop and for each

row just make sure that you close all these brackets

so in between these we will make a simple table row tag inside table data

we will embed the variable which is row.

page so in the similar way we will print out all these

variables so if I go to the browser and uh

it is saying that unexpected token else in that dashboard EGS temp while

compiling EGS so I think we haven't closed the

bracket just make sure that you close all the

brackets so let me re reexecute

it is saying that or URL is not

defined uh let me check

is saying all

URL oh sorry we pasted I think the wrong

okay we need to paste only this we pasted

the only the HTML

code so yeah this is now correct and if I again

refresh so now if you see you will have your select list here you can select

your website that you want to Target so let me select this free mediat tools.com website and let me pick a date so 27

January 2024 and the ending date is today 27 November

2024 in between that we need to fetch the data as I click the button you will see it will fetch all the statistics in

between the starting date and the ending date we have the top pages of the website listed out we have the queries

we have the number of clicks Impressions CTR position all this data that you see

the top 50 pages of the website is listed out so we are fetching all this information from the Google search

console API so it's a free full web app that we developed from scratch

so this data will be different for you if you have the website you can analyze

it so you can change if you want more pages uh we are fetching the top 50

Pages if you want top 100 Pages you can change this value and uh in this way you can use

this API to fetch the results and display it in the browser so thank you very much for watching this video and do

check out my website as well free mediat tools.com which contains thousands of free tools regarding audio video and

image and I will be seeing you in the next video