Cut your Kubernetes cloud bill in half with CAST AI - Product Showcase

Hi everyone. Welcome back to C.Shapcona live show. I'm a host Stephen Simon and we are back with another episode of a product showcase. And in fact, this episode is going to be very different and very exciting. As this time, I'm not your only host. I have co-host with me, Annie. She's joining. She is from Cast A.I. Team. Without any further ado, let's invite Annie. Hi, welcome to the live show

Hi, thank you so much for having me. So Annie, I have been doing you for a while now and you have been doing amazing stuff for the community

So thanks for the time today. Would you quickly like to go ahead and introduce yourself what who you are and what you do

Yeah, that would be interesting. Yeah, of course, super happy too. And I'm so excited to be talking with CSOP Shore Corner again

Very excited to chat once again here. But yeah, hi to everyone. I'm Annie. I am the product marketing manager at Cast AI

joined rather recently. It's a very nice company in general. And as far as my background goes

I have been, as Simon said, really highly involved with community activities for a long time now

I run my own podcast called Cloud Gossip. I've been organizing the CNCF and Kubernetes meetup for

over three years now. And I also speak a lot at international tech events. But yes

nowadays the product marketing manager at Cast AI. Absolutely amazing. So, Annie, you did say that you recently moved to Cast AI

Could you give a brief introduction about what is Cast AI and what's the product all about

Yes, very much so. So Cast AI combines instant AI driven cloud optimization with Kubernetes

It's one platform that cuts your cloud bill in half, boosts the power of DevOps 510

and guarantees business continuity by preventing downtime. And that's kind of the gist of the gist of

but what it actually means that we are building the industry leading cost optimization platform for Kubernetes

And at the moment, we have the best offering essentially for AWSE

And we will come up with the GKE and the AKS version very soon

So stay tuned for updates on that phone. Wow, that's great. Now, if everyone's watching now, if they look at the title of today's show

it clearly says that Cut Your Kubernetes Cloud. within half with Cast AI, right? That's a pretty bold statement to make, right, to cut to

have and I've been visiting your website on past couple of weeks. First, absolutely amazing website

I highly encourage everyone to visit Castor AI website. They have built it very nicely. And you say

that we're going to build it to half. How does this actually work? Can you give a brief idea about that

Yeah, of course. Well, essentially for like a quick brief, It is the wonderful AI we have created within the company

The company itself is a bit over a year old, one year and a half about

We launched the product during KubeCon Europe a month ago. But I think the best way to really experience the product

we get to know with the value, we get to know what happens and how it happens will be within the demo

that our amazing CEO has agreed to do quite soon to us

Yeah, exactly. So everyone's watching. Stay tuned as you're going to have a hands-on demo by Mike Leon

He's going to do a demo on Cast AI. Before we moved that in just a couple of minutes

it did seem that you have a community background, you have been speaking it, and now when you move to Cast AI

definitely the community part is very, very important too. So how would someone get connected with the community

If you have any channels or platforms, people can go and connect with other people

who work on in the same way. Yeah, of course. We are really loving the community that we're building and we already have up and running

and very much welcome everyone to join. We're super healthy to help you get started, to help you get involved with Castaa as well

We have a Discord and a Slack channel that everyone can join. If you need help joining or you need help using the product, always happy

I'm very happy to receive your questions. You can reach out to me on Twitter or obviously do the Castega official accounts as well

We have a really wonderful customer success team really very happy and ready to help you out as well

So either to me or the cast AI official ones if you have any problems

And I think the Slack community is also a really great place to start to get involved as well

Absolutely amazing. So what would you just did say? I did drop all the links in the comments on all platforms

So people can just go ahead. Don't click on this right now because then you're going to leave the live show

click on these links after we end the live show and it will always be there so go ahead and be a part of that

community and and yeah connect with other members too having seen that annie i think it's time to go

ahead and invite leon who has been very kind for waiting behind the scene so uh yeah let's let me add

hi leon and welcome to the live show hey guys what's going on how's everyone doing

We're doing great. Annie, I've been talking about some Cast AI, an overview

It looks pretty promising. And yeah, so it looks like next 15, 20 minutes of hands on

on Cast AI. It should be absolutely amazing. And I'm going to leave you

both, and I'm going to go back and grab a tea here in India. Then I'll watch

your demo. Fantastic. So we're doing a live demo. Excellent. Good. Good. Why don't we get

started? Any, can you see my screen okay is that is is the screen shared perfect okay so welcome everyone I'm

gonna take you through the story kind of from the very genesis of a Kubernetes

cluster so we'll just very quickly walk through I've pre-created a cluster because

it takes about 25 minutes to create on a AWS and then we'll walk through

like deploying an application and like kind of sizing your cluster and kind of

the estimates that DevOps engineers make and then we'll see why the vast majority of clusters are actually way over provisioned and poorly configured

right? Because we're all using our past experience and best efforts estimates to decide on

what our initial scaling should look like and what our overall auto scaling should look like

And this is where we see a ton of room for optimization and improvement. So I'm going to get into

the details in just a second. So let's go back and I'll show you what we've done

I created an EKS cluster using AWS's EKS-CTL tool. So if I bring up just my console here

hopefully you guys can see this, yeah. So this is kind of the completion of the EKS-CTL tool

I use the command to create a very vanilla default cluster with six nodes

I want some availability and some capacity. And then now we're kind of ready to

to use this cluster to deploy an application. So just a couple of setup notes

One thing I've done is I've run this really cool tool called OpsView, and I've connected Opsview to the cluster

So what does Ops View look like? It's this image right here

So I think that's pretty visible for everyone. And if you haven't used the tool before

it's basically a kind of a graphical representation of all of the nodes

There are no master nodes in this visualization because the master nodes actually sit in the AWS control plane They don sit in your tendency as an AWS customer So all we see here is the data plan nodes or the worker nodes right So we got about six nodes here no application deployed

So our next step will be, let's go ahead and deploy the application

So I'm going to use the command line tool to do that

I've got my little C sharp corner demo here. And let's do this

Let's apply. And I think it, oh, no, let's see what the demo is boutique

And so this is an e-shop demo that we're going to use

It kind of comes from Google Open Source. I'll show you the app in just a sec

Very, very simple e-commerce app. But let's go ahead and apply it

So I'm applying all of the objects that are about eight or nine microservices that are being deployed

Some of them are well replicated. And if we actually look at the OpsView view right now

you'll see that all of those pods or deployments are being deployed

So in just a second, this whole app will get deployed. If you wanted to take a look at what the actual app is

it's demo.coma.coma.i.com.com. And it's just a very simple app that basically runs a shopping card, payment microservice

a recommendation microservice, and so forth. Cool. So if we go back to the operational view, we should see the app kind of fully deployed here

But there's a problem. What's the problem? You may say, well, Leon, okay, I deployed my app

I've got six notes. Like, what better can we do? Well, the problem is that we, the Kubernetes by default

and the default scheduler is attempts to, utilize all the resources at its disposal

So it just deploys all of the apps across all of the notes

So if I had 10 nodes, I would be using 10, the app would be distributed on 10 notes

Well, what happens there? Well, everything becomes severely underutilized. So in terms of requests for CPU and memory

we're really just scratching, we're not using hardly any of the cluster

So this is where CASC comes in. So now we've got a vanilla E.K.S. cluster, we've got our application deployed

Let's now take the next step, which is to really connect the cluster and get some insight

into what we could be doing better. Cool. So to do that, you register on cast

You'll get a, my view as kind of lots of clusters that are already been used

So you're going to get a walkthrough view as you do this

And what we're going to do is we're going to add a cluster to our configuration, but instead

creating a cluster, I'm just going to connect an existing one. So this is like an existing cluster you've had

CAST had nothing to do with its creation. So we're going to go ahead and connect cluster

Excellent. So now what we have is we have a very simple set of instructions

And by the way, you'll see there are different tabs for GKE for COPS

which is Kubernetes Operations Tool and AKS is coming soon. Like the GKE and COPS pieces were just literally released a couple days

ago, maybe this morning. So I personally haven't tried GKE yet. I've heard the results are pretty cool compared to AWS as well

Cool. So let's go ahead and grab this curl command. And then we'll go back to my console

And we're just going to apply it. And what is this going to do

It's going to make an API call to cast, and it's going to deploy an agent

So if we go back to our ops view, we should see, oh, yeah, there's an agent

a little damon that's deployed. And what's the response? And this is a read only

Notice how we didn't ask you for access keys, we didn't ask you for any permissions

This is a pure read only agent. And it's open source. So you can see exactly what we're doing in your environment

One of our principles is if we install software in your Kubernetes cluster

you should be able to see exactly what that software is doing

So right now, what the agent is doing, it's collecting all your pod information

All your node information, it's using AWS's instance metadata service to go and pull for all of the details around the financials, the instance type, is it versatile

What's the architecture? Is it arm? Is it Intel and so forth? And if we go back to the console, we'll see now that we've got our C-sharp demo savings report is available to see. Excellent. So let's first time

I'm running this today, so let's give it a whirl and make sure we've done the appropriate

sacrifices to demo gods. Awesome. So what is this saving screen saying? It's saying that you should be paying

less than, you should be paying 66.5% less than you're paying now for this cluster. So let's take a look

Right now at the bottom, there's a bottom table here that says, look, you're currently paying

for six M5 large nodes. If M5 stands for memory optimized, right

So it's eight gigs of RAM relative to two CPUs. You're paying about 10 cents an hour, which sounds pretty good

It's all reasonable, but your cluster is ending up costing you $414.72 a month, right

So that's not great, especially since Cass is saying, look, we can get you down to $138 a month

and your monthly savings should be around $275. If you add that up for the year

you're saving yourself $3,300. That's a pretty good MacBook with an M1 chip

if you can save that money. So I would save these dollars anywhere I could

So now you may be saying, okay, Leon, great. So what are you suggesting here

Well, the first thing we're suggesting is these M5 instances aren't right for you

after the ysis of your workloads, and we've done all of the kind of precalculations

looked at all the permutations of the nodes that we can possibly use in HADPS

We're saying we believe that you should be running on two C5A large nodes

and then you should be using a spot instance. So very quickly, for those of you who don't know

what a spot instance is, only about 7% of AWS customers use spot instances

And here's why. A spot instance is a computer that you get from AWS

at a severe discount. You can get, depending on the popularity of the computer

it could be 50% off, 60% off, 70% off, and there are some obscure instance types

that are like 80% off. So why doesn't every customer use spot instances you may be asking

Well, the reason why nobody, or very few people use spot instances

is because it creates chaos. When a real customer, someone who hasn't bought that instance type as a spot

comes to you and comes to AWS and says, I need a computer urgently. I'm willing to pay on-demand prices

AWS gives you two minutes or will remove that instance from your environment within two minutes

and you have to terminate everything and wrap up all your work within

those two minutes and give the computer back or it'll be forcefully taken from you

With Google, it's even worse. You only have one minute to relinquish control of the computer

So one of the key kind of findings that we've had and the key operational urecas we have is we can fully automate the interruption process

So we, and I'm going to show you this guys as part of the demo, we basically intercept the lifecycle of these spot instances

And we replace that instance with either another spot instance, maybe a different instance type, maybe an on-demand instance, depending on market pressures

So if there are no spot instances available at the moment, not to worry

We just go back to inventory. We pull the next most appropriate on-demand instance

And when the market opens up again, we'll just replace it again. So it a very hard problem to solve if you automating it yourself This platform makes it super simple Like you just turn on a policy which I show you in a second and all of the spot instances all of that management will get automated for you

So why are we recommending three nodes? Last point on the state in this report

We're recommending three months because it's kind of the minimal node count you need for a reasonably highly available setup, right

So you don't want to have a single point of failure. You don't want to have one node

So we kind of recommend an odd number of nodes, you have three or five. And in this case, we're recommending three

because of the size of the workload. So enough to yammering, let's get started and actually do the savings

So what I'm going to do is I'm going to click on the start savings button

And sure enough, a CASP platform does need to have some greater level of access to your cluster

So what we've done is we've pre-created a script that you can run in Shell that will create a role

and let's let me describe it while we're running so that we're not wasting time here

So what will this do? It's going to create a role. It's going to make sure that my current access

he has all of the valid access to touch the CPS cluster

It's going to then create very specific permissions for our, and I'm going to kind of flash this and then I'll delete it very quickly afterwards

but it'll create a very specific user with very specific policies that will only

only be scoped to the VPC and the cluster that we're considering

So in other words, it's not a, it's not got access. We don't have access to your whole account

with this set of credentials. We just have access to manage this cluster

And very specifically, it's managing the node groups. And it's also able to create new EC2 instances

and add them to the cluster. Cool. So while that's creating, let's go to our EPS cluster

I just want to make sure that we look at the auto scaling group for a second

Cool. And then the auto scaling group that we have is this one. This auto scaling group has a desired capacity of six and the minimum capacity of six

So we're going to just change the minimum capacity to be zero so that we don't get

we don't let the auto scaling group kind of get in the way of this particular

script. Cool. So now we've got our access key and we've got our secret. I'm going to go ahead and

plug that into the cast console. So we've got our access key. We've got our secret. And then the only

thing that we have to do is give access. This script is going to go, this service is going to go ahead and

validate. But yeah, oh, we've got a bad request. So let me just make sure

that I've copied this correctly and I've copied this correctly. One thing I cannot control is the EWS control link

So let's see if this works and then we'll be able to kind of take you to the

yeah it's it's not very happy right now unfortunately. No worries. There was a few questions from the audience as well if we want to take them while we see. Oh, did it

Yeah. Well, do you want to take a couple questions or do you want to kind of continue the floor anyway? What do you think

Yeah, maybe we could have a few questions and then we can continue just because there was quite a lot already

Awesome. I think we covered this a bit already, but I think it's good to repeat which clouds are supported by Casta

Cool. So AWS, EKS is supported right now. Within a couple of weeks, you're going to be able to do the exact same process on Google, so on GKE. These are Kubernetes managed services on those clouds. And then we have a, in a couple of weeks, you're also going to be able to, if you've created your own clusters using COPS, which is Kubernetes operations tools, you'll be able to use past as well for COPS. And then within a couple of months, we're also going to be able to use CAST as well for COPS. And then within a couple of months, we're also

going to be supporting Azure's managed Kubernetes platform, which is called A-K-S or Azure Kubernetes service. Perfect

Then the other one is, will Azure and Cast AI bill be separate or together or I guess

any cloud bill and the Cast-A-A-A. Great question. So we don't buy the compute on your behalf, right

These are your accounts. So in the Azure context, it's called a resource group and your bill remains a, it remains

inside of Azure, and then CASS just charges you a very small management fee

So it's not nearly in the scope of the cloud bill. So you'll get a bill from CASS, and you'll get a bunch of billing reports from CASS

showing how your clusters are trending in terms of their spend, but the actual bill you pay

is for Azure or AWS or Google, any cloud that you work with today

Perfect. And the last one for this middle Q&A session, am I giving my data to a third party

So you are, so it depends. So are you giving user data to a third party

No. Your data stays in your data planes. If you've got a database with user data in it that needs to be protected from a PI perspective

or from a data residency perspective, nothing changes. Are you giving any data to cast

Yes, you are. You're giving some data to cast. What are you giving us? You're giving us access to your

your pod configurations, right? You're giving us access to your node configurations

but you're not giving us access to any of your secrets and Kubernetes. In fact, we make sure that we scrape all of those out

before we send any data. So there is a filtering process to make sure that no sensitive data makes it to cast

So we're really looking at your workload characteristics. We don't care about anything else

And if you have secret data that is actually exposed unknowingly like, let me say in a config map

or something like that, our part plan is to warn you about those things. So on top of the savings report, you saw, we're going

to actually have a conformance and security report that will tell you, hey, you've got some weak

links here, you've got a config map with a database password that shouldn't be there, take care of it

and remove it. So yes, you are revealing some workload-related data to cast, but you're not

revealing any user data to cast, if that makes sense. Perfect. I think then we can continue with

the demo. Awesome. So let's see. start optimizing. So we got our cluster connected now and we really want to see some of these

savings, right? So I'm going to walk through some of these steps a little faster now. There are four

basic policies that you have to set. The first one is just the limits policy. I don't know

Let's just update the up this to 40. So we don't have a cluster that grows too crazy based on

workload. We want to enable the node autoscaler. So when unscheduled, pods occur in your environment

we are going to then create infrastructure to manage those on schedule pods

We also want to enable preemptive instances or spot instances in AWS case

so that we can take advantage of the cheapest possible inventory. And then finally, no good to scale up a cluster

if you can't remove it. So this is kind of where some of the magic comes in

So what you want to do is enable the node deletion policy

Before we do that, we're going to install this little add-on called in the Victor, right

So what I'm going to do is I'm going to open our documentation

and I'm going to go into this little command. And I'm going to, I think the helm repo is already added

but I'm going to install a helm chart in this cluster that's going to install our evictor component

and we're going to watch it do its magic in just a second. So actually, maybe let's do this

Let's enable the node deletion policy here, real quick. And now let's go ahead and install the Victor

This goes back to I going to clear the screen So my access keys aren on the screen and I going to add the Helm repo And then I going to install the Evictor

So you see a Victor does not exist, installing now. Awesome. So it's up and running

And in fact, remember if we go back to our Kubernetes ops view. Oh, cool

Now we see the Evictor is installed on one of the nodes

And actually, if you look into the Evictor logs, you'll see it's doing leader election

is highly available and after leader election, it's going to start working on the following problem

You see how we have this nice distribution of pods all over the place

Well, this is just too much, too much distribution. So what the evictor does is it works with our SaaS data

collection platform and it says, which one of these nodes are the worst from a cost effectiveness perspective

We're just going to pick one victim now. And that victim is going to get drained

And the evictor is going to do its magic in terms of moving the pods to another

schedulable node. And one of the things that we do is a pre-flight check to say, look, if we bring down

this node or if we drain this particular set of pods out of the node, are we going to

bring down a service? If the answer to that is yes, the evictor will reject the victim and we'll move on to find

another victim. So you guys see the red here that's shown up. Basically what it's done is it's

targeted that node and it said, look, we'll distribute the work on other nodes. This node is gone

And it does it through by painting the node. So no other scheduled work can occur on that node

while we're doing this kind of bin packing process. So really this is a bin packing problem

How do we get the best and most efficient use set of a cluster by packing in all of the workloads into the fewest possible nodes without sacrificing uptime and without sacrificing availability

So if we wait another couple seconds, what we'll see probably, let me make sure that I did

Now I'm like, yeah, I did turn on the node deletion policy. If we wait another couple seconds, what we'll see is the node deletion policy will kick in

And there is a TTR because you don't want to take too drastic action on this cluster

Within about 10 minutes, you're going to see that that node will disappear and that it'll probably choose at least two more victims

And our whole goal is to get some, let me open the screen kind of in split mode here

And our whole goal is to get the savings report as close to as close to optimize as possible

So what we see here is the saving report has already said

cool, you've already enabled all four of your policies. You started the journey and we're looking for this green bar kind of to flip into the optimal zone

I don't know if we're going to get there because of the time we have, but I think we're certainly going to see some marked improvement as this node goes away

So what do we see here? Oh, now we have one, two, three, four, five nodes

These two little pods are damien sets, they're going to go away. And then within a few seconds, the savings report progresses

And what we call this the slow and low migration strategy, because our whole goal here is to do the migration and the node reduction and node creation very slowly

We don't want to do anything that will jar your production or your pre-production environments

So our whole goal here is to go slow and low. And then within a few hours, you have a really nicely optimized cluster

So let's take a look at the savings report here. and see what we got because I think we should have a little bit of savings coming up now

Cool. So you guys see that the cluster cost dropped from $414 to $345

and now you see that now there are five nodes, and eventually that will kind of trickle down to three

And by the time we are finished with the process and go through the lifecycle

of adding nodes and deleting, adding pods and deleting, bleeding floods your normal life cycle, we're going to get to this $138 kind of optimized environment

So Annie, I don't think we have time for another cycle, but hopefully this gives folks a great

idea of how the product works through the covers. And we'll be able to take folks through this

in more detail, come onto our Slack, and we'll, or Discord, and we'll walk you guys through it

Perfect. I think that's a great next step. Before the final kind of

comments. There was another question on, is there any white papers available? And I can at least say that there will be actually a white paper published in few days. So no worries. Keep your eye out for that. But obviously, if you only have anything to add, you can do so. Well, you are much more versed on that side than I am. So, yeah, I think we have a whole bunch of stuff in the pipeline case studies and white papers. It's going to be really exciting when it all comes out

So stay tuned for those. But that's starting to be it, but I think I really want to emphasize to everyone that you can go to test AI and

get your free trial and the always free yzer to see and yze your cluster cost and

after that optimize it. Is there any more? Yes, I think the link will be up on all of our feeds so you can click through that and get started on

on using and optimizing your clusters. Yeah, and just as we were talking, another note got signaled for as a victim

It's getting it's getting cleaned up right now and then we'll start to see that cluster shrink

to its full optimized capacity. Perfect. Okay. Absolutely amazing. Well, it goes all over my head now, but it looks like people really enjoyed your session

People did ask. So, Leon and Annie, it looks like this is an amazing and promising product, right

I already see your website. Apart from all this community and all, do you have like docs or something or a place

Any other call to action resources if someone wants to go and learn more about it

Would you like to go and talk about that and then we'll wrap it up

Yeah, of course. I think the best place to find we do have extensive documentation in the site as well where you can see

All the things. You can reach out to any of us, and we will help you get started, join the Slack group

And I think one of the best links is the EKS Optimizer link that you can use to straight jump into the ETS landing page

and find out more about that product there that we showcased today

All right, everyone. So I think that's the wrap. Thank you everyone for joining

I see people really love this session. Lauren says thank you. Asipa says, a great session

Thanks for joining in Asipo. who just says interesting session. Static world says

interesting session. So it looks like people are looking at a lot of your demo and the product

Beyond, thank you so much for time. Once again, I really appreciate it. Any final thing you want to plug in before we close the show

Yeah, I mean, guys, from my perspective, this is all about saving your time

Our mission here is to get you out of the plumbing and into the higher order thinking

there's no reason for us in 2021 to be doing these low-level operations manually

So really encourage you to try the automation. And I'm really excited about kind of the next steps that we're going to be taking very

I believe they will be industry changing even more so than what we've shown you today

So really looking forward to showing that. And Simon, maybe when we have some of those interesting demos that come up

we'll revisit and have a chat with the community again. Yeah, we have an Azure Summit coming up in this September

That's going to, that's with a 120 speakers. If time perverts, we would love to host you back there too

It's an amazing event that we are doing. So thank you once again, Leon and Annie. And thank you very much joining

That was it. This episode of Product Showcase. My name is Stephen Simon. This time, co-host with Annie and an amazing guest, Leon

We'll see you in the next episode of Product Showcase. Until then, take good care of yourself and get vaccinated