> Works out to be about ~$2/hr all said

What GPU you end up with for that price? Vast.ai (https://cloud.vast.ai/?priceInstanceHourlyMax=2) has a bunch of setups available to reach 192GB VRAM under $2 :) Quick skim showed 4x48, 2x96 and 8x24, all for under 2 buckaroos or around there.

It's actually a bit less than $2 but I rounded up just to make the math easier.

And yeah, did a lot of work with Vast AI at a past job and it's pretty wild the variety of prices/hardware that they have.

> It's actually a bit less than $2 but I rounded up just to make the math easier.

That's cool, what actual GPU though? I'm still curious :P

gpu-6000adax1-48gb RTX 6000 Ada GPU Droplet - 1X

Awesome, I used Claude to write a small python script to do the same with Linode's API. The only difference is I setup a persistent drive, and with Linode you can boot off of it. So my biggest start up lag is ~ 2 minutes to deploy + boot, then maybe 2 more to warm the model.

I actually dislike LLMs. But I'm a realist, and on-demand compute like this is massive cost saving measure.

(persistent drives are relatively cheap, compared to a box with several GPUs.. or even one. I find it worth the expense)

> the only difference is I setup a persistent drive,

Going to try this out vs the snapshot!

I also really like this experiment b/c it's a mix of LLMs and old school IaaC/DevOps.