Forgive my insolence, I'm a server engineer, not an ai specialist, so the following might have already been answered a million times already. I know how to set up the infrastructure, but not the differences in models or agents that run against them. With that being said, I need assistance with the following.
My buddy wants to localize his "vibecoding" and "chat" ai models after spending so much money monthly on claude credits etc, and we've settled on putting a gpu in my server that has monstrous amounts of ram(512gb ddr4 ecc). He has set his sights on Gemma 4, and currently is doing this on a dell precision 7790 with 64gb of ram and an rtx 5000 ada gpu(16gb). This is his work laptop, not personal, hence wanting to switch away from it(among other reasons). His wants are to be able to use gemma4 with 20b(as thats what he thinks he is doing right now). I know there are way more complexities regarding ai, setup, and tuning, but we need something to start with for now, before we spend 5k on a gpu(a100 80gb).
The budget is around 700$ for now, and I would like some feedback on best gpu to get our foot in the door, and give a way better experience than his work laptop. My server specs are below:
- supermicro x10dri-f
- 2x e5-2680 v4's
- 512gb ddr4 ecc
- rosewill ls4500(case)
- truenas(os on host, will be running in a windows 11 vm. he will connect over rdp when he wants to use solidworks/lightshot etc. he is a mechanical graphic designer)
I've looked at the widely popular mi50's, but they are from 2019 and lack some of the instruction sets i know modern models can make use of. The 5070 ti is also enticing, although is lower in vram(16gb vs 32) but if i can get away with vgpu I'd rather do that. I've thought about the intel arc cards, but not sure where they stand currently if all they are doing is using vulkan. I'm fine with used hardware, and am preferable to tesla/quadro due to their vgpu nature. Primary use is ai, with secondary being solidworks/lightshot rendering. Thanks for any responses!
[link] [comments]