Good value but a 12GB card isn't going to let you do too much given the low quality of small models. Curious what "home AI" use cases small models are being used for?
It would be nice to see a best value home AI setups under different budgets or RAM tiers, e.g. best value configuration for 128 GPU VRAM, etc.
My 48GB GPU VRAM "Home AI Server" cost ~$3100 from all parts on eBay running 3x A4000's in a Supermicro 128GB RAM, 32/64 core Xeon 1U rack server. Nothing amazing but wanted the most GPU VRAM before paying the premium Nvidia tax on their larger GPUs.
This works well for Ollama/llama-server which can make use of all GPU VRAM unfortunately ComfyUI can't make use of all GPU VRAM to run larger models, so on the lookout for a lot more RAM in my next GPU Server.
Really hoping Intel can deliver with its upcoming Arc Pro B60 Dual GPU for a great value 48GB option which can be run 4x in an affordable 192GB VRAM workstation [1]. If it runs Ollama and ComfyUI efficiently I'm sold.
Agreed, 12 GB does not seem useful. For coding LLM, it seems 128 GB is needed to be even close to the frontier models.
For generative image processing (not video), it looks like one can get started with 16GB.
I use a Proxmox server with RTX 3060 to generate paintings (I have a couple of old jailbroken Amazon Kindle's attached to walls for that purpose), and to run ollama, which is connected to Home Assistant & their voice preview device, allowing me to talk with LLM without transmitting anything to cloud services.
Admittedly with that amount of VRAM the models I can run are fairly useless for stuff like controlling lights via Home Assistant, occasionally does what I tell it to do but usually not. It is pretty okay for telling me information, like temperature or value of some sensors I have connected to HA. For generating AI paintings it's enough. My server also hosts tons of virtual machines, docker containers and is used for remote gameplay, so the AI thing is just an extra.
That's good to know. The model I was using supported function calling, but seemed to get the calls often wrong. Perhaps I should try a more fine-tuned model for the purpose.
It's really not going to let you train much which IMO is the only reason I'd personally bother with a big GPU. Gradients get huge and everything does them with single/half precision floating point.
just wondering, why that one? it looks small, which is probably doesn't require a gpu.
The dataset seems to be images of high production value (e.g. limited races, staged poses, etc). If I have time, I will compare it with Bumble's model, but I think the images I'm trying to identify are closer to Bumble's training set.
It would be nice to see a best value home AI setups under different budgets or RAM tiers, e.g. best value configuration for 128 GPU VRAM, etc.
My 48GB GPU VRAM "Home AI Server" cost ~$3100 from all parts on eBay running 3x A4000's in a Supermicro 128GB RAM, 32/64 core Xeon 1U rack server. Nothing amazing but wanted the most GPU VRAM before paying the premium Nvidia tax on their larger GPUs.
This works well for Ollama/llama-server which can make use of all GPU VRAM unfortunately ComfyUI can't make use of all GPU VRAM to run larger models, so on the lookout for a lot more RAM in my next GPU Server.
Really hoping Intel can deliver with its upcoming Arc Pro B60 Dual GPU for a great value 48GB option which can be run 4x in an affordable 192GB VRAM workstation [1]. If it runs Ollama and ComfyUI efficiently I'm sold.
[1] https://www.servethehome.com/maxsun-intel-arc-pro-b60-dual-g...