> Are there models that will run on 3 series that will not run on 4 series cards due to feature incompatibility?
No, it is all the same matrix multiplication.
> if you have a 2x 3090 rig, is that about as fast as a single 4090?
Yes, but double the VRAM
> But can you train e.g. a model that requires 48gb of space with 2 3090s on the same rig, or do you need a large card with 48gb, like an a4000?
Yes, 2x 3090 would be able to handle anything that a 48Gb card can. For training you will need space for weights and gradients. If using Adam optimizer, that would be 2x-4x model size. Plus weights, plus activations, plus inputs times batch size.
So a 24Gb card can train approximately a 3B model without compromises.
No, it is all the same matrix multiplication.
> if you have a 2x 3090 rig, is that about as fast as a single 4090?
Yes, but double the VRAM
> But can you train e.g. a model that requires 48gb of space with 2 3090s on the same rig, or do you need a large card with 48gb, like an a4000?
Yes, 2x 3090 would be able to handle anything that a 48Gb card can. For training you will need space for weights and gradients. If using Adam optimizer, that would be 2x-4x model size. Plus weights, plus activations, plus inputs times batch size. So a 24Gb card can train approximately a 3B model without compromises.
I am not a lawyer.