Considering they're really eeking every last bit of juice out of the box, I'm doubtful that distributing it would be cheaper.

Also they're not maxxing the CPU out, they just need memory bandwidth and the Mellanox nics. Storage would be more expensive on more boxes since they can't distribute the storage, they have to use local storage to reach the performance we're after.

Not necessarily. To a first approximation, if they can get 800Gbps out of the disks in a single box, they could split those disks over say 8 100 Gbps boxes and get the same performance out of the disks for the same price. Once you split it into 8 boxes, maybe instead of each box getting 2x 16 TB pci-e 4.0x4 drives, you get them 4x 8 TB pci-e 3.0 x 4 drives. Half size and older generation drives are likely to be less than half the cost. Netflix does have ways to segment cache among their appliances, so they wouldn't need to have the same storage capacity on each box as they do on the combined box.

It's certainly a procurement analysis to see if those savings will add up to overall savings, and there's a good chance it won't; especially if you need to add a 800 Gbps network switch. You do often get a pretty good cost savings by having two single socket servers vs one dual socket server though; again though, probably not if you have to add a switch.