You mentioned VLC playback is choppy on your desktop or phone. Have you tried local playback (VLC on your Raspberry Pi that's doing the capturing), is it smooth there? Also, what does CPU and memory usage look like when you a) play locally, b) stream RTSP over the network to your desktop/phone. These should steps should help narrow this down.