I'll second this - my experience is pretty much the same (though I did some basic Fourier transforms at school), studying Physics at college. I did 3 Maths A levels along with Physics and Chemistry (and "General Studies" :) however, so I too may be atypical.