No - any toxic materials in a solar panel are bound up in the silicon and unable to leech out. Rain is generally good, as it cleans dust off and increases efficiency. Pollution is not a concern at all with solar.
There's a concerted disinformation campaign out there to try to insinuate that solar panels break and then poison the ground. Here's a podcast covering some of it and the funding mechanisms that help push it:
People will believe anything with the right confusing language. Throw in "rare earth minerals" and everybody gets as scared as if you call water "dihydrogen monoxide."
