To your first point I think that's actually where we are. Poking in the dark with no real understanding of the tools at our disposal. And the algorithms DO have power. We don't know WHY, except that we are emulating solutions given to us by biological evolution.
To your second point, that's basically what evolution did. Poked in the dark for several billions of years and ended up with solution machines (us).
Yes, I'm aware, that's in large part why it worries me. And I think you undersell the importance of online learning. NNs are far from "plug in and forget."
To your second point, that's basically what evolution did. Poked in the dark for several billions of years and ended up with solution machines (us).