What's also hidden behind the numbers is that a lot (most?) of H1-Bs come from India and China, both of which have insanely long green card backglogs. Because changing jobs while on H1-B is a perilous affair (and employers know it), those folks end up basically in indentured servitude to their original H1-B sponsor employer, especially if they have families and/or property in the US.
BTW 500k people on the H1B1 visa is a hell lot of "skilled people": https://www.nytimes.com/2020/05/12/us/foreign-workers-visas-...