Ah yes but I mean the first time I don't connect to the internet or even VPN. I just connect to my git server on the local LAN.
But good point, as GitHub Cloud is not local. In that case I would verify the key indeed.
I don't think I'd use github for stuff that needs to stay private though. At work we have a self-hosted internal GitHub enterprise server for all closed source projects.
And store the keys on a yubikey (OpenPGP or more modern Fido2) so they can't be exfiltrated. A huge benefit over a password.
And if you really want, SSH certificates are a thing too.