Sure, that's one option, and I don't have any insight into what nyt actually does with regards to it's handling of googlebot traffic.

But if I were implementing filtering, I might prefer a solution that doesn't require keeping a whitelist up to date.

The whitelist can be updated asynchronously

Maybe we could use GCP infra and the trick will work better ?