Is that really Googlebot? Check by reverse DNS

Why the user agent is not enough

Scrapers and attack tools often call themselves Googlebot, because sites wave Googlebot through. If you block by user agent you may block the real one and lose crawling. If you trust by user agent you let the fakes through. The address is what tells them apart.

Check one address by hand

1. Look up the name of the address (reverse DNS). getent is on every Linux system; host or dig give the same answer if you have them.
2. A real Googlebot ends in googlebot.com or google.com (Google's other fetchers can also end in googleusercontent.com). A real Bingbot ends in search.msn.com.
3. Look that name up again (forward DNS). It must give back the same address. Without this step anyone who controls the reverse zone of their own addresses could name them crawl-something.googlebot.com.

Check the busiest ones in your logs

List the addresses that sent a Googlebot user agent, busiest first, then check each one. This works on the usual combined log format of nginx and Apache:

Or use the published address lists

Google publishes Googlebot's address ranges as JSON (developers.google.com/static/search/apis/ipranges/googlebot.json) and Bing does the same for Bingbot (www.bing.com/toolbox/bingbot.json). Checking against them needs no DNS lookups, which is better for a firewall or rate-limit rule that runs on every request. Fetch the lists again regularly, because the ranges change.

What to do with the answer

Fake ones: treat them like any other bot. Rate-limit or block them if they cost you something. Real ones: never block them. If real Googlebot is heavy, that is a crawl budget question (faceted navigation, endless filter URLs, slow pages), not an attack. In rate-limit rules, exempt verified crawlers rather than anything that says Googlebot.