Slammed with Google crawl requests

One of our hosted web sites was disabled due to their bandwidth quota. The site had exceeded the 50gb per month limit. This was notable because the site usually uses around 500mb of bandwidth each month.

The site was disabled on February 17th. We tried to access the site and it came up blank. We checked out the index.php file and it had some malware in it. We cleaned up the malware and scanned the entire site and cleaned malware from one other page.

We could see from cPanel that the excessive bandwidth seemed to start on February 12th, when 4.32 GB was used.

Date	Bandwidth
9th	18.26mb
10th	26.55mb
11th	545.78mb
12th	4.32.gb
13th	13.91gb

It ramped up fairly quickly from there.

We checked the Visitors log on cPanel and saw hundreds of accesses like this. Performing a reverse IP lookup on 69.249.77.94 showed that it was crawl-66-249-77-94.googlebot.com, so it was Google crawling the site. Each hit returned 40k or so of a themed 404 page. So it appeared that Google crawl was causing our bandwidth issues.

We think that the site was compromised, and malware was installed so those odd URLs would return a page that the malware author could use for monetization.

Once our site was compromised to return those pages, they created links from another site they controlled to those odd URLs, and told Google to crawl them and get them into the Google search index.

We were unknowingly hosting pages the malware author was using to boost their Google crawl results.

So, we added the site to Google Search console and uploaded a site map that showed the real URL’s hosted on the site. We also set the Google crawl speed to as slow as it could be. Hopefully Google will use the sitemap to decide that these pages don’t exist and stop hitting them.

Also, we replaced the 404 page with a single HTML file with the word 404 in it (29 bytes total) to minimize the impact of the crawl trying to download non-existent pages.