Return to previous page

Googlebot over-crawls our website and create lot of PHP execution request

This topic has 10 replies, 2 voices, and was last updated 1 months, 3 weeks ago ago by Luca Rossi

slw250@hotmail.com

Participant

July 28, 2024 at 20:48

Our web host giving us notifications that google bot over crawling our website, and using lot of resources, as we are on shared plane so we have limited quota for php executions, would you please help us to over come this problem
The response from hosting company is in the private content area

best regards,

Please, contact administrator
for this information.

#407433 Copied Theme version: 9.3.15

9 Answers

Luca Rossi

Support staff

July 29, 2024 at 08:02

Dear @slw250@hotmail.com,

We hope this message finds you well. We are currently in the process of updating our robots.txt file and would appreciate your assistance. Could you kindly provide us with the list of URLs that GoogleBot crawls on your website? This information will help us ensure optimal website performance and search engine interaction.

Thank you for your cooperation.

Best regards,
8Theme’s Team

#407483 Copied

slw250@hotmail.com

Participant

July 29, 2024 at 09:52

Thank you Rossi for you quick response
Here is the screen shot for the bots
https://prnt.sc/TIaqrqL7uw6v

We already have robots.txt file on hosting server
as you can see the following screen shot
https://prnt.sc/YxWxVSF2wj4U
But it still over crawling our website
We temporarily block the googlebot in cloudflare by creating a custom rule under the WAF protection
More over we activate I am on Attach Mode in Cloudflare

Please contact administrator
for this information.

#407517 Copied
Luca Rossi

Support staff

July 29, 2024 at 09:57
Hi @slw250@hotmail.com,

Could you please help us to add the following rules to your robots.txt file then check again?
```
Disallow: /my-account/ 

Disallow: /*?et-wishlist-page 

Disallow: /*add_to_wishlist=* 
```
Let us know how it goes!
#407519 Copied
slw250@hotmail.com

Participant

July 29, 2024 at 10:04

now our robot.txt file look like this

User-agent: *
Disallow: /

User-agent: Googlebot
Disallow:

Disallow: /my-account/

Disallow: /*?et-wishlist-page

Disallow: /*add_to_wishlist=*

https://wholesalebuddy.net/sitemap.xml

Do you want me to remove the WAF rules from the cloudflare and disable I am on attack mode as well?

#407521 Copied

Luca Rossi

Support staff

July 29, 2024 at 10:07

Hi @slw250@hotmail.com,

Yes, let do that. We need to check how is it going after.

Thank you!

#407524 Copied

slw250@hotmail.com

Participant

July 29, 2024 at 10:10

done
How do we check the stat to ensure everything going smoothly

#407525 Copied

Luca Rossi

Support staff

July 29, 2024 at 10:55

Dear @slw250@hotmail.com,

We hope this message finds you well. We would like to kindly suggest that you consult with your hosting provider’s support team regarding the GoogleBot’s requests. It is important to ensure that these requests do not become excessively frequent.

We trust that this issue will be efficiently resolved and that there will be no further concerns regarding large requests from GoogleBot.

Thank you for your attention to this matter.

Best regards,
The 8Theme Team

#407548 Copied

slw250@hotmail.com

Participant

July 29, 2024 at 11:05

Hello,
They confirm that the executions decreased but they need 24hours to observe to conclude the result. I will let you know after 24 hours

#407550 Copied

Luca Rossi

Support staff

July 29, 2024 at 14:35

Thanks @slw250@hotmail.com.

If you need anything else, please let us know.

#407605 Copied
Tagged: googlebot over-crawl, improve site performance, php execution request, reduce website crawling, woocommerce templates, wordpress optimization
Viewing 10 results - 1 through 10 (of 10 total)