Crawler budget – Bots also have a budget

Many people mistakenly believe that Google is searching for sites that are relevant to the user's search query – on the Internet itself.

The truth is that Google manages its own unique index, where it maintains web pages that seem important enough to them, which will be relevant to any future search results, and periodically updates this index.

So when users search the search engine, Google returns them results not from the network itself, but from the index it has built.

Theoretically, it may well be that the pages Google presents for a user's query have changed significantly since indexing them and are no longer relevant to the user's query. This is exactly what Google is doing for crawlers, whose entire job is to find and index new changes or pages that Google doesn't know.

גוגל בוט


What is a crawl budget?

Officially, there is no such concept for Google called a crawl budget.

Google announces that it invests crawl resources to find new or renewable pages based on the site's ability to serve Google bots without compromising user experience.

This means that faster sites will most likely be crawled by Google crawlers in a given amount of time.

Also, in order to invest its resources wisely, the amount of resources Google invests in crawling to find pages on a particular site depends on the rate of changes that occur on the site, the popularity of the pages on the site, and many other parameters.


How can the crawl budget be improved to fit your site?

To know how to improve the crawl budget on your site, it is useful to understand the cases where the crawl budget is wasted, and try to avoid these scenarios as much as possible.

Here are a few examples of a crawl budget that doesn't take full advantage of its potential:

Parameter pages on large sites

This usually refers to product pages, for example, to reach them through various filters on the site's work interface. In fact, the site has a large collection of URLs, which actually describe exactly the same content. If the site is a large site that contains thousands of such pages, then Google crawlers should crawl all these URLs. Too bad for the more wastage.

Yaost settings Correct


Site speed

The greater the speed of the site, the more Google will feel comfortable crawling the site without allegedly harming the user experience. Improve your site speed, upgrade the quality of your site's storage (Upress is a good option) – and you can actually increase your site's crawl budget.


Incorrect navigation on the site

One of the methods that Google uses to find new pages on your site is through the internal links within the site. An optimized link hierarchy will cause Google to miss important pages, especially if it is internal pages, which do not receive links from central pages on the site.

Exists pages that have been hacked and contain malicious code

Google will probably not want to index, or update, its site pages that have been compromised and endanger the privacy of the site.


Having low-level content

Or having little content – Pages that have very little content, or that the content is low-level and will not be displayed to a user who queries the search engine – are in fact nothing but a waste of the crawl time and budget of the bot, these pages are also called "content bucket pages".


Multiple 404 pages on the site

Pages that have been removed and not redirected to new pages will only make it more difficult for the bots to crawl the site pages, wasting an important crawl budget, which could be directed to important pages that have been updated on the site. Fix broken links on the site, and try to avoid leaving irrelevant pages on the site and deleting such pages as much as possible without dealing with the links that came in. Learn more about page 404.


AMP Pages

Google's AMP project allows publishers, under certain conditions, to significantly improve the loading time of certain pages on the site when using mobile devices. To do this, publishers create special pages on the site that fulfill certain criteria, in the AMP-Accelerated mobile page format.

When creating the new format, publishers make sure to handle the site's script and design files and manage the various resources needed for page load time on the site, asynchronously. All of this is nice and good, but it's important to understand that creating AMP pages on the site also forces Google to crawl those pages as part of its crawl budget.


In conclusion

The crawl budget is set based on various parameters that, on the one hand, aim to maintain a high level of Google up-to-date on the content of the site and, on the other hand, not harm the user experience of the site, by activating the bots.

Large website managers who want to significantly improve their crawl budget should avoid duplicate content on the site and unnecessary site filtering, improve site speed, remove broken links and handle 404 pages, consider the site hierarchy, update pages containing limited content or pages that has of no value to the users, and of course to secure the site from various offenders.

שלחו לחבר קישור למאמר, הוא חייב לקרוא אותו!
שלחו לחבר קישור למאמר
הוא חייב לקרוא אותו!
עשהאל דרייר
עשהאל דרייר

מנכ"ל חברת קידום אתרים ואפיליאצייה המונה כ-125 עובדים ומאות לקוחות בפריסה עולמית, מייסד דיגיפארם - חנות אונליין להזמנת תכנים וקישורים עבור אנשי SEO, מייסד רובולינק - אפליקצייה לבדיקת מיקומים חינם לאנדרואיד ולאייפון, מוביל את קבוצת אנשי ה-SEO הגדולה בישראל בפייסבוק ובווטסאפ.

מאמרים נוספים מומלצים לקריאה עבורך:

המדריך המלא להקמת אתרי לידים

בואו נשים את הדברים על השולחן: אין מי מאיתנו שלא מחפש דרכים קלות, יחסית, וחוקיות, כמובן, לייצר כסף. גם אם לא מדובר בהכנסה שניתן להסתמך

כדאי להשקיע ב-SEO בזמן מלחמה?

מדינת ישראל נמצאת במלחמה (נכון להיום – 6.11.23). אני כותב את זה בגוף ראשון כמי שלוקח בה חלק פעיל. אין ספק שמלחמה היא עניין מורכב

בדיקת מהירות אחסון אתרים קלאודוויז
עשהאל דרייר
היי כאן עשהאל ואני רוצה להמליץ לך באופן אישי לפתוח שרת בקלאודוויז ולהנות מאתרים מהירים, תמיכה 24/7 וממשק נוח. אני מאחסן למעלה מ-400 אתרים בעצמי וממליץ בחום ובלב שלם, לאחר מספר שנות ניסיון
תוכנת מעקב הדירוגים הטובה בעולם!
בואו לעקוב אחרי מיקומים של כל אתר, מכל מקום בעולם, כולל הפרדה בין מיקומי דסקטופ ומובייל, תנועה, תיעוד עבודה ואפילו תוכלו לשלוח ללקוחות שלכם לינק לדשבורד שמתעדכן אונליין!
הטבה לגולשי האתר:
14 ימי ניסיון ו-25% הנחה לחודשיים