Learn how to increase your website crawl budget and increase your online visibility

Posted on by Lani Haque

Website crawl budget is the number of pages Googlebot crawls and indexes on a website within a given time frame.

Crawl budget = crawl rate + crawl demand,

where,

Crawl rate = # of requests per second Googlebot makes to your site when it is crawling it

Crawl demand = how much Google wants to crawl your site based on popularity, how long since it has been crawled

Why is crawl budget important?

If Google doesn’t index a page, it won’t rank for anything.  Most sites don’t have to worry about crawl budget.  So, when do you have to worry about crawl budget?

  1. When you run a big site e.g. ecommerce site, enterprise site with 10K+ pages; Google may have trouble finding all the pages.
  2. When you just added a whole bunch of pages.  E.g. you added a section to your site with hundreds of pages, this is when you want to ensure you have the crawl budget to index them all.
  3. You have a lot of redirects.  E.g many redirects and/or redirects simultaneously can use up crawl budget.

How to maximize crawl budget

Improve site speed.

  • Slow loading pages eat up crawl budget.
  • A faster site means better user experience and increasing crawl rate.

Use internal links.

  • Googlebot prioritizes sites with more internal links.
  • Eliminate broken links.
  • Internal links send the Googlebot crawling around your site increasing the probability more of your site’s webpages will be crawled and indexed.

Flat website architecture

  • Flat architecture sets things, pages, up so that all your site’s pages have some link authority.
  • URLs that are more popular on the internet are crawled more often to keep them fresher.  So, to Google, popular = link authority.  A flat website architecture helps with this.

Avoid orphaned pages.

  • Pages with no internal or external links
  • Google has a hard time finding orphaned pages so try to avoid them.  Have some or at least one internal or external link to every page.

Limit duplicate content.

  • There are many reasons why limited duplicate content is wise.Duplicate content can hurt crawl budget because Google doesn’t want to waste resources crawling the same content.
  • Unique, quality content

Factors affecting crawl budget.

  1. Low value URLs affect crawl budget.
  2. Faceted navigation and
  3. Session identifiers
  4. Onsite depreciated content
  5. Soft error pages
  6. Hacked pages
  7. Infinite spaces and proxies
  8. Low quality and spam content

Whether you choose to optimize your crawl budget or want to optimize your website for online visibility, you will opimize the other. This will benefit your website in the long run.

Reference:

  1. https://backlinko.com/hub/seo/crawl-budget  (There is a video in thei article for crawl budget optimization.)
Lani Haque
Latest posts by Lani Haque (see all)
Avatar for Lani Haque
Lani Haque

I enjoy learning and sharing that knowledge. Sharing has been in many forms over the years, as a teaching assistant, university lecturer, Pilates instructor, math tutor and just sharing with friends and family. Throughout, summarizing what I have learnt in words has always been there and continues to through blog posts, articles, video and the ever growing forms of content out there!

You May Also Like

More From Author