Site Audit Crawled Pages Report

The Crawled Pages section of your Site Audit lists all of the URLs that were crawled by our bot.  This gives you an easy way to look up every page on your site that was crawled, and analyze the status of your website on a page-by-page basis.

Crawled Pages Table

Here are the elements of the report (please see below for details):

(1) The page’s Internal LinkRank

(2) Number of unique pageviews

(3) Filters menu

(4) View switches

(5) Page’s crawl depth

(6) Total number of issues

(7) Metric selection

(8) Audit rerun for a single page

The table provides in-depth data about every crawled page, including:

• HTTP Status Codes
• HTML Load Time
• Markups
• Canonical Tags
• Presence in Sitemap
• Incoming and Outgoing Links
• Links to AMP (Accelerated Mobile Page) Versions
• Hreflang Usages
• JS and CSS Number and Sizes

To work through this report, you can filter the pages by URL or, for example, by status (pages with issues, broken pages, redirects, blocked pages, and healthy pages). You can also combine multiple filters at once to get the exact results you need.

For example, add two filters to the report to locate pages that have issues and have a Internal LinkRank over 70 (see below), then you can sort the pages by unique pageviews, and start fixing the most important first.

To re-crawl a specific page, click on the circular icon in the far-right hand column, under “Re-audit URL.” This will send our crawler to just that one page and check for issues. This is an efficient way to follow the progress of site maintenance without using up an unnecessary amount of your crawl budget.

To see your account’s crawl budget, go to your Subscription Info Summary and look for “Pages to crawl.”

Why are these page attributes important for SEO? 

Crawl depth of a page refers to the number of clicks needed to reach a specific page from the domain’s homepage using the shortest path. The homepage will have a depth of 0 and any page linked from the homepage has a depth of 1.

For the most important content on your website, it’s best practice to have a crawl depth of 3 and less.

Similarly, it’s beneficial for your most important pages to be present in your sitemap as this makes it easier for crawlers to locate your content.

The Internal LinkRank (ILR) is based on the number of incoming internal links and the page crawl depth. Pages with higher Internal LinkRanks are more accessible, as they have a lot of incoming internal links and low crawl depth.

An Accelerated Mobile Page (AMP) is a webpage with simplified HTML that experiences faster loading speed so that users can gain quicker access to the page content from a mobile device.

If you have different versions of a page, you can use a canonical tag to point search engines to the preferred one; this way you will avoid negative impact on rankings for identical or duplicate content.

For example, if you have both a non-AMP and an AMP version of the same page, you will need to inform crawlers about it with link tags connecting these pages. If you only have an AMP version, it will still require a canonical tag which points to itself.

The Hreflang tag specifies lingual and geographical targeting of a page. If you have localized versions of your website for different countries, you should carefully check the hreflang tags, to ensure that your audiences get the right content.  

The HTTP Status Codes represent a server’s response to a user’s request. Pay close attention to the 4xx and 5xx status codes as these show that a page is unavailable due to some error.

HTML Load Time shows how long it takes to load HTML code of the page. Website loading speed is a big part of the user experience, and an important ranking factor. Pages’ slow loading can be caused by too big or too many JavaScript (JS) and CSS files.

Using proper markups in your HTML can help search engines and social networks identify the entities on your website and index your content more accurately.

By connecting your Google Analytics account you can get the information on the number of unique pageviews in the report. Identify the most viewed pages and fix them first.

You can find more detailed information about each of these attributes in the Statistics report of your Site Audit.

Individual URL report

To see that page’s individual URL report, click on any URL in the table. The next page will list the errors, warnings, and notices as well as the number of incoming internal links discovered for each crawled page. Hover over the grayed-out metrics and notes for more information.

You can hover over the info button for an explanation of the issue, click on each issue for more details, or click on the link in the right column to see how many other pages on the site had the same issue.

Or you can select the blue square and arrow icon beside the URL to open up the webpage in a new tab.

Site Structure View

Switch to the Site Structure to get an overview of your website’s subdomains and subfolders, and see which parts require more work.

Note that Site Audit composes your site’s structure from the pages it has crawled, so if you have limited the number of pages to check, or selected a specific  part of your website for audit, the result might differ from your actual site structure. You can always change the settings of your campaign to get the real picture.