Why only a few pages of my website are being crawled?

If you’ve noticed that only 4-6 pages of your website are being crawled (your home page, sitemaps URLs and robots.txt), most likely this is because our bot couldn’t find internal links on your Homepage. There can be two possible reasons for this:

1. Your Home page is more than 2 MB. Site Audit is currently only equipped to parse pages not larger than 2MB.
In this case you will see the large HTML page size error:

2. Your Homepage is built on JavaScript. Our bot cannot parse JavaScript content, so if your homepage has links to the rest of your site hidden in JavaScript elements, we will not be able to read them and crawl those pages. Although we cannot crawl JavaScript content, we can crawl the HTML of a page with JS elements, and we can review the parameters of your JS and CSS files with our Performance checks.

In both cases there is a way to ensure that our bot will crawl your pages. To do this, you need to change the crawl source from “website” to “sitemap” or “URLs from file” in your campaign settings:

“Website” is a default option. It means we will crawl using a breadth-first search algorithm and navigating through the links we see on your page’s code - starting from the homepage.

If you choose one of the other options, we will crawl links that are found in the sitemap or in the file you upload.