Crawling: Can Search Engines Find Your Pages?<\/a><\/h2>Before your website can appear in the search engine results, it must appear in the search engine’s index. The first task of optimising your site should be to establish if any issues prevent your site from being fully indexed. Many sites are constructed in a way that prevents search engines from indexing some or all the pages on the site.<\/p>
Analysing Current Performance<\/a><\/h3>It is essential to establish the proportion of your site’s pages that are indexed. Do this by comparing the number of pages on your website (as measured by your content management system) with the number appearing in the search engines index. To find the number of pages from your website which Google has indexed, consult the Google Search Console (see below). To get an estimate to the proportion of indexed pages, divide the number of indexed pages by the total number of pages.<\/p>
It takes time for a search engine to refresh its index, and so if your site is large or often changes, 100% inclusion is unlikely. However, if the number of pages indexed is way below the expect this should be investigated (see below). If your site is not appearing at all, then there could be a few reasons for this:<\/p>
- New site that has not yet been crawled.<\/li>
- No links from external websites.<\/li>
- Site navigation makes it difficult for a robot to crawl it.<\/li>
- ‘Crawler directives’ are blocking search engines. E.g., robots.txt file<\/li>
- Site penalised by Google for ‘Black Hat’ tactics<\/li><\/ul>
Google Search Console and Google Sitemaps<\/a><\/h2>Google has created a suite of tools called Google Search Console (GSC) with the mission of ‘helping you measure your site’s search traffic and performance, fix issues and make your site shine in Google Search results. These tools supplied include:<\/p>
- Coverage reports<\/strong>. See the number of pages that Google is indexing.<\/li>
- Search performance<\/strong>. Natural search clicks and performance as measure by Google.<\/li>
- Website analysis<\/strong>. Google’s analysis of the quality of your website, e.g., usability and page speed.<\/li>
- Google sitemaps<\/strong>. When submitted to Google, an XML file will provide them with details of the pages on your site.<\/li><\/ul>
Submitting a Google sitemap is crucial to improve your indexing and be a priority soon after a site launch. These will be generated automatically from all decent CMS and eCommerce systems and can be quickly submitted to GSC.<\/p>
Indexing problems<\/a><\/h2>If your site is not being indexed or has a low percentage of indexed pages, one of the following problems may be present:<\/p>
Robots.txt file<\/a><\/h3>There are some good reasons why you would not want a search engine to index sections of your site, for example, admin and checkout pages and duplicate content.<\/p>
Robots.txt files can be found in the root directory of each website (e.g., vendlab.com\/robots.txt) and tells Google which parts of your site it should and should not crawl. A problem with the robots.txt may cause Google to stop crawling your site.<\/p>
Inaccessible Content<\/a><\/h3>Your content could be inaccessible to search engine crawlers for several reasons:<\/p>
- Search required<\/strong>. If the content is only accessible via a search, the content will not be accessible to the crawler.<\/li>
- Non-text content<\/strong>. Do not use non-text media formats (images, video, GIFs, etc.) to display text that you wish to be indexed.<\/li>
- Non-HTML menus<\/strong>. Search engines have difficulty reading menus not written using HTML, e.g., JavaScript.<\/li>
- Navigation dead ends<\/strong>. Crawlers only find pages linked to other pages.<\/li><\/ul>
It is essential to establish the proportion of your site’s pages that are indexed. Do this by comparing the number of pages on your website (as measured by your content management system) with the number appearing in the search engines index. To find the number of pages from your website which Google has indexed, consult the Google Search Console (see below). To get an estimate to the proportion of indexed pages, divide the number of indexed pages by the total number of pages.<\/p>
It takes time for a search engine to refresh its index, and so if your site is large or often changes, 100% inclusion is unlikely. However, if the number of pages indexed is way below the expect this should be investigated (see below). If your site is not appearing at all, then there could be a few reasons for this:<\/p>
- New site that has not yet been crawled.<\/li>
- No links from external websites.<\/li>
- Site navigation makes it difficult for a robot to crawl it.<\/li>
- ‘Crawler directives’ are blocking search engines. E.g., robots.txt file<\/li>
- Site penalised by Google for ‘Black Hat’ tactics<\/li><\/ul>
Google Search Console and Google Sitemaps<\/a><\/h2>
Google has created a suite of tools called Google Search Console (GSC) with the mission of ‘helping you measure your site’s search traffic and performance, fix issues and make your site shine in Google Search results. These tools supplied include:<\/p>
- Coverage reports<\/strong>. See the number of pages that Google is indexing.<\/li>
- Search performance<\/strong>. Natural search clicks and performance as measure by Google.<\/li>
- Website analysis<\/strong>. Google’s analysis of the quality of your website, e.g., usability and page speed.<\/li>
- Google sitemaps<\/strong>. When submitted to Google, an XML file will provide them with details of the pages on your site.<\/li><\/ul>
Submitting a Google sitemap is crucial to improve your indexing and be a priority soon after a site launch. These will be generated automatically from all decent CMS and eCommerce systems and can be quickly submitted to GSC.<\/p>
Indexing problems<\/a><\/h2>
If your site is not being indexed or has a low percentage of indexed pages, one of the following problems may be present:<\/p>
Robots.txt file<\/a><\/h3>
There are some good reasons why you would not want a search engine to index sections of your site, for example, admin and checkout pages and duplicate content.<\/p>
Robots.txt files can be found in the root directory of each website (e.g., vendlab.com\/robots.txt) and tells Google which parts of your site it should and should not crawl. A problem with the robots.txt may cause Google to stop crawling your site.<\/p>
Inaccessible Content<\/a><\/h3>
Your content could be inaccessible to search engine crawlers for several reasons:<\/p>
- Search required<\/strong>. If the content is only accessible via a search, the content will not be accessible to the crawler.<\/li>
- Non-text content<\/strong>. Do not use non-text media formats (images, video, GIFs, etc.) to display text that you wish to be indexed.<\/li>
- Non-HTML menus<\/strong>. Search engines have difficulty reading menus not written using HTML, e.g., JavaScript.<\/li>
- Navigation dead ends<\/strong>. Crawlers only find pages linked to other pages.<\/li><\/ul>
- Non-text content<\/strong>. Do not use non-text media formats (images, video, GIFs, etc.) to display text that you wish to be indexed.<\/li>
- Search performance<\/strong>. Natural search clicks and performance as measure by Google.<\/li>
- Coverage reports<\/strong>. See the number of pages that Google is indexing.<\/li>
![\"\"](\"https:\/\/i0.wp.com\/vendlab.com\/wp-content\/uploads\/2021\/05\/Crawling.jpg?resize=582%2C331&ssl=1\")
<\/figure><\/div>