Every website consists of chunks of text segments and not all those segments are significant for the readers. As a website owner, you might find out that some segments of your website content are irrelevant, duplicate, or aren’t SEO-friendly. Often it is found that a particular portion of a web page is not valuable from SEO perspectives. Rather, that portion might lead the website downward in the SERP ranking. In such a case, does the webmaster need to exclude the entire page from Google’s indexing list?
That is not an ideal thing to do as the important and SEO-friendly segments of that page might be overlooked by search engine bots in that way.
So, what’s the solution in such a case? Well, you can block the particular segment of a web page that you don’t want Google to index. This way, you’ll prevent Google from indexing unnecessary portions of your web pages and you can boost your site’s SEO ranking at the same time.
What are the ways to exclude parts of pages from the indexing lists of Google and other search engines?
Robots.txt files are the ultimate solutions to this question
Generally, robots.txt files come into play when an entire web page needs to get blocked from Google’s or other search engines’ indexing lists. The main purpose of using robots.txt files is to protect a website from overloading requests. However, these files can be used for blocking specific portions of web pages as well.
Are you familiar with robots.txt files? Are you confused about how to create robots.txt files and imply those in your web pages? The following snippet will help you understand the standard structure of robots.txt files-
# Group 1
User-agent: Googlebot
Disallow: /nogooglebot/
# Group 2
User-agent: *
Allow: /
Sitemap: http://www.example.com/sitemap.xml
As you place a robots.txt file in the HTML header of a specific page, that prevents Google and other search engines from crawling into that page.
Meta tags can help a lot
- Numerous meta tags turn out to be very handy in terms of excluding specific articles and portions of your web pages from search engines’ indexing lists.
- If you want to exclude a particular segment of your web page from Google’s indexing, you should use the following meta tag-
- <meta name=”googlebot” content=”unavailable_after: 27-Aug-2020 18:30:00 IST”>.
- You should change the date portion in the meta tag at your convenience.
- <meta name=”robots” content=”noindex, nofollow”>.
You should add the meta tag mentioned above to make a specific article of your website non-indexable for Google bots and other search engine bots.
Apart from these two, there are multiple meta tags that might help you exclude certain segments of your website from all search engines’ indexing lists.
Putting the excludable segments in iframes is an easy solution
Though this technique would be a bit difficult unless you’re tech-savvy, this is a handy option to solve your purpose. Adding the iframe meta tag to a particular segment of your website makes the portion non-crawlable for Google and other search engines. In this process, the webmaster needs to put the specific content piece in a separate HTMLl file. After doing so, an iframe tag should be utilized for the display of that file on the host page. The following snippets will clarify the entire thing to you in a better way-
This text is crawlable, but the following is text that search engines can’t see: <iframe src=”/nocrawl/content.html” width=”100%” height=300 scrolling=no>
<iframe src=”excluded.html”>
Utilizing AJAX is also a convenient option
As iframes tags don’t work appropriately in all search engines, utilizing jquery AJAX comes in handy to exclude parts of a web page from search engines’ indexing list.
Let’s check out how the job is done using jquery AJAX-
this text is crawlable, but the following is text that search engines can’t see:
<div id=”hidden”></div>
<script>
$.get(
“/nocrawl/content.html”,
function(data){$(‘#hidden’).html(data)},
);
</script>
You may use a directive nonsnippet
Google generally shows a snippet of your website content at the top of your website’s URL header in the search results. If you use a directive nonsnippet, you can prevent Google from showing a specific portion of your web content in the snippet. Here’s an example of how you do it-
<p>This text can be shown in a snippet
<span data-nosnippet>and this part would not be shown</span>.</p>
<div data-nosnippet>not in snippet</div>
<div data-nosnippet=”true”>also not in snippet</div>
<div data-nosnippet=”false”>also not in snippet</div>
<!– all values are ignored –>
<div data-nosnippet>some text</html>
<!– unclosed “div” will include all content afterwards –>
<mytag data-nosnippet>some text</mytag>
<!– NOT VALID: not a span, div, or section 鈫
These are the most efficient ways to exclude specific portions of your website content from Google’s and other search engines’ indexing lists.
Why not JavaScript or Flash?
Previously, webmasters used to exclude specific portions of their web pages from Google’s indexing list. After the up-gradation of Google algorithms and indexing features, JavaScript and Flash don’t work appropriately. At present, Google can crawl into your website and index content even if you’ve used JavaScript or Flash for excluding certain segments of your web content.
Final words
This post described all the possible procedures of excluding parts of web pages from Google’s and other search engines’ indexing lists. Now it’s time for you to execute the knowledge that you gathered from this article! Choose one of these techniques and hide specific portions of your website content from Google and other search engines.
