.Google.com's John Mueller responded to an inquiry about why Google marks web pages that are prohibited from creeping by robots.txt and also why the it is actually safe to overlook the relevant Look Console documents about those crawls.Crawler Web Traffic To Inquiry Parameter URLs.The person talking to the concern recorded that robots were actually developing links to non-existent concern specification URLs (? q= xyz) to pages along with noindex meta tags that are likewise obstructed in robots.txt. What prompted the inquiry is actually that Google.com is creeping the hyperlinks to those web pages, obtaining shut out through robots.txt (without watching a noindex robotics meta tag) after that acquiring shown up in Google Explore Console as "Indexed, though obstructed through robots.txt.".The individual asked the observing concern:." But listed below's the huge inquiry: why would Google.com index pages when they can't even find the material? What's the perk because?".Google.com's John Mueller verified that if they can not crawl the web page they can not view the noindex meta tag. He also creates an exciting reference of the web site: hunt driver, recommending to neglect the outcomes given that the "typical" consumers will not view those outcomes.He created:." Yes, you are actually correct: if our company can not crawl the web page, our team can not find the noindex. That claimed, if our experts can't crawl the pages, after that there is actually not a lot for our company to mark. Therefore while you may view some of those webpages with a targeted web site:- inquiry, the common customer won't see all of them, so I definitely would not fuss over it. Noindex is additionally fine (without robots.txt disallow), it merely indicates the URLs will definitely find yourself being actually crept (and end up in the Search Console report for crawled/not indexed-- neither of these statuses induce problems to the rest of the web site). The vital part is actually that you do not produce all of them crawlable + indexable.".Takeaways:.1. Mueller's answer validates the restrictions in using the Web site: search evolved hunt driver for diagnostic causes. Some of those reasons is actually since it is actually not linked to the regular search mark, it is actually a distinct factor altogether.Google.com's John Mueller discussed the site hunt driver in 2021:." The brief solution is actually that a web site: concern is certainly not indicated to become complete, neither used for diagnostics reasons.An internet site concern is actually a specific kind of hunt that limits the end results to a particular website. It's essentially simply the word internet site, a bowel, and then the web site's domain.This concern confines the outcomes to a certain internet site. It is actually certainly not indicated to be a detailed assortment of all the web pages from that site.".2. Noindex tag without utilizing a robots.txt is fine for these sort of situations where a crawler is linking to non-existent web pages that are obtaining found by Googlebot.3. URLs along with the noindex tag are going to produce a "crawled/not indexed" entry in Search Console and also those will not have a negative result on the remainder of the website.Go through the concern and also answer on LinkedIn:.Why would Google.com mark webpages when they can't even observe the web content?Featured Graphic by Shutterstock/Krakenimages. com.