How to correctly keep a page hidden from Google – the difference between robots.txt and meta tag “noindex”
Do you know how to effectively keep Google from displaying your secret page in the search results? If not, you’re not alone! – Don’t worry! The graphics below should clear things up.
I decided to write this blog post and create the “info”graphic below because a former developer colleague double checked some requests and a “fact sheet” he had recieved from a client’s “SEO consultant” (I am using quotation marks since a lot of the requests based on the ‘fact sheet’ were wrong or ineffective).
One of these requests were that the website’s internal search results pages should be blocked from Google’s indexing by using robots.txt.
Preventing Google from indexing the internal search results is a good idea 99% of the times BUT the solution from the SEO consultant would NOT have prevented indexing! Had my former colleague not questioned the advice from the consultant, the client was at risk of spending money on a solution that would not have had the desired effect!
In order to correct this widely spread misconception – that blocking Google from certain pages using robots.txt will prevent Google from indexing the blocked pages – I chose to create the graphical explanation below.
Hopefully this will clear things up a bit – and prevent clients and developers waste their money and time.
If you find this helpful then please do share it 🙂