You are viewing this forum as a guest. Login to an existing account, or create a new account, to reply to topics and to create new topics.
How do I stop a robot from crawling into some of the pages I have? Such as my contact us, company policies, FAQ's ... etc. I already have <meta name="robots" content="ALL"> on my layout. THere will be two <head></head> code on those pages if I put <meta name="robot" content="noindex"> on the page itself. Validator says that was an error. Is there a better way of doing this? Any inputs will be greatly appreciated.
Offline
The default behavior of major search engine spiders is to follow and index so robots meta tags like or don't really serve a useful purpose and can be omitted.
Using a file can centralize it to some extent.
That aside, those pages - contact, faq, etc. - can be good candidates for optimization.
Offline