Even before the concept of search engine optimization was introduced, robots.txt has been a fixture on websites. With or without SEO, its purpose was clear: to request search engines to ignore crawling certain files and folders as specified in the protocol. These pages may have been placed on our servers for convenient access to certain people, but with the concept of open Web, whatever we uploaded can now be seen by anyone who has access to the Internet through search engines. That's where robots.txt comes into picture.
With SEO we have become more aware of what search engines can do. Our purpose of doing SEO is to make our pages more visible, thereby attracting more visitors who see its more prominent search engine placement. But prominent search engine placement for wrong keywords and pages is not
