You have a website that is not meant for everyone, only a particular audience. You do not want certain directories to be crawled by a spider. How can you limit (in theory at least) what search engines will index?
A) .htaccess file
B) robots.txt file
C) set permissions on the directories to deny spiders read access
D) Require a login page

Question

Quizplus · Accepted Answer

Robots.txt file in placed at the root directory of the website. The .htaccess file is used in apacher webservers as an authentication feature and was at one time placed in the root directory but is no longer with newer versions. Login pages sometimes protect "The dark net" but search enignes might still be able to reach the files. EC-Coincil states that robots.txt will work, but in the real world keep in mind that compliance is up to the developer of the spider.

You Have a Website That Is Not Meant for Everyone