What is a Robots.txt File?
Robots.txt is a plain text file that Search Engines use to see if there are areas within your website which should not be indexed. 
This file must be placed in your 'www' (home directory) in order for a search engine to see it. 

The sample below tells search engines not to index pages in the specified folders. 

---- robots.txt  (contents follow)

User-agent: * 
Disallow: /cgi-bin/
Disallow: /images
Disallow: /secret
Disallow: /members
Disallow: /rodnev
Disallow: /htaccess
Disallow: /invoices
Disallow: /mofcart
Disallow: /OLDmofcart
Disallow: /Scripts
Disallow: /report
Disallow: /cgibin