Magento robots.txt

Magento comes without robots.txt functionality. It can be useful to add one yourself to tell the search engines where they are not allowed to index. It will hide your javascript files, hide SID parameters and prevent some duplicate content. It will help your SEO process and reduces resources on your server. In this blogpost explain you how to set your own robots.txt using an existing example and using an extension. Both solutions are easy to handle. 

 Very good explanation :

http://www.byte.nl/blog/2012/06/11/magento-robots-txt/