- ASP(3477)
- ASP.NET(1519)
- Ajax(293)
- C and C++(487)
- CFML(272)
- CGI and Perl(4376)
- Flash(2133)
- HTML5(1)
- Java(1807)
- JavaScript(2961)
- PHP(16078)
- Python(128)
- Remotely Hosted(2495)
- Ruby on Rails(82)
- Tools and Utilities(4427)
- Web Hosting(24)
- XML(269)
- 1,251 categories
- 40,829 scripts
Categories
Updated: 03/07/2008
robots.spider(tm)
- robots.spider(tm) will read a robots.txt file and compare it with the pages on the site. Search engines spider the site. If there are no references to a directory or file, it will not be indexed. robots.txt files should not refer to unreferenced content because it allows other viewers to identify areas of your site that you don''t want indexed. The read depth limits the number of pages scanned.
- Categories
- Platforms
- Linux
- Licenses
- GPL
- Author