Every webmaster would like to have a lot of visitors on his website. Therefore he is happy if his site can be found via searchengines. Searchengines uses webspider, crawler or robots to increase and refresh their database. In this way these webspiders browse the whole internet and follow internal and external links at the websites. In so doing all sites you can find in the internet are submitted though links at the searchengineindex. There exist data which should not be admited to searchengines. These are for example:
- unfinished sites
- technical sites (cgi-bin directories, logfiles)
- contents of data privacy laws
- installationdirectories, configurationsfiles
and so on.
To avoid an indexation of such kind of data it has been arranged a standard called robots-exclusion protokoll. This standard should enable the decline of websites at searchengines. Therefore is a file needec called "robots.txt". The content of this file are the names of sites which don't should be visited by the searchengines. Most famous searchengines like Google, MSN and Yahoo keep this standard. Their webcrawler is optimizied in this way. Spiders are used by searchengines not only to create a searchindex, but also to gathering data. There are differnet factors for collecting data
- Readout of mailto:address on the websites to generate spam mails
- Theft of pictures (Mister Pix etc)
- "Grabbing" of content (content theft)
- Spiders which are searching for infringement of trademark laws
- Spamattacks on a server through a multiplicity of sitecalls
These used spiders are called "bad bots". "Spider-Trap" trys to stopp the machinations of such bad bot and to bar them from thise site. Roughly speaking the bots has no more access on a website. This script is used on this website and can be tested by you. At the left head of this website you can see a spider at a prohibitory sign. If you click at this link you will have no more access on thise website. Spider Trap can't give you a 100% guarantee but you can stop the visit of many unpopular bots. The Spider Trap Software you can download for free. At the point Functionality you can find a technical back of Spider Trap. Installationshelp, support and further developement you can find at the Spider Trap forum. At these forum youn can discuss called topics. If you are convinced of Spider Trap, so you can set a link on this site.
Greetings your Spider-Trap Team
Help to make our site more popular! Implement the following HTML-code on your website.