Just ran across this "I have both a spider.txt and htaccess set up to prevent bad spiders from running amuck."

Can ANYONE explain what the heck this person is talking about? Seems you need a degree these days in computer lingo or computer language/programming to even understand topics discussed on forums. I am just trying to learn how to promote a new website and am constantly running across stuff like this, and it is just frustrating me more! I do know that the search engines use "spiders" to "crawl" sites to consider them for submission, but this just threw me for a loop....the person went on to list a LARGE list of "spiders", (I think!)...2 examples: "vandex" and "worldlight" ARRRRGGGHHH!! What are they???????!!!!