If you check the log, you'll find a number of them are spider. At any one point 10%-30% of visitors to my forums are usually search engine spider. You have to be careful, as they continuosly refresh your pages and use your bandwidth.

You can stop them, but then it can effect rankings you have. It's a lose/lose situation, unless you have unlimited bandwidth