I'm aware the server has been up and down more than a politicians rent boy (allegedly).
I've just found the problem........YAHOO SLURP.....the Yahoo search engine spider, throwing in 100's of searches a second. As Yahoo supplies less than 1% of traffic, that's a bit of liberty, so I've set it up so they can only scan 1 per second from now on.
On top of that, the server is having another GB of memory added in the next 24hrs.
Keith - Administrator
Someone has the clap
So much for robots.txt files.....Yahoo came back...so I've banned the IP for now, can't leave it that way though
Keith - Administrator
Did you get more hits due to the AVG issue? Or are we all to clever not to use that application?
What's AVG got to do with a Linux server?
Keith - Administrator
http://www.theregister.co.uk/2008/06...affic_numbers/
Many people not happy.
As part of my remit in my new role will be internet and tings i have to look intrested
So much for that. Everything fine last night, get up this morning and theYahoo bot is back and hanging the server with so many requests.....so I've BANNED it's IP.
Keith - Administrator
Seems like that you didn't ban it too good.......Originally Posted by Win2Win
So much for that. Everything fine last night, get up this morning and the
"The b**ch is back".......
My current subscription to Eset NOD32 AV expires next month, and I'm looking around for a decent free AV as a replacement. Having seen the link scanner nonsense that AVG supplies, I can discount that piece of rubbish straightaway. I don't like AVG anyway as it takes up so much CPU. NOD32 uses very little system resources, updates itself 6 times a day and can be fully configured.http://www.theregister.co.uk/2008/06...affic_numbers/
Many people not happy.
As part of my remit in my new role will be internet and tings i have to look intrested
Memory upgrade now complete
Should run like a well oiled rent boy!
Keith - Administrator
Whats your obsession with rent boys?Should run like a well oiled rent boy!
They aren't covered in wool?Whats your obsession with rent boys?
Keith - Administrator
Sheep?They aren't covered in wool?
Your's probably aren't as you tear it out while being violently passionate
Keith - Administrator
thought my connection is slowing down
You couldn't make it up with Yahoo.
I allowed it back on all the sites bar the one it was basically doing a "Denial of Service Attack" on, and banned it in the Robots.txt
It completely ignored that, and brought the server down again last night
So I've banned it now via its IP on the article site. I know what it's done. I added software to list sitemaps for Google last week for better indexing, and Yahoo managed to find them and seems to want to index all 107,000 articles at once!!!
Keith - Administrator
Oh that's cool everything is fine now in here.. Coz this morning is really crap as much as I want to visit the site it's not really working. But now so far the connections is good and it's fast. Big applause for the BIG BOSS^^, Keep up the good work BOSS, Thanks
PS...Don't use YAHOO
Keith - Administrator
thank god,its working now...im really angry this morning
to my hubby,i blame him for what happen,coz his the last
person who touch my computer...
i think he deserve an apology.
Keith - Administrator
this place has gone down hill since you moved the server to manc land scouser keith
it's like the old days connecting at 300 baud
this is what happens when you cut corners and save a few pennies
did you tell them you were a scouser
It cos I moved from Yankland, they are trying to bring me down from California.....well they can't get in now
Server runs at an average load of 0.5% without the Yahoo bot.....and over 30% with it.....all for an extra 5-10 bonus visitors a day
Keith - Administrator
Oooo ... I remember those old days! A Teletype 43 and an acoustic coupler. I ran up a £500 phone bill in the first quarter ... and the neighbour with whom I shared a party line complained that she could never use her phone!
Couldn't she walk to the phone box in the rain?
Keith - Administrator
Think we need to host it from your Ngagethis place has gone down hill since you moved the server to manc land scouser keith
it's like the old days connecting at 300 baud
this is what happens when you cut corners and save a few pennies
did you tell them you were a scouser
i'm sad to say that i had some thing similar if not the same , it was made by Texas instruments and in those days there was only a few bbs, and all input/output was sent to the built in thermal printer, it was the size of a medium suitcase and was pretty heavy to move around
The Yahoo Slurp should not load the server anymore than Google and other spiders. Yahoo appears more in the online users page because it hits from a range of IP addresses and vb considers each IP a different "guest" whereas each new Google index comes from the same IP address so vb considers it as one IP.
This Yahoo was one IP, and was bringing down the article directory trying to read over 100,000 pages as quick as it could load them, nothing to do with vB.
I left it banned for a few days, and the server ran at an average of 0.5, I let it in again then to test it and it brought the server to a crashing halt again, with an average of 86!!!
Keith - Administrator
You can slow down the rate that Yahoo hits the server via the robots.txt file - I am sure that you know that though (Assuming you use a proper HTTP Server that is!)
I know that but it ignores it.
It's my server, all 100% of it, and Yahoo is no loss on that site with only around 200 hits a month from Yahoo, and 10,000's from Google.
Keith - Administrator
There are currently 2 users browsing this thread. (0 members and 2 guests)
Posting Permissions
