Server Slowdown

[KeithD]

I'm aware the server has been up and down more than a politicians rent boy (allegedly :rolleyes:).

I've just found the problem........YAHOO SLURP.....the Yahoo search engine spider, throwing in 100's of searches a second. As Yahoo supplies less than 1% of traffic, that's a bit of liberty, so I've set it up so they can only scan 1 per second from now on. :xxgrinning--00xx3:

On top of that, the server is having another GB of memory added in the next 24hrs. :)

[telford]

:appl:

[KeithD]

Someone has the clap :Erm:

So much for robots.txt files.....:censored: Yahoo came back...so I've banned the IP for now, can't leave it that way though :NoNo:

[andypaul]

Did you get more hits due to the AVG issue? Or are we all to clever not to use that application?

[KeithD]

What's AVG got to do with a Linux server? :)

[andypaul]

http://www.theregister.co.uk/2008/06/13/avg_scanner_skews_web_traffic_numbers/

Many people not happy.

As part of my remit in my new role will be internet and tings i have to look intrested:D

[KeithD]

:furious3::furious3::furious3::furious3::furious3:

So much for that. Everything fine last night, get up this morning and the :furious3: Yahoo bot is back and hanging the server with so many requests.....so I've BANNED it's IP. :NoNo:

[aromulus]

:furious3::furious3::furious3::furious3::furious3:

So much for that. Everything fine last night, get up this morning and the :furious3: Yahoo bot is back and hanging the server with so many requests.....so I've BANNED it's IP. :NoNo:

Seems like that you didn't ban it too good.......:omg:

"The b**ch is back".......:D

[walesrob]

http://www.theregister.co.uk/2008/06/13/avg_scanner_skews_web_traffic_numbers/

Many people not happy.

As part of my remit in my new role will be internet and tings i have to look intrested:D

My current subscription to Eset NOD32 AV expires next month, and I'm looking around for a decent free AV as a replacement. Having seen the link scanner nonsense that AVG supplies, I can discount that piece of rubbish straightaway. I don't like AVG anyway as it takes up so much CPU. NOD32 uses very little system resources, updates itself 6 times a day and can be fully configured.

[KeithD]

Memory upgrade now complete :xxgrinning--00xx3:

Should run like a well oiled rent boy! :action-smiley-081:

[walesrob]

Should run like a well oiled rent boy! :action-smiley-081:

Whats your obsession with rent boys? :omg:

[KeithD]

Whats your obsession with rent boys? :omg:

They aren't covered in wool? :Erm:

[walesrob]

They aren't covered in wool? :Erm:

Sheep?

[KeithD]

Sheep?

Your's probably aren't as you tear it out while being violently passionate :Hellooo:

[doonat]

thought my connection is slowing down

[KeithD]

:doh You couldn't make it up with Yahoo.

I allowed it back on all the sites bar the one it was basically doing a "Denial of Service Attack" on, and banned it in the Robots.txt

It completely ignored that, and brought the server down again last night :action-smiley-081:

:furious3::furious3::furious3: YAHOO :butthead:

So I've banned it now via its IP on the article site. I know what it's done. I added software to list sitemaps for Google last week for better indexing, and Yahoo managed to find them and seems to want to index all 107,000 articles at once!!! :Brick:

[amyburple]

Oh that's cool everything is fine now in here.. Coz this morning is really crap as much as I want to visit the site it's not really working. But now so far the connections is good and it's fast. Big applause for the BIG BOSS^^, Keep up the good work BOSS, Thanks:icon_win::REDancedancer08::REGamblMoney01HL1::appl::appl::appl:

[KeithD]

PS...Don't use YAHOO :)

[alicat]

thank god,its working now...im really angry this morning
to my hubby,i blame him for what happen,coz his the last
person who touch my computer...
i think he deserve an apology.

[KeithD]

:icon_lol:

[joebloggs]

:icon_lol:

this place has gone down hill since you moved the server to manc land scouser keith :NoNo:

it's like the old days connecting at 300 baud :cwm23:

this is what happens when you cut corners and save a few pennies

did you tell them you were a scouser :doh

:xxparty-smiley-004:



:D

[KeithD]

It cos I moved from Yankland, they are trying to bring me down from California.....well they can't get in now :action-smiley-081:

Server runs at an average load of 0.5% without the Yahoo bot.....and over 30% with it :Erm: ...hit 89% last night....:yikes:.....all for an extra 5-10 bonus visitors a day

[PeterB]

it's like the old days connecting at 300 baud

Oooo ... I remember those old days! A Teletype 43 and an acoustic coupler. I ran up a £500 phone bill in the first quarter ... and the neighbour with whom I shared a party line complained that she could never use her phone!

[KeithD]

.....and the neighbour with whom I shared a party line complained that she could never use her phone!
Couldn't she walk to the phone box in the rain? :D

[andypaul]

this place has gone down hill since you moved the server to manc land scouser keith :NoNo:

it's like the old days connecting at 300 baud :cwm23:

this is what happens when you cut corners and save a few pennies

did you tell them you were a scouser :doh

:xxparty-smiley-004:



:D

Think we need to host it from your Ngage:D

[joebloggs]

Oooo ... I remember those old days! A Teletype 43 and an acoustic coupler. I ran up a £500 phone bill in the first quarter ... and the neighbour with whom I shared a party line complained that she could never use her phone!

i'm sad to say that i had some thing similar if not the same , it was made by Texas instruments and in those days there was only a few bbs, and all input/output was sent to the built in thermal printer, it was the size of a medium suitcase and was pretty heavy to move around :doh

:icon_lol:

[Banana]

The Yahoo Slurp should not load the server anymore than Google and other spiders. Yahoo appears more in the online users page because it hits from a range of IP addresses and vb considers each IP a different "guest" whereas each new Google index comes from the same IP address so vb considers it as one IP.

[KeithD]

The Yahoo Slurp should not load the server anymore than Google and other spiders. Yahoo appears more in the online users page because it hits from a range of IP addresses and vb considers each IP a different "guest" whereas each new Google index comes from the same IP address so vb considers it as one IP.
This Yahoo was one IP, and was bringing down the article directory trying to read over 100,000 pages as quick as it could load them, nothing to do with vB.

I left it banned for a few days, and the server ran at an average of 0.5, I let it in again then to test it and it brought the server to a crashing halt again, with an average of 86!!! :omg:

[Banana]

You can slow down the rate that Yahoo hits the server via the robots.txt file - I am sure that you know that though (Assuming you use a proper HTTP Server that is!)

[KeithD]

I know that but it ignores it. :doh

It's my server, all 100% of it, and Yahoo is no loss on that site with only around 200 hits a month from Yahoo, and 10,000's from Google.

[Banana]

I agree with you that it's Yahoo loss. It's something they need to sort out. Surprised that they ignore the robots.txt file though :( Perhaps they are adopting Microsoft attitude to good practice in advance of any takeover.

[KeithD]

Reading the net the 'odd' Yahoo IP does ignore the robots file, and according to Yahoo, each IP only checks the robots file now & again, not every visit, so it could be months between readings. Stupid idiots!

They still have access to my other 50+ sites, and the article site is duplicated twice elsewhere (with SEO changes) so they still have access in a way. I'll just avoid putting a sitempa on them.

[PeterB]

What slows down my reading of this site is the time taken to fetch the little adverts which appear at the top of each page! :(

Perhaps I need to check the configuration settings in FireFox.

[KeithD]

Works fine on broadband :)

You can switch the Pics off in any browser, or install an Ad Blocker

[PeterB]

Hey, I AM on broadband ... Philippines broadband!

Rob suggested I try Adblocker, and it works a treat ... many thanks Rob!

[KeithD]

Nice of Rob to make sure the site earns bugger all :Erm:

[PeterB]

Sorry Keith, I know that it's how you fund the site, but I really did find it was a pain to wait 30 seconds, or more, every time I open a new page.

Perhaps we need a little script which will keep fetching the ads when I'm not trying to read the forum!

[KeithD]

:Erm: I don't even wait 1/100th of a second per page. One reason I'll never live in the Phil, couldn't trust the utilities to keep my business going.

[aromulus]

As it happens, mornings, often I cannot log on at all.:cwm23:

"cannot find the page"......:Erm:

[LEAHnew]

:Erm:Page cannot display for about 5 hours:Erm:
Is it about my WGA:Erm::D I need to disable soon:doh:Cuckoo:

[aposhark]

thank god,its working now...im really angry this morning
to my hubby,i blame him for what happen,coz his the last
person who touch my computer...
i think he deserve an apology.

That's funny :icon_lol:
I asked my Asawa to log-in to the site and have a look last night for the first time.
She probably thinks I've been giving her bad links :doh

[KeithD]

I ran a full backup during the early hours UK time which can prevent logins, as the Phil is in a different time zone....hard luck :icon_lol:



to the moderator:

Is it possible to change my username? pls.... thanks!
Email me.

[LEAHnew]

I ran a full backup during the early hours UK time which can prevent logins, as the Phil is in a different time zone....hard luck :icon_lol:



:doh An avid reader eh:xxgrinning--00xx3:, so boring when it's off:rolleyes::Rasp:

Thanks for the info boss:xxgrinning--00xx3: