Posts: 14768
From: Saturn next to Charmainec
Registered: Apr 2009

posted January 14, 2012 03:05 PM            

---

I'm wondering if a search engine spidering a site can cause a sudden spike in server resources during the spidering?

IP: Logged

Posts: 3495
From:
Registered: Sep 2011

posted January 14, 2012 04:07 PM            

---

What kind of server resources?

If you're thinking whether the server slow-down may be caused by the spiders/crawlers/robots of the search engines, I doubt it. At least with reputable search engines, you shouldn't have to worry about that. With the amount of indexing they do, if servers were affected by them, they would have been abandoned as a search&index solution!
However it's an interesting question since they index the real thing on the fly. For instance for sites like this one with frequent updates of pages, the crawlers are very busy! But again it wouldn't make sense that they slow down the server! That would be completely unproductive for them and any other website they index!

PS: I'm not an SEO specialist.

IP: Logged

Posts: 14768
From: Saturn next to Charmainec
Registered: Apr 2009

posted January 14, 2012 09:16 PM            

---

If it crawls the site, it has to increase the server load. The question is how much.

IP: Logged

Posts: 426
From:
Registered: Apr 2009

posted January 14, 2012 09:37 PM            

---

About six hog heads to the gallon

------------------
Proud Member of the Vatican Assassin Warlock Attack Squad!

You can't handle my level of Tinfoil! ~ {;,;}

IP: Logged

Posts: 9155
From: Still out looking for Schrodinger's cat.......& LEXIGRAMMING.♥.. is my Passion!
Registered: Apr 2009

posted January 14, 2012 11:39 PM            

---

Could be spidering or mining.
Every post will be crawled.
A well behaved bot should NOT be doing that.
Your administration tools should have a feature to disable spidering.

------------------

~The present time is theirs, but the future is mine.~Никола Тесла
~I remember, therefore I am immortal~LEXX
}><}}}(*>~♥ ♥ ♥ ♥ ♥~

IP: Logged

Posts: 14768
From: Saturn next to Charmainec
Registered: Apr 2009

posted January 15, 2012 12:13 AM            

---

Don't want to stop it--just figure out a way to deal with the surge on the server (if that is even the issue). Most people find us via search engines. We are number one or two (sometimes Wikipedia outranks us).

------------------
"Never mentally imagine for another that which you would not want to experience for yourself, since the mental image you send out inevitably comes back to you." Rebecca Clark

IP: Logged

Posts: 3495
From:
Registered: Sep 2011

posted January 15, 2012 04:11 AM            

---

I really don't think it has to do with server-side resources. The way the crawlers work (I used to do them in Perl) is to index every page on your site. It's like you have an extra user or group of users going through the pages. That's the effect on the server. Ergo, I would not blame your server bottle-up's to the bots.

The only window for doubt here is a bot of a "malicious" engine; *that* could be/become virulent. Can you select which engines are "seeing" you and only limit yourself to the reputable ones?

IP: Logged

Posts: 14768
From: Saturn next to Charmainec
Registered: Apr 2009

posted January 15, 2012 03:30 PM            

---

But a single user couldn't go through every page. Don't they do it rather quickly? Just trying to figure this out.

------------------
"Never mentally imagine for another that which you would not want to experience for yourself, since the mental image you send out inevitably comes back to you." Rebecca Clark

IP: Logged

Posts: 3495
From:
Registered: Sep 2011

posted January 15, 2012 03:34 PM            

---

I guess how quickly they crawl will depend on their resources, not yours.

IP: Logged

Posts: 14768
From: Saturn next to Charmainec
Registered: Apr 2009

posted January 17, 2012 11:09 AM            

---

But it's my resources that display the information to them. Maybe I'm getting confused because the internal search engine definitely pulls on the server. Memory and bandwidth are handled well by the server. It's the CPU that spikes for no apparent reason...and then posts vanish and the site is slow.

------------------
"Never mentally imagine for another that which you would not want to experience for yourself, since the mental image you send out inevitably comes back to you." Rebecca Clark

IP: Logged