A Simple Explanation Of How Search Engine Spiders Work


You can use search engines to your advantage if you understand how their “Spiders” or “Web Crawlers” work.   Learning how to use the search engines to get greater visitor visibility to your blog or web site is called Search Engine Optimization.    SEO is the skill or process of improving the visibility of your web site by natural or unpaid methods.


The search engines use precise mathematical formulas or programs to process data.   These formulas are called algorithms.

Web Crawlers – Or Spiders:

A “spider” is an automatic program (algorithm) that is run by a search engine system that crawls your site for information.  A very simple explanation of how it works is:  A “spider” visits your site and crawls your pages, reads your Title Tag, the content of your post, looks for keywords, and follows the links your pages contain.

It then returns all that information to a central depository, where the information is stored and indexed.


An index is almost like a huge book that contains a table of contents.  When some one searches for a particular word or term or phrase… (which are called “keywords”), the search term is looked up in the index and the information is delivered to the searcher.

Millions of pages are indexed every day and you want to make sure your blog pages are indexed with a high ranking so readers can find you easily  That is the reason you need to be SEO smart.

Web Crawler Are Smart:

Spiders or Web crawlers (algorithms) are really smart.  They not only can crawl your page and see your title tag, content, and links… they scan for relevant keywords, their frequency and location in the content of  your pages.  They can also detect artificial “keyword stuffing”… or spamming.

Revisit Schedule:

A spider can also detect how frequently you post on your blog.  After they visit your site numerous times, an algorithm sets up a “revisit” schedule.  This is the reason, it is wise to be consistent in the frequency of your posting.

If you post 3 times a week… it’s best to post on the same days and about the same time.   The spiders will be by to check on your site based on your “revisit” schedule.

When your posting is erratic and not uniform, many of your pages will not be indexed because your “revisit schedule” or the web crawlers have been disrupted.

There is no SEO benefit to blogging everyday for 30 days when you have been posting three times a week.   The search engine will not index many of your blog pages during the 30 days because your “revisit schedule” will not change unless the blogging pattern remains the same for some time.

If you’re into “30 Day” blog challenges… evaluate your goal.  Are you blogging for SEO, networking or social reasons?  If you’re trying to build your SEO… you need to continue blogging everyday… even after the challenge if over for optimum SEO results.

Forming a blogging frequently pattern and being consistent with that schedule will bring you the best SEO benefits.

Happy Marketing and Promotion!

Til Later,

Kathy Griffiths

P.S.  The reason I love WordPress is that so many SEO advantages are already built into the system.  If you utilize the plug in… All in One Seo… you are far above and beyond most bloggers who don’t take advantage of this fantastic plug in.

Technorati Tags: search engines, SEO, spiders, web crawlers

Categories : SEO


  1. I have alerts set up so I can see when I’m getting indexed. During the last challenge, I started getting crawled every day. That was the motivation I needed to continue with that posting schedule. As long as I’m there, they are there. If I quit showing up, so will they.

  2. Kathryn says:

    Angie…way to go… You’ve found what works best for you. As long as you are blogging everyday… blog challenges are great for you.

    I participated in a blog challenge because my blog was new and I needed instant blog readers. However, I dropped the everyday blogging for another schedule to free up time for more marketing and promotion time for this blog and other niches. This schedule works best for me.

  3. sue White says:

    Kathryn – these are great easy to understand explanations. This is an area that is so often misunderstood. Thanks for sharing.

  4. grace says:

    Very informative and helpful. Thank you!

  5. Kathy,

    I bet the search engines are LOVING this post! You’ve got some killer keywords in here that the spiders are just going to deliciously and voraciously devour. 🙂

    Wow – there’s so much good information to digest. And I want to thank you for sharing the “revisit” schedule facts. I wasn’t aware of this principle and it makes total sense.

    You’re a champ!

  6. Lisa Raymond says:

    Wonderful post, Kathy! I finally know how those spiders really work, so my next challenge is to post at nearly the same time each day and post a little more often. Thanks for the tips!

  7. Cool, what an excellent explanation. This is another reason why the spiders LOVE WordPress websites. WordPress owns “ping-o-matic” so when you publish a post, your WordPress site pings Google and says “Hey! Come to this site! It has new stuff!”. Because of this, I always find a googlebot on my site just after I post a new blog, regardless of when it’s posted.

    And it’s not just the spiders which are smart, as Google has a team of staff also manually checking websites for keyword stuffing, which I think is excellent. I prefer a level playing field.

    Thanks for sharing Kathy 🙂

  8. Kathy, I am starting to come to your site more and more because the content is so fantastic! You have a way of breaking down subjects and explaining them in simple-to-use explanations that really benefit your readers. You are awesome!

  9. Kathy,
    I learned a lot from this post and I’m scratching my head wondering what I can do differently. This is a huge subject right now and we’ll never get to the bottom of it, but we can certainly try! Thanks so much for your research.


  10. Thanks Kathy,
    That is simply one of the best, easy to understand, explanations of algorithms that I have ever seen. 🙂

  11. Bonnie says:

    Such great and simple explanations…thank you so much! Despite my best efforts as trying to get a handle on SEO this is the first time I have understood the revisit schedule so clearly. I didn’t realize how important posting the same time is. Thanks for the tips.

    keep well 🙂

  12. Thanks for the tips. I’m just starting to blog and get into the social media world. I look forward to more posts from you.

  13. CaptRob says:

    Ahoy Kathy.

    It’s been a while since we have chatted…I miss those days.

    But like the automatic algorithm spiders, I crawl your pages. 😉



  14. Kathy,

    I remember reading this post even though I did not leave a comment at the time and will be including a link to it in my post later today as I really appreciated this simple, easy-to-understand explanation of what can be a confusing topic, and I’d like to share this with my readers.

    You also confirmed something I’ve observed but was not sure why. I’ve posted to my blog by 9:00 a.m. and received a Google alert within an hour or so, while other posts have taken a couple of days for me to receive a notification, if any. I now understand that I should keep posting at approximately the same each day or at least before a certain time. Hmm, I’ll submit that post tomorrow morning instead. 🙂

    Thank you, Kathy.