Search engines and Robots
Starting at the beginning
Robots are pieces of software that explore the Internet to discover, review and index what they find. You have very little control over them and are also known as trawlers, crawlers, spiders, wanderers and worms.Robots come in 3 varieties, the good, the bad and the ugly. The good ones search and index the Internet and register the information it finds. The most common being the search engines such as Google or Yahoo which sends them out to explore. These then go through your website to assist in classifying and applying a rank to your site. Other robots simply check your web site for errors and report back on omissions, errors on links and any content that has been updated – there are others but the list goes on and on. That’s it for the goodies, what about the baddies…….Bad and ugly ones look to cause havoc by searching for weaknesses in your code and then extract information from you without your knowledge. The most common occurrence of this being to spam you or 'harvest' your email address to then sell on to other people….very annoying. As per the goodies there are others and the list goes on and on.Very interesting, but so what ?
Well, in order to get listed in the search engines naturally and get as much targeted traffic to you as possible, you need to know this, so read on…..
Most indexing is done automatically with the notable exception of two really important directories, DMOZ and the Yahoo Directory. Believe it or not, these are reviewed by humans and as a result, take longer. Note: The DMOZ submission is free but it can take many months to get listed. Robots are a good thing so make your site available for them
What should you do?
On the assumption you do want a robot to visit your site, you must encourage them and make it easy for them to trawl your site. Finally see below for the top must do points:
Add your URL to the Search Engines
Submit to DMOZ (it free! But takes months)
Submit to the Yahoo Directory (Not free but can be listed in a couple of weeks)
Make a site map (search for Xml sitemaps)
Link it to your Home Page
………wait
Remember that robots like to see
· Links (internal and external)
· Text (lots of it)
· ALT Tags on your pictures
· Site Maps (on your pages + uploaded to the search engines)
· Page titles
· Meta descriptions
· Meta keywords
Robots don’t like to see
· Java script
· FLASH
· Most animations
· Images (for the moment)
I’m scared of spiders and don’t want them
Probably not the best thing in the world, but you can prevent spiders coming to see you. If you create a file called robots.txt, you have the ability to tell the robots what not to spider. This does have its uses, for example, if you have a telephone list on your site that you didn’t realize was suddenly going to made public once the spiders came to town, add the exclusion to this file.
How do I know if they have been?
The easiest way is to look at your server logs, certain IP addresses can even tell you what type of spider came to visit. There are third party utilities that can extract your logs and make them more readable and friendly.
Author: 4little1s.com
Terms: This document can be copied as long as the links below stay intact.
4little1s.com is an online UK baby gifts, baby clothes and Izziwotnot nursery furniture web site located at http://www.4little1s.com
