Search engines are websites that allow users to find information on the Internet based on topics of interest. They work like a library where users can enter keywords, phrases and terms related to their topic of interest, and the search engine will generate a list of relevant Web pages with appropriate links. Most likely, you have probably used a search engine such as Google, MSN or AltaVista, or a search directory such as Yahoo, DMOZ or Global Business Listing, in order to help you find specific information on the Internet that is of interest to you. The first and foremost thing is to understand the important differences between a search engine and a directory.

Search Engine can be defined as a system that “indexes” all the pages in the web. It process millions of web pages at a time and stores relevant information about those web pages. All three search engines works in similar fashion. Though there are many search engines, the main three Search engines are Yahoo! (wwww.yahoo.com), Google (www.Google.com), and MSN (www.msn.com).

Search Directory can be defined as a system which groups web sites and not web pages. The directory has no idea about the content present in any of the pages within the site; it only knows what the site is about. In terms of Directory,
Yahoo! had the first massive directory on the Internet (dir.yahoo.com)
Google also has a directory (dir.google.com);
MSN also has a directory specials.msn.com.

So how do these search engines work?

Each search engine relies on a database containing the full text of Web pages selected from the billions of Web pages out there. These huge list of web pages are added and constantly updated using special automated software programs / robot progrmas known as Crawlers or Spiders. These spiders crawl across the whole Internet, adding pages they visit to the list of pages to their index.

If you want your website to be included in the search engines, you have three main options:

1. You can wait and hope that the search engine spiders find your site on their own by following links pointing to your website.

2. You can submit your website – request inclusion in the search engine database (search engine index) – which is more effective than relying on spiders alone.

3. You can pay a fee for guaranteed inclusion – this leaves nothing to chance and your Web page is indexed in a timely manner.

Automated spiders are capable of reading a Web page and finding links to other pages or websites to visit. In this fashion, spiders can often travel across the Web finding new or updated pages to add or update their index. In the case of Google, "Freshbot" (the name of one of its most popular crawlers) visits a large number of websites every 24 or 48 hours, and many times will write the date it updated that site in its index directly on the results pages in Google.

Each search engine uses its own "secret technology" to find and rank pages, but most base their technology on the frequency and location of the search, as well as certain parameters such as keyword density and other factors. For example, Google uses it's patented and very popular PageRank algorithm, which it calculates as a site's importance in the number of "votes" from other on-topic, related web pages. The way it works is, the higher the Page Rank, thus the number of votes, the higher that site should be ranked in it's index.

Once your Web page has been included in a search engine’s database (indexed), the search engine then uses an algorithm to determine the ranking of results for each keyword search. At this point, the on-page optimization of your website and off-page optimization is pivotal to your success in the search engine ranking process. Professional optimization is geared towards understanding and adapting your website to best present your Web pages for these algorithms. As the algorithms search engines use frequently change, on- and off-page optimization needs to be an ongoing process.

It is evident that the processes of submission and optimization work hand in hand. One without the other seriously inhibits your search engine ranking potential. Remember, the top 30 results receive over 90% of the search traffic.