A number of articles and white papers have been written on this topic. Before the days of Google an Web 2.0 there was little attention paid to Search Engine Optimization (SEO). SEO refers t “optimizing” web sites to rank very high in the results of search engines such as Google, Yahoo an MSN. SEO helps structure your website content so it is easily found and indexed by search engines. In the early days of the Internet Era you could get your site listed on Yahoo or Altavista by merel submitting it. As long as it wasn't too complicated for search engines to understand your site, you could get it listed within 2 to 3 days. But Yahoo and Altavista couldn't keep up with the flood of UR submissions. As web evolved, companies began to embrace the online medium and the potential for using it as a marketing tool. In the late 1990's something dramatic happened. Google introduced PageRank, a system of ranking Web pages based on their importance and other factors. Google search spiders crawl through the web following links, indexing and categorizing pages based on their content and relevancy. When the search spiders crawls and finds backlinks to your site along the way, each backlink translates to a positive confidence of vote to your site. It indicates popularity or importance of your website to the search engine which ultimately helps place you in the top of the search results. But this is one of the many factors that Google or other search spiders take into account while ranking sites. In order to allow spiders to understand the site content, web masters needed to follow certain rules like using proper keywords, title tags and content in each web page. At this point search became more of an art than a science. People can no longer expect their sites to appear in the top of the search engine results by simply adding their sites to major directories.
What Is a Search Engine?
Okay, so you know the basic concept of a search engine. Type a word or phrase into a search box and click a button. Wait a few seconds, and references to thousands (or hundreds of thousands) of pages will appear. Then all you have to do is click through those pages to find what you want. But what exactly is a search engine, beyond this general concept of “seek and ye shall find”? It’s a little complicated. On the back end, a search engine is a piece of software that uses applications to collect information about web pages. The information collected is usually key words or phrases that are possible indicators of what is contained on the web page as a whole, the URL of the page, the code that makes up the page, and links into and out of the page. That information is then indexed and stored in a database.
On the front end, the software has a user interface where users enter a search term — a word or phrase — in an attempt to find specific information. When the user clicks a search button, an algorithm then examines the information stored in the back-end database and retrieves links to web pages that appear to match the search term the user entered.
· You can find more information about web crawlers, spiders, and robots
The process of collecting information about web pages is performed by an agent called a crawler, spider, or robot. The crawler literally looks at every URL on the Web, and collects key words and phrases on each page, which are then included in the database that powers a search engine. Considering that the number of sites on the Web went over 100 million some time ago and is increasing by more than 1.5 million sites each month, that’s like your brain cataloging every single word you read, so that when you need to know something, you think of that word and every reference to it comes to mind.
In a word . . . overwhelming.
v Anatomy of a Search Engine
By now you probably have a fuzzy picture of how a search engine works. But there’s much more to it than just the basic overview you’ve seen so far. In fact, search engines have several parts. Unfortunately, it’s rare that you find an explanation for just how a search engine is made — and that information is vitally important to succeeding with search engine optimization (SEO)
The query interface is what most people are familiar with, and it’s probably what comes to mind when you hear the term “search engine.” The query interface is the page that users see when they navigate to a search engine to enter a search term. There was a time when the search engine interface looked very much like the Ask.com page shown in Figure 1-1. The interface was a simple page with a search box and a button to activate the search.
Today, many search engines on the Web have added much more personalized content in an attempt to capitalize on the real estate available to them. For example, Yahoo! Search, shown in Figure 1-2, allows users to personalize their pages with a free e-mail account, weather information, news, sports, and many other elements designed to make users want to return to that site to conduct their web searches. One other option users have for customizing the interfaces of their search engines is a capability
like the one Google offers. The Google search engine has a customizable interface to which users can add different gadgets. These gadgets allow users to add features to their customized Googl search home that meet their own personal needs or tastes.