How Do Search Engines Work

 

The invention of the internet and the world wide web is akin to the invention of the printing press or internal combustion engine. It has revolutionised the way we do business and interact in a little over 20 years and it’s scope is so vast it has changed the way we speak; if you asked somebody to ‘Google it’ 20 years ago people wouldn’t have a clue as to what you were talking about.

The fact that we can find out information on virtually every conceivable subject, from any device connected to the internet, from anywhere in under a second is an amazing feat. Search engines such as Google, Bing and Yahoo amongst others, allow us to select one web page out of trillions, which is relevant to a given query. Just how they do this is down to a complex algorithm they use to grade pages in terms of relevancy and importance.

The 3 Main Functions Of Search Engines

  1. Crawling – Search engines employ automated robots called spiders which crawl the web from document to document. Spiders crawl the web using links.
  2. Indexing – Once a spider reaches a web document, it looks at the code to determine what it’s about, then stores it in massive data centres owned by the search engines.
  3. Serving results – When a user enters a query into a search engine, it will return a list of relevant pages from those stored in it’s database. The list of pages the search engine returns is ordered in terms of relevance and importance.

How Search Engines Provide Relevant Results

The relevance of a web page is the degree to which it matches a user’s query. Early search engines circa late 1990’s would look at a document’s text, such as titles, headings and content, to determine the relevance of a page for a specific query. Because unscrupulous SEO’s would use techniques such as keyword stuffing to trick search engines into thinking a page was relevant when it wasn’t, engines have become much more advanced, and there are now 1000’s of additional characteristics which will determine the relevance of a page.

How Search Engines Provide Important Results

To a search engine, the more popular a page is, the more important it is deemed to be. And the more important it is, the higher it will rank for relevant queries. A question you may be asking yourself now is just how do I go about making my web page or site more popular or important. One way this can be accomplished is by building links from relevant sites to your web page. In the eyes of the search engines, links from external sites count as votes, so if the relevance of two pages is the same, the one with more links (votes) to it will rank more highly.

SERP example

Searching for ‘cheap holidays’ includes these results ordered by relevance and importance.

 

The reality is, of course, much more complex than this and search engines use hundreds of ranking factors to determine where a page should be on the search engine results page. These factors are all part of a complex algorithm, search engines use to serve up results which satisfy user queries.

You Can’t Fool A Search Engine

Or rather you used to be able to. Early search engines such as Infoseek, Lycos and early Google were very simplistic and only looked to a document’s text to ascertain whether a site was relevant. A whole industry was set up around fooling the engines, to ensure a site came at the top of the search engine results. As this sometimes led to poor results being presented to users, Google and other search engines, have had to develop increasingly complex algorithms, to ensure users get the best search experience possible. Over the past two years, Google has introduced two major updates to it’s algorithm – Panda and Penguin. These updates badly affected sites which were using unethical tactics to rank, some being de-indexed by Google.

The bottom line is, today you can’t fool the search engines, but if you work with them, within their guidelines, you will benefit from higher and more sustainable rankings.