In its most basic form, Search Engines operate in three distinct steps and in this blog I’m going to break down each so you have a basic understanding of how it all works.
#1 Crawling
Search engines trawl through the internet finding sites, pages and documents. Obviously given the scale of the WWW this is no task for a human, so Google (there are others, but let’s be honest Google is the one we’re the most familiar with) has robots, sometimes referred to as search engine spiders, that crawl the web to discover pages, PDFs, videos images and anything else there might be out there.
Unlike humans, these crawlers can very quickly crawl an unimaginable number of different sites simultaneously and frequently; so Google is constantly looking at your website for updates and changes. This is one of the reasons why linking from old content to new is a great idea as crawlers can find the pages much faster.
Crawlers resources aren’t unlimited, so you will find they crawl larger well-known websites far more than lesser-known smaller sites. Another great reason for having backlinks from authority sources.
#2 Indexing
Once the crawlers have found a site, page, PDF etc. the information gets added to an index. This is essentially a huge database containing all the web pages the crawlers have found.
When you type a query into Google, it queries this database and returns results accordingly. As you can imagine, the scale of the data that Google has to deal with is astronomical and without the next step, it would be total chaos.
#3 Ranking
The final step is to order or ‘rank’ the content in the index based on hundreds of different factors. Google updates the criteria for how things are ranked hundreds of times a year with very complex algorithms.
Some of the factors Google will share, but a lot of what we know is reverse engineered. So, by looking at the number one ranked website is for a given keyword we can see what it does differently/better to the website is position two and so on.
It could be the volume of content, structure, backlinks, domain authority and a huge number of other factors. But being able to monitor this with tools such as SEMrush we can try to mimic what the top sites have done on our own.
Anyway, that’s a whole other industry let alone another blog post. Hopefully now you have a better idea of how search engines work.
