Seeing What a Search Engine Sees

Make sure you can be found: seeing site content and navigation from the point of view of a search engine.

I’ve mentioned a few times on this blog how compelling content and links from other sites are the most important factors if you want to show up in search engines. That’s true, but they matter little if a search engine can’t even find the pages on your site. This may sound obvious, but all too often I look at a web site and find that the navigation is actually hindering or preventing search engines from getting around. I could go into a long-winded explanation of how search engines crawl sites at this point, but thankfully there is a much easier way to demonstrate. There are a number of tools available that give you a view of how a search engine sees your site. Once you’ve seen a site the way a search engine sees it, it all becomes pretty clear. The simplest, most accessible tool is provided by Google itself. To use this tool, search for a page in Google. As an example, I will search for one of my recent posts by copying its URL into the Google search bar: https://twooctobers.com/2010/03/rocky-mountain-viral-3282010/

Here is the result in Google. The next step is to click on the “Cached” link to the right of the URL in the listing.

Once there, click on the “Text only version” link in the grey bar at the top. Voilá! You are now seeing the page pretty much the way Google sees it. Here is the text only version of my post:

Search engines have a pretty boring view of the world, huh? Overall, this page looks good in terms of how the content is laid out. Things to consider when evaluating the search engine’s view:

• Does the page include links to other important pages on your site? The links on the page are like signs telling search engines where to go. Just as highway signs list major landmarks, your navigation should point to the pages you most want found.
• Is it obvious what the page is about? All too often, sites include boilerplate content at the top of pages, or try to tackle too many things at once. Pages that have a clear focus tend to rank better. If you have to scroll down to find unique content, you have a problem.
• Is anything important missing? Page elements such as forms, Flash, JavaScript and iframes can hide content from search engines. You don’t need to know what all of this means, but if there is important content missing from the search engine view, you need to talk to a developer.

If you find yourself wanting to see a search-engine-view of a lot of pages, using the method described above can get kind of tedious. I use the Foxy SEO Tool Add-on for Firefox instead. It is free and has a lot of great functionality. In particular, it has a “Search Engine Spider Simulator” that enables you to quickly switch to a search engine view of any page.

More on Navigation

To the right is a sign that sits outside of the Rock Rest Lounge in Golden, Colorado. Can you imagine having to find your way around if all signs looked like that? At the opposite extreme would be a post with no sign at all. The first would be so confusing as to be useless. The second would be just plain useless.

This is what we often put search engines through. Search engines use links to find content, and as an indication of what is important. Too many links and a search engine can’t tell what matters. Too few and it can’t get around.

Also, search engines look at the text used in a link as an indication of what a page is about. Imagine if road signs just said “city” or “big city”, instead of “Colorado Springs” or “Denver”. You’d have a hard time finding your way around. Or imagine if you were crossing the border into Utah and saw a sign that said “Sigurd 200 Miles”. Sigurd is about 200 miles from the border, but you probably don’t know or care what Sigurd is unless you are one of the 400 people that live there. From far away, it is much better to signpost major landmarks that everyone will know. Conversely, what if you are in Denver and trying to get to the Rock Rest Lounge in Golden, and you see a sign for “Salt Lake City 535 Miles”? Maybe you know that Golden and Salt Lake City are in the same direction, but still not a very useful sign. When smaller towns and cities are close by, it’s good to include them on signs.

Think about navigation on your site like road signs for search engines. Link to your most important pages from all over, and make sure the links clearly describe the content they are pointing to. Link to more specific content from close by, again using descriptive text. And don’t put too many links on a page. There is no hard-and-fast rule for how many is too many, but more than one hundred is definitely too much. For most kinds of content you should try to keep it well below that.

Lastly, look at your pages the way a search engine does: a colorless world where text is all that matters. It may be a bleak view, but the view is even bleaker if you can’t be found.