Website Optimisation and Link Structure
A search engine requires links to get from one page to the next and the way your website is structured is important. Your website may, either deliberately or accidentally, contain links that confuse or stop a search engine spider altogether. Additionally your website may entirely lack the type of link a search engine requires to access certain pages within your website.
Speed Bumps
These can slow down, confuse or possibly stop a search engine as it crawls through your website. Complex types of links or pages that require the search engine to follow four or five links before it finds a page are examples of speed bumps, other possibilities are:
- Dynamic websites that produce links with two or more parameters i.e. http://www.dynamicwebsite.com/page.php?id=4&CK=34rr&User=%Tom%.
- Pages with a large number of links (+100) to other pages on the website. The SE spider might get tired of following all these links and give up.
- Pages buried deep within the website that require three or more links to access.
- Pages requiring a session ID or cookie to enable access.
- Pages that contain frames.
Road Blocks
These may possibly stop a spider in its tracks.
- Pages only accessible by completing a form and clicking on a submit button.
- Pages requiring a drop down menu before they can be accessed.
- Pages deliberately blocked by use of a robots meta tag or robots.txt file.
- Pages protected by a login box.
- Pages that redirect to another page before a SE spider can read any content on the page.
The last road block is a well known black hat method that the search engines hate. Redirects can be used to trick a search engine into thinking it is on a page with one kind of content while the redirect switches humans to another page with content of an entirely different kind, often porn or some kind of medication. These techniques are known as bait and switch or cloaking.
The best way of ensuring that a page on your website is fully accessible by a search engine is to provide direct links, preferably starting from the home page with no more than a total of three links to get to it. Another way is to provide a sitemap structured for the search engines.