By using this site you agree to the use of cookies by Brugbart and our partners.

Learn more

Avoiding duplicate content

How to avoid duplicate content issiues, resulting from poorly configured servers, and poorly developed CMS systems.

Edited: 2012-10-15 18:47

Content duplication usually happens as a result of badly configured servers, or webmasters using poorly developed CMS systems. A good CMS system will never produce identical URLs with the same content, given that it has been installed properly.

But it can also happen if someone steals your content, and keeps it on their own website.

Note. Such violations should be reported to the relevant search engines.

Effect on Search Engines

Some search engines might try to correct these issues. But that dose not change the fact, that its your own problem, its your server that is messed up, and its your CMS system that is messed up.

If a search engine encounters the same content at two pages, it often has to guess which one is the real one. The below URLs holds exactly the same content, but are entirely different, and could very well have different content.

Same as below URL:

To solve that problem, Brugbart recommends that you 404 out your index page

If however you have been using the index page for a long time, it may be better to do a 301 redirect to your domain. Brugbart recommends that you don't use a index.html, home.html, or whatever you may have called it. This is mainly to keep things as clean as possible to the users. When a user links to your website, they are more likely to just type your domain, then they are to type in index.html. For example, Brugbart uses the below URL format for our front page:

URLs are case sensitive

Not all servers are configured for case sensitivity, which could create duplicate content if someone suddenly gets the idea to link to your site, using caps in the URL, where there are no caps.

Above example has the same content as the below, but is two different urls.


This can especially be a problem for database driven sites, since they would often just request whatever used as page ID, and return it to the user, without checking for caps.

This can however be easily fixed, by doing a 301 redirect to the correctly capitalized/un-capitalized URL. One way to do this, is to check if the title of the database entry, has a different capitalization then the requested URL.

Keep in mind that this only becomes a problem, if the title was used as a part of the URL.

Slashes in URLs

Even the slashes used at the end of directories, can result in duplicate content.

The above is not really a directory, but more likely to be a page, requested from the database. Normally the first example would just be redirected to the latter, if there was a directory with that name, and if the server isn't configured to do something else, like requesting the page from the database.

You can read much more about this in the article: Trailing slashes in URLs


The protocol used to access the page, is also considered part of the URL. So this means that http, is different from https.