What is duplicate content and why should you care? Google’s Matt Cutts has said many times, there is no duplicate content “penalty” (you won’t be kicked out of the results), but it can substantially impact your rankings. When you have duplicate content search engines can’t determine which page should rank for a given phrase. Perhaps worse, if some people link using one address (URL), while others link using another address, neither page will have the credibility that it would if all the links pointed to a single address.
This has long been a problem on large database driven sites where only a minor difference in the address (URL) of the content (perhaps only a session ID) makes it appear that the same information is on more than one page. However this is also a common problem on Blogs where a Category listing, Tag listing, or Archive listing can present the exact same content, but with differing addresses.
This looks at some ways that duplicate content can occur.
1) True Duplicates – Any page that is 100% identical to another page other than the page’s address (URL).
This could result from that session ID id issue I mentioned:
2) Near Duplicates – Pages that differs only slightly, perhaps only an image or sidebar is different.
This could occur from differing sorts of the same information (category, tag, archive, …):
3) Cross-domain Duplicates -Occurs when two websites share the same piece of content.
This could occur from just shear theft (someone scraping your content) or from syndicating articles:
While the SEO crowd has been preaching the pitfalls of duplicate content for sometime, the farmer or “panda” updates that occurred throughout 2011 have pushed the importance of keeping duplicate content off your site this to the forefront. Peter Meyers covers the topic in more detail here, along with some ways to address these problems.
Duplicate Content has long been an issue with webmasters and is cause for concern because duplicating content dilutes your ranking in search results. Perhaps the most common cause of having duplicate content is in allowing both www. and non www versions of your site to exist. However, another place where I see duplicate content occur is when a customer has multiple domain names pointing to the same content. When you use multiple domain names, and there are good reasons to do this, you should pick a preferred URL and ensure that any secondary domains are 301 redirected to that URL.
In this video, Google’s Greg Grothaus does a great job explaining duplicate content problems and solutions ranging from 301 redirects to canonical tags.
I have previously blogged about how 301 redirects can prevent common problems from diluting the search engine ranking for your site. However, there are however a number of situations that create duplicate content concerns that are more difficult address. Webmasters now have additional tools to deal with duplicate content through Google’s Webmaster Tools panel. Through this addition to webmaster tools, Google Lets You Tell Them Which URL Parameters To Ignore. This will a great help, especially for many large dynamic sites.
This article has been updated to address additional questions received on this topic.
I am frequently asked about whether to use the www or not in the URL. While many of us are predisposed to typing the www, most of the time including the www is immaterial as typing either www.mywebsite.com or mywebsite.com will usually take you to the same place. So which should you use and when?
The answer is either, but not both and here are specific cases to consider:
Declaring one or the other is important as search engines will see www.mywebsite.com and mywebsite.com as different websites and index each independently (Google lets you declare a preference in Webmaster Tools). This has the undesirable effect of dividing the rank for your content between the two sites. While this may seem odd, www is technically a sub-domain and legitimate sub-domains such as jobs.yourdomain.com, support.mydomain.com, and so on are common.
So why does it matter, wouldn’t having two versions of your site in the engine double you chances of being found?
Let’s say you have a product page called product.html and three clients place links on their websites to the page as www.mywebsite.com/product.html, while two others link to to the page as mywebsite.com/product.html. All other things being equal, if a competitor had five links to a single page www.competitorwebsite.com/product.html, the competitor’s page would outrank yours.
To avoid having a duplicate version of your website in the search engines, decide whether to use the www or not and create 301 re-directs (typing the undesired one will redirect to the desired version) to enforce that decision. Then add both to Google’s webmasters tools and declare the preference in site settings.
