1. If Your website is a very popular site in your niche and getting lots of traffic from search engines, it means that your website URLs are crawled very highly and this makes it easy for scrapers to steal the content and make a MADE FOR ADSENSE(MFA) sites putting your content.

2. Some are marketing analytics for advertising companies to gather data about you and your company and sell it to advertisers for profit. The marketing strategies involve continuous observations on following factors

In the recent times, Many people seem to post about sitemap.xml suffering a problem with content. In the sitemap you give a title, description and URL of the webpages in your website

Is the new content title and meta tag scraped before the sitemap is submitted to google by sitemap generators? And the Answer is YES

The sitemap.xml file hands over a list of urls of website directly to any scraper who wants to make use of it for cloaking

Cloaking is primarily used to show an optimized page to the search engines and a different page to humans

Excessively scraped sites can struggle in the SERPs- This means that When someone mirrors your content it’s possible for your page/site to get hit with a duplicate content penalty.

Some Ideas to make it hard for Scrapers

But….

Any time you give scrapers a clear path to avoid honey pots and spider traps they’ll use it. With that said, the scrapers can simply scrape a search engine first using site:mydomain.com to get the equivalent of a sitemap and avoid your spider traps anyway.

But the number of keyword phrases that can cause the penalty depends on other factors like

1. Indexing
2. Weighting
3. Ranking
4. Duplicate content
5. Spam detection and weighting (yes – not all spam is bad?)
6. Back Links/Link Profiles
7. Personalized Search – code can be added on the page by spammer to lead the user to a network to a spam(”keyword stuffing page“) websites.

So What does Spam filter of Search engines do?

Spam Assassin
A Phrase Based Indexing and Retrieval (PaIR) system could set about identifying a web page (or document) as a spam page by comparing the actual number of related phrases present in the document with the expected number of related phrases. To high or to low an expected phrase rate or density, could flag a given document for further algorithmic inspection. Also expectations of plurals and singular occurrences can be valuated as part of the spam detection process. It is then added to a list of spam documents.

The whole process takes place both at indexing and retrieval. In essence the document gets its spam score at indexation and then upon retrieval, should that page be included in the results, weighting is then removed and the page is devalued during the ranking process for previously calculated Spam threshold scoring/weighting.

According to the folks that drafted it, a normal related, topical keyword phrase occurrence (or related phrases) is in the order of 8-20 whereas the typical Spam document would contain between 100-1000 related phrases. So by looking for statistical deviations in related phrase occurrences the system can flag an item as Spam. Once again it is mostly for the high end, but a low deviation count can also be used as a flag for a low occurrences (which could be compared to the link profile for link spam)

So how to identify the right keyword phrases from artificially created keyword phrases?

The goal is to classifying each potential keyphrase as either “a good phrase or a bad phrase” depending on it’s usage and frequency; then using those ‘good’ phrases in predicting the usage of other ‘good phrases’ in the collection of web pages.

E.G: If a normal webpage contains about 8-20 repetitions of a popular keyphrase, the same keyphrase could be present about 100- 1000 times on a spam document.

It is also important which blocks of the webpage influence the search engine to detect it as spam document -

For example, the phrase “President of the United States” is a phrase that predicts other phrases such as “George Bush” and “Bill Clinton.” However, other phrases are not predictive, such as “fell down the stairs” or “top of the morning,” “out of the blue,” since idioms and colloquisms like these tend to appear with many other different and unrelated phrases. Thus, the keyword phrase identification phase determines which phrases are good phrases and which are bad (i.e., lacking in predictive power).

But I suspect that phrase density can be very high, maybe 90% but still be deemed relevant. For example, a recruitment page about ‘Telesales jobs in London’ may have the phrase 50 times as it defines each job advertised and this could be a very good page. If the phrase density is very high then google will look for the associated phrases in pages clustered with that page. If the clustered page has the phrase in a low density, e.g. on a specific job description page, then the associated phrases should be on that page. If they are, then the page with a high density which is clustered with it, is still deemed to be OK.

So I believe it is applied on a cluster basis. It’s all about website navigation and offering the user the best entry point into a website, rather than automatically offering the highest scoring page which may be out of date.
I think it can be good method to identify “good phrases” in pages’s clusters to detect spam because google engineers can detect a bunch of spam related terms.

For those of us not interested in spam prevention but in rankings, do you think this model can be used with a few billion of pages with a zillion of terms in a few dozens of languages?