Under the Iceberg
Over the past two years we have been trying to bring attention to the real danger of click fraud. It is a real problem that is getting worse not better. Since we began reporting our Click Fraud Index, the overall rate has climbed over 20%. This problem has been highlighted in mainstream publications including Business Week, USA Today and the Wall St. Journal. No one today denies that click fraud is a problem and that it is having a negative effect on the growth of the online industry.
What may be less obvious is that click fraud is only the tip of the iceberg. The bigger issue for our industry is the overall decline of traffic quality. Advertisers want to get what they pay for and know that the traffic they buy has value. Problems including: the growth of botnets, out of country traffic and other low quality traffic sources like made-for-ad sites and parked domains are hurting ROI. Advertisers know this and have been demanding action from ad providers. One recent example is the poor quality traffic that comes from social network sites like MySpace. Google surprised Wall St by missing Q4 earnings due, in part, to their inability to monetize MySapce traffic. Social network sites are notorious for having very low SiteScore’s.
So what is happening? Smart ad providers are taking matters into their own hands. They can’t afford to lose business and they are listening to their customers. By using real time tools to detect invalid traffic, publishers can block it, redirect it or re-price it. This is the way the industry is moving and we are working hard to lead the way. It is encouraging to see ad providers like Yahoo see the dangers ahead and help their clients steer clear. On the other hand, it is concerning others are on a collision course.
Why Yahoo! Matters
A couple of weeks ago at Search Engine Strategies in New York, Click Fraud made big news. Yahoo announced a partnership with Click Forensics that changes the tone of the ongoing “Click Fraud Debate”. Since late 2005 there has been denial, litigation, finger pointing, 17 page reports and lots and lots of media coverage around the topic of click fraud. In March of 2006 I wrote that, “It will take a community approach to solve the problem”. Since then the community of advertisers, agencies, third parties, publishers, ad networks, industry organizations and search providers has grown. We have all been drawn closer together to help solve the problem; not deny its existence. A visible result of that progress was the Yahoo announcement.
It was just over a year ago that I first met Reggie Davis. Yahoo was the first search engine to name a Vice President over Marketplace Quality. Reggie’s approach has been refreshing. He listened carefully and took notes to what advertisers were saying. He worked with his team to implement solutions that helped improve transparency. And now, less than a year later, Yahoo is the first search engine to work with a third-party to fight click fraud.
This is real progress and a sign of things to come. For quite some time we have been drawing the distinction between traditional media (TV and Radio) and online. Advertisers are used to having standards, auditable invoices and notarized affidavit’s confirming they get what they paid for. It’s especially important because as pay-per-click industry expert, Dr. Tuzhilin, has noted: third-parties have access to data search engines don’t have. And that information is helpful for identifying quality issues such as click fraud. Yahoo! understands this and we’re now able to share this information to help Yahoo! help its advertisers.
Stealing Clicks
Forbes reporter Andy Greenberg recently published a story with dueling perspectives from my friend Shuman, of Google, and me highlighting our differing views of the click fraud problem. It got me thinking about how far the community has come toward solving the click fraud problem and how far we still have to go.
It was just a little over a year ago when search engine giants – Google, Yahoo! and Microsoft – publicly pronounced their support for the development of industry standards for measuring click fraud with the Interactive Advertising Bureau (IAB). Around the same time they also promised to work directly with third-party click fraud auditors on solutions to the click fraud problem. Although many were skeptical at the time, we thought these promises were important steps in the right direction. After all, just a few months earlier search engines, like Google, labeled click fraud “immaterial.”
Looking back, however, it’s clear not much has been done by search engines to turn these two important promises to advertisers into action. The standards development process, which we also joined to help ensure the voice of advertisers was heard, has lumbered along. We’re still waiting for the IAB to deliver its recommendations for click fraud measurement standards and open them up for public comment. We’re also waiting for Google and the other search engines to work with third-parties. While they have launched traffic quality resource centers and done lots of talking, there has been little meaningful action.
Meanwhile, back at the ranch, traffic in the publisher networks (including Google AdSense) is abysmal. Over 70% of the sites that make up these networks are made-for-ad sites or parked domains. Well over 60% of the traffic from these types of sites is traffic advertisers should not be paying for. Instead, advertisers should be actively excluding poor performing sites and avoiding low quality ad networks. But it’s is daunting without the help of search engines. Knowing which ones to exclude is a dynamic process that is tough to do when your information is limited.
So the question is, why the delay by search engines on all these fronts? There are many possible reasons but I believe the main one is economic incentive. When a company doesn’t have an immediate or near-term financial incentive to change, then it won’t. One only needs to look at the stock prices of certain search engines to see the economic incentive just isn’t there.
I do believe there is a light at the end of the tunnel. Advertisers are starting to pull back on their online spending because of quality issues. It’s only a matter of time before search engines feel the pinch. The spark may just well have been lit by Google’s recent success.
Related Article:
Forbes reporter Andy Greenberg - 09.24.07, 6:00 AM ET
Profiting from Click Fraud
A day or so ago, Shuman Ghosemajumder from Google posted two blogs on his personal site questioning Click Forensics data, methodologies and motivation. While it is really unfortunate that we have to take time to address these issues, I will. It is unfortunate because Shuman and I have had numerous phone conversations, email exchanges and meetings over the past nine months and we discussed many of the topics he brought up in his post. Let me begin…
1) Many third parties have not counted clicks properly Click Forensics counts clicks properly. We do not count back clicks. We count page views correctly. As a matter of fact, we use click ID’s (including Google’s) and other methods to ensure that our reports are the most accurate in the industry. 2) Inflated click counts result in even more inflated “click fraud” estimates Google calls them “invalid clicks.” Click fraud is a subsection of this category and does include malicious attacks, competitive clicks and affiliate fraud. Shuman has said on many occasions that Google does not charge for invalid clicks. This is a game of semantics. Our report focuses on what we call “high threat level clicks.” These are clicks that our rating engine and algorithm identified as being unwanted by the advertiser. These are clicks that advertisers should not be paying for and the reality is that in Q4 of 2006 14.2% of all clicks we scored fell into this category. 3) Even if they fixed those problems, they’re not actually measuring click fraud Actually the truth is Click Forensics measures and reports on all suspicious click activity and attempted click fraud across search engines and their content network advertising channels. Google has admitted that is has not charged for some clicks. Shuman told Andy Beal that this number was .2% of all clicks. Our advertisers (and Google’s) can see the number of clicks that they credited and it is far lower than the number of unwanted clicks search engines credit them for in their campaigns. As a matter of fact, many of the 3,000 advertiser members of the Click Fraud Network report that they have never been discounted more than 2 percent for the invalid clicks. It is clear search engines are not catching all bad click activity. Part of the reason is their lack of advertiser-side information and the difficulty in policing content networks alone. 4) Industry metrics are not necessarily the same as Google’s metrics True. Our industry metrics a representative of virtually every search provider large and small in the industry. This is why we have a statistically accurate view of the true rate of industry click fraud levels. We used to separate our click fraud figures by search engines tiers but stopped reporting this way this quarter because there is a growing disparity between individual search providers as to the click fraud levels. We didn’t think it was fair to lump the good with the bad so we will begin to report on individual providers later this year. The results may surprise you. 5) ROI on the content network is the same as it is on search That is an interesting way to spin the story. The fact is that content networks produce a higher amount of fraudulent and invalid click activity. We reported that 19.2% of clicks coming from content networks are fraudulent and fall into the high threat category. I’m glad Google agrees with us on this one. Its clear advertisers have been justified in their views on the click quality of content networks because the threat level is so high. We help advertisers identify publishers in the content networks that they should avoid. This approach actually helps improve the ROI on all content networks and gives advertisers peace of mind, thereby accelerating the growth of pay per click advertising. Shuman’s first post ends with this, “The key point here is not that their (Click Forensics) numbers are "too high". The point is that their data collection methods are inherently flawed and any resemblance their numbers could have to reality would be coincidental. Even so, given that they are not measuring click fraud, they apparently don't intend their numbers to reflect reality.”
At Click Forensics we stand by our numbers, our methodology, our people, our clients, our partners and the 3,000 plus members of the Click Fraud Network. And we are not alone in defending our approach. In March 2006, our methodology was examined by Dr. Tuzhilin, the same expert who examined Google's click fraud practices and wrote the report as part of the $90 million click fraud settlement case. Dr. Tuzhilin said:
“Click Forensics has good data and this is a source of their advantage over the search engines. My role is to work with them to refine the scoring methodology to improve accuracy. Their approach is to incorporate as much data as possible to improve accuracy. The search providers simply don't have enough data to have the most accurate approach."
The final false accusation from Shuman, responding to a post, charged us with releasing data to “…promote flawed industry estimates in order to attract business.”
Nothing could be further from the truth.
We are not compensated based on how much fraud we find. But each time a fraudulent click occurs and is missed by a search engine, they get paid for it.
Our goal is to continue to build a great company than helps advertisers ensure they get what they pay for in the tradition of Nielsen and Arbitron. Google would do well to take a less confrontational and more cooperative approach embraced by their competitors. That said, I look forward to discussing these issues with Shuman and continuing to work together to accelerate the growth of online advertising by fostering trust and accountability.
SES Chicago 2006
I am on my way back from Search Engine Strategies (SES) in Chicago. What could have been more fun than another click fraud panel this time in sunny Chicago in December! Honestly, I have been looking forward to participating with the same group we had in San Jose last summer. The discussion was lively and I have a deep respect for the other panelists.
Jeff Rohrs did another exceptional job laying out the issues surrounding click fraud. Since we last met there have been several major developments around this topic. I thought I would update you on those as well as provide you a recap of SES.
First is the work of the IAB and the Click Measurement Working Group. The process is underway to help define industry standards surrounding click fraud. Click Forensics is deeply involved, having met in person with both the IAB and the Media Rating Council (MRC). We take our role seriously and have dedicated time and resources to help this important initiative. All four major search engines are engaged as well as other industry players. John Slade from Yahoo! gave a brief recap saying that two meetings have been held and all the stakeholders are working well together.
Secondly, the formation of the Click Quality Council (CQC) signals advertiser interest in the issue of click fraud. The CQC held its first meeting the week of SES and will be focused on developing solutions from an advertiser perspective to improve click quality for our industry. The CQC and the 3,000+ members of the Click Fraud Network represent a large percentage of all online spending. It is critical that their collective voices are heard as we work together to improve the industry. I pointed out on the panel that the Wall Street Journal recently reported that online advertising revenue growth is flattening. Clearly, part of this is due to a lack of standards, clear measurement standards and a deficit of transparency into the process. My hope is ’07 will provide some movement on these issues.
As you know, Q4 represents the most active quarter of the year for pay per click advertising. The Click Fraud Index released November numbers indicating the click fraud activity increased to above 14%. This is not surprising given the increase in activity. If you have looked at the increased spam in your inbox you will understand! Stay tuned for more information on this.
There was a great deal of discussion around the “friction” that exists for advertisers who find click fraud in their campaigns in submitting that information. Several questions from audience members made known their frustration. While both Shuman and John commented that they want to work with advertisers, it was not enough. Our feeling is that there needs to be a process for advertisers to easily submit findings, understand the levels of invalid activity and have a defined reconciliation process. Click Forensics will continue to work on this.
Lori Weiman and Jessie Stricchiola did a great job discussing the lawsuit updates. Jeff mentioned the recent BusinessWeek article titled “The Vanishing Click-Fraud Case” (http://www.businessweek.com/technology/content/dec2006/tc20061204_923336.htm) and that sparked some good discussion.
We all survived the 14 degree temperature in Chicago and all in all felt that SES was well worth the trip. Thanks to my fellow panelists and I look forward to New York!
Announcing the Click Quality Council
The Google Guy
The Importance of the MIT Sutdy
The Number One Issue
The discussion around click fraud is no longer an advertising industry topic. There has been a growing interest from the financial community over the past few months. I have had several analyst meetings, spoken to numerous reporters and have met with leading advertiser and agency executives. There is one common thread to these meetings… they are all concerned about the impact of click fraud on our industry. This raised awareness is a great step forward, benefiting the entire industry by helping to create a positive dialogue between search providers and advertisers to begin partnering towards a solution to the problem of click fraud.
In the past few weeks there have been numerous articles discussing everything from bot attacks to lawsuits. I have appeared on CNBC twice to discuss the impact of the Click Fraud Index reporting at a 14% threat level. While this coverage is good for the industry, in my opinion the most critical issue facing our industry is not how we define click fraud, but how we communicate concerns to the search providers. We believe Click Forensics has the most complete dataset, the most advanced algorithm and the most accurate approach to identifying unwanted clicks. But what can an advertiser do with that information? Currently, just about all they can do is send in log files, excel sheets or hope the ROI on their campaigns will improve over time. It is this situation that is causing the growing frustration in the industry between advertisers and the search providers. In other media (TV, radio and print) standards exist for advertisers and publishers to agree on delivery. The answer for the online community is to establish and agree on a way for advertisers to submit their concerns to the providers. Becky Quick from CNBC asked me if I thought there would be a deal in place before the end of the year. The answer is yes. Stay tuned as I will have more to say on this topic at the Bear Stearns Internet Roundtable in New York next week. Tom CuthbertAdTech Update
Mark Cuban: Agree or Disagree
Dr. Tuzhilin Visits Click Forensics
Click Forensics has been working with Dr. Alexander Tuzhilin from NYU to further enhance our rating engine and statistical modeling approach. Dr. Tuzhilin visited with the Click Forensics’ team in San Antonio last week. The company has a goal of producing reports with the highest degree of accuracy in the industry. [caption id="attachment_693" align="alignright" width="248" caption="Dr. Tuzhilin works with the Click Forensics executive team at the San Antonio headquarters"][/caption] Dr. Tuzhilin said, "Click Forensics has good data and this is a source of their advantage over the search engines. My role is to work with them to refine the scoring methodology to improve accuracy. Their approach is to incorporate as much data as possible to improve accuracy. The search providers simply don't have enough data to have the most accurate approach." The work of Dr. Tuzhilin has helped Click Forensics build a road map for the future. We appreciate his insight and expertise and look forward to working with him for years to come.
Click Fraud Network Update
I want to take a moment and provide you some feedback from the Click Fraud Index and share some recent updates. The overall “high threat level” click rate is leveling off. As the number of advertisers increases in the Network, we are seeing this number settle down to around 14% of all clicks from all categories. Tier 1 search providers have a significantly lower threat level than those in Tiers 2 and 3. This would make sense because they have more internal resources to focus on the problem. While this is good news it still means that click fraud could account for over $750,000,000 in 2006. The search providers simply do not have the data available to resolve this issue. There is a growing call for an independent third party to validate clicks in the same way the Nielsen, Arbitron and the ABC exist in traditional media. We will be commenting more on this topic in the coming weeks.
Another interesting development has been the spyware issues relating to affiliates sites as pointed out by Ben Edelman. (His comprehensive studies have validated what we have been watching for a long time) Affiliate networks can be a risky place to advertise without a monitoring tool in place. This issue will grow more complex as time goes by and technologies advance. The exposure and opportunity for click fraud using spyware, bots and organized “clickers” is a real threat that is increasing daily.
We are thrilled by the response to the Click Fraud Network and that so many of you are taking advantage of our CF Analytics tool, offered to advertisers at no charge. I hope you will encourage others to “Join the Network”
Plan to visit us at the upcoming AdTech show in San Francisco April 26th to 28th. We are exhibiting at booth #6366. I am also looking forward to being part of the Click Fraud panel discussion lead by Jessie Stricchiola on Thursday April 27th at 3:15 PM. I hope to see you all there for some lively discussion!
Best regards,
Tom
The Real Problem with Click Fraud
The core of the issue is indeed that “no real data” exists. How big a problem is click fraud and is it growing? While we don’t claim to have all the answers I do believe we have a more statistically significant dataset than anyone else published at the Click Fraud Network site. Our member base has grown significantly since the Network was launched a few weeks ago. We are reporting that the instance of “high threat level” clicks is currently at 16.4%. As our Network grows we will be able to announce threat levels by vertical market, search provider and other relevant subsets. The industry will see that there is a big disparity across industries and providers.
Looking at individual campaigns or advertisers and trying to draw conclusions is pointless. Our approach is to aggregate the data by providing our tool at no cost to the 90% of PPC advertisers who have less than 100,000 paid clicks per month. This “collective intelligence” provides protection for them while helping the industry by revealing the scope and nature of click fraud. Similar to the approach taken by spam filters or anti virus tools, the Click Fraud Network data powers the rating engine making our algorithm as accurate as possible.
Henry is on to something. We want to help answer the questions. Click fraud is not “immaterial”, so rather than speculate, let’s work together to peel back the onion. Advertisers should Join the Network and let’s solve the problem.