“Privacy” is a hot topic in the world of marketing analytics, driven primarily by shifting consumer (and, in turn, regulatory) sentiment on the subject. That shifting sentiment, I think, is largely being driven by the increasing integration of social media into our lives and our online behavior.

The WAA stepped up and put together a Code of Ethics a few months ago, and privacy is going to be a recurring topic at eMetrics and other conferences for the foreseeable future. Following the San Francisco eMetrics conference, Stéphane Hamel put together three scenarios and asked the #measure community to vote as to the ethics and allowability of each situation. He then revealed the results and added his own thoughts. Towards the end of that second post, Stéphane noted that he was disappointed by the lack of interest in the exercise, given the generally accepted importance of the topic.

Emer Kirrane responded in the comments:

It’s interesting that there seems to be a correlation between legality and ethics in the minds of your respondents. To me, the Code of Ethics is there as a flag against practices that are deemed unethical by the community, rather than deemed unethical by law.

Stéphane’s concern and Emer’s response have been bouncing around in my brain for several weeks. My conclusion: “ethics vs. legality” is going to continue to give us fits.

I realize this isn’t the first time that “ethics” and “the law” haven’t perfectly aligned (they almost never do, actually, even though that, from a purist point of view, is the goal), but bear with me — it’s worth using that lens to explore the issue and outline the challenges we’re going to have to deal with. These are two very different dimensions of the privacy debate, and one of them is in flux on several fronts.

Why 2.5 Dimensions?

Obviously, there is a legal/regulatory dimension, and there is an ethical dimension. But, really, the legal/regulatory dimension is heavily driven and influenced by consumer perceptions and fears. I actually wrote some thoughts on that a couple of years ago. With high-profile Facebook snafus and high-profile media outlets reporting on cookies and cross-site tracking, politicians have found an issue that their constituents care about (or can be prodded to care about). So, in a sense, the legal/regulatory dimension has some added “oomph” of consumer concerns behind it; I’m calling that “consumer perspective” another half a dimension.

It’s possible that “consumer perception” should be a third dimension in and of itself. But, oh boy, that would make for some hairy sketching in the remainder of this post. I’m pretty sure I’m not just punting, though — the will of the consumer when it comes to something like privacy does generally get manifested through some form of government regulation.

Start with the Basics

Two dimensions: legal and ethical. We can look at them like this:

Various practices raise privacy questions. In theory, we can plot each of them on this (conceptual) grid — there are more than shown here, but I’m just laying out the basic idea of the framework:

In Theory, We’d Have Harmonious Dimensions

If life was simple, we would have perfect clarity for each dimension, and perfect alignment between dimensions:

Notice the shaded quadrants at top left and bottom right — there would be no practices that were ethical but not legal, nor would there be any practices that were legal but unethical.

Alas! Privacy is Rife with Gray Areas!

Reality is more like this — gray areas rather than hard lines along both dimensions:

Ugh. Things get messy. There are more activities that are questionable — they may or may not be legal and/or they may or may not be ethical! Argh!

But Wait! There’s More!

Ever since the web went mainstream, it’s been a more global medium than anything that came before. And, we’ve all run into cases and concerns that our standard web analytics implementation runs afoul of the law in some country somewhere. This grid illustrates that wrinkle, too — the legal/regulatory gray areas live in different places depending on the country (only the U.S. and the E.U. are shown here — it’s an illustrative diagram, people! Not a comprehensive one!):

And the big blue arrow shows where pressure is being applied (back to that half-dimension of consumer fears mentioned at the beginning of this post). It’s a little counterintuitive that the arrow is pointing upward, isn’t it? How could it be that things are trending towards “allowed?” They’re not. Rather, the “interpretation zone” is moving upward — practices that used to be “clearly allowed” aren’t inherently changing what they are, but those practices are moving from “in the clear” towards the gray area.

Helpful?

This was definitely one of those situations where, when I initially had a rough picture in my mind that would represent these two dimensions, it was simple and clear. It was only as I put pen to paper to sketch it out that it turned out to be tricky. Shortly after I finished writing this post (but, obviously, before I published it…as I’m adding this comment at the end), Jason Thompson made a really good case as to what is (misguidedly) driving the legal dimension out of alignment with the ethical perspective. That reminded me that I keep meaning to go back and re-read the last chapter (chapter 9?) of Jim Sterne’s Social Media Metrics book, as I recall that it was an intriguing non-sequitur that considered turning the entire “tracking” model on its head. Food for thought for another post, that.

What do you think? Is this an effective representation of the shifting privacy landscape we’re dealing with? What does it miss?Similar Posts:

• Data Portability vs. Privacy

© Tim for Gilligan on Data by Tim Wilson, 2011. | Permalink | 3 comments | Add to del.icio.us
Post tags: privacy

Feed enhanced by Better Feed from Ozh

• Banner ad clickthroughs reported by the ad server don’t match the clickthroughs reported by the web analytics tool
• Visits reported by one web analytics tool don’t match visits reported by another web analytics tool running in parallel
• Site registrations reported by the web analytics tool don’t match the number or registrations reported in the CRM system
• Ecommerce revenue reported by the web analytics tool doesn’t match that reported from the enterprise data warehouse

In most cases, the “don’t match” means +/- 10% (or maybe +/- 15%). And, seasoned analysts have been rattling off all the reasons the numbers don’t match for years. Industry guru Brian Clifton has written (and kept current) the most comprehensive of white papers on the subject. It’s 19 pages of goodness, and Clifton notes:

If you are an agency with clients asking the same accuracy questions, or an in-house marketer/analyst struggling to reconcile data sources, this accuracy whitepaper will help you move forward. Feel free to distribute to clients/stakeholders.

It can be frustrating and depressing, though, to watch the eyes of the person who insisted on the “match” explanation glaze over as we try to explain the various nuances of capturing data from the internet. After a lengthy and patient explanation, there is a pause, and then the question: “Uh-huh. But…which number is right?” I mentally flip a coin and then respond either, “Both of them” or “Neither of them” depending on how the coin lands in my head. Clifton’s paper should be required reading for any web analyst. It’s important to understand where the data is coming from and why it’s not simple and perfect. But, that level of detail is more than most marketers can (or want to) digest.

After trying to educate clients on the under-the-hood details…I almost wind up at a point where I’m asked the “Well, which number is right?” question. That leads to a two-point explanation:

• The differences aren’t really material
• What matters in many, many cases is more the trend and change over time of the measure — not its perfect accuracy (as Webtrends has said for years: “The trends are more important than the actual numbers. Heck, we put ‘trend’ in our company name!”

This discussion, too, can have frustrating results.

I’ve been trying a different tactic entirely of late in these situations. I can’t say it’s been a slam dunk, but it’s had some level of results. The approach is to list out a handful of familiar situations where we get discrepant measures and are not bothered by it at all, and then use those to map back to the data that is being focussed on.

Here’s my list of examples:

• Compare your watch to your computer clock to the time on your cell phone. Do they match? The pertinent quote, most often attributed to Mark Twain, is as follows: “A man with one watch knows what time it is; a man with two watches is never quite sure.” Even going to the NIST Official U.S. Time Clock will yield results that differ from your satellite-synched cell phone. Two (or more) measures of the time that seldom match up, and with which we’re comfortable with a 5-10 minute discrepancy.

Photo courtesy of alexkerhead

• Your bathroom scale. You know you can weigh yourself as you get out of the shower first thing in the morning, but, by the time you get dressed, get to the doctor’s office, and step on the scale there, you will have “gained” 5-10 lbs. Your clothes are now on, you’ve eaten breakfast, and it’s a totally different scale, so you accept the difference. You don’t worry about how much of the difference comes from each of the contributing factors you identify. As long as you haven’t had a 20-lb swing since your last visit to the doctor, it’s immaterial.

Photo courtesy of dno1967

• For accountants…”revenue.” If the person with whom your speaking has a finance or accounting background, there’s a good chance they’ve been asked to provide a revenue number at some point and had to drill down into the details: bookings or billings? GAAP-recognized revenue? And, within revenue, there are scads of nuances that can alter the numbers slightly…but almost always in non-material ways.

Photo courtesy of alancleaver_2000

• Voting (recounts). In close elections, it’s common to have a recount. If the recount re-affirms the winner from the original count, then the results is accepted and moved on from. There isn’t a grand hullabaloo about why the recount numbers differed slightly from the original account. In really close races, where several recounts occur, the numbers always come back differently. And, no one knows which one is “right.” But, once there is a convergence as to the results, that is what gets accepted.

Photo courtesy of joebeone

That’s my list. Do you have examples that you use to explain why there’s more value in picking either number and interpreting it rather than obsessing about reconciling disparate numbers. I’m always looking for other analogies, though. Do you have any?

© Tim for Gilligan on Data by Tim Wilson, 2010. | Permalink | 3 comments | Add to del.icio.us
Post tags: match

Feed enhanced by Better Feed from Ozh

The quote that really jumped out for me in the post, though, was this one:

Anyone who understands BI, however, knows that no interface, no matter how magical, will give you access to data that isn’t available, will clean data that is dirty, or will simplify the navigation of complicated operational databases.

That quote alone warranted a mini-blog post, if for no other reason than to allow me to quickly get my hands on it in the future when I’m in the midst of bashing my head against a cinder block wall of requests for crisp, clean analytic insights from a messy, messy world of data.

Mr. Few, I already had you placed high on a pedestal. Please accept this footstool upon which you can perch to be raised up just a little bit higher thanks to the clarity and insight within that one sentence you have penned!Similar Posts:

• The Best Little Book on Data

© tgwilson_php for Gilligan on Data by Tim Wilson, 2010. | Permalink | 2 comments | Add to del.icio.us
Post tags: Stephen Few

Feed enhanced by Better Feed from Ozh

With that in mind, I started thinking about all of the sources of data that marketers now have available to them to drive their decisions. The challenge is that almost all of the data sources out there are good tools — while they all claim competitive advantage and differentiation from other options…I believe in the free markets to the extent that truly bad tools don’t survive (do a Google search for “SPSS Netgenesis” and the first link returned is a 404 page — the prosecution rests!). To avoid getting caught up in the shiny baubles of any given tool, it seems worth organizing the range of available data some way — put every source into a discrete bucket.  It turns out that that’s a pretty tricky thing to do, but one approach would be to put each data source available to us somewhere on a broad spectrum. At one end of the spectrum is data from secondary research — data that someone else has gone out and gathered about an industry, a set of consumers, a trend, or something else. At the other end of the spectrum is the data we collect on our customers in the course of conducting some sort of transaction with them — when someone buys a widget from our web site, we know their name, how they paid, what they bought, and when they bought it!

For poops and giggles, why not try to fill in that spectrum? Starting from the secondary research end, here we go…!

Secondary Research (and Journalism…even Journalism 2.0)

This category has an unlistable number of examples. From analyst firms like Forrester Research and Gartner Group, to trade associations like the AMA or The ARF, to straight-up journalists and trade publications, and even to bloggers. Specialty news aggregators like alltop.com fall into this category as well (even if, technically, they would fit better into a “tertiary research” category, I’m going to just leave them here!).

I stumbled across iconoculture last week as one interesting company that falls in this category…although things immediately start to get a little messy, because they’ve got some level of primary research as well as some tracking/listening aspects of their offer.

Listening/Collecting

Moving along our spectrum of data sources, we get to an area that is positively exploding. These are tools that are almost always built on top of a robust database, because what they do is try to gather and organize what people — consumers — are doing/saying online. As a data source, these are still inherently “secondary” — they’re “what’s happening” and “what’s out there.” But, as our world becomes increasingly digital, this is a powerful source of information.

One group of tools here are sites like compete.com, Alexa, and even Google’s various “insights” tools: Google Trends, Google Trends for Websites, and Google Insights for Search. These tools tend to not be so much consumer-focussed as site-focussed, but they’re getting their data by collecting what consumers are doing. And they are darn handy.

“Online listening platforms” are a newer beast, and there seems to be a new player in the space every day. The Forrester Wave report by Suresh Vittal in Q1 2009 seems like it is at least five years old. An incomplete list of companies/tools offering such platforms includes (in no particular order…except Nielsen is first because they’re the source of the registration-free PDF of the Forrester Wave report I just mentioned):

• Nielsen Buzzmetrics
• BuzzLogic
• Radian6
• Alterian/Techrigy SM2
• Filtrbox
• Crimson Hexagon
• Collective Intellect
• Spiderfly

And the list goes on and on and on… (see Marshall Sponder’s post: 26 Tools for Social Media Monitoring). Each of these tools differentiates itself from their competition in some way, but none of them have truly emerged as a  sustained frontrunner.

Web Analytics

I put web analytics next on the spectrum, but recognize that these tools have an internal spectrum all their own. From the “listening/collecting” side of the spectrum, web analytics tools simply “watch” activity on your web site — how many people went where and what they did when they got there. Moving towards the “1:1 transactions” end of the spectrum, web analytics tools collect data on specifically identifiable visitors to your site and provide that user-level specificity for analysis and action.

Google Analytics pretty much resides at the “watching” end of this list, as does Yahoo! Web Analytics (formerly IndexTools). But, then again, they’re free, and there’s a lot of power in effectively watching activity on your site, so that’s not a knock against them. The other major players — Omniture Sitecatalyst, Webtrends, Coremetrics, and the like — have more robust capabilities and can cover the full range of this mini-spectrum. They all are becoming increasingly open and more able to be integrated with other systems, be that with back-end CRM or marketing automation systems, or be that with the listening/collecting tools described in the prior section.

The list above covered “traditional web analytics,” but that field is expanding. A/B and multivariate testing tools fall into this category, as they “watch” with a very specific set of options for optimizing a specific aspect of the site. Optimost, Omniture Test&Target, and Google Website Optimizer all fall into this subcategory.

And, entire companies have popped up to fill specific niches with which traditional web analytics tools have struggled. My favorite example there is Clearsaleing, which uses technology very similar to all of the web analytics tools to capture data, but whose tools are built specifically to provide a meaningful view into campaign performance across multiple touchpoints and multiple channels. The niche their tool fills is improved “attribution management” — there’s even been a Forrester Wave devoted entirely to tools that try to do that (registration required to download the report from Clearsaleing’s site).

Primary Research

At this point on the spectrum, we’re talking about tools and techniques for collecting very specific data from consumers — going in with a set of questions that you are trying to get answered. Focus groups, phone surveys, and usability testing all fall in this area, as well as a plethora of online survey tools. Specifically, there are online survey tools designed to work with your web site — Foresee Results and iPerceptions 4Q are two that are solid for different reasons, but the list of tools in that space outnumbers even the list of online listening platforms.

The challenge with primary research is that you have to make the user aware that you are collecting information for the purpose of research and analysis. That drops a fly in the data ointment, because it is very easy to bias that data by not constructing the questions and the environment correctly. Even with a poorly designed survey, you will collect some powerful data — the problem is that the data may be misleading!

Transaction Data

Beyond even primary research is the terminus of the spectrum — it’s customer data that you collect every day as a byproduct of running your business and interacting with customers. Whenever a customer interacts with your call center or makes a purchase on your web site, they are generating data as an artifact. When you send an e-mail to your database, you’ve generated data as to whom you sent the message…and many e-mail tools also track who opened and clicked through on the e-mail. This data can be very useful, but, to be useful, it needs to be captured, cleansed, and stored in a way that sets it up for useful analysis. There’s an entire industry built around customer data management, and most of what the tools and processes in that industry focus on is transaction data.

What’s Missing?

As much as I would like to wrap up this post by congratulating myself on providing an all-encompassing framework…I can’t. While there are a lot of specific tools/niches that I haven’t listed here that I could fit somewhere on the spectrum of tools as I’ve described it, there are also sources of valuable data that don’t fit in this framework. One type that jumps out to me is marketing mix-type data and tools (think Analytic Partners, ThinkVine, or MarketShare Partners). I’m sure there are many other types. Nevertheless, it seems like a worthwhile framework to have when it comes to building up a portfolio of data sources. Are you getting data from across the entire spectrum (there are free or near-free tools at every point on the spectrum)? Are you getting redundant data?

What do you think? Is it possible to organize “all data sources for marketers” in a meaningful way? Is there value in doing so?

© Tim for Gilligan on Data by Tim Wilson, 2009. | Permalink | One comment | Add to del.icio.us
Post tags: data sources, Malcolm Gladwell

Feed enhanced by Better Feed from Ozh

• status quo — the way things are, as opposed to the way they could be
• inertia — The property of a body that resists any change to its uniform motion
• cognitive dissonance — a conflict or anxiety resulting from inconsistencies between one’s beliefs and one’s actions or other beliefs

The first two of these can be applied to any sort of technology or process change being introduced to an organization — entire careers and companies are built around trying to figure out how to effectively drive change within organizations. In the case of data management, the third defintion — cognitive dissonance — comes into play as well.

As a brilliant and phenomenally handsome man* once said, “Customers are people, and people are messy.” Customer data is inherently incomplete and imperfect. Any process or system that captures and stores customer data stores flawed data as soon as it rolls out for two reasons:

• It is not reasonable to add to any process all of the overhead required to rigorously capture and validate all attributes of a customer — it’s a balancing act between the efficiency of the process and the quality of the data captured
• Customer data decays, and it decays a lot more quickly than we like to admit; customer data maintenance tends to be an afterthought that gets addressed only after time has degraded the data to the point that it starts causing the company real problems

Once we hit the point where we really need to tackle our customer data management head on, we have two options, of which one option is completely inviable:

1. Throw out all of our customer data, customer data processes, and customer data systems and start over, but “do it right this time”
2. Identify the most broken parts of our processes and start fixing them — going after the lowest cost and highest benefit ones first and then working our way down the list until we hit a satisfactory point (which is, typically, never)

Clearly, the first option is not an option. No company would survive if they tossed out their customer base, barred their doors, and conducted no business for a year or two while they rebuilt their process and technology infrastructure.

That leaves us with the second option (technically, “do nothing” is an option as well, but that’s only an option if the pain hasn’t reach the point where it’s not an option!), and, thus, we reach a cognitive dissonance conundrum:

We know our customer data is dirty — customer service reps complain about the number of duplicate records in their systems, sales reps complain of the incomplete pictures they have of their customers (which hinders their ability to prep for and conduct customer visits), marketing complains that they can’t effectively segment and target their database because the customer data is bad, customers complain because the company keeps screwing things up in one way or another…

BUT

…as we start to explore and design replacement processes, we realize that these processes are going to be inherently imperfect, too. We may accept that the new process will be better (even significantly so), but we obsess about the flaws.

We don’t want to repeat the mistakes of the past and roll out something that is not bulletproof — a chink in the data management armor is a chink, no matter how small. So we obsess about the chinks. We propose process changes to accomodate the identified gaps. Even for the gaps that are purely theoretical (“yes, I see, but what if the poles reversed at the exact same point that pigs learned to fly — the process would break!”) We’re trying to do the right thing. We’re aiming for perfection — for a flawless process.

But we’re talking about customer data, and customers are people, and people are messy.

We find ourselves (and/or the people who will ultimately need to adopt the process) paralyzed, caught in an endless cycle of Visio vetting and process rework, perpetually getting halfway to the “perfect” process, but never actually getting there. At some point, due to impatience or frustration, someone stands up and yells, “Enough! Just build what you’ve got!”

And then we realize we’ve designed a process that is so complex and unwieldy that the cost to implement it would wipe out any hope of the company having a profitable year for the ensuing decade.

Of course you’d like a more tangible example:

Let’s say we’re trying to clean up our customer’s mailing addresses (which, thankfully, is now an exercise from my past, but that’s more a digression for a discussion over drinks than for a blog post!). Let’s say that, for any 1,000 customer addresses, we have conclusively demonstrated that at least 50 of them are bad — the postal service is going to struggle to deliver mail sent to them, and the postal service is going to fail more often than not. Now, let’s also say that we’ve demonstrated that, by introducing some automated cleansing processes, we can: 1) identify those 50 addresses, 2) “fix” 30 of them, and 3) flag the remaining 20 as being known problems that need some sort of manual intervention. Let’s say that, rather than 1,000 records, we’re talking about 10 million.

“Hurray!”

“Sounds great!”

“Awesome!”

“Gimme some of that!”

Ah…BUT…

…we have also  determined that, as part of those automated cleansing processes, we might actually take 1 of the 950 addresses that were already good…and make it worse.

Logically, the project should still be a go. We’re making 30 addresses better and only might be making a single address worse!

Ohhhhh…that single address. That molehill that eats its Wheaties, regularly applies cream provided by a shady character, and injects itself in the buttocks with a substance its cousin purchased over the counter in the Dominican Republic. The molehill grows. It grows quickly. Suspiciously quickly…yet no one seems to notice. It becomes a hill, and then a big hill, and then a mountain! The project manager is left scratching his head and wondering how a theoretical aside in a meeting three weeks ago has now become a virtually insurmountable issue that has put the entire project at risk of ever being implemented!

Cognitive dissonance — simultaneously recognizing that things are bad and must be fixed, but also accepting that the status quo is “right.”

The answer? I’d like to say it’s just a matter of putting the dissonant perspectives side by side and forcing objectors to reconcile them. That should work, right?

Alas!

As it happens, the current debate about healthcare reform in the U.S. prompted James Surowiecki to right a column on Status-Quo Anxiety in The New Yorker a couple of weeks ago. Surowiecki discusses the “endowment effect:”

“…the mere fact that you own something leads you to overvalue it. A simple demonstration of this was an experiment in which some students in a class were given coffee mugs emblazoned with their school’s logo and asked how much they would demand to sell them, while others in the class were asked how much they would pay to buy them. Instead of valuing the mugs similarly, the new owners of the mugs demanded more than twice as much as the buyers were willing to pay.”

Surowiecki goes on to relate this effect to the healthcare debate: “What this suggests about health care is that, if people have insurance, most will value it highly, no matter how flawed the current system.”

The same applies to customer data management all too often — we know we have a flawed system, but it’s the system we have, gosh darn it, and I don’t want your new system if I can find any imperfections in it!

This really has been a farewell post of sorts. Rambling, yes. Academic, yes. Lacking any prescriptive solution. But, hopefully at least a little entertaining, and maybe even with an insight or two that may come in handy to you. Look for a topical shift to measuring digital media going forward.

So long, and thanks for the fish!

* Dramatic license — I said that in this post, and “brilliant and phenomenally handsome” is perhaps a bit of an overstatement.

Photos courtesy of Olof S and bobster855

© Tim for Gilligan on Data by Tim Wilson, 2009. | Permalink | No comment | Add to del.icio.us
Post tags: cognitive dissonance, intertia, James Surowiecki, status quo

Feed enhanced by Better Feed from Ozh

Let’s Start with Definitions

Wikipedia has a nice write-up that goes into much deeper detail, but here are my quick and crude definitions:

• Type I Error = α (Alpha) Error = False Positive = you mistakenly judge something to be true that, in fact, is false
• Type II Error = β (Beta) Error = False Negative = you mistakenly judge something to be false that, in fact, is true

It’s easy to get mired in statistics-speak and make a snap judgment that this distinction between types of data errors are solely of interest and use to statisticians. That’s not the case. Depending on the situation, one type can be critical while the other can be barely consequential.

Customer Data Example: Automated Customer Merges

Most companies of any size battle duplicates in their customer data. I have yet to work at or with a company where the sales force and call center staff don’t complain about the number of duplicates that exist in internal systems. In this situation, it’s pretty common to try to automate a deduplication routine of some sort (some would say that this is the biggest benefit of a customer data integration, or CDI, initiative).

Unfortunately, customer data is messy. For every “identical match” in the system, there are generally 5 to 10 “probable matches.” After all, if the data was truly identical, then the likelihood that a duplicate would have gotten created in the first place would have been greatly diminished! That means that the automated matching system has to apply various business rules to determine which matches are true matches and which ones are not — the system makes an educated guess. Many CDI systems apply a scoring system — the higher the score, the more likely the two records are a true match. The lower the score, the less likely. It’s then up to the user to establish the threshold above which the records will automatically be merged (and, possibly, a lower threshold that will trigger a manual review of the records by a human being).

Where that threshold gets set is purely a Type I vs. Type II error decision:

• The higher the threshold gets set, the greater the likelihood of a Type II error — a false negative where two records could have been merged because they were, in fact, duplicates, but they were not identified as such
• The lower the threshold gets set, the greater the likelihood of a Type I error — a false positive where two records get merged, even though, in reality, they represent two different customers

It would be wonderful if the matching logic was such that a threshold existed that eliminated all errors, but no such threshold will ever exist. The question then becomes: which type of error is “more” acceptable? That will influence where you set your threshold. It’s a situational question! For example:

• If your customer data includes extremely sensitive information (think medical records, social security numbers, credit card numbers), then you will want to err on the side of Type II / False Negative errors — deal with the messiness of more duplicates and put the burden on your call center / sales force to trigger merges if and only if they have confirmed the merge needs to happen through a manual inspection and/or an interaction directly with the customer
• If, on the other hand, you are only using the post-merged customer data to send marketing promotions to the customers, then the added cost of the direct mail may push you to err on the side of Type I / False Positive errors — some of your customers may miss out on some marketing promotions, but very few of them will receive multiple copies of the same direct mail piece, and your overall postage costs will be lower

This is a very real example — a single company may use the former mentality as its core matching logic, but then maintain a separate data store with more aggressive matching for purposes where Type I errors are not critical.

Type I and Type II in the News

A friend and former colleague shared a recent story in The Washington Post about the Social Security Administration running into trouble when trying to deny social security benefits to certain felons. It sounds like there were some technical gaffs, for sure — when people with minor offenses on their record got caught up in the denial of benefits. Overall, this reads like it was a case of an unacceptable number of Type I errors — people being denied benefits because they were mis-identified as “fleeing felons” (the article reads like some of the stories of people being banned from flying because they had the same name as someone on the terrorist watch list).

Both types of errors will occur. As important as being clear on which type are “better” given the situation (and developing your processes accordingly), ensuring that you have processes in place to fix the errors when they do occur and are identified is just as critical — something that the federal government was apparently missing in this case! The federal government? Lacking in customer service? Shocker!

So, there you have it — more thought than you ever wanted to see on Type I and Type II errors. I also promised in my earlier post that a future post would cover my observations on cognitive dissonance and the status quo when it comes to customer data management, but that’s going to have to wait for another day!Similar Posts:

• Your Customer Data Is Dirtier than You Think

© Tim for Gilligan on Data by Tim Wilson, 2009. | Permalink | No comment | Add to del.icio.us
Post tags: CDI, customer data integration, false negative, false positive, Type I error, Type II error

Feed enhanced by Better Feed from Ozh

The math is pretty simple:

99% x 99% x 99% x 99% x 99% x 99% x 99% x 99% x 99% x 99% = 90.4%

Only 90.4% of the finished products would work? That doesn’t seem good at all! Considering that there are very few manufactured products — especially electronic ones — that have only ten critical parts, it was an eye-opening insight (albeit obvious in hindsight).

The point the professor was making was that there are many cases where “99% perfect” really isn’t good enough when that one part is considered in a larger context.

Of late, I’ve had a few run-ins with the opposite insight. Stick with me — it’ll be fun!

For starters, customer data management processes are not automated manufacturing processes. Customers are people, and people are messy!

In a manufacturing environment, a key way to drive quality is to remove as much variability as possible by strictly controlling the environment. Customers (people) are none too keen about being “strictly controlled.” From a pure (read: manufacturing) customer data perspective, what we’d like is:

• To have every human assigned a unique ID
• To have every human log into a system once a week and update all sorts of meta data about themselves:
  • Who they are related to and how (using those people’s unique IDs)
  • What products they own
  • How old they are
  • How much they weigh
  • What their favorite flavor or ice cream is
  • What political party they support
  • …and so on
• To enter all of this information from drop-down lists so that all of the data is structured
• To have them be very, very careful when they update this information, maybe even swearing that the data is perfectly accurate, under penalty of severe consequences

Obviously, that ain’t gonna happen.

Our processes to manage customer data are very different from manufacturing processes for one simple reason: the data does not have to be perfect. It has to be good enough for us to effectively interact with our customers.

Here is where a similar example to the one that started this post comes into play. When working on processes that deal with customer data — creating or maintaining it — we all develop use cases and scenarios to ensure that we are keeping the data as accurate as possible. It is exceedingly easy (and awfully tempting) to start working with scenarios that are theoretically ossible but not very probable. If we’re not wearing our Hat of Practicality, we will find ourselves developing processes that are so inordinately complex that one of two things happen:

• We never get the new process implemented because it collapses under its own developmental weight, or
• We implement it, but it is so complex that it collides with itself and starts generating bad customer data!

Is this sounding theoretical? I’ll illustrate with an example.

A couple of weeks ago, I ran into an issue that had to do with a new third-party data cleansing process that we are introducing that involved sending customer name and address data to a third party service (all over obscenely secure channels and with no more personal information than could easily be found in a phone book or through Yahoo! People). During testing, we came across some unexpected behavior as to how the third party vendor handled hyphenated last names. The initial proposal was to throw out responses for any customer who had a hyphenated last name. Something seemed amiss with that approach.

I thought up the most plausible scenario I could where the returned data would actually be incorrect, and it looked like this (I’ll spare you the details as to why this scenario was the most plausible — just trust me):

• John Smith marries Mary Jones and they both keep their original surnames
• They have a son named John Smith-Jones
• When John Smith-Jones is a teenager, he becomes a customer of the same company of which his dad is a customer
• When John Smith-Jones graduates from high school, he moves out of the house, while both he and his father remain customers of that same company

In this scenario…the process would be a little broken — in a way that the customer (the father, in this case) would probably understand and would definitely be able to easily correct.

So, here comes the math. Without doing any research beyond my own gut-based estimates from 37 years of experience on planet Earth, I made conservative estimates for all of the variables involved:

• The percent of all married couples in the U.S. where both parties have kept their original surnames: 1%
• The percent of all kids in the U.S. with hyphenated surnames: 0.5%
• The percent of all kids in the U.S. who share the same name as their mother or father: 2%
• How often a kid in the U.S. is a separate, distinct customer of the same company that his parents are (in the particular space this company is in) at the point that he/she leaves home: 75%

Then comes the math. It works just the opposite from the original equation, in that it is an “AND” situation rather than an “OR” situation — all of these factors had to be met in order for the process to make an erroneous customer data update (as opposed to any one of the components having to be defective in order for the final product to be defective):

1% x 0.5% x 2% x 75% = 0.000075% (!!!)

If my estimates were accurate, which they almost assuredly were not, then we would make this customer data error roughly once for every one million customers. If you think about it, you realize that the absolute accuracy of the small percentages just doesn’t really matter once those small percentages start multiplying. Let’s say I was off by a factor of four on my estimate of the percent of kids with hyphenated last names, so the formula above should have 2% where it had 0.5%. That ups the likelihood of this data error occurring to 3 times in a million rather than less than one — given the highly non-catastrophic nature of the error, this is still an “almost never” when it comes to looking at the types of other, more critical customer data errors that occur day in and day out.

In this case, there was another factor that I could have applied, and that was, for those one million customers, how many would be affected in any given year? 13% is a fair estimate of how many people move each year in the U.S., which means we would need to apply that percentage to the original result…and we’re back to “effectively never” for our likelihood of occurence.

There are a couple of caveats here, and they’re important:

• I came up with one scenario. If there were four other plausible scenarios that were all equally likely to occur, then I would need to multiply the final result by five. In this case, we’re still talking a very small number, but there may be cases where a particular process gap could cause problems in a long tail’s worth of scenarios and may need to be viewed differently
• It’s worth vetting the estimates somewhat — not through extensive research, necessarily, but at least by running them by a couple of sharp people to see if they pass the sniff test

In this example, we were deep into testing — well past the point where code updates could be made without introducing risk to the overall implementation. To me, it was a no-brainer — proceed as planned!

The pushback I’ve received in other, similar situations, has been: “Well, yeah, that’s only one person in a million. But…what about that one person?!” THAT gets us to my next post, which will be about Type 1 vs. Type 2 errors and cognitive dissonance when it comes to both knowing that the status quo is bad but also assuming the status quo is right. More on that next time!

Photo by Bill Burris

© Tim for Gilligan on Data by Tim Wilson, 2009. | Permalink | 5 comments | Add to del.icio.us
Post tags: equation

Feed enhanced by Better Feed from Ozh

In my current job, I’m a full-time customer data management guy. It is not sexy. Like many large companies, we’ve got customer data that is created and managed in a wide range of disparate systems on diverse platforms, each with multiple decades of system evolution. It’s important. It’s painful.

There are some great opportunities in our increasingly electronic and e-based world to make some real headway with data management. In the case of the guest blog post, I focussed on opportunities to use marketing automation tools and your web site to drive improvements in the quality of your customer data. As for how exactly I made the “high quality mattress” analogy? Click on over and check out the post!

© Tim for Gilligan on Data by Tim Wilson, 2009. | Permalink | One comment | Add to del.icio.us
Post tags: Eloqua, marketing automation, mattress, Steve Woods

Feed enhanced by Better Feed from Ozh

I had a professor in business school who used to explain the relationship between the stock market and the bond market as a teeter-totter (in rural southeast Texas, I grew up knowing this as a see-saw): as the yields on one went up, the yields on the other went down and vice versa. 

Managing your customer data can be like that, too — the more of a burden you put on your customers and prospects to keep your data about them clean, the less of a burden you put on yourself. And, likewise, the more of a burden you take on yourself, the less of a burden you’re putting on your customer.

While bouncing through links from a tweet, I stumbled across Steve Woods’s original Contact Washing Machine post, and it set some alarm bells off. Steve’s a damn sharp guy — he was a co-founder and remains the CTO of Eloqua, and he is pretty much an undisputed visionary when it comes to marketing automation technology. Yet, this post sparked an immediate reaction, as well as teeter-totter imagery. Since then, Steve has clarified…and I think I misread his initial premise. His point is that data cleansing should happen as early in the data acquisition process as possible — cleanse the data as it comes in, rather than crossing your fingers and waiting to run batch processes after the fact in the hopes that the data will get cleaned up.

That’s a valid point, but, after digging deeper into the cross-links in the post, I still think there’s some under-estimating of what it takes to “fix” dirty data as it comes in. For starters, when it comes to customer/prospect data, there are typically a range of incoming data entry points:

Web Data Entry

In the world o’ the web, data can come into your systems directly as typed by a visitor to your site — when a user is filling out a web form, for instance. On the surface, that’s a great place to do data validation, because you’ve got the actual user right there to clarify anything that has gone amiss. If he’s fat-fingered his phone number or put in an e-mail address that is clearly not valid, it’s best to prompt him right then and there to correct the mistake. But, the teeter-totter comes into play: if that piece of data is really not germaine (as perceived by the user), it doesn’t take long for your cleansing to lead to a frustrated visitor to your. Worse, if you don’t allow the user to bypass the validation step (with a “I don’t care what you think, I’ve entered the information correctly, so just keep it that way and let me move on” option), there is a very good chance that you will keep some visitors from ever getting to where they and you want them to!

If you include field validation on your web forms, and if you don’t allow the user to override that validation, it behooves you to include detailed form abandonment tracking in your web analytics to make sure you haven’t set up an insurmountable barrier for some of your customers.

Human Data Entry

Call centers almost always serve a data entry function as part of the customer service process. In addition, many companies have dedicated data entry staff to translate mail, fax, tradeshow-collected leads, or other transactions. This can be a great opportunity to clean your data up front, as you can certainly place a higher burden of getting the data right and enforced data validation on employees of your own company than you can on your customers and prospects.

BUT, this turns out to be a stickier wicket than it seems at first blush. If I had a nickel for every time I heard someone living in world of backend data propose data augmentation or enhancement by updating the human data entry processes to “just add one more quick step,” I’d be able to buy a Starbucks Venti Caramel Frapuccino^® blended coffee (which is a lot of nickels, if you think about it). Two reasons that there should be a proceed-with-extreme-caution label placed prominently on any solution that heads down this path:

• Call centers typically live and die by the average handle time (AHT) for their calls; yes, they want to meet the customer’s needs, but they also, out of necessity, can save big dollars by cutting the AHT by a few seconds on average. Adding 5 or 10 seconds to every call can have a very real impact (and can make you some quick enemies with call center managers)
• It’s easy to identify the benefits of more, more complete, or cleaner data…when it comes to backend processes and data analysis. But, is that benefit readily evident to the people whom you’re relying on to capture it? Does it benefit them directly, either through smoothing the immediate next steps in their process or by impacting their compensation? Due to the high-volume nature of call center and data entry work, data that is “just another field you need to fill out” is data that is at risk of falling prey to shortcuts (the first value in the dropdown, “aaa” in a text field, etc.). The most successful introductions of process changes have a net-no-change or net decrease in the number of steps/time/complexity of the process into which it is being introduced.

Human data entry offers opportunities to get data that is more complete and cleaner…but those opportunities don’t come automatically.

There are many other ways that data can enter your systems: provided by an intermediary (often semi-independent sales channels: distributors, resellers, etc.), sourced from a third-party lead sourcing company, passed in from another system within your company (often a system that doesn’t store the data in the same format or even have the same definitions for what specific fields mean and are used for), etc. There’s value in inspecting the sources of your customer data, assessing how clean the data is that comes from those different sources, and then, with the teeter-totter firmly in mind, investigating where and how to get that data coming in cleaner!

Photo courtesy of jhirtz.

© Tim for Gilligan on Data by Tim Wilson, 2009. | Permalink | 2 comments | Add to del.icio.us
Post tags: cleansing, teeter-totter

Feed enhanced by Better Feed from Ozh

• Using behavioral data to drive timely and relevant emails
• Integrating information across different customer touchpoints/channels to deduce customers’ and prospects’ preferred communications medium
• Building analytic models to predict which customers are most likely to churn and making special offers to retain them

Those are all admirable goals. And, they’re all attainable. AND, they’re all going to be expected baseline capabilities within five years.

Before you tackle these higher order applications, it’s worth grounding yourself in an understanding of how rapidly customer data decays. Here are a couple of fun facts to wrap your head around on that front:

• The U.S. Postal Service processes over 40 million address changes annually [source]
• The population of the United States is estimated as being just north of 300 million [source]

Clearly, this isn’t an apples-to-apples comparison. But, we tend to imagine that our customers and prospects are more static than, in reality, they are — who they work for, what their job title is, and, yes, even where they live.

© Tim for Gilligan on Data by Tim Wilson, 2009. | Permalink | 2 comments | Add to del.icio.us
Post tags: change of address, population

Feed enhanced by Better Feed from Ozh