Gilligan on Data by Tim Wilson

You keep hearing the buzz about this “multi-dimensional lead scoring” thing. You’ve seen some pictures in a blog entry. You might even have read the white paper. You might have even heard rumors that there is a 10-minute video version of the white paper in the works. Maybe you even read up on how Bulldog Solutions incorporated multi-dimensional lead scoring in their lead reengagment program.

But, what if none of these cover your preferred mode of learning? By my tally, we’ve missed three channels for getting out and communicating on the subject:

• Text messages
• Fax
• Radio

Well, you’re simply out of luck on the first two, but radio is coming! I’ll be spending an hour next week — Wednesday, May 21st from noon to 1:00 on the American Marketing Association’s Marketing News Radio. The experience promises to be…nothing at all like the picture depicted above. I’ll be sitting in my office on the phone. But, the AMA does a great job with the production of these shows.

The challenge for me is that the subject plays right into a nice, simple picture. And there won’t be any pictures. That’s had me thinking about some analogies and anecdotes that will help make the core concepts resonate a bit more. We’ll see how it goes.

Photo by yarra64

Connie Bensen has a Social Media Measurement post that is worth a read. While the post is focussed on measuring social media specifically, she hits on some areas that, all too often, are overlooked when it comes to developing metrics and then reporting on them over time.

The post includes a lot of resources for measuring social media — going well beyond simply web analytics data — as well as a list of examples of things that can be measured. What really struck me, though, was the list at the end of the post of what a community manager’s monthly report should include. First, the fact that it is a monthly report is somewhat refreshing — real-time on-demand reports are way overrated, and really are not practical when it comes to providing the sort of context that Connie describes.

On to Connie’s list of report elements — the bold text is from her list, and the non-bold description is my own take on the item:

• Ongoing definition of objectives — the framework of any recurring report should be the objectives that it is attempting to measure, so I love that this is the first bullet on the list. I would qualify it just a bit — it does not seem right to be making the defining of objectives an ongoing exercise; rather, objectives should be established, reiterated on an ongoing basis (so that everyone remembers why we’re tackling this initiative in the first place), and revisited periodically (objectives can and should change).
• Web analytics — this is the “easy” data to provide on a recurring basis, it’s data that most people are getting comfortable with, and, even though there is a lot of noise in the data, it is still reasonably objective; the key here is to focus on the web analytics data that actually matters, rather than including everything.
• Interaction - Trends in members, topics, discovery of new communities – this is a somewhat community-specific component, but it’s a good one; the “discovery of new communities” actually implies an objective regarding the role of a community manager; what a great metric, though, to drive behavior within the role.
  
• Qualitative Quotes - helpful for feedback & marketing – to broaden this list to beyond reporting for social media, let’s change “Quotes” to “Data;” make the report real by providing tangible, but qualitative, examples of what is going well (or not); reporting on lead generation activity, for instance, can include selected comments that were made by attendees at a webinar — highlighting what resonated with the audience (and what did not).
• Recommendations - Based on interactions with the customers – recommendations, recommendations, recommendations! What is the point of pulling all of this information together if nothing gets done with it? I sometimes like to include recommendations at the beginning of a report — they’re a great way to engage the report consumer by making statements about a course of action right up front.
• Benchmark based on previous report – my preference is to use stated targets (where it makes sense) as the benchmark, rather than simply looking for the delta of the data over a prior reporting period. But, sometimes, that is simply not feasible. Including “here’s the measurement…and here’s the direction it is heading” is definitely a good thing. But, it’s also important to not look at a 2-month span and jump to “we have a trend!”

Having recently relaunched the Bulldog Solutions blog, I’ve got a good opportunity to put Connie’s post into practice. Oh, dear…that’s going to require re-opening the, “What are our objectives for this thing…clearly stated, please?!” Stay tuned…

Avinash Kaushik has a great post today titled The “Action Dashboard” (An Alternative to Crappy Dashboards. As usual, Avinash is spot-on with his observations about how to make data truly useful. He provides a pretty interesting 4-quadrant dashboard framework (as a transitional step to an even more powerful dashboard). I’ve gotten red in the face more times than I care to count when it comes to trying to get some of the concepts he presents across. It’s a slow process that requires quite a bit of patience. For a more complete take on my thoughts check out my post over on the Bulldog Solutions blog.

And, yes, I’m posting here and pointing to another post that I wrote on a completely different blog. We’ve recently re-launched the Bulldog Solutions blog — new platform, and, we hope, with a more focussed purpose and strategy. What I haven’t fully worked out yet is how to determine when to post here and when to post there…and when to post here AND there (like this post).

It may be that we find out that we’re not quite as ready to be as transparent as we ought to be over on the corporate blog, in which case this blog may get some posts that are more “my fringe opinion” than will fly on the corporate blog. I don’t know. We’ll see. I know I’m not the first person to face the challenge of contributing to multiple blogs (I’ve also got my wife’s and my personal blog…but that one’s pretty easy to carve off).

Tamara Gielen has a post titled Triggered Email Is Only As Good As Your Data over on the B2B E-mail Marketing blog. She describes a scenario where you want to send a satisfaction survey to customers 90 days before their contract expires with your company, and adds on some logic of resending the survey to customers who do not respond to the initial request. Her point is — that may be a lot trickier than it sounds if you’re actually living in the real world:

• The data needed to trigger and send these e-mails lives in different systems
• You don’t have a good way to determine whom at the account should receive the e-mail (you probably don’t want to send the survey to everyone you have in your systems for the acccount)
• You don’t have a mechanism for updating your data when someone leaves the account and is no longer an employee there
• The list goes on…

The real world would put you in a position of needing to make some unpleasant decisions:

• Do you cast the net broadly and risk sending a non-applicable e-mail to a bunch of people in your database, or do you cast the net very narrowly and only send to people you are absolutely sure are the right folk…but then limit the impact of doing the survey in the first place?
• Do you limit the amount of manual cleanup on the data, or do you engage your sales and account management groups to manually flag who should receive the invitation (or something in between)?
• Do you try to explain all of the caveats of the data to the person who initiated the project, or do you just make a series of judgment calls and be prepared to defend/explain them if asked later?

The truth is that, in most cases, this sort of initiative does make sense, but it also requires making a long list of trade-offs, assumptions, and judgment calls to balance the expected impact with the effort required.

The entry brought to mind some interesting data integrity snafus that I’ve come across in my personal life:

• For years, the phone company thought my name was “Jim” rather than “Tim” — initially, this only impacted caller ID, but, over time, that list got sold to various direct marketers, and “Jim” started getting junk mail
• Several months after we moved to Ohio, I went into REI in Austin, and when they looked up my account, they had my Ohio address with a phone number of my former employer — none of this was information that we had ever explicitly provided
• After moving to Ohio, we signed up for a Giant Eagle grocery store card, with our new address and phone number; somehow, a couple of months later, when I didn’t have my card with me and they had to look me up…they had the address as our address in Austin!
• I recently shifted my cell phone plan from my wife’s and my joint account with T-Mobile to my company’s account with AT&T; I found out last week that I show up as “Julie Wilson” on caller ID when I make calls with my cell phone
• For years, Microsoft was convinced that I was a high-level IT manager responsible for my company’s system administration and infrastructure — I’ve never been within a light-year of that sort of role

All of this is to say that the data is messy. It’s never going to be perfect. Spending time and energy on data integrity initiatives makes sense, but that has to be balanced with the practical reality of the world — short of an Orwellian society, the data is always going to have some level of inaccuracy. Understand the data. Understand what you’re trying to accomplish with it. And then make a judgment call.

Photo by Daquella manera

One of the areas that I’ve spent a lot of time focusing over the past six months is lead scoring. Specifically, multidimensional lead scoring. I’ve written about it here before, and my white paper is still available on the Bulldog Solutions web site. We shot a 10-minute video white paper a few weeks ago, which should be available soon. The slides I talk to in that are available below courtesy of slideshare.net.

| View | Upload your own

And, just yesterday, Laura Ramos posted a topic diving into lead scoring over on the Forrester Interactive Marketing blog.

All this is to say that the topic remains very much on the top of my mind. And, as is no great surprise, I’ve got additional thoughts that I’m still working out on the subject — mostly driven by our experiences putting multidimensional lead scoring into practice at Bulldog Solutions. I expect I’ll be working through some of these over on the Bulldog blog in the coming weeks. But, I thought I’d start here to at least get a little bit of a list going.

To start with, we haven’t uncovered anything that invalidates any of the concepts in the white paper. Quite the contrary, actually. It has been very well received by everyone to whom it has been presented and has sparked some interesting conversations. And, as we’ve put it into practice, it’s holding up to be as viable an approach as we expected.

The two areas that I’m most looking to flesh out my thinking are as follows:

• The Buying Cycle Dimension — Partly because the paper itself was getting to be darn long, and partly because we were not yet in the process of implementing a third dimension, the paper only briefly speaks to the “buying cycle position” dimension of lead scoring. The paper was initially titled “2-D” lead scoring rather than “Multidimensional” lead scoring, and our CMO pointed out that this was overly limiting — we were only implementing two dimensions at the time, but we knew there were others that we would be moving on to. Specifically, he was referring to the buying cycle position. This is a bit dicey to work out, but it’s a straightforward concept: a lead who is checking out detailed product specifications and pricing on your web site is likely much closer to making a purchase decisions than a lead who is simply reading white papers or analyst articles on a topic.
• Multiple Offering Types — this was flat out an oversight on my part, largely due to the fact that I’m at a startup that does not yet have a wildly extensive set of offerings. But, in my prior life, this issue very much was at the fore, so missing it was definitely a whiff. The issue is that one lead may be highly qualified for one type of offer but very much not qualified for another offer. In our case, we have offerings where the economic buyer is a marketing executive. We have other offerings where the economic buyer is a marketing manager. The marketing executive is actually not a qualified lead when it comes to some offerings. And vice versa. So, how does this get handled with the lead scoring? I think that this situation is an indication that you may have multiple profile scores — one for each broad offer type. I don’t think the engagement score changes from offer to offer — this is a measure of engagement with the company / the brand and is offering-independent. And, I don’t think the buying cycle position will differ, either. But, I’m still working this out. We’re testing it out manually with some upcoming activities.

One of the benefits of multidimensional lead scoring is that it does make it manageable to add additional nuances and complexity over time. And, while the lead scoring is primarily a lead qualification mechanism, it serves other purposes: it’s one factor in segmenting your database for different nurturing programs (or content within the nurturing programs) — based on what quadrant in the multidimensional matrix the lead falls in; and, it can be used for lead routing — again, based on what quadrant in the lead matrix the lead is in may dictate whether the lead is routed to an inside sales organization or directly to another organization within Sales.

As always, let me know if you disagree or can add color from your own experiences.

Stéphane Hamel, it seems, is a bit peeved with Eric Peterson. These are two pretty big names in web analytics — Eric as one of the fathers of web analytics, and Stéphane as both a thought leader in the space as well as the creator of one of the most practical, useful web analytics supplemental tools out there — WASP: The Web Analytics Solution Profiler plugin for Firefox. With the plugin, you visit any site, and a sidebar will tell you what web analytics solutions it looks like it’s running. It’s pretty cool.

I don’t know the full background of the current back-and-forth between these two guys, but I’m a huge fan of Stéphane, and my ears perked up when I read this observation in the post:

Business Process Analysis implies understanding & improving a collection of interrelated tasks which solve a particular issue. Nothing new here… Most businesses face complex and “hard” processes, and the way to make them “easy” is by decomposing them into smaller sub-processes until they are manageable.

For one thing, for a period of ~8 months, my job title was “Director of Business Process Analytics.” And, frankly, I was never sure what that meant. In hindsight, if I’d had these two sentences from Stéphane and if I’d replaced “Analytics” with “Analysis,” I would have seen a much clearer mapping from my label to what I was actually doing in the role.

More important, though, is the concept of “decomposition.” I find myself preaching the Decomposition Doctrine regularly. And I believe in it strongly.

As an example, when it comes to the Holy Grail of Marketing Analysis — calculating the ROI of your marketing spend — many, many B2B marketers start out looking for the correlation between leads generated and revenue. I have yet to see a case in B2B where this can be found with a sufficiently tight, sustained correlation to be meaningful. That actually makes sense. It’s like looking for a correlation between the state someone is born in and the achievement of a PhD. There’s a lot going on over time between Point A and Point Z.

In the case of B2B marketing, decomposition makes sense. Decompose the process:

• The lead-to-qualified lead sub-process
• The qualified lead to sales accepted lead sub-process
• The sales accepted lead to sales qualified lead sub-process
• The sales qualified lead to close sub-process

Each of these sub-processes have people who proceed to the next sub-process as well as people who do not — put simplistically: people who “fall out of the funnel.” But, you can further decompose — of the people who fell out, where did they fall out and why? And does that mean they are gone forever, or are there processes/subprocesses that can be used to reengage them in the future?

The key here is that, from a theoretical perspective, if you link together all of the simpler sub-processes, then you’ve got an accurate representation of the more complex master process. The problem is that this is mostly true. There are probably other sub-processes that are unknown — those pesky “corner cases” that the real world insists on throwing at us. And, each sub-process likely experiences various anomalies over time. Add those together, and you’ve got a complex process that verges on the unanalyzable.

On the other hand, if you focus on a sub-process, you can analyze what is going on, including accounting for the anomalies. “But, isn’t there a risk that you’ll be missing the forest for the trees?” you ask. Absolutely. That’s why it’s important to start with a high-level view of the whole process, with a clear picture of the components that go into it. If you simply pick a “simple sub-process” to focus on, without understanding how and where that fits into the big picture, you run the risk of rearranging deck chairs on the Titanic. On the other hand, if you simply try to “analyze the Titanic,” without some level of decomposition, you’re equally doomed.

Yep. I’m living under a rock.

I’d re-stumbled across Hans Rosling and Trendalyzer a couple of months ago. I made a comment regarding if Trendalyzer hits the business world. Well, in a way, it sort of has. It’s hanging around under the hood in some fashion, I’m almost sure, of Google’s Visualization API.

Must. Find. Time. To. Play. With. Google Spreadsheets and visualization gadgets.

At the top list of blogs I follow is Jeremiah Owyang’s Web Strategist blog. He posts frequently, with depth, and with insight. However, I was in the midst of a hectic week in Austin when he posted his Analysis of the Zuckerberg Lacy Interview, and, frankly, while the title persisted in a couple of places (not only in his feed, but in my Yahoo! Pipe feed on data from non-data blogs because of the word “analysis” in the title), I’d been pretty much Zuckerberg-Lacy’d out, and I thought this was going to be a “my take on what happened” type of “analysis.” I was wrong.

The one-sentence recap of the actual incident: Sarah Lacy, of Business Week, interviewed Mark Zuckerberg, founder and CEO of Facebook, at SXSW Interactive in Austin, and the crowd, which was none too happy with the way the interview was going, pretty much took over using Twitter as a backchannel of communication. Google it and you can read scads more as well as see video of the event.

Well, I finally got around to looking at Jeremiah’s post, and I’ll be damned if it wasn’t a brief post linking to a really interesting TechnoSocial post: Anatomy of a Mob: The Lacy/Zuckerberg Interview. Kee Hinckley sifted through a bunch of Twitter data to try to get some insight into what really went on through Twitter during the keynote. It’s a fascinating read.

What I want to point out, though, is not so much the results of the analysis, but some pretty darn noteworthy aspects of what went into it.

First, I immediately started wondering how on earth Hinckley figured out which Twitter users were at the keynote. As it turns out, he explains it — recognizing that it’s imperfect, but, by golly, still pretty clever! And, it took a mix of tools, some level of clunky automation (no one likes to do screen scraping), and quite a bit of flat-out manual effort. He revised what he included/excluded as he got into the manual part of the exercise. What’s Noteworthy: the data Hinckley wanted was not easily accessible (the data always requires more prep work than most people realize), and it required some judgment when it came to getting it. That’s stepping out of a formulaic approach to analysis of “pull the data that’s available and present it.”

Second, the visualization. I am almost always opposed to 3D representations of data. Categorically when it’s two dimensions of data presented with “depth” — that’s just silly. But, even when it’s three dimensions presented in three dimensions, more often than not, the result is uninterpretable. Not the case here! Hincklely steps outside of the box to think about ways to effectively visualize the data — much, much more thought than simply “How do I get all of the data displayed?” He even includes two different charts, one with bubbles and one with a color spectrum, of the same data — clearly grappling with how best to show the information clearly (both work, IMHO). What’s Noteworthy: All too often, I see analysts go through all of the hurdles of prepping the data and “running the numbers” only to take shortcuts when it comes to the visual representation of the results. That’s the equivalent of running 25.5 miles of a marathon really hard…and then going home.

Finally, Hinckley puts a lot of text-based interpretation behind his analysis. In this case, he clearly had the question, ran with trying to find the answer, and took responsibility for explaining the whole process and the results. And, he did all three swimmingly! What’s Noteworthy: In many situations, one person is asking the question, while an analyst tries to find the answer. It’s that third area — explaining the process and results — where many analysts decide not to tread. Rather, they “do the analysis” and turn the “results” (mostly the data, including charts) over to the original requestor to interpret and explain. This bothers me. I much prefer to see an analyst actually draw conclusions and provide real context and interpretation. Whether they are expected to or not! It’s up to the original requestor to decide whether to use that information and how. More often than not, it gets used. To good results.

Overall, it’s a fascinating read. Top. Notch. Work!

I realized that my Data Posts from Non-Data Blogs Yahoo! pipe wasn’t working correctly, and when I fixed it, a recent post from Debbie Weil at BlogWrite for CEOs popped up: More on the ROI of Social Media: Return on Influence. Ordinarily, I’m a big fan of Weil’s thoughts, but this one had me wondering if I ought to try to track down some blood pressure medication. Weil by no means invented the phrase (and does not claim to have), “When it comes to social media, ROI really means ‘return on influence,’” but, sadly, she has jumped right on that misguided bandwagon.

Maybe it’s that I was raised in a house where one parent was an engineer and the other was an English major. Maybe it’s because I’ve got a contrarian bent — a slight one (I like “alternative” music but not “experimental” music). For whatever reason, “ROI is return on influence” has stuck in my craw from the first time I heard it. And it still makes me twitch whenever I stumble across a post where someone waxes eloquently about the genius of the phrase.

Weil has a couple of “short answers” for why return on influence makes sense. Her first is that it makse sense “because the return is soft. The benefits of incorporating social media strategies into your marketing are real (and can no longer be ignored) but they’re not normally measured in dollars.” I have no argument with any part of that assertion after the word “because.” Weil points out that the return is soft. So, why isn’t the “return” being replaced in this platitude? “Influence from (social media) investment” I get. And that is something that you should try to measure.

Are you still with me? No one who has picked up this phrase has stopped to think that it doesn’t make sense! If you develop influence in your market, then you will get a return, which may or may not be soft. But, are you trying to measure the return on that influence, or are you trying to measure the influence that you garnered by engaging in social media?

Marketers really are freaked out by the increasing focus on Marketing ROI. That focus is driven by CEOs and CFOs. In my experience, CFOs are pretty sharp people. They get that Marketing is important. What they want is accountability, efficiency, and effectiveness from Marketing. They want to know that the chunk of the company’s budget that is being invested in Marketing is being well-used. Unfortunately, they communicate that imperative in financial terms: “What’s the ROI?” They’re Finance people, folks! What would you expect?

Marketers, rather than getting to the heart of delivering business value — driving improvements in efficiency and effectiveness, and demonstrating results — have instead gone nutso with, “I have to show ROI!” Return on Influence is a headless-chicken response to this belief. And, almost comically, it has resulted in a classic marketing response: “Let’s spin and message it! Let’s talk about how, for Marketing in the social media world, ROI really stands for ‘Return On Influence.’”

Oh, man oh man, what I would pay to sit in the room when a Fortune 1000 CMO proudly rolls out that explanation to the CFO. It completely, utterly, totally, and ridiculously misses the point.

Accountability and continuous improvement, people: the executives in your company are not stupid (if you think they are, then they either are, or they aren’t but you think they are: in either case, find a new company). Understand what you are trying to accomplish with your social media strategy. Is it to build your brand? Is it to engage with your most avid customers? Is it to position your company as being full of cutting-edge thought leaders? Articulate that. Measure whether you are making headway with your efforts.

Am I right?

The topic of “Marketing ROI” has crossed my inbox and feed reeder on several different fronts over the past few weeks. I don’t know if the subject actually has peaks and valleys, or if it’s just that my biorhythms periodically hit a point where the subject seems to bubble up in my consciousness.

The good news is that the recent material I’ve seen has had a good solid theme of, “Don’t focus too much on truly calculating ROI.” The bad news is that that message has been in response — directly or indirectly — to someone who is trying to do just that.

One really in-depth post came from — no surprise — My Hero Avinash Kaushik. He did a lengthy post, including five embedded videos, each 4-9 minutes long: Standard Metrics #5: Conversion / ROI Attribution.  What the post does is walk through a series of scenarios  where a Marketer might be trying to calculate the ROI for their search engine marketing (SEM) spend. He starts with the “ideal” scenario: a visitor does a search, clicks on a sponsored link, comes to the site, moves through and makes a purchase. In that case, calculating/attributing ROI is very simple. But, that’s just a setup for the other scenarios…which are wayyyyyy closer to reality. The challenge is that, as Marketers, it’s we all too often ignore our own typical behavior and common sense so that we can assume that most of our potential customers behave in an overly simplistic way. When was the last time you did a search, clicked on a sponsored link, and then, during that visit, made a purchase?

Unfortunately, very, very, very few Marketing executives would ever actually spend the 45 minutes it would take to truly consume all of Avinash’s post.  And, honestly, that’s not really “the solution.” The smart Marketing executive will find the Avinashes of the world and will hire them and trust them. Avinash (and John Marshall) really make the case that “time on site” is a more useful metric for assessing the effectiveness of your SEM spend — ROI just brings in too many variables and too much complexity.

In short: Don’t treat ROI as the Holy Grail and try to tie every one of your marketing tactics to “revenue generated.” For one thing, you will head down so many rat holes that you’ll start drooling whenever someone says, “cheese.” For another thing, you will find yourself facing decisions that seem right based on your ROI calculation…but that you just know are wrong.

Another place where this topic came up was in a thread titled ROI Models - High Level Thinking on the webanalytics Yahoo! group. I responded, but others chimed in as well. Some of those responses, in my mind, are still a bit too accepting of the premise that “I need to calculate a hard ROI.” But, other responses go more to a “back up and don’t look at ROI as the be-all/end-all.”

And, finally, ROI crossed my inbox last week by way of a CMO Council press release from back in January. I saw this when it came out, but a colleague forwarded it along last week, which prompted me to re-read it. The press released emphasized how much marketers are focussing on accountability when it comes to their marketing investments. One data point that jumped out was “34 percent [of marketers] said they were planning to introduce a formal ROI tracking system.” This is an alarming statistic. Marketers absolutely should be focusing on accountability – finding ways that they can measure and analyze the results of their efforts. But, if they truly are framing this as the need for “a formal ROI tracking system,” then that means 34 percent of marketers are going to be largely chasing their tails rather than driving business value.