Fairly regularly, I’m drawn into work where the team has unrealistic expectations of the degree to which I can accurately quantify the impact of their initiatives on their top (or bottom) line. I’ve come at these discussions from a variety of angles:

• Acknowledging why expectations are inflated — what I’ve termed The Frustration Gap
• Articulating the myriad ways that media mix modeling (MMM) has started to crumble
• Developing a handy analogy based on the Mississippi River to explain the complexity of today’s measurement reality
• Putting forth a framework for social media measurement that makes the distinction between measuring the performance of individual channels and measuring the overall brand outcomes that result from these channels working in concert

This post is largely an evolution of the last link above. It’s something I’ve been exploring over the past six months, and which was strongly reinforced when I read John Lovett’s recent book. As I’ve been doing measurement planning (measurement strategy? marketing optimization planning?) with clients, it’s turned out to be quite useful when I have the opportunity to apply it.

Initially, I referred to this approach as developing a “logical model” (that’s even what I called it towards the end of my second post that referenced John’s book), but that was a bit bothersome, since “logical model” has a very specific meaning in the world of database design. Then, a couple of months ago, I stumbled on an old Harvard Business Review paper about using non-financial measures for performance measurement, and that paper introduced the same concept, but referred to it as a “causal model.” I like it!

How It Works

The concept is straightforward, it’s not particularly time-consuming, it’s a great exercise for ensuring everyone involved is aligned on why a particular initiative is being kicked off, it sets up meaningful optimization work as individual tactics and campaigns are implemented, and it positions you to be able to demonstrate a link (correlation) between marketing activities and business results.

This approach acknowledges that there is no existing master model that shows exactly how a brand’s target consumers interact and respond to brand activity. The process starts with more “art” than “science” — knowledge of the brand’s target consumers and their behaviors, knowledge of emerging channels and where they’re most suited (e.g., a QR code on a billboard on a busy highway…not typically a good match), and a hefty dose of strategic thought.

The exact structure of this sort of model varies widely from situation from situation, but I like to have my measurable objectives — what we think we’re going achieve through the initiative or program that we believe has underlying business value — listed on the left side of the page, and then build linkages from that to a more definitive business outcome on the right:

It should fit on a single page, and it requires input from multiple stakeholders. Ultimately, it can be a simple illustration of “why we’re doing this” for anyone to review and critique. If there are some pretty big leaps required, or if there are numerous steps along the way to get to tangible business value, then it begs the question: “Is this really worth doing?” It’s an easy litmus test as to whether an initiative makes sense.

What I’ve found is that this exercise can actually alter the original objectives in the planning stage, which is a much better time and place to alter them than once execution is well under way!

Once the model is agreed to, then you can focus on measuring and optimizing to the outputs from the base objectives — using KPIs that are appropriate for both the objective and the “next step” in the causal model.

And, over time, the performance of those KPIs can be correlated with the downstream components of the causal model to validate (and adjust) the model itself.

This all gets back to the key that measurement and analytics is a combination of art and science. Initially, it’s more art than science — the science is used to refine, validate, and inform the art.

© Tim for Gilligan on Data by Tim Wilson, 2011. | Permalink | One comment | Add to del.icio.us
Post tags: causal model

Feed enhanced by Better Feed from Ozh

A recent Wall Street Journal article titled Business Schools Plan Leap Into Data covered the recognition by business schools that they are sending their students out into the world ill-equipped to handle the data side of their roles:

Data analytics was once considered the purview of math, science and information-technology specialists. Now barraged with data from the Web and other sources, companies want employees who can both sift through the information and help solve business problems or strategize.

That article spawned a somewhat cranky line of thought. It’s been a standard part of presentations and training I’ve given for years that there is a gap in our business schools when it comes to teaching students how to actually use data. And, the article includes a quote from an administrator at the Fordham business school: “Historically, students go into marketing because they ‘don’t do numbers.’” That’s an accurate observation. But, what is “doing numbers?” In the world of digital analytics, it’s a broad swath of activities:

• Consulting on the establishment of clear objectives and success measures (…and then developing appropriate dashboards and reports)
• Providing regular performance measurement (okay, this should be fully automated through integrated dashboards…but that’s easier said than done)
• Testing hypotheses that drive decisions and action using a range of analysis techniques
• Building predictive models to enable testing of different potential courses of action to maximize business results
• Managing on-going testing and optimization of campaigns and channels to maximize business results
• Selecting/implementing/maintaining/governing data collection platforms and processes (web analytics, social analytics, customer data, etc.)
• Assisting with the interpretation/explanation of “the data” — supporting well-intended marketers who have found “something interesting” that needs to be vetted

This list is neither comprehensive nor a set of discrete, non-overlapping activities. But, hopefully, it illustrates the point:

The “practice of data analytics” is an almost impossibly broad topic to be covered in a single college course.

What bothered me about the WSJ article are two things:

• The total conflation of “statistics” with “understanding the numbers”
• The lack of any recognition of how important it is to actually be planning the collection of the data — it doesn’t just automatically show up in a data warehouse

On the first issue, there is something of an on-going discussion as to what extent statistics and predictive modeling should be a core capability and a constantly applied tool in the analyst’s toolset. Michael Healy made a pretty compelling case on this front in a blog post earlier this year – making a case for statistics, econometrics, and linear algebra as must-have skills for the web analyst. As he put it:

If the most advanced procedure you are regularly using is the CORREL function in Excel, that isn’t enough.

I’ve…never used the CORREL function in Excel. It’s certainly possible that I’m a total, non-value-add reporting squirrel. Obviously, I’m not going to recognize myself as such if that’s the case. I’ve worked with (and had work for me) various analysts who have heavy statistics and modeling skills. And, I relied on those analysts when conditions warranted. Generally, this was when we were sifting through a slew of customer data — profile and behavioral — and looking for patterns that would inform the business. But this work accounted for a very small percentage of all of the work that analysts did.

I’m a performance measurement guy because, time and again, I come across companies and brands that are falling down on that front. They wait until after a new campaign has launched to start thinking about measurement. They expect someone to deliver an ROI formula after the fact that will demonstrate the value they delivered. They don’t have processes in place to monitor the right measures to trigger alarms if their efforts aren’t delivering the intended results.

Without the basics of performance measurement — clear objectives, KPIs, and regular reporting — there cannot be effective testing and optimization. In my experience, companies that have a well-functioning and on-going testing and optimization program in place are the exception rather than the rule. And, companies that lack the fundamentals of performance management that try to jump directly to testing and optimization find themselves bogged down when they realize they’re not entirely clear what it is they’re optimizing to.

Diving into statistics, econometrics, and predictive modeling in the absence of the fundamentals is a dangerous place to be. I get it — part of performance measurement and basic analysis is understanding that just because a number went “up” doesn’t mean that this wasn’t the result of noise in the system. Understanding that correlation is not causation is important — that’s an easy concept to overlook, but it doesn’t require a deep knowledge of statistics to sound an appropriately cautionary note on that front. 9 times out of 10, it simply requires critical thinking.

None of this is to say that these advanced skills aren’t important. They absolutely have their place. And the demand for people with these skills will continue to grow. But, implying that this is the sort of skill that business schools need to be imparting to their students is misguided. Marketers are failing to add value at a much more basic level, and that’s where business schools need to start.

© Tim for Gilligan on Data by Tim Wilson, 2011. | Permalink | 8 comments | Add to del.icio.us
Post tags:

Feed enhanced by Better Feed from Ozh

Basically…We’re Moving in a Good Direction!

At the core of the release is a shift away from “Likes” and “Impressions” and more to “exposed and engaged people.” There are now a slew of metrics available at both the page level and the individual post level that are “unique people” counts. That…is very fine indeed! It’s progress!

Visually Explaining the New Metrics

As I sifted through the new Facebook Page Insights product guide (kudos to Facebook for upping the quality of their documentation over the past year!) with some co-workers, it occurred to me that a visual representation of some of the new terms might be useful. I settled on a Venn diagram format, with one diagram for the main page-level metrics and one for the main post-level metrics.

Starting with page-level metrics:

Defining the different metrics — heavily cribbed from the Facebook documentation:

• Page Likes – The number of unique people who have liked the page; this metric is publicly available (and always has been) on any brand’s Facebook page.
• Total Reach – The number of unique people who have seen any content associated with a brand’s page. They don’t have to like the page for this, as they can see content from the page show up in their ticker or feed because one of their friends “talked about it” (see below).
• People Talking About This – The number of unique people who have created a story about a page. Creating a story includes any action that generates a News Feed or Ticker post (i.e. shares, comments, Likes, answered questions, tagged the page in a post/photo/video). This number is publicly available (it’s the “unique people who have talked about this page in the last 7 days”) on any brand’s Facebook page.
• Consumers – The number of unique people who clicked on any of your content without generating a story.

A couple of things to note here that are a little odd (and likely to be largely inconsequential), but which are based on a strict reading of the Facebook documentation:

• A person can be counted in the Total Reach metric without being counted in the Page Likes metric (this one isn’t actually odd — it’s just important to recognize)
• A person can be counted as Talking About This without being included in the Reach metric. As I understand it, if I tag a page in a status update or photo, I will be counted as “talking about” the page, and I can do that without being a fan of the page and without having been reached by any of the page’s content. In practice, this is probably pretty rare (or rare enough that it’s noise).
• Consumers can also be counted as People Talking About This (the documentation is a little murky on this, but I’ve read it a dozen times: “The number of people who clicked on any of your content without generating a story.” Someone could certainly click on content — view a photo, say — and then move on about their business, which would absolutely make them a Consumer who did not Talk About the page. But, a person could also click on a photo and view it…and then like it (or share it, or comment on the page, etc.), in which case it appears they would be both a Consumer and a Person Talking About This.
• A person cannot be Consumer without also being Reached…but they can be a Consumer without being a Page Like.

Okay, so that’s page-level metrics. Let’s look at a similar diagram for post-level metrics:

It’s a little simpler, because there isn’t the “overall Likes” concept (well…there is…but that’s just a subset of Talking About, so it’s conceptually a very, very different animal than the Page Likes metric).

Let’s run through the definitions:

• Reach – The number of unique people who have seen the post
• Talking About – The number of unique people who have created a story about the post by sharing, commenting, or liking it; this is publicly available for any post, as Facebook now shows total comments, total likes, and total shares for each post, and Talking About is simply the sum of those three numbers
• Engaged Users – The number of unique people who clicked on anything in the post, regardless of whether it was a story-generating click

And, there is a separate metric called Virality which is a simple combination of two of the metrics above:

That’s not a bad metric at all, as it’s a measure of, for all the people who were exposed to the post, what percent of them actively engaged with it to the point that their interaction “generated a story.”

The Reach and Talking About metrics are direct parallels of each other between the page-level metrics and the post-level metrics. However (again, based on a close reading of the limited documentation), Consumers (page-level) and Engaged Users (post-level) are not analogous. At the post-level, Talking About is a subset of Engaged Users. It would have made sense, in my mind, if, at the page-level Talking About was a pure subset of Consumers…but that does not appear to be the case.

KPIs That I Think Will Likely “Matter” for a Brand

There have been several posts that have jumped on the new metrics and proposed that we can now measure “engagement” by dividing People Talking About by Page Likes. The nice thing about that is you can go to all of your competitors’ pages and get a snapshot of that metric, so it’s handy to benchmark against. I don’t think that’s a sufficiently good reason to recommend as an approach (but I’ll get back to it — stick with me to the end of this post!).

Below are what I think are some metrics that should be seriously considered (this is coming out of some internal discussion at my day job, but it isn’t by any means a full, company-approved recommendation at this point).

We’ll start with the easy one:

This is a metric that is directly available from Facebook Insights. It’s a drastic improvement over the old Active Users metric, but, essentially, that’s what it’s replacing. If you want to know how many unique people are receiving any sort of message spawned from your Facebook page, Total Reach is a pretty good crack at it. Oh, and, if you look on page 176 of John Lovett’s Social Media Metrics Secrets book…you’ll see Reach is one of his recommended KPIs for an objective of “gaining exposure” (I don’t quite follow his pseudo-formula for Reach, but maybe he’ll explain it to me one of these days and tell me if I’m putting erroneous words in his mouth by seeing the new Facebook measure as being a good match for his recommended Reach KPI).

Another possible social media objective that John proposes is “fostering dialogue,” and one of his recommended KPIs for that is “Audience Engagement.” Adhering pretty closely to his formula there, we can now get at that measure for a Facebook page:

Now, I’m calling it Page Virality because, if you look up earlier in this post, you’ll see that Facebook has already defined a post-level metric called Virality that is this exact formula using the post-level metrics. The two are tightly, tightly related. If you increase your post Virality starting tomorrow by publishing more “engage-able” posts (posts that people who see it are more like to like, comment, or share), then your Page Virality will increase.

There’s a subtle (but important this time) reason for using Total Reach in the denominator rather than Page Likes. If you have a huge fan base, but you’ve done a poor job of engaging with those fans in the past, your EdgeRank is likely going to be pretty low on new posts in the near term, which means your Reach-to-Likes ratio is going to be low (keep reading…we’ll get to that). To measure the engage-ability of a post, you should only count against the number of people who saw the post (which is why Facebook got the Virality measure right), and the same holds true for the page.

Key Point: Page Virality can be impacted in the short-term; it’s a “speedboat measure” in that it is highly responsive to actions a brand takes with the content they publish

This is all a setup for another measure that I think is likely important (but which doesn’t have a reference in John’s book — it’s a pretty Facebook-centric measure, though, so I’m going to tell myself that’s okay):

I’m not in love with the name for this (feel free to recommend alternatives!). This metric is a measure (or a very, very close approximation — see the messy Venn diagram at the start of this post) of what percent of your “Facebook house list” (the people who like your page) are actually receiving messages from you when you post a status update. If this number is low, you’ve probably been doing a lousy job of providing engaging content in the past, and your EdgeRank is low for new posts.

Key Point: Reach Penetration will change more sluggishly than Page Virality; it’s an “aircraft carrier measure” in that it requires a series of more engaging posts to meaningfully impact it

(I should probably admit here that this is all in theory. It’s going to take some time to really see if things play out this way).

Those are the core metrics I like when it comes to gaining exposure and fostering dialogue. But, there’s one other slick little nuance…

Talking About / Page Likes

Remember Talking About / Page Likes? That’s the metric that is, effectively, publicly available (as a point in time) for any Facebook page. That makes it appealing. Well, two of the metrics I proposed above are, really, just deconstructing that metric:

This is tangentially reminiscent of doing a DuPont Analysis when breaking down a company’s ROE. In theory, two pages could have identical “Talking About / Page Likes” values…with two very fundamentally different drivers going on behind the scenes. One page could be reaching only a small percentage of its total fans (due to poor historical engagement), but has recently started publishing much more engaging content. The other page could have historically engaged pretty well (leading to higher reach penetration), but, of late, has slacked off (low page virality). Cool, huh?

What do you think? Off my rocker, or well-reasoned (if verbose)?Similar Posts:

• Have You Picked Up a Copy of “Social Media Metrics Secrets” Yet?
• “Demystifying” the Formula for Social Media ROI (Spoiler: There Isn’t One)

© Tim for Gilligan on Data by Tim Wilson, 2011. | Permalink | 10 comments | Add to del.icio.us
Post tags: John Lovett, Reach

Feed enhanced by Better Feed from Ozh

One of the more directly usable sections of the book is in Chapter 5, where Lovett lays out pseudo formulas for KPIs for various possible (probable) social media business objectives. This post started out to be about my experiences drilling down into some of those formulas…but then the content took a turn, and one of Lovett’s partners at Web Analytics Demystified wrote a provocative blog post…so I’ll save the formula exploration for a subsequent post.

Instead…Social Media ROI

Lovett explicitly notes in his book that there is no secret formula for social media ROI. In my mind, there never will be — just as there will never be unicorns, world peace, or delicious chocolate ice cream that is as healthy as a sprig of raw broccoli, no matter how much little girls and boys, rationale adults, or my waistline wish for them.

Yes, the breadth of social media data available is getting better by the day, but, at best, it’s barely keeping pace with the constant changes in consumer behavior and social media platforms. It’s not really gaining ground.

What Lovett proposes, instead of a universally standard social media ROI calculation, is that marketers be very clear as to what their business objectives are – a level down from “increase revenue,” “lower costs,”and “increase customer satisfaction” – and then work to measure against those business objectives.

The way I’ve described this basic approach over the past few years is using the phrase “logical model,” – as in, “You need to build a logical link from the activity you’re doing all the way to ultimate business benefit, even if you’re not able to track those links all the way along that chain. Then…measure progress on the activity.”

Unfortunately, “logical model” is a tricky term, as it already has a very specific meaning in the world of database design. But, if you squint and tilt you’re head just a bit, that’s okay. Just as a database logical model is a representation of how the data is linked and interrelated from a business perspective (as opposed to the “physical model,” which is how the data actually gets structured under the hood), building a logical model of how you expect your brand’s digital/social activities to ladder up to meaningful business outcomes is a perfectly valid  way to set up effective performance measurement in a messy, messy digital marketing world.

No Wonder These Guys Work Together

Right along the lines of Lovett’s approach comes one of the other partners at Web Analytics Demystified with, in my mind, highly complementary thinking. Eric Peterson’s post about The Myth of the “Data-Driven Business” postulates that there are pitfalls a-looming if the digital analytics industry continues to espouse “being totally data-driven” as the penultimate goal. He notes:

…I simply have not seen nearly enough evidence that eschewing the type of business acumen, experience, and awareness that is the very heart-and-soul of every successful business in favor of a “by the numbers” approach creates the type of result that the “data-driven” school seems to be evangelizing for.

What I do see in our best clients and those rare, transcendent organizations that truly understand the relationship between people, process, and technology — and are able to leverage that knowledge to inform their overarching business strategy — is a very healthy blend of data and business knowledge, each applied judiciously based on the challenge at hand. Smart business leaders leveraging insights and recommendations made by a trusted analytics organization — not automatons pulling levers based on a hit count, p-value, or conversion rate.

I agree 100% with his post, and he effectively counters the dissenting commenters (partial dissent, generally – no one has chimed in yet fully disagreeing with him). Peterson himself questions whether he is simply making a mountain out of a semantic molehill. He’s not. We’ve painted ourselves into corners semantically before (“web analyst” is too confining a label, anyone…?). The sooner we try to get out of this one, the better — it’s over-promising / over-selling / over-simplifying the realities of what data can do and what it can’t.

Which Gets Back to “Is It Easy?”

Both Lovett’s and Peterson’s ideas ultimately go back to the need for effective analysts to have a healthy blend of data-crunching skills and business acumen. And…storytelling! Let’s not forget that! It means we will have to be communicators and educators — figuring out the sound bites that get at the larger truths about the most effective ways to approach digital and social media measurement and analysis. Here’s my quick list of regularly (in the past…or going forward!) phrases:

• There is no silver bullet for calculating social media ROI — the increasing fragmentation of the consumer experience and the increasing proliferation of communication channels makes it so
• We’re talking about measuring people and their behavior and attitudes — not a manufacturing process; people are much, much messier than widgets on a production line in a controlled environment
• While it’s certainly advisable to use data in business, it’s more about using that data to be “data-informed” rather than aiming to be “data-driven” — experience and smart thinking count!
• Rather than looking to link each marketing activity all the way to the bottom line, focus on working through a logical model that fits each activity into the larger business context, and then find the measurement and analysis points that balance “nearness to the activity” with “nearness to the ultimate business outcome.”
• Measurement and analytics really is a mix of art and science, and whether more “art” is required or more “science” is required varies based on the specific analytics problem you’re trying to solve

There’s my list — cobbled from my own experience and from the words of others!Similar Posts:

• Have You Picked Up a Copy of “Social Media Metrics Secrets” Yet?
• Complex Processes and Analyses Therein
• The New Facebook Insights — One More Analyst’s Take

© Tim for Gilligan on Data by Tim Wilson, 2011. | Permalink | 4 comments | Add to del.icio.us
Post tags: Eric Peterson, John Lovett

Feed enhanced by Better Feed from Ozh

As such, it’s simply not practical to provide an effective recap of the entire book. Rather, I’ll give my take on the general topics the book tackles, and then likely have some subsequent posts diving in deeper as I try to put specific sections into action.

Part I

The first three chapters of the book are foundational material, in that they lay out a lot of the “why you should care about social media,” as well as set expectations for what isn’t possible with social media data (calculating a hard ROI for every activity) as well as what is possible (moving beyond “counting metrics” to “outcome metrics” to enable meaningful and actionable data usage). Early on, Lovett notes:

Analytics solutions and social media monitoring tools are often sold with the promise that “actionable information is just a click away,” a promise that an increasing number of companies have now realized is not usually the case.

That encapsulates, by extension, much of the theme of the first part of the book — that it requires that a range of emerging tools, skills, processes, and organizational structures to come together to make social media investments truly data-driven activities. In addition to the social analytics platforms that Lovett discusses in greater detail later in the book, he makes a case for data visualization as a key way to make reams of social media data comprehensible, and he paints a picture of a “social media virtual network operations center” — a social media command center that harnesses the right streams of near real-time social media data, presents that data in a way that is meaningful, and has the right people in place with effective processes for putting that information to use.

Part II

In Part II of the book, Lovett starts with some basics that will be very familiar to anyone who operates in the world of performance measurement — aligning key metrics to business objectives, using the SMART (Specific, Measurable, Attainable, Relevant, Times) methodology (although Lovett extends this to be “SMARTER” by adding “Evaluate” and “Reevaluate) for establishing meaningful goals and objectives, understanding the difference between accuracy and precision, and so on. This material is presented with a very specific eye towards social media, and then extended to provide a list common/likely business objectives for social media, which each objective drilled into to identify meaningful measures.

These objectives build directly on the work that Lovett did with Jeremiah Owyang of Altimeter Group in the spring of 2010 when they published their Social Marketing Analytics: A New Framework for Measuring Results in Social Media paper. In the book, Lovett substantially extends his thinking on that framework — broadening from four common social media objectives to six, laying out the “outcome measures” that apply for each objective, and then providing pseudo-formulas for getting to those measures (pseudo-formulas only because Lovett emphasizes the need for social media strategies to not be premised on a single channel such as Facebook or Twitter, and he also didn’t want the book to be wholly outdated by the time it was published — the formulas are explicitly not channel-specific, but anyone who is familiar with a given channel will be well-armed with the tools to develop specific formulas that ladder up to appropriate outcome measures). In short, Chapter 5 is one area that warrants a highlighter, a notepad, and multiple reads.

Part III

Part III of the book really covers three very different topics:

• Actually demonstrating meaningful results — looking at how to get from the ask of “what’s the hard ROI?” to an answer that is satisfactory and useful, if not a “simple formula” that the requestor wishes for; Lovett devotes some time to explaining the now-generally-accepted realization that the classic marketing funnel no longer applies, and then extends that thinking to demonstrate what will/will not work when it comes to calculating social media ROI
• Social analytics tools — while Lovett makes the point repeatedly that there are hundreds of tools out there, which can be overwhelming, he nonetheless managed to narrow down a list of seven leading platforms (Alterian SM2, Converseon, Cymfony, Lithium, Radian6, Sysomos, and Trendrr) and conducted an extensive evaluation of them. He includes how that evaluation was organized and the results of the analysis in Chapter 8. While the information is sufficiently detailed that a company could simply take his list and choose a platform, the evaluation is set up as an illustration of what should go into a selection process, so it’s a boon to anyone who has been handed the task of “picking the best tool (for our unique situation).”
• Consumer privacy — this is a very hot topic, and it’s a messy area, so Lovett tries to lay out the different aspects of the situation and what needs to happen to get to some reasonably workable resolution over the next few years. It’s a portion of the book that I’ve already referenced and quoted internally, as it is very easy for marketers and vendors to get caught up in the cool ways they can make content more relevant…without thinking through whether consumers would be okay with those uses of the data

After reading the book once, I’ve already found myself flipping back to certain sections to the point that I’ve got Post Its coming out of it to mark specific pages. Overall, the book is sufficiently modular that individual chapters (and even portions of chapters) stand alone.

Buy it. Buy it now!

 Similar Posts:

• “Demystifying” the Formula for Social Media ROI (Spoiler: There Isn’t One)
• The New Facebook Insights — One More Analyst’s Take

© Tim for Gilligan on Data by Tim Wilson, 2011. | Permalink | 5 comments | Add to del.icio.us
Post tags: book review, John Lovett

Feed enhanced by Better Feed from Ozh

The Basics

A fairly succinct explanation of what game mechanics is can be found in a paper published last October by Bunchball, a gamification platform provider:

At its root, gamification applies the mechanics of gaming to nongame activities to change people’s behavior. When used in a business context, gamification is the process of integrating game dynamics (and game mechanics) into a website, business service, online community, content portal, or marketing campaign in order to drive participation and engagement.
:
The overall goal of gamification is to engage with consumers and get them to participate, share and interact in some activity or community. A particularly compelling, dynamic, and sustained gamification experience can be used to accomplish a variety of business goals.

The key here — and this is actually the biggest detriment of the term itself — is that “gamification” is not simply “playing games.” All too often, I have conversations with people who immediately think XBox, Playstation, Kinect, Farmville, or any number of other “traditional games” when the topic of gamification comes up. That’s an entirely appropriate starting point, but it’s by no means the whole story.

Gamification is about using human nature’s inherent interest in being engaged with others, being rewarded, achieving goals, and, yes, having some fun in the process.

A Recent (and Simple) Example

During a #measureX trip to New Orleans, one of the other people on the trip mentioned that she had been doing a lot of travelling lately, and she tries to fly American Airlines, because they have good flights to most of the places she goes, and she is close to reaching the Gold Level of their Frequent Flier Program. Frequent flier programs are an example of gamification applied for the direct benefit of the brand, allowing travelers to earn points towards different levels, at which they are awarded with different perks. These programs don’t directly drive engagement with other consumers, but that’s another key to gamification — it’s not a one-size-fits-all deal.

And an Even More Recent #measure Example

Even the elusive @AnalyticsFTW has indulged in some gamification of late, with an infographic-creating contest to win a pass toeMetrics in NYC, <shamelessplug>where I will be speaking on Twitter analytics </shamelessplug>. It’s simply a matter of offering a prize (a valuable one, in this case), and then letting the #measure community spread the word, with entrants being challenged to come up with something original and amusing. On the one hand, it’s a “simple contest,” but it’s a simple contest that:

• Forces all potential entrants to actually stop and think about the value of eMetrics
• Requires an investment of time and energy to illustrate that value in a clever way (which causes them to thinkmore about the value of eMetrics)
• Generates marketing collateral for the event that others will come and look at (user-generate content, baby! Not a single designer finger on the paid eMetrics team was lifted to generate the material)

It’s brilliant, really.

An Entirely Different (and More Involved) Example

I was tapped/volunteered to teach a “Microsoft Excel Tips & Tricks” brown bag lunch at work a couple of months ago. It was content that I knew attendees would get value out of…but with a title that didn’t exactly have a “Cowboys & Aliens”-type mystique that would be a natural attendance draw (and, while personable enough, I’m not exactly the office equivalent of Daniel Craig or Harrison Ford).

I applied some game mechanics to promote the event by distributing a series of cards around our various offices (physical cards as well as digital versions to our remote locations):

The cards led to a video (PowerPoint with low-fi voiceover) with details as to the “game,” which required participants to do a little searching and a little collaboration with another office before posting “the answer” on the wall of a Facebook group.

Here’s what I hoped to achieve:

• Engage as many employees as possible just enough with the type of content that I would be presenting that they would have an opportunity to pause and think, “Hmmm… I might actually get something useful out of this”
• Extend that engagement beyond our main office in Columbus to our satellite offices and remote workers
• Find out if I could apply game mechanics without consulting a gamification expert and achieve good results

My KPI for the effort was pretty simple: a “healthy turnout” at the brown bag. I had a handful of additional measures in place:

• Whether or not anyone actually managed to complete the challenge and, if so, how long it took for that to happen
• The number of clicks on the goo.gl link/QR code link driving to the YouTube video
• The number of views of the video
• The number of people who walked by my desk and either chuckled or shook their head

In the end, we had a full room for the brown bag. KPI achieved!

We’re over 300 employees now, and my other measures played out as follows: 159 clicks on the link, 131 video views, and a half-dozen people who chuckled and shook their heads as they walked by my desk. Not bad.

Most surprisingly, though, was how quickly and to what extent people got into the activity. I launched on a Wednesday evening after almost everyone was gone for the day. At 8:29 AM on Thursday morning…9 seconds apart…two people (from two different offices, and they’d both colluded with the same person in a third office) posted the winning answer on the Facebook group’s wall. Considering that I was a little concerned that the whole thing would be a total dud, I certainly didn’t expect to have winners before 8:30 AM on the first day!

Different from “Games”

So, gamification is not simply “playing games.” It’s using the aspects of human nature that make playing games fun and engaging…and then leveraging those to drive interest and engagement around a brand, a product, an event, or something else. It’s an utterly intriguing concept, and it’s not hard to spot examples of marketers putting these ideas to good use.

Another paper/presentation on the subject published late last year by Resource Interactive has some additional good nuggets on the topic:

Measuring the Results

Any marketing initiative should be measured. Campaigns that rely heavily on game mechanics are easier to measure than a lot of always-on social media activities (a Twitter feed, a Facebook page, etc.). That is, they’re easier to measure if there is a clear objective for the effort, and if that objective is something that gamification is good at supporting: driving engagement and/or driving awareness (and education) through word-of-mouth. Meaningful KPIs may include:

• The number of people exposed to the campaign
• The number of people who participated in the game mechanics aspects of the game
• The number of people who reached a certain level of engagement with the campaign

Now, this sets me up for the criticism, “Well, yeah, but did it drive business results.” In some cases, CTAs can be embedded in the game that can lead to conversions that can be measured as results. But, there is, admittedly, some requirement that the entire campaign has been designed with a logical link to business value. For instance, for a low-awareness brand targeted at a niche audience, then a campaign that grows awareness across a community that represents that niche, and that does so at a relatively low cost will often be a no-brainer when compared with low-engagement paid media.

Benchmarks will seldom be available for these types of campaigns. Get over it! If you’re developing a compelling campaign, it’s going to need some degree of originality, which means there won’t be a sea of comparable campaigns at your fingertips for benchmarking. That makes establishing targets a bit scary. Set a target anyway. Think through what would be acceptable and what would be clearly awesome based on other, more traditional ways you could have chosen to invest those same dollars. More often than not, if it’s a well-designed game-mechanics-applied campaign, you will know whether you are on to a good idea early in the planning, and you will be very pleasantly surprised by the results.

© Tim for Gilligan on Data by Tim Wilson, 2011. | Permalink | No comment | Add to del.icio.us
Post tags: game mechanics, gamification

Feed enhanced by Better Feed from Ozh

I’ve always felt that “web analytics” is a poor label for what most of us who spend a significant amount of our time with web behavioral data do day in and day out. I see three different types of information-providing:

• Reporting — recurring delivery of the same set of metrics as a critical tool for performance monitoring and performance management
• Analysis —  hypothesis-driven ad hoc assessment geared towards answering a business question or solving a business problem (testing and optimization falls into this bucket as well)
• Analytics — the development and application of predictive models in the support of forecasting and planning

My dander gets raised when anyone claims or implies that our goal should be to spend all of our time and effort in only one of these areas.

Reporting <> (Necessarily) Data Puking

I’ll be the first person to decry reporting squirrel-age. I expect to go to my grave in a world where there is still all too much pulling and puking of reams of data. But (or, really, BUT, as this is a biggie), a wise and extremely good-looking man once wrote:

If you don’t have a useful performance measurement report, you have stacked the deck against yourself when it comes to delivering useful analyses.

It bears repeating, and it bears repeating that dashboards are one of the most effective means of reporting. Dashboards done well (and none of the web analytics vendors provide dashboards well enough to use their tools as the dashboarding tool) meet a handful of dos and don’ts:

• They DO provide an at-a-glance view of the status and trending of key indicators of performance (the so-called “Oh, shit!” metrics)
• They DO provide that information in the context of overarching business objectives
• They DO provide some minimal level of contextual data/information as warranted
• They DON’T exceed a single page (single eyescan) of information
• They DON’T require the person looking at them to “think” in order to interpret them (no mental math required, no difficult assessment of the areas of circles)
• They DON’T try to provide “insight” with every updated instance of the dashboard

The last item in this list uses the “i” word (“insight”) and can launch a heated debate. But, it’s true: if you’re looking for your daily, weekly, monthly, or real-time-on-demand dashboard to deliver deep and meaningful insights every time someone looks at it, then either:

• You’re not clear on the purpose of a dashboard, OR
• You count, “everything is working as expected” to be a deep insight

Below is a perfectly fine (I’ll pick one nit after the picture) dashboard example. It’s for a microsite whose primary purpose is to drive registrations to an annual user conference for a major manufacturer. It is produced weekly, and it is produced in Excel, using data from Sitecatalyst, Twitalyzer, and Facebook. Is this a case of, as Avinash put it, us being paid “an extra $15 an hour to dump the data into Excel and add a color to the table header?” Well, maybe. But, by using a clunky Sitecatalyst dashboard and a quick glance at Twitalyzer and Facebook, the weekly effort to compile this is: 15 minutes. Is it worth $3.75 per week to get this? The client has said, “Absolutely!”

I said I would pick one nit, and I will. The example above does not do a good job of really calling out the key performance indicators (KPIs). It does, however, focus on the information that matters — how much traffic is coming to the site, how many registrations for the event are occurring, and what the fallout looks like in the registration process. Okay…one more nit — there is no segmentation of the traffic going on here. I’ll accept a slap on the wrist from Avinash or Gary Angel for that — at a minimum, segmenting by new vs. returning visitors would make sense, but that data wasn’t available from the tools and implementation at hand.

An Aside About On-Dashboard Text

I find myself engaged in regular debates as to whether our dashboards should include descriptive text. The “for” argument goes much like Avinash’s implication that “no text” = “limited value.” The main beef I have with any sort of standardized report or dashboard including a text block is that, when baked into a design, it assumes that there is the same basic word count of content to say each time the report is delivered. That isn’t my experience. In some cases, there may be quite a bit of key callouts for a given report…and the text area isn’t large enough to fit it all in. In other cases, in a performance monitoring context, there might not be much to say at all, other than, “All systems are functioning fine.” Invariably, when the latter occurs, in an attempt to fill the space, the analyst is forced to simply describe the information already effectively presented graphically. This doesn’t add value.

If a text-based description is warranted, it can be included as companion material. <forinstance> “Below is this week’s dashboard. If you take a look at it, you will, as I did, say, ‘Oh, shit! we have a problem!’ I am looking into the [apparent calamitous drop] in [KPI] and will provide an update within the next few hours. If you have any hypotheses as to what might be the root cause of [apparent calamitous drop], please let me know” </forinstance> This does two things:

1. Enables the report to be delivered on a consistent schedule
2. Engages the recipients in any potential trouble spots the (well-formed) dashboard highlights, and leverages their expertise in understanding the root cause

Which…gets us to…

Analysis

Analysis, by [my] definition, cannot be something that is scheduled/recurring/repeating. Analysis is hypothesis-driven:

• The dashboard showed an unexpected change in KPIs. “Oh, shit!” occurred, and some root cause work is in order
• A business question is asked: “How can we drive more Y?” Hypotheses ensue

If you are repeating the same analysis…you’re doing something wrong. By its very nature, analysis is ad hoc and varied from one analysis to another.

When it comes to the delivery of analysis results, the medium and format can vary. But, I try to stick with two key concepts — both of which are violated multiple times over in every example included in Avinash’s post:

• The principles of effective data visualization (maximize the data-pixel ratio, minimize the use of a rainbow palette, use the best visualization to support the information you’re trying to convey, ensure “the point” really pops, avoid pie charts at all costs, …) still need to be applied
• Guy Kawasaki’s 10-20-30 rule is widely referenced for a reason — violate it if needed, but do so with extreme bias (aka, slideuments are evil)

While I am extremely wordy on this blog, and my emails sometimes tend in a similar direction, my analyses are not. When it comes to presenting analyses, analysts are well-served to learn from the likes of Garr Reynolds and Nancy Duarte when it comes to how to communicate effectively. It’s sooooo easy to get caught up in our own brilliant writing that we believe that every word we write is being consumed with equal care (you’re on your third reading of this brilliant blog post, are you not? No doubt trying to figure which paragraph most deserves to be immortalized as a tattoo on your forearm, right? You’re not? What?!!!). “Dumb it down” sounds like an insult to the audience, and it’s not. Whittle, hone, remove, repeat. We’re not talking hours and hours of iterations. We’re talking about simplifying the message and breaking it up into bite-sized, consumable, repeatable (to others)  chunks of actionable information.

Analysis Isn’t Reporting

Analysis and reporting are unquestionably two very differing things, but I don’t know that I agree with assertions that analysis requires an entirely different skillset from reporting. Meaningful reporting requires a different mindset and skillset from data puking, for sure. And, reporting and analysis are two different things, but you can’t be successful with the latter without being successful with the former.

Effective reporting requires a laser focus on business needs and business context, and the ability to crisply and effectively determine how to measure and monitor progress towards business objectives. In and of itself, that requires some creativity — there are seldom available metrics that are perfectly and directly aligned with a business objective.

Effective analysis requires creativity as well — developing reasonable hypotheses and approaches for testing them.

Both reporting and analysis require business knowledge, a clear understanding of the objectives for the site/project/campaign/initiative, a better-than-solid understanding of the underlying data being used (and its myriad caveats), and effective presentation of information. These skills make up the core of a good analyst…who will do some reporting and some analysis.

What About Analytics?

I’m a fan of analytics…but see it as pretty far along the data maturity continuum. It’s easy to poo-poo reporting by pointing out that it is “all about looking backwards” or “looking at where you’ve been.” But, hey, those who don’t learn from the past are condemned to repeat it, no? And, “How did that work?” or “How is that working?” are totally normal, human, helpful questions. For instance, say we did a project for a client that, when it came to the results of the campaign from the client’s perspective, was a fantastic success! But, when it came to what it cost us to deliver the campaign, the results were abysmal. Without an appropriate look backwards, we very well might do another project the same way — good for the client, perhaps, but not for us.

In general, I avoid using the term “analytics” in my day-to-day communication. The reason is pretty simple — it’s not something I do in my daily job, and I don’t want to put on airs by applying a fancy word to good, solid reporting and analysis. At a WAW once, I actually heard someone say that they did predictive modeling. When pressed (not by me), it turned out that, to this person, that meant, “putting a trendline on historical data.” That’s not exactly congruent with my use of the term analytics.

Your Thoughts?

Is this a fair breakdown of the work? I scanned through the comments on Avinash’s post as of this writing, and I’m feeling as though I am a bit more contrarian than I would have expected.Similar Posts:

• Measurement Strategies: Balancing Outcomes and Outputs
• The “Action Dashboard” — Avinash Mounts My Favorite Soapbox
• ROI — the Holy Grail of Marketing (and Roughly as Attainable)
• Put-in-Play Percentage: A “Great Metric” for Youth Baseball?
• What is “Analysis?”

© Tim for Gilligan on Data by Tim Wilson, 2011. | Permalink | 7 comments | Add to del.icio.us
Post tags: Avinash Kaushik

Feed enhanced by Better Feed from Ozh

Let’s say you’re Chipotle. You’re not nearly as awesome as Freebirds, but you’ve got a much larger market presence, and you’ve embraced social media. So, what do you do when it comes to measuring and monitoring Twitter?

Online Listening Tools, Perhaps?

Chipotle, I’m sure, is already using one of the major online listening platforms — Radian6, Nielsen Buzzmetrics (or whatever their current platform name is), Converseon, SM2, etc. Online listening platforms purport to index all social media so that queries can be built to assess the volume, sentiment, and influencers related to any topic in any and all social media channels. Here’s the challenge when it comes to Twitter:

The Physics of Twitter apply!

Only two entities have unfettered access to all tweets, and neither have made their full datasets available to anyone:

• Twitter
• The Library of Congress

Twitter has never allowed full access to all tweets through its API. I ran a test using TweetReach and one of the three online listening platform leaders according to Forrester, and the results bore this out. For the 3-day period where I compared two simple queries (it was a brand that ran a TV ad during the Oscars, and that date was the middle day in the 3-day period I evaluated), the listening platform reported 513 total tweets, while TweetReach reported 1,225. Comparing the tweets, there were 6 that the listening platform included that were missing from TweetReach, and a staggering 718 tweets that TweetReach included that the listening platform was missing.

I wasn’t surprised, but it illustrates the point. Online listening platforms are good for what they do, but Twitter detail does not count.

So, What Tool Should I Use Instead?

There really is no single tool that tells me everything I want to know about a brand’s Twitter presence. But, I do think two tools complement each other well: Twitalyzer and TweetReach. These are both paid tools, but the costs are nominal. There are scads of other paid tools, as well as free tools (for a pretty good run-down on Twitter tools, check out Michele Hinojosa’s Twitter Analytics presentation), but I keep finding myself coming back to these two.

The key distinction between the two tools is that Twitalyzer is user-centric, and TweetReach is tweet-centric tool. The platforms have some overlapping capabilities, but their underlying orientation is quite different. To illustrate, think about how Chipotle might view and manage Twitter. The Venn diagram below is a possible representation of the relevant Twitterverse from their perspective:

Twitalyzer

Twitalyzer provides a range of metrics related to @chipotletweets (and competitors, such as @tacobell) – it’s oriented towards the green circles above. These include both straight-up measures — follower count, number of lists that include the account, number of tweets, number of times the account is replied to and retweeted, etc. — and calculated metrics that take into account how often the account is retweeted/replied to, the number of followers of followers of the account (and the number of followers of users that reference the account), and so on to estimate things like the overall impact of the account, its influence, and it’s potential and effective reach (click on the image to see the full range of available metrics, as well as to see current data):

Using Twitalyzer, you can track an account’s performance over time — be that your own account or competitor accounts. Within the tool, you can identify members of an account’s overarching network who have the highest impact in the channel, and you can do the same for a particular keyword or topic. The key, though, is that the tool operates primarily as a user-oriented measurement tool. For deeper thoughts on Twitalyzer, check out Michele Hinojosa’s post on the tool, and, if you still want more, I wrote a post dedicated to the tool as well.

One key note about Twitalyzer is that it’s not able to violate the Physics of Twitter any more than any other platform. It relies on the Twitter API to pull user data, and an unlimited user history is not directly available. So, with the paid Twitalyzer accounts, the tool goes out and checks Twitter every day and updates its metrics — keeping a running history from that point forward. If someone else is already having those daily updates pulled for a user, and you go in and “twitalyze” the user, you’ll get a nice set of historical data. If someone twitalyzed the user 2 months ago and no one has twitalyzed it since, then, when you twitalyze (twitalyze, twitalyze, twitalyze!!!) that user, you’ll see choppy data as Twitalyzer tries to fill in the holes in its dataset.

In short, don’t think that Twitalyzer keeps some sort of running record of all Twitter users over all time. The volume of daily tracked users they do have is growing, but you really have to load in the accounts you care about to get consistent data from that point forward. It’s kinda’ like web analytics that way — if you don’t tag it, you can’t necessarily expect to measure it effectively!

TweetReach

TweetReach is, in some ways, more like a traditional online listening platforms (Hah! Traditional! A “tradition” going allllll the way back to 2006 or so), in that it uses keyword-driven queries to tap into the Twitterverse (via the Twitter search API) and then pull and analyze the subsets of tweets that fit those query criteria. Basically, TweetReach is geared towards the blue bubbles in the Venn diagram above. And, not just the blue bubbles, but particular areas of blue bubble intersections (“fast food” AND “burritos,” for instance). With TweetReach, you set up a “tracker” using Boolean logic to include and exclude tweets based on keywords and phrases. The tracker then taps into Twitter and starts retrieving and analyzing those tweets. Part of the output you get in near-real-time looks something like this:

For the date range you choose, the tool provides the total reach (the sum of unique people following people who generated one or more tweets that met the tracker criteria) of the topic, the total exposure (similar to reach, but not de-duped to unique people), total tweets, and total contributors. That’s useful in and of itself, because it quickly shows the impact of a campaign or event, as well as the volume of Twitter conversation occurring around a given topic over time.

In addition, the platform provides all of the tweets that drove those numbers and allows them to be sorted by “highest exposure” — the tweets that generated the most impressions, through a combination of followers of the person who tweeted and the volume of retweets (and the number of followers of the people who retweeted). The tool also provides a simple list of users who generated the highest number of impressions. Plug some of those people into Twitalyzer, and you can get a quick read as to whether they’re someone you may want to keep an eye on and even engage directly.

Since TweetReach is aimed squarely at Twitter, it does a better job of comprehensively capturing relevant tweets than other tools — paid listening platforms (as discussed early in this post) or free Twitter archiving services. By downloading the raw tweets for a relevant timeframe, you can do some basic text analysis — generate a word cloud, for instance — and get a quick view of the sentiment, tone, and word usage of these tweets. TweetReach posted a couple of great examples of word clouds from trackers after the Super Bowl. Neat stuff, huh?

You Said There Was Overlap?

The tools do overlap. Twitalyze can do some hashtag and topic analysis, but it doesn’t have the Boolean querying capabilities of TweetReach. TweetReach can do some user-based analysis by running reports and trackers using “from:<username>.” Both tools report “reach” numbers. But, with Twitalyzer, it’s the reach of the user, while, with TweetReach, it’s the reach of the topic. Both are valuable, but they’re different.

I’m sure there are other ways that they overlap, but I wouldn’t want to run solo with either tool! They complement each other fantastically, though!

© Tim for Gilligan on Data by Tim Wilson, 2011. | Permalink | One comment | Add to del.icio.us
Post tags:

Feed enhanced by Better Feed from Ozh

In a discussion last week, I grabbed a dry erase marker and sketched out a quick diagram on an 8″x8″ square of nearby whiteboard to try to illustrate the landscape of social media measurement tools. A commute’s worth o’ pondering heading home that evening, followed by a similar commute back in the next morning, and I realized I might have actually gotten a reasonable-to-comprehend picture that showed how and wear the myriad social media measurement tools fit.

Here it is (yep — click on the image to view a larger version):

‘Splain Yourself, Lucy

The first key to this diagram is that it makes a distinction between “individual channel performance” and “overall brand results.” Think about the green box as being similar to a publicly traded company’s quarterly filing. It includes an income statement that shows total revenue, total expenses, and net income. Those are important measures, but they’re not directly actionable. If a company’s profitability tanks in any given quarter, the CEO can’t simply say, “We’re going to take action to increase profitability!”  Rather, she will have to articulate actions to be taken in each line of business, within specific product lines, regarding specific types of expenses, etc. to drive an increase profitability. At the same time, by publicly announcing that profitability is important (a key objective) and that it is suffering, line of business managers can assess their own domains (the blue boxes above) and look for ways to increase profitability. In practice, both approaches are needed, but the actions actually occur in the “blue box” area.

When it comes to marketing, and especially when it comes to the fragmented consumer world of social media, things are quite a bit murkier. This means performance measurement should occur at two levels — at the overall ecosystem (the green box above), which is akin to the quarterly financial reporting of a public company, and at the individual channel level, which is akin to the line of business manager evaluating his area’s finances. I use a Mississippi River analogy to try to explain that approach to marketers.

Okay. Got It. Now, What about These “Measurement Instruments?”

Long, long, LONG gone are the days when a “web analyst” simply lived an breathed a web analytics tool and looked within that tool for all answers to all questions. First, we realized that behavioral data needed to be considered along with attitudinal data and backend system data. Then, social media came along introduced a whole other set of wrinkles. Initially, social media was simply “people talking about your brand.” Online listening platforms came onto the scene to help us “listen” (but not necessarily “measure”). Soon, though, social media channels became a platform where brands could have a formally managed presence: a Facebook fan page, a Twitter account, a YouTube channel, etc. Once that happened, performance measurement of specific channels became as important as performance measurement of the brand’s web site.

When it comes to “managing social media,” brand actions occur within a specific channel, and each channel should be managed and measured to ensure it is as effective as possible. Unfortunately, each of the channels is unique when it comes to what can be measured and what should be measured. Facebook, for instance, is an inherently closed environment. No tool can simply “listen” to everything being said in Facebook, because much of users’ content is only available to members of their social graph within the environment, or interactions they have with a public fan page. Twitter, on the other hand, is largely public (with the exception of direct messages and users who have their profile set to “private”). The differing nature of these environments mean that they should be managed differently, that they should be measured differently, and that different measurement instruments are needed to effectively perform that measurement.

Online listening platforms are not a panacea, no matter how much they present themselves as such. Despite what may be implied in their demos and on their sites, both the Physics of Facebook and the Physics of Twitter apply — data access limited by privacy settings in the former and limited by API throttling in the latter. That doesn’t mean these tools don’t have their place, but they are generalist tools and should be seen primarily as generalist measurement platforms.

Your Diagram Is Missing…

I sketched the above diagram in under a minute and then drew it in a formal diagram in under 30 minutes the next morning. It’s not comprehensive by any means — neither with the three “social media channels” (the three channels listed are skewed heavily towards North America and towards consumer brands…because that’s where I spend the bulk of my measurement effort these days) nor with the specific measurement instruments. I’m aware of that. I wasn’t trying to make a totally comprehensive eye chart. Rather, I was trying to illustrate that there are multiple measurement instruments that need to be implemented depending on what and where measurement is occurring.

As one final point, you can actually wipe out the “measurement instrument” boxes and replace those with KPIs at each level. You can swap out the blue boxes with mobile channels (apps, mobile site, SMS/MMS, mobile advertising). I’m (clearly) somewhat tickled with the construct as a communication and planning tool. I’d love to field some critiques so I can evolve it!Similar Posts:

• Social Media Measurement: A Practitioner’s Practical Guide

© Tim for Gilligan on Data by Tim Wilson, 2011. | Permalink | 2 comments | Add to del.icio.us
Post tags: social media measurement

Feed enhanced by Better Feed from Ozh

The three presentations of the evening were:

• Social Search by Dave Culbertson of Lightbulb Interactive
• Campaign measurement planning by Bryan Cristina of Nationwide Insurance
• Web analytics platforms comparison by yours truly of Resource Interactive

Dave’s presentation was the most informal and focussed on the various developments across Facebook/Bing and Google when it comes to incorporating social graph and social profile data into search results. The Google video he showed was pretty interesting, and he illustrated how rapidly the space is evolving. But, overall, I took lousy notes, so I don’t know that I’ll manage to get a full blog post up on the subject.

As for Bryan’s presentation, I had the benefit of previewing the material and, as such, getting to have a mini-Q&A with Bryan via e-mail.

Campaign Measurement Planning

Bryan and I are both pretty passionate about measurement planning. His presentation really nails some key points about the topic and has a fantastic list on slide 8 as to elements to consider including in a measurement plan:

In addition, Bryan provided a (click to download) measurement plan example Word document (it’s an auto insurance company example, so it’s obviously grounded in reality, but he worked it over pretty thoroughly on several fronts in preparation for the presentation, so it is an entirely fictional example).

I asked Bryan a couple of questions offline about his approach prior to the event:

Q: Under “Targets and Benchmarks,” you note, “Don’t be afraid to put ‘TBD’ or ‘No Data’ for some benchmarks.” If that is the case, do you support not setting a target, or should you still try to set a target (even noting that it is a bit of a swag) in the absence of a benchmark?

Bryan’s response: I try to set a target no matter what because it gets people at least thinking about it and TRYING to set up some kind of expectation.  It makes sure that people are at least estimating.  Maybe they don’t know the CPC for the search terms yet, aren’t sure on the demand, and aren’t sure on the completion rates, but it’s at least a start.  We were completely off for one of our last campaigns because we had no idea on all those factors.  It still gave the agency something to report on for their % of goal and it drove an informed discussion mid-campaign.

[I, of course, loved this answer...because I totally agreed with it]

Q: You’re at a company that uses agencies for much of the campaign execution, and, clearly, you have put in a process whereby you develop this sort of plan partly as a tool to drive clarity and alignment with the agencies and their work. As an agency analyst, we are increasingly including “measurement planning” as a non-optional part of the scope of our engagements. In those cases, we (the agency) actually do the discovery and documentation of the measurement plan (which clients provide input to, review, and approve). I actually would love to have clients coming to us with this level of forethought, but, in the absence of that, what are your thoughts on having the accountability for the creation of a measurement plan reside with an agency?

Bryan’s response: I think this varies on the relationship between the agency and company.  For us, we’re very capable, we all know how to do the campaign execution, but we just don’t have the time or bodies to do it.  I’m sure there are many companies that have no clue how to do it, so the agency does both the execution and the strategy, or the execution, strategy, tracking, plus reporting, or whatever else. It really depends on the analytics maturity of the client as to whether it makes more sense for the agency or the client to own the creation of the plan.  If it’s the agency, you’d have to be absolutely be sure to talk to the client in depth about all of it and make sure they’re on board with all the points.  In the end, the outcome should be the same, the only difference really being the author of the document would be the agency instead of the business, and I’m sure some of the reporting responsibilities would change based on that.

Bryan joked during his presentation about how “exciting” the topic of measurement planning is. Obviously, it can seem like a pretty dry topic, but, in both of our experiences, measurement planning can drive some tough and interesting discussions. More importantly, it’s a foundational element of marketing — without it, you wind up looking back after the fact and wondering if what you executed was successful, whether you captured the right data, and whether you learned anything that can be meaningfully applied to the next initiative.

Hey…I also cleaned up my “sharing” options on my blog this weekend. Go ahead. Give it a try! See how easy it is to Like or Tweet (or…er…whether I really got those implemented and functioning correctly). Who knows, maybe Facebook Insights will start giving me some interesting web site data!Similar Posts:

• Monish Datta Learns All about Facebook Measurement

© Tim for Gilligan on Data by Tim Wilson, 2011. | Permalink | One comment | Add to del.icio.us
Post tags: measurement planning, WAW Columbus

Feed enhanced by Better Feed from Ozh