Fairly regularly, I’m drawn into work where the team has unrealistic expectations of the degree to which I can accurately quantify the impact of their initiatives on their top (or bottom) line. I’ve come at these discussions from a variety of angles:

• Acknowledging why expectations are inflated — what I’ve termed The Frustration Gap
• Articulating the myriad ways that media mix modeling (MMM) has started to crumble
• Developing a handy analogy based on the Mississippi River to explain the complexity of today’s measurement reality
• Putting forth a framework for social media measurement that makes the distinction between measuring the performance of individual channels and measuring the overall brand outcomes that result from these channels working in concert

This post is largely an evolution of the last link above. It’s something I’ve been exploring over the past six months, and which was strongly reinforced when I read John Lovett’s recent book. As I’ve been doing measurement planning (measurement strategy? marketing optimization planning?) with clients, it’s turned out to be quite useful when I have the opportunity to apply it.

Initially, I referred to this approach as developing a “logical model” (that’s even what I called it towards the end of my second post that referenced John’s book), but that was a bit bothersome, since “logical model” has a very specific meaning in the world of database design. Then, a couple of months ago, I stumbled on an old Harvard Business Review paper about using non-financial measures for performance measurement, and that paper introduced the same concept, but referred to it as a “causal model.” I like it!

How It Works

The concept is straightforward, it’s not particularly time-consuming, it’s a great exercise for ensuring everyone involved is aligned on why a particular initiative is being kicked off, it sets up meaningful optimization work as individual tactics and campaigns are implemented, and it positions you to be able to demonstrate a link (correlation) between marketing activities and business results.

This approach acknowledges that there is no existing master model that shows exactly how a brand’s target consumers interact and respond to brand activity. The process starts with more “art” than “science” — knowledge of the brand’s target consumers and their behaviors, knowledge of emerging channels and where they’re most suited (e.g., a QR code on a billboard on a busy highway…not typically a good match), and a hefty dose of strategic thought.

The exact structure of this sort of model varies widely from situation from situation, but I like to have my measurable objectives — what we think we’re going achieve through the initiative or program that we believe has underlying business value — listed on the left side of the page, and then build linkages from that to a more definitive business outcome on the right:

It should fit on a single page, and it requires input from multiple stakeholders. Ultimately, it can be a simple illustration of “why we’re doing this” for anyone to review and critique. If there are some pretty big leaps required, or if there are numerous steps along the way to get to tangible business value, then it begs the question: “Is this really worth doing?” It’s an easy litmus test as to whether an initiative makes sense.

What I’ve found is that this exercise can actually alter the original objectives in the planning stage, which is a much better time and place to alter them than once execution is well under way!

Once the model is agreed to, then you can focus on measuring and optimizing to the outputs from the base objectives — using KPIs that are appropriate for both the objective and the “next step” in the causal model.

And, over time, the performance of those KPIs can be correlated with the downstream components of the causal model to validate (and adjust) the model itself.

This all gets back to the key that measurement and analytics is a combination of art and science. Initially, it’s more art than science — the science is used to refine, validate, and inform the art.

© Tim for Gilligan on Data by Tim Wilson, 2011. | Permalink | One comment | Add to del.icio.us
Post tags: causal model

Feed enhanced by Better Feed from Ozh

A recent Wall Street Journal article titled Business Schools Plan Leap Into Data covered the recognition by business schools that they are sending their students out into the world ill-equipped to handle the data side of their roles:

Data analytics was once considered the purview of math, science and information-technology specialists. Now barraged with data from the Web and other sources, companies want employees who can both sift through the information and help solve business problems or strategize.

That article spawned a somewhat cranky line of thought. It’s been a standard part of presentations and training I’ve given for years that there is a gap in our business schools when it comes to teaching students how to actually use data. And, the article includes a quote from an administrator at the Fordham business school: “Historically, students go into marketing because they ‘don’t do numbers.’” That’s an accurate observation. But, what is “doing numbers?” In the world of digital analytics, it’s a broad swath of activities:

• Consulting on the establishment of clear objectives and success measures (…and then developing appropriate dashboards and reports)
• Providing regular performance measurement (okay, this should be fully automated through integrated dashboards…but that’s easier said than done)
• Testing hypotheses that drive decisions and action using a range of analysis techniques
• Building predictive models to enable testing of different potential courses of action to maximize business results
• Managing on-going testing and optimization of campaigns and channels to maximize business results
• Selecting/implementing/maintaining/governing data collection platforms and processes (web analytics, social analytics, customer data, etc.)
• Assisting with the interpretation/explanation of “the data” — supporting well-intended marketers who have found “something interesting” that needs to be vetted

This list is neither comprehensive nor a set of discrete, non-overlapping activities. But, hopefully, it illustrates the point:

The “practice of data analytics” is an almost impossibly broad topic to be covered in a single college course.

What bothered me about the WSJ article are two things:

• The total conflation of “statistics” with “understanding the numbers”
• The lack of any recognition of how important it is to actually be planning the collection of the data — it doesn’t just automatically show up in a data warehouse

On the first issue, there is something of an on-going discussion as to what extent statistics and predictive modeling should be a core capability and a constantly applied tool in the analyst’s toolset. Michael Healy made a pretty compelling case on this front in a blog post earlier this year – making a case for statistics, econometrics, and linear algebra as must-have skills for the web analyst. As he put it:

If the most advanced procedure you are regularly using is the CORREL function in Excel, that isn’t enough.

I’ve…never used the CORREL function in Excel. It’s certainly possible that I’m a total, non-value-add reporting squirrel. Obviously, I’m not going to recognize myself as such if that’s the case. I’ve worked with (and had work for me) various analysts who have heavy statistics and modeling skills. And, I relied on those analysts when conditions warranted. Generally, this was when we were sifting through a slew of customer data — profile and behavioral — and looking for patterns that would inform the business. But this work accounted for a very small percentage of all of the work that analysts did.

I’m a performance measurement guy because, time and again, I come across companies and brands that are falling down on that front. They wait until after a new campaign has launched to start thinking about measurement. They expect someone to deliver an ROI formula after the fact that will demonstrate the value they delivered. They don’t have processes in place to monitor the right measures to trigger alarms if their efforts aren’t delivering the intended results.

Without the basics of performance measurement — clear objectives, KPIs, and regular reporting — there cannot be effective testing and optimization. In my experience, companies that have a well-functioning and on-going testing and optimization program in place are the exception rather than the rule. And, companies that lack the fundamentals of performance management that try to jump directly to testing and optimization find themselves bogged down when they realize they’re not entirely clear what it is they’re optimizing to.

Diving into statistics, econometrics, and predictive modeling in the absence of the fundamentals is a dangerous place to be. I get it — part of performance measurement and basic analysis is understanding that just because a number went “up” doesn’t mean that this wasn’t the result of noise in the system. Understanding that correlation is not causation is important — that’s an easy concept to overlook, but it doesn’t require a deep knowledge of statistics to sound an appropriately cautionary note on that front. 9 times out of 10, it simply requires critical thinking.

None of this is to say that these advanced skills aren’t important. They absolutely have their place. And the demand for people with these skills will continue to grow. But, implying that this is the sort of skill that business schools need to be imparting to their students is misguided. Marketers are failing to add value at a much more basic level, and that’s where business schools need to start.

© Tim for Gilligan on Data by Tim Wilson, 2011. | Permalink | 8 comments | Add to del.icio.us
Post tags:

Feed enhanced by Better Feed from Ozh

One of the more directly usable sections of the book is in Chapter 5, where Lovett lays out pseudo formulas for KPIs for various possible (probable) social media business objectives. This post started out to be about my experiences drilling down into some of those formulas…but then the content took a turn, and one of Lovett’s partners at Web Analytics Demystified wrote a provocative blog post…so I’ll save the formula exploration for a subsequent post.

Instead…Social Media ROI

Lovett explicitly notes in his book that there is no secret formula for social media ROI. In my mind, there never will be — just as there will never be unicorns, world peace, or delicious chocolate ice cream that is as healthy as a sprig of raw broccoli, no matter how much little girls and boys, rationale adults, or my waistline wish for them.

Yes, the breadth of social media data available is getting better by the day, but, at best, it’s barely keeping pace with the constant changes in consumer behavior and social media platforms. It’s not really gaining ground.

What Lovett proposes, instead of a universally standard social media ROI calculation, is that marketers be very clear as to what their business objectives are – a level down from “increase revenue,” “lower costs,”and “increase customer satisfaction” – and then work to measure against those business objectives.

The way I’ve described this basic approach over the past few years is using the phrase “logical model,” – as in, “You need to build a logical link from the activity you’re doing all the way to ultimate business benefit, even if you’re not able to track those links all the way along that chain. Then…measure progress on the activity.”

Unfortunately, “logical model” is a tricky term, as it already has a very specific meaning in the world of database design. But, if you squint and tilt you’re head just a bit, that’s okay. Just as a database logical model is a representation of how the data is linked and interrelated from a business perspective (as opposed to the “physical model,” which is how the data actually gets structured under the hood), building a logical model of how you expect your brand’s digital/social activities to ladder up to meaningful business outcomes is a perfectly valid  way to set up effective performance measurement in a messy, messy digital marketing world.

No Wonder These Guys Work Together

Right along the lines of Lovett’s approach comes one of the other partners at Web Analytics Demystified with, in my mind, highly complementary thinking. Eric Peterson’s post about The Myth of the “Data-Driven Business” postulates that there are pitfalls a-looming if the digital analytics industry continues to espouse “being totally data-driven” as the penultimate goal. He notes:

…I simply have not seen nearly enough evidence that eschewing the type of business acumen, experience, and awareness that is the very heart-and-soul of every successful business in favor of a “by the numbers” approach creates the type of result that the “data-driven” school seems to be evangelizing for.

What I do see in our best clients and those rare, transcendent organizations that truly understand the relationship between people, process, and technology — and are able to leverage that knowledge to inform their overarching business strategy — is a very healthy blend of data and business knowledge, each applied judiciously based on the challenge at hand. Smart business leaders leveraging insights and recommendations made by a trusted analytics organization — not automatons pulling levers based on a hit count, p-value, or conversion rate.

I agree 100% with his post, and he effectively counters the dissenting commenters (partial dissent, generally – no one has chimed in yet fully disagreeing with him). Peterson himself questions whether he is simply making a mountain out of a semantic molehill. He’s not. We’ve painted ourselves into corners semantically before (“web analyst” is too confining a label, anyone…?). The sooner we try to get out of this one, the better — it’s over-promising / over-selling / over-simplifying the realities of what data can do and what it can’t.

Which Gets Back to “Is It Easy?”

Both Lovett’s and Peterson’s ideas ultimately go back to the need for effective analysts to have a healthy blend of data-crunching skills and business acumen. And…storytelling! Let’s not forget that! It means we will have to be communicators and educators — figuring out the sound bites that get at the larger truths about the most effective ways to approach digital and social media measurement and analysis. Here’s my quick list of regularly (in the past…or going forward!) phrases:

• There is no silver bullet for calculating social media ROI — the increasing fragmentation of the consumer experience and the increasing proliferation of communication channels makes it so
• We’re talking about measuring people and their behavior and attitudes — not a manufacturing process; people are much, much messier than widgets on a production line in a controlled environment
• While it’s certainly advisable to use data in business, it’s more about using that data to be “data-informed” rather than aiming to be “data-driven” — experience and smart thinking count!
• Rather than looking to link each marketing activity all the way to the bottom line, focus on working through a logical model that fits each activity into the larger business context, and then find the measurement and analysis points that balance “nearness to the activity” with “nearness to the ultimate business outcome.”
• Measurement and analytics really is a mix of art and science, and whether more “art” is required or more “science” is required varies based on the specific analytics problem you’re trying to solve

There’s my list — cobbled from my own experience and from the words of others!Similar Posts:

• Have You Picked Up a Copy of “Social Media Metrics Secrets” Yet?
• Complex Processes and Analyses Therein
• The New Facebook Insights — One More Analyst’s Take

© Tim for Gilligan on Data by Tim Wilson, 2011. | Permalink | 4 comments | Add to del.icio.us
Post tags: Eric Peterson, John Lovett

Feed enhanced by Better Feed from Ozh

To give credit where credit is due, the discussion really took off when Eric Peterson posted an excerpt and a link in Google+ and solicited thoughts from the Google+/#measure community. That turned into the longest thread I’ve participated in to date on Google+, and subsequently led to a Google+ hangout that Eric set up and then moderated yesterday.

This post is an attempt to summarize the highlights of what I saw/heard/learned over the past week.

What I Learned about the #measure Community

Overall, the discussion brought back memories of some of the threads that would occasionally get started on the webanalytics Yahoo! group back in the day. That’s something we’ve lost a bit with Twitter…but more on that later.

What I took away about the group of people who make up the community was pretty gratifying:

• A pretty united “we” – everyone who participated in the discussions was contributing with the goal of trying to move the discussion forward; as a community, everyone agrees that we’re at some sort of juncture where “web analytics” is an overly limiting label, where the evolution of consumer behavior (read: social media and mobile) and consumer attitudes (read: privacy) are impacting the way we will do our job in the future, and where the world of business is desperately trying to be more data-driven…and floundering more often than succeeding. There are a lot of sharp minds who are perfectly happy to share every smart thought they’ve got on the subject if it helps our industry out — the ol’ “a rising tide lifts all boats” scenario. That’s a fun community with whom to engage.
• Strong opinions but small egos – throughout the discussion that occurred both on Google+ and on Twitter (as well as in several blog posts that the discussion spawned, like this one by Evan LaPointe and Nancy Koon’s inaugural one and Eric’s post), there were certainly differing points of view, but things never got ugly; I actually had a few people reach out to me directly to make sure that their thoughts hadn’t been taken the wrong way (they hadn’t been)
• 100s of years of experience — we have a lot of experience from a range of backgrounds when it comes to trying to figure out the stickiest of the wickets that we’re facing. That is going to serve us well.
• (Maybe) Agencies and vendors leading the way? – I don’t know that I learned this for sure, but an informal tally of the participants in the discussion showed a heavy skewing towards vendor and agency (both analytics agencies and marketing/creative/advertising agencies) representation with pretty limited “industry” participation. On the one hand, that is a bit concerning. On the other hand, having been in “industry” for more of my analytics career than I’ve been on the agency side, it makes sense that vendors and agencies are exposed to a broader set of companies facing the same challenges, are more equipped to see the patterns in the challenges the analytics industry is facing, and are being challenged from more directions to come up with answers to these challenges sooner rather than later.

These were all good things to learn — the people in the community are one of the reasons I love my job, and this thread demonstrated some of the reasons why that is.

Highlights of the Discussion

Boiling down the discussion is bound to leave some gaps, and, if I started crediting individuals with any of the thoughts, I’d run the serious risk of misrepresenting them, so feel free to read the Google+ thread yourself in its entirety (and the follow-up thread that Eric started a few days later). I’ve called out any highlights that came specifically from the hangout as being from there (participants there were Adam Greco, John Lovett, Joseph Stanhope, Tim Wilson, Michael Helbling, John Robbins, Emer Kirrane, Lee Isensee, Keith Burtis, and me), since there isn’t a reviewable transcript for that.

Here goes:

• Everyone recognizes that a “just plug it in and let the technology spit out insights” solution will likely never exist — the question is how much of the technical knowledge (data collection minutia, tool implementation nuances, reporting/analysis interface navigation) can be automated/hidden. A couple of people (severalpublicly, one privately) observed that we want (digital) analytics platforms to be a like a high-performance car — all the complexity as needed under the hood, but high reliability and straightforward to operate. Pushing that analogy — how far and fast it runs will still be highly dependent on the person behind the wheel (the analyst).
• Adobe/Omniture and Google Analytics had near-simultaneous releases of their latest versions; both companies touted the new features being rolled out…but both companies have stressed that there was a lot more about the releases that were under-the-hood changes that were positioning the products for greater advances in subsequent releases; time will tell, no? And, several people who have actually been working  with SC15 (I’ve only seen a couple of demos, watched some videos, and read some blog posts — the main Omniture clients I support are over a year out from seeing SC15 in production), have pointed out that some of the new features (Processing Rules and Context Data, specifically) will really make our lives better
• There was general consensus that Omniture has gotten much, much better over the years about listening to customer feedback and incorporating changes based on that feedback; there is still a Big Question as to whether customer-driven incremental improvements (even improvements that require significant updates on the back end) will get to true innovation — the “last big innovations” in web analytics were pointed out as being a decade ago (I would claim that the shift from server logs and image beacons to Javascript-based page tags was innovative and wasn’t much older) — or whether “something else” will have to happen was a question that did not get resolved
• Getting beyond “the web site” is one major direction the industry is heading — integrating cross-channel data and then getting value from it – introduces a whole other level of complexity…but the train is barrelling along on a track that has clearly been laid in that direction
• We all get sucked into “solving the technical problem” over “focusing on the business results” — the tools have enough complexity that we count it a “win” when we solve the technical issues…but we’re not really serving anyone well when we stop there; this is one of those things, I suspect, that we all know and we constantly try to remind ourselves…and yet still get sucked into the weeds of the technology and forget to periodically lift our heads up and make sure we’re actually adding value; John Lovett has been preaching about this conundrum for years (and he hits on it again in his new book)
• Marketing/business are getting increasingly complex, which means the underlying data is getting more complex (and much more plentiful — another topic John touches on in his book), which means getting the data into a format that supports meaningful analysis is getting tougher; trying to keep up with that trend is hard enough without trying to get ahead!
• Tag management — is it an innovation, or is it simply a very robust band-aid? Or is it both? No real consensus there.
• Possible areas where innovation may occur: cross-channel integration, optimization, improved conversion tracking (which could encompass both of the prior two areas), integration of behaviora/attitudinal/demographic data
• [From the hangout] “Innovation” is a pretty loaded term. Are we even clear on what outcome we’re hoping to drive from innovation?
• [From the hangout] Privacy, privacy, privacy! Is it possible to educate the consumer and/or shift the consumer’s mindset such that they are informed about why that “tracking” them isn’t evil? Can we kill the words “tracking” and “targeting,” which both freak people out? Why are consumers fine with allowing the mobile or Facebook application access to their private data…but freak out about no-PII behavioral tracking (we know why, but it still sucks)?
• [From the hangout] How did a conversation about where and how innovation will occur devolve into the nuts and bolts of privacy? Why does that happen so often with us? Is that a problem, or is it a symptom of something else?

Yikes! That’s my attempt to summarize the discussion! And it’s still pretty lengthy!

What I Learned about Google+

I certainly didn’t expect to learn anything about Google+ when I wrote the post — it was focusing on plain ol’ web (site) analytics, for Pete’s sake! But, I learned a few things nonetheless:

The good:

• Longer-form (than 140 characters) discussions, triggered by circles, with the ability to quickly tag people, are pretty cool; Twitter sort of forced us over to blog posts (and then comments on the posts) to have discussions…and Google+ has the potential to bring back richer, more linear dialogue
• Google+ hangouts…are pretty cool and fairly robust; we had a few hiccups here and there, but I was able to participate reasonably well from inside a minivan traveling down the highway that had the other four members of my family in it (Verizon 4G aircard, in case you’re wondering); and, as the system detects who is speaking, that person’s video jumps to the “main screen” pretty smoothly. It’s not perfect (see below), but we had a pretty meaty conversation in a one-hour slot (and credit, again, to Eric Peterson for his mad moderation skills — that helped!)

The not-so-good:

• Discussions aren’t threaded, and the “+1″ doesn’t really drive the organization of the discussion — multiple logical threads were spawned as the discussion continued, but the platform didn’t really reflect that, which many discussion forums have supported for years
• Linking the blog post to the discussion was a bit clunky. Who knows what long tail search down the road would benefit from seeing the original post and the ensuing conversation? I added a link to the Google+ discussion to the post after the fact…but it’s not the same as having a string of comments immediately following a post (and if Google+ fizzles…that discussion will be lost; I’ve made a PDF of the thread, but that feels awfully 2007)
• Google+ hangouts could use some sort of “hand-raising” or “me next” feature; everyone who participated in the hangout worked hard to not speak over anyone else, but we still had a number of awkward transitions

So, that’s what I took away. It was a busy week, especially considering I was knocking out the first half of John Lovett’s new book book (great stuff there) at the same time!

© Tim for Gilligan on Data by Tim Wilson, 2011. | Permalink | 3 comments | Add to del.icio.us
Post tags: Google+

Feed enhanced by Better Feed from Ozh

I’ve always felt that “web analytics” is a poor label for what most of us who spend a significant amount of our time with web behavioral data do day in and day out. I see three different types of information-providing:

• Reporting — recurring delivery of the same set of metrics as a critical tool for performance monitoring and performance management
• Analysis —  hypothesis-driven ad hoc assessment geared towards answering a business question or solving a business problem (testing and optimization falls into this bucket as well)
• Analytics — the development and application of predictive models in the support of forecasting and planning

My dander gets raised when anyone claims or implies that our goal should be to spend all of our time and effort in only one of these areas.

Reporting <> (Necessarily) Data Puking

I’ll be the first person to decry reporting squirrel-age. I expect to go to my grave in a world where there is still all too much pulling and puking of reams of data. But (or, really, BUT, as this is a biggie), a wise and extremely good-looking man once wrote:

If you don’t have a useful performance measurement report, you have stacked the deck against yourself when it comes to delivering useful analyses.

It bears repeating, and it bears repeating that dashboards are one of the most effective means of reporting. Dashboards done well (and none of the web analytics vendors provide dashboards well enough to use their tools as the dashboarding tool) meet a handful of dos and don’ts:

• They DO provide an at-a-glance view of the status and trending of key indicators of performance (the so-called “Oh, shit!” metrics)
• They DO provide that information in the context of overarching business objectives
• They DO provide some minimal level of contextual data/information as warranted
• They DON’T exceed a single page (single eyescan) of information
• They DON’T require the person looking at them to “think” in order to interpret them (no mental math required, no difficult assessment of the areas of circles)
• They DON’T try to provide “insight” with every updated instance of the dashboard

The last item in this list uses the “i” word (“insight”) and can launch a heated debate. But, it’s true: if you’re looking for your daily, weekly, monthly, or real-time-on-demand dashboard to deliver deep and meaningful insights every time someone looks at it, then either:

• You’re not clear on the purpose of a dashboard, OR
• You count, “everything is working as expected” to be a deep insight

Below is a perfectly fine (I’ll pick one nit after the picture) dashboard example. It’s for a microsite whose primary purpose is to drive registrations to an annual user conference for a major manufacturer. It is produced weekly, and it is produced in Excel, using data from Sitecatalyst, Twitalyzer, and Facebook. Is this a case of, as Avinash put it, us being paid “an extra $15 an hour to dump the data into Excel and add a color to the table header?” Well, maybe. But, by using a clunky Sitecatalyst dashboard and a quick glance at Twitalyzer and Facebook, the weekly effort to compile this is: 15 minutes. Is it worth $3.75 per week to get this? The client has said, “Absolutely!”

I said I would pick one nit, and I will. The example above does not do a good job of really calling out the key performance indicators (KPIs). It does, however, focus on the information that matters — how much traffic is coming to the site, how many registrations for the event are occurring, and what the fallout looks like in the registration process. Okay…one more nit — there is no segmentation of the traffic going on here. I’ll accept a slap on the wrist from Avinash or Gary Angel for that — at a minimum, segmenting by new vs. returning visitors would make sense, but that data wasn’t available from the tools and implementation at hand.

An Aside About On-Dashboard Text

I find myself engaged in regular debates as to whether our dashboards should include descriptive text. The “for” argument goes much like Avinash’s implication that “no text” = “limited value.” The main beef I have with any sort of standardized report or dashboard including a text block is that, when baked into a design, it assumes that there is the same basic word count of content to say each time the report is delivered. That isn’t my experience. In some cases, there may be quite a bit of key callouts for a given report…and the text area isn’t large enough to fit it all in. In other cases, in a performance monitoring context, there might not be much to say at all, other than, “All systems are functioning fine.” Invariably, when the latter occurs, in an attempt to fill the space, the analyst is forced to simply describe the information already effectively presented graphically. This doesn’t add value.

If a text-based description is warranted, it can be included as companion material. <forinstance> “Below is this week’s dashboard. If you take a look at it, you will, as I did, say, ‘Oh, shit! we have a problem!’ I am looking into the [apparent calamitous drop] in [KPI] and will provide an update within the next few hours. If you have any hypotheses as to what might be the root cause of [apparent calamitous drop], please let me know” </forinstance> This does two things:

1. Enables the report to be delivered on a consistent schedule
2. Engages the recipients in any potential trouble spots the (well-formed) dashboard highlights, and leverages their expertise in understanding the root cause

Which…gets us to…

Analysis

Analysis, by [my] definition, cannot be something that is scheduled/recurring/repeating. Analysis is hypothesis-driven:

• The dashboard showed an unexpected change in KPIs. “Oh, shit!” occurred, and some root cause work is in order
• A business question is asked: “How can we drive more Y?” Hypotheses ensue

If you are repeating the same analysis…you’re doing something wrong. By its very nature, analysis is ad hoc and varied from one analysis to another.

When it comes to the delivery of analysis results, the medium and format can vary. But, I try to stick with two key concepts — both of which are violated multiple times over in every example included in Avinash’s post:

• The principles of effective data visualization (maximize the data-pixel ratio, minimize the use of a rainbow palette, use the best visualization to support the information you’re trying to convey, ensure “the point” really pops, avoid pie charts at all costs, …) still need to be applied
• Guy Kawasaki’s 10-20-30 rule is widely referenced for a reason — violate it if needed, but do so with extreme bias (aka, slideuments are evil)

While I am extremely wordy on this blog, and my emails sometimes tend in a similar direction, my analyses are not. When it comes to presenting analyses, analysts are well-served to learn from the likes of Garr Reynolds and Nancy Duarte when it comes to how to communicate effectively. It’s sooooo easy to get caught up in our own brilliant writing that we believe that every word we write is being consumed with equal care (you’re on your third reading of this brilliant blog post, are you not? No doubt trying to figure which paragraph most deserves to be immortalized as a tattoo on your forearm, right? You’re not? What?!!!). “Dumb it down” sounds like an insult to the audience, and it’s not. Whittle, hone, remove, repeat. We’re not talking hours and hours of iterations. We’re talking about simplifying the message and breaking it up into bite-sized, consumable, repeatable (to others)  chunks of actionable information.

Analysis Isn’t Reporting

Analysis and reporting are unquestionably two very differing things, but I don’t know that I agree with assertions that analysis requires an entirely different skillset from reporting. Meaningful reporting requires a different mindset and skillset from data puking, for sure. And, reporting and analysis are two different things, but you can’t be successful with the latter without being successful with the former.

Effective reporting requires a laser focus on business needs and business context, and the ability to crisply and effectively determine how to measure and monitor progress towards business objectives. In and of itself, that requires some creativity — there are seldom available metrics that are perfectly and directly aligned with a business objective.

Effective analysis requires creativity as well — developing reasonable hypotheses and approaches for testing them.

Both reporting and analysis require business knowledge, a clear understanding of the objectives for the site/project/campaign/initiative, a better-than-solid understanding of the underlying data being used (and its myriad caveats), and effective presentation of information. These skills make up the core of a good analyst…who will do some reporting and some analysis.

What About Analytics?

I’m a fan of analytics…but see it as pretty far along the data maturity continuum. It’s easy to poo-poo reporting by pointing out that it is “all about looking backwards” or “looking at where you’ve been.” But, hey, those who don’t learn from the past are condemned to repeat it, no? And, “How did that work?” or “How is that working?” are totally normal, human, helpful questions. For instance, say we did a project for a client that, when it came to the results of the campaign from the client’s perspective, was a fantastic success! But, when it came to what it cost us to deliver the campaign, the results were abysmal. Without an appropriate look backwards, we very well might do another project the same way — good for the client, perhaps, but not for us.

In general, I avoid using the term “analytics” in my day-to-day communication. The reason is pretty simple — it’s not something I do in my daily job, and I don’t want to put on airs by applying a fancy word to good, solid reporting and analysis. At a WAW once, I actually heard someone say that they did predictive modeling. When pressed (not by me), it turned out that, to this person, that meant, “putting a trendline on historical data.” That’s not exactly congruent with my use of the term analytics.

Your Thoughts?

Is this a fair breakdown of the work? I scanned through the comments on Avinash’s post as of this writing, and I’m feeling as though I am a bit more contrarian than I would have expected.Similar Posts:

• Measurement Strategies: Balancing Outcomes and Outputs
• The “Action Dashboard” — Avinash Mounts My Favorite Soapbox
• ROI — the Holy Grail of Marketing (and Roughly as Attainable)
• Put-in-Play Percentage: A “Great Metric” for Youth Baseball?
• What is “Analysis?”

© Tim for Gilligan on Data by Tim Wilson, 2011. | Permalink | 7 comments | Add to del.icio.us
Post tags: Avinash Kaushik

Feed enhanced by Better Feed from Ozh

Back in the early aughts [I'm not Canadian, but I know a few of 'em], I was the business owner of the web analytics tool for a high tech B2B company. We were running Netgenesis (remember Netgenesis? I still have nightmares), which was a log file analysis tool that generated 100 or so reports each month and published them as static HTML pages. It took a week for all of the reports to process and publish, but, once published, they were available to anyone in the company via a web interface. One of the product marcoms walked past my cubicle one day early in the month, then stopped, backed up, and stuck his head in: “Did you see what happened to traffic to <the most visited page on our site other than the home page> last month?” I indicated I had not. We pulled up the appropriate report, and he pointed to a step function in the traffic that had occurred mid-month — traffic had jumped 3X and stayed there for the remainder of the month.

“I made a couple of changes to the meta data on the page earlier in the month. This really shows how critical SEO is! I shared it with the weekly product marketing meeting [which the VP of Marketing attended most weeks].”

I got a sinking feeling in my stomach, told him I wanted to look into it a little bit, and sent him on his way. I then pulled up the ad hoc analysis tool and started doing some digging and quickly discovered that a pretty suspicious-looking user-agent seemed to be driving an enormous amount of traffic. It turned out that Gomez was trying to sell into the company and had just set up their agent to ping that page so they could get some ‘real’ data for an upcoming sales demo. Since it was a logfile-based tool, and since the Gomez user agent wasn’t one that we were filtering out, that traffic looked like normal, human-based traffic. When the traffic from that user-agent was filtered out, the actual overall visits to the page had not shown any perceptible change. I explained this to the product marcom, and he then had to do some backtracking on his claims of a wild SEO success (which he had continued to make in the course of the few hours since we’d first chatted and I’d cautioned him that I was skeptical of the data). The moral of the story: If the data looks too dramatic to be true, it probably is!

This anecdote is an example of The Myth of the Step Function (planned to be covered in more detail in Chapter 10 of the book I’ll likely never get around to writing) — the unrealistic expectation that analytics can regularly deliver deep and powerful insights that lead to immediate and drastic business impact. And, the corollary to that myth is the irrational acceptance of data that shows such a step function.

Any time I do training or a presentation on measurement and analytics, I touch on this topic. In an agency environment, I want our client managers and strategists to be comfortable with web analytics and social media analytics data. I even want them to be comfortable exploring the data on their own, when it makes sense. But, (or, really, it’s more like “BUT“), I implore them that, if they see anything that really surprises them, to seek out an analyst to review the data before sharing it with the client. More often than not, the “surprise” will be a case of one of two things:

• A misunderstanding of the data
• A data integrity issue

All of this is to say, I know this stuff. I have had multiple experiences where someone has jumped to a wholly erroneous conclusion when looking at data that they did not understand or that was simply bad data. I’d even go so far as to say it’s one of my Top Five Pieces of Personal Data Wisdom!

And yet…

When I did a quick and simple data pull from an online listening tool last week, I had only the slightest of pauses before jumping to a conclusion that was patently erroneous.

Maybe it’s good to get burned every so often. And, I’m much happier to be burned by a frivolous data analysis shared with the web analytics community than to be burned by a data analysis for a paying client. It’s tedious to do data checks — it’s right up there with proof-reading blog posts! — and it’s human nature to want to race to the top of the roof and start hollering when a truly unexpected result (or a more-dramatically-than-expected affirming result) comes out of an analysis.

For me, though, this was a good reminder that taking a breath, slowing down, and validating the data is an unskippable step.

© Tim for Gilligan on Data by Tim Wilson, 2010. | Permalink | No comment | Add to del.icio.us
Post tags:

Feed enhanced by Better Feed from Ozh

• As we discussed an internal “Analytics 101″ class  – the bulk of the class focusses on the ins and outs of establishing clear objectives and valid KPIs — a senior executive observed: “The class may be mislabeled. The subject is really more about effective client service delivery — the students may see this as ‘something analysts do,’ when it’s really a a key component to doing great work by making sure we are 100% aligned with our clients as to what it is we’re trying to achieve.”
• A note added by another co-worker to the latest updated to the material for that very course said: “If you don’t set targets for success up front, someone else will set them for you after the fact.”
• Finally, a third co-worker, while working on a client project and grappling with extremely fuzzy objectives, observed: “If you’ve got really loose objectives, you actually have subjectives, and those are damn tough to measure.”

It struck me that these comments were three sides to the same coin, and it got me to thinking about how often I find myself talking about performance measurement as a critical fundamental building block for conducting meaningful analysis.

“Reporting” is starting to be a dirty word in our industry, which is unfortunate. Reporting in and of itself is extremely valuable, and even necessary, if it is done right.

Before singing the praises of reporting, let’s review some common reporting approaches that give the practice a bad name:

• Being a “report monkey” (or “reporting squirrel” if you’re an Avinash devotee) — just taking data requests willy-nilly, pulling the numbers, and returning them to the requestor
• Providing “all the data” — exercises of listing out every possible permutation/slicing of a data set, and then providing a many-worksheeted spreadsheet to end users so that they can “get any data they want”
• Believing that, if a report costs nothing to generate, then there is no harm in sending it — automation is a double-edged sword, because it can make it very easy to just set up a bad report and have it hit users’ inboxes again and again without adding value (while destroying the analyst’s credibility as a value-adding member of the organization)

None of these, though, are reasons to simply toss reporting aside altogether. My claim?

If you don’t have a useful performance measurement report, you have stacked the deck against yourself when it comes to delivering useful analyses.

Let’s walk through a logic model:

1. Optimization and analysis are ways to test, learn, and drive better results in the future than you drove in the past
2. In order to compare the past to the future (an A/B test is a “past vs. future” because the incumbent test represents the “past” and both the incumbent and the challenger represent “potential futures”), you have to be able to quantify “better results”
3. Quantifying “better results” mean establishing clear and meaningful measures for those results
4. In order for measures to be meaningful, they have to be linked to meaningful objectives
5. If you have meaningful objectives and meaningful measures, then you have established a framework for meaningfully monitoring performance over time
6. In order for the organization to align and stay aligned, it’s incredibly helpful to actually report performance over time using that framework, quod erat demonstrandum (or, Q.E.D., if you want to use the common abbreviation — how in the hell the actual Latin words, including the correct spelling, were not only something I picked up in high school geometry in Sour Lake, TX, but that has actually stuck with me for over two decades is just one of those mysteries of the brain…)

So, let’s not just bash reporting out of hand, okay? Entirely too many marketing organizations, initiatives, and campaigns lack truly crystallized objectives. Without clear objectives, there really can’t be effective measurement. Without effective measurement, there cannot be meaningful analysis. Effective measurement, at it’s best, is a succinct, well-structured, well visualized report.

Photo: Greymatterindia

© Tim for Gilligan on Data by Tim Wilson, 2010. | Permalink | One comment | Add to del.icio.us
Post tags: optimization, reporting

Feed enhanced by Better Feed from Ozh

Selling Tweet Access

On Wednesday, Twitter announced they would be selling access to varying volumes of tweets, with 50% of all tweets being available for the low, low price </sarcasm> of $360,000/year. It appears there will be a variety of options, with “50%” being the maximum tweet volume, but with other options in the offing to get 5% of all tweets, 10% of all tweets, or all tweets/references/retweets that are tied to a specific user. All of these sound like they’re going to come with some pretty tight usage constraints, including that they can’t be resold and that the actual tweet content can’t be published.

Twitter has made an API available almost from the moment the service was created. That’s one of the reasons the service grew so explosively — developers were able to quickly build a range of interfaces to the tool that were better than what Twitter’s development team was able to create. But, the API came with limitations — a very tight limit on how often an application could get updates, and a tight limit on just how many updates could be pushed/pulled at once.

As various Twitter analytics-type services began to crop up, Twitter opened up a “garden hose” option — developers could contact Twitter, show that they had a legitimate service with a legitimate need, and they could get access to more tweets more often through the API. Services like Twitalyzer, TweetReach, and Klout jumped all over that option and have built out robust and useful solutions over the course of the last 6-12 months. Now it looks like Twitter is looking to coil up the garden hose, which could spell a permanent end to the growing season for these services. This will be a shame if it comes to pass.

For a steep price, these paid options from Twitter will have limited use: limited to some basic monitoring/listening and some basic performance measurement. Even with the $360K/year option, providing half of the tweets seems problematic when you consider Twitter from a social graph perspective — in theory, half of the network ripple from any given tweet will be lost, or, more confusingly, will crop up as a 2nd or 3rd degree effect with no ability to trace it back to its source because the path-to-the-source passes through the “unavailable 50%!”

This data also won’t be of much use as a listen-and-respond tool. Imagine a brand that has a fantastic ability to monitor Twitter and appropriately engage and respond…but appears schizophrenic because they’re operating with one eye closed (and paying a pretty penny to do even that!). To be clear, for any given brand or user, only a tiny fraction of all tweets are actually of interest, but that tiny fraction is going to be spread across 100% of the Twitterverse, so only having access to a 5%, 10%, or even 50% sample means that relevant tweets will be missed.

Online listening platforms — Radian6, SM2, Buzzmetrics, Crimson Hexagon, Sysomos, etc. — may actually have deep enough pockets to pay for these tweets to improve their own underlying data…but they will have to significantly alter the services they provide in order to comply with the usage guidelines for the data.

Ugh.

Twitter Analytics

On Thursday, Mashable reported that Twitter Analytics was being tested by selected users. Unfortunately, I’m not one of those users (<sniff><sob>), so I’m limited to descriptions in the Mashable article. Between that article and Pete Cashmore’s (Mashable CEO) editorial on cnn.com, I’ve got pretty low expectations for Twitter Analytics.

Both pieces seem somewhat naive in that they overplay the value to brands that Facebook has delivered with Facebook Insights, and they confuse “pretty graphs” with “valuable data.” All I can think to do is rattle off a series of reactions from the limited information I’ve been able to dig up:

• Replies/references over time: um…thanks, but that’s always been something that’s pretty easy to get at, so no real value there.
• Follows/unfollows: this seems to be taking a page directly from Facebook Insights with it’s new fans/removed fans reporting (which, by the way, never agrees with the “Total Fans” data available in the same report, but I digress…); this has marginal value — in practice, unless a user is really pissing off followers or baiting them to follow with a very specific promotional giveaway (“Follow us and retweet this and you’ll be entered to win a BRAND NEW CAR!!!”), there’s probably not going to be a big spike in unfollows, and it isn’t that hard to trend “total followers” over time, so I can’t get too excited about this, either
• Unfollows (cont’d.): “tweets that cause people to unfollow” is another apparent feature of Twitter analytics. Really? Was that something that someone living on planet Earth came up with? This sounds nifty initially, but, in practice, isn’t going to be of much use. If a user posts offensive, highly political (for a non-political figure user), or obnoxiously self-promoting tweets…he’s going to lose followers. I don’t think “analytics” will really be needed to figure out the root cause (if it was a single tweet) driving a precipitous follower drop. Common sense should suffice for that.
• Retweets: this is like references, in that it’s not really that hard to track, and I wouldn’t be surprised at all if Twitter Analytics only counts retweets that use the official Twitter retweet functionality, rather than using a looser definition that includes “RT @<username>” occurrences (which are retweets that are often more valuable, because they can include additional commentary/endorsement by the retweeters)
• Impressions: I’m expecting a simplistic definition of impressions that is based just on the number of followers, which is misleading, because most users of Twitter see only a fraction of the tweets that cross their stream. Twitalyzer calculates an “effective reach” and Klout calculates a “true reach” — both make an attempt to factor in how receptive followers are to messages from the user. None of these measures is going to be perfect, but I’m happier relying on companies whose sole focus is analytics trying to tinker with a formula than I am with the “owner” of the data coming up with a formula that they think makes sense.

With the screen caps I’ve seen, there is no apparent “export data” button, and that’s a back-breaker. Just as Facebook Insights is woefully devoid of data export capabilities (the “old interface” enables data export…but not of some of the most useful data, and API access to the Facebook Insights data doesn’t exist, as best as I’ve been able to determine), Twitter looks like they may be yet another technology vendor who doesn’t understand that “their” dashboard is destined to be inadequate. I’m always going to want to combine Twitter data with data that Twitter doesn’t have when it comes to evaluating Twitter performance. For instance, I’m going to want to include referrals from Twitter to my web site, as well as short URL click data in my reporting and analysis.

Ikong Fu speculated during an exchange (on Twitter) that Twitter may also, at some point, include their internal calculations of a user’s influence in Twitter Analytics:

I didn’t realize that Twitter was calculating an internal reputation score. It makes sense, though, that that would be included when they make recommendations of who else a user might want to follow. I found a post from Twitter’s blog back in July that announced the rollout of  “follow suggestions,” and that post indicated these were based on “algorithms…built by our user relevance team.” The only detail the post provided was that these suggestions were “based on several factors, including people you follow and the people they follow.” That sounds more like a social graph analysis (“If you’re following 10 people who are all following the same person who you are not following, then we’re going to recommend that you follow that person”) than an analysis of each user’s overall influence/quality. Again…I’m more comfortable with third party companies who are fully focussed on this measurement and who make their algorithms transparent providing me with that information than I am with Twitter in that role.

So, Where Does This Leave Us?

Maybe, for once, I’m just seeing a partially filled glass of data as being half empty rather than half full (okay, so that’s the way I view most things — I’m pessimistic by nature). In the absence of more information, though, I’m forced to think that, just as I was headed towards analytics amour when it came to Twitter data, Twitter is making some unfortunate moves and rapidly smudging the luster right off of that budding relationship.

Or, maybe, I’m unfairly pre-judging. Time will tell.

© Tim for Gilligan on Data by Tim Wilson, 2010. | Permalink | No comment | Add to del.icio.us
Post tags:

Feed enhanced by Better Feed from Ozh

Many companies rely on some form of media mix modeling (or “marketing mix modeling”) to determine the optimal mix of their advertising spend. With the growth of “digital” media and the explosion of social media, these models are starting to break down. That puts many marketing executives in a tough bind:

1. Marketing, like all business functions, must be data-driven — more so now than ever
2. Digital is the “most measurable medium ever” (although their are wild misperceptions as to what this really means)
3. Ergo, digital media investments must be precisely measured to quantify impact on the bottom line

For companies that have built up a heavy reliance on media mix modeling (MMM), the solution seems easy: simply incorporate digital media into the model! What those of us who live and breathe this world recognize (and lament over drinks at various conferences for data geeks), is that this “simple” solution simply doesn’t work. Publicly, we say, “Well…er…it’s problematic, but we’re working on it, and the modeling techniques are going to catch up soon.”

My take: don’t hold your breath that MMM is going to catch up — even if it catches up to today’s reality, it will already be behind, because digital/social/mobile will have continued its explosive evolution (and complexity to model).

Believe it or not, I’m not saying that MMM should be completely abandoned. It still has it’s place, I think, but there are a lot of things it’s going to really, really struggle to address. I’d actually like to see companies who provide MMM services weigh in on what that is. At eMetrics earlier this month, I attended a session where the speaker did just that. Skip ahead to the last section to find out who!

Geographic Test/Control Data

Both traditional and digital marketing have a mix of geo-specific capabilities. The cost of TV, radio, print, and out-of-home (OOH) marketing provides an imperative to geo-target when appropriate (or simply to minimize the peanut butter effect of spreading a limited investment so thinly that it doesn’t have an impact anywhere). Many digital channels, though, such as web sites and Facebook pages, are geared towards being “available to everyone.” Other channels – SEM, banner ads, and email, for instance – can be geo-targeted, but there often isn’t a cost/benefit reason to do so. Without different geographic placements of marketing, the impact on sales in “exposed areas” vs. “unexposed areas” cannot be teased out:

Cross-Channel Interaction

While marketers have long known that multi-channel campaigns produce a whole that is greater than the sum of the parts, the sheer complexity that digital has introduced into the equation forces MMM to guess at attribution. For example, we know (or, at least, we strongly suspect) that a large TV advertising campaign will not only provide a lift in sales, but it will also produce a lift in searches for a brand. Those increased searches will increase SEM results, which will drive traffic to the brand’s web site. Consumers who visit the site can then be added to a retargeting campaign. Those are four different marketing channels that all require investment…but which one gets the credit when the consumer buys?

This is both data capture and a business rules question. Entire companies (Clearsaleing being the one that I hear the most about) have been built just to address the data capture and application of business rules. While they provide the tools, they’re a long way from really being able to capture data across the entire continuum of a consumer’s experience. The business rules question is just as significant — most marketers’ heads will explode if they’re asked to figure out what the “right” attribution is (and simply trying different attribution models won’t answer the question — different models will show different channels being the “best”). Is this a new career option for Philosophy majors, perhaps?

Fragmentation of Consumer Experiences

This one is related to the cross-channel interaction issue described above, but it’s another lens applied to the same underlying challenge. Consumer behavior is evolving — there are exponentially more channels through which consumers can receive brand exposure (I picked up the phrase “cross-channel consumer” at eMetrics, which is in the running for my favorite three-letter phrase of 2010!). Some of these channels operate both as push and pull, whereas traditional media is almost exclusively “push” (marketers push their messages out to consumers through advertising):

We’re now working with an equation that has wayyyyyyyy more variables, each of which has a lesser effect than the formulas we were trying to solve when MMM first came onto the scene. HAL? Can you help? This is actually beyond a question of simply “more processing power.” It’s more like predicting what the weather will be next week — even with meteoric advancements in processing power and a near limitless ability to collect data, the models are still imprecise.

Self-Fulfilling Mix

Finally, there is a chicken-and-egg problem. While there are reams of secondary research documenting the shifting of consumer behavior from offline to online consumption…many brands still disproportionately invest in offline marketing. It’s understandable — they’re waiting for the data to be able to “prove” that digital marketing works (and prove it with an unrealistic degree of accuracy — digital his held to a higher standard than offline media, and the “confusion of precision with accuracy” syndrome is alive and well). But, when digital marketing investments are overly tentative (and those investments are spread across a multitude of digital channels), the true impact of digital can’t be detected because it’s dwarfed by the impact of the massive — if less efficient — investments in offline marketing:

If I shoot a pumpkin simultaneously with a $1,500 shotgun and a $30 BB gun and ask an observer to tell me how much of an impact the BB gun had…

So, Should We Just Start Operating on Faith and Instinct?

I wrote early in this post that I think MMM has its place. I don’t fully understand what that place is, but the credibility of anyone whose bread is buttered by their MMM book of business who stands up and says, “Folks, MMM has some issues,” immediately skyrockets. That’s exactly what Steve Tobias from Marketing Management Analytics (MMA) did at eMetrics. In his session, “Marketing Mix Modeling: How to Make Digital Work for a True ROI,” he talked at length about many of the same challenges I’ve described in this post (albeit in greater detail and without the use of cartoon-y diagrams). But, he went on to lay out how MMA is using traditional MMM in conjunction with panel-based data (in his examples, he used comScore for the analysis) to get “true ROI” measurement. All I’ve seen is that presentation, so I don’t have direct experience with MMA’s work in action, but I liked what I heard!

© Tim for Gilligan on Data by Tim Wilson, 2010. | Permalink | 4 comments | Add to del.icio.us
Post tags: MMA, MMM, Steve Tobias

Feed enhanced by Better Feed from Ozh

What Kind of Analysis Are We Talking About?

I’m primarily focussed on the measurement and analysis of consumer brands on Twitter rather than on the measurement of one’s personal brand (e.g., @tgwilson). While there is some overlap, there are some things that make these fundamentally different. With that in mind, there are really three different lenses through which Twitter can be viewed, and they’re all important:

• The brand’s Twitter account(s) — this is analysis of followers, lists, replies, retweets, and overall tweet reach
• References of the brand or a campaign on Twitter — not necessarily mentions of @<brand>, but references to the brand in tweet content
• References to specific topics that are relevant to the brand as a way to connect with consumers — at Resource Interactive, we call this a “shared passion,” and the nature of Twitter makes this particularly messy, but, to whatever level it’s feasible, it’s worth doing

While all three of these areas can also be applied in a competitor analysis, this is the only mention (almost) I’m going to make of that  – some of the techniques described here make sense and some don’t when it comes to analyzing the competition.

And, one final note to qualify the rest of this post: this is not about “online listening” in the sense that it’s not really about identifying specific tweets that need a timely response (or a timely retweet). It’s much more about ways to gain visibility into what is going on in Twitter that is relevant to the brand, as well as whether the time spent investing in Twitter is providing meaningful results. Online listening tools can play a part in that…but we’ll cover that later in this post.

Capturing Tweets?

When it comes to Twitter analysis, it’s hard to get too far without having a nice little repository of tweets themselves.  Unfortunately, Twitter has never made an endless history of tweets available for mining (or available for anything, for that matter). And, while the Library of Congress is archiving tweets, as far as I know, they haven’t opened up an API to allow analysts to mine them. On top of that, there are various limits to how often and how much data can be pulled in at one time through the Twitter API. As a consumer, I suppose I have to like that there are these limitations. As a data guy, it gets a little frustrating.

Two options that I’ve at least looked at or heard about on this front…but haven’t really cracked:

• Twapper Keeper — this is a free service for setting up a tweet archive based on a hashtag, a search, or a specific user. In theory, it’s great. But, when I used it for my eMetrics tweet analysis, I stumbled into some kinks — the file download format is .tar (which just means you have to have a utility that can uncompress that format), and the date format changed throughout the data, so getting all of the tweets’ dates readable took some heavy string manipulation
• R — this is an open source statistics package, and I talked to a fellow several months ago who had used it to hook into Twitter data and do some pretty intriguing stuff. I downloaded it and poked around in the documentation a bit…but didn’t make it much farther than that

I also looked into just pulling Tweets directly into Excel or Access through a web query. It looks like I was a little late for that — Chandoo documented how to use Excel as a Twitter client, but then reportd that Twitter made a change that means that approach no longer works as of September 2010.

So, for now, the best way I’ve found to reliably capture tweets for analysis is with RSS and Microsoft Outlook:

1. Perform a search for the twitter username, a keyword, or a hashtag from http://search.twitter.com (or, if you just want to archive tweets for a specific user, just go to the user’s Twitter page)
2. Copy the URL for the RSS for the search (or the user)
3. Add a new RSS feed in MS Outlook and paste in the URL

From that point forward, assuming Outlook is updating periodically, the RSS feeds will all be captured.

There’s one more little trick: customize the view to make it more Excel/export-friendly. In Outlook 2007, go to View » Current View » Customize Current View » Fields. I typically remove everything except From, Subject, and Received. Then go to View » Current View » Format Columns and change the Received column format from Best Fit to the dd-Mmm-yy format. Finally, remove the grouping. This gives you a nice, flat view of the data. You can then simply select all the tweets you’re interested in, press <Ctrl>-<C>, and then paste them straight into Excel.

I haven’t tried this with hundreds of thousands of tweets, but it’s worked great for targeted searches where there are several thousand tweets.

Total Tweets, Replies, Retweets

While replies and retweets certainly aren’t enough to give you the ultimate ROI of your Twitter presence, they’re completely valid measures of whether you are engaging your followers (and, potentially, their followers). Setting up an RSS feed as described above based on a search for the Twitter username (without the “@”) will pick up both all tweets by that account as well as all tweets that reference that account.

It’s then a pretty straightforward exercise to add columns to a spreadsheet to classify tweets any number of ways by some use of the IF, ISERROR, and FIND functions. These can be used to quickly flag each tweet  as a reply, a retweet, a tweet by the brand, or any mix of things:

• Tweet by the brand — the “From” value is the brand’s Twitter username
• Retweet — tweet contains the string “RT @<username>“
• Reply — tweet is not a retweet and contains the string “@<username>“

Depending on how you’re looking at the data, you can add a column to roll up the date — changing the tweet date to be the tweet week (e.g., all tweets from 10/17/2010 to 10/23/2010 get given a date of 10/17/2010) or the tweet month. To convert a date into the appropriate week (assuming you want the week to start on Sunday):

=C1-WEEKDAY(C1)+1

To convert the date to the appropriate month (the first day of the month):

=DATE(YEAR(C1),MONTH(C1),1)

C1, of course, is the cell with the tweet date.

Then, a pivot table or two later, and you have trendable counts for each of these classifications.

This same basic technique can be used with other RSS feeds and altered formulas to track competitor mentions, mentions of the brand (which may not match the brand’s Twitter username exactly), mention of specific products, etc.

Followers and Lists

Like replies and retweets, simply counting the number of followers you have isn’t a direct measure of business impact, but it is a measure of whether consumers are sufficiently engaged with your brand. Unfortunately, there are not exactly great options for tracking net follower growth over time. The “best” two options I’ve used:

• Twitter Counter — this site provides historical counts of followers…but the changes in that historical data tend to be suspiciously evenly distributed. It’s better than nothing if you don’t have a time machine handy. (See the Twitalyzer note at the end of this post — I may be changing tools for this soon!)
• Check the account manually — getting into a rhythm of just checking an account’s total followers is the best way I’ve found to accurately track total followers over time; in theory a script could be written and scheduled that would automatically check this on a recurring basis, but that’s not something I’ve tackled

I also like to check lists and keep track of how many lists the Twitter account is included on. This is a measure, in my mind, of whether followers of the account are sufficiently interested in the brand or the content that they want to carve it off into a subset of their total followers so they are less likely to miss those tweets and/or because they see the Twitter stream as being part of a particular “set of experts.” Twitalyzer looks like it trends list membership over time, but, since I just discovered that it now does that, I can’t stand up and say, “I use that!” I may very well start!

Referrals to the Brand’s Site

This doesn’t always apply, but, if the account represents a brand, and the brand has a web site where the consumer can meaningfully engage with the brand in some way, then measuring referrals from Twitter to the site are a measure of whether Twitter is a meaningful traffic driver. There are fundamentally two types of referrals here:

• Referrals from tweeted links by the brand’s Twitter account that refer back to the site — these can be tracked by a short URL (such as bit.ly), by adding campaign tracking parameters to the URL so the site’s web analytics tool can identify the traffic as a brand-triggered Twitter referral, or both. The campaign tracking is what is key, because it enables measuring more than simply “clicks:” whether the visitors are first-time visitors to the site or returning visitors, how deeply they engaged with the site, and whether they took any meaningful action (conversions) on the site
• “Organic” referrals — overall referrals to the site from twitter.com. Depending on which web analytics tool you are using on your site, this may or may not include the clickthroughs from links tweeted by the brand.

By looking at referral traffic, you can measure both the volume of traffic to the site and the relative quality of the traffic when compared to other referral sources for the site.

(If the volume of that traffic is sufficiently high to warrant the effort, you may even consider targeting content on the landing page(s) for Twitter referral traffic to try to engage visitors more effectively– you know the visitor is engaged with social media, so why not test some secondary content on the page to see if you can use that knowledge to deliver more relevant content and CTAs?)

Word Clouds with Wordle

While this isn’t a technique for performance management, it’s hard to resist the opportunity to do a qualitative assessment of the tweets to look for any emerging or hot topics that warrant further investigation. Because all of the tweets have been captured, a word cloud can be interesting (see my eMetrics post for an example). Hands-down, Wordle makes the nicest word clouds out there. I just wish it was easier to save and re-use configuration settings.

One note here: you don’t want to just take all of the tweet content and drop it straight into Wordle, as the search criteria you used for the tweets will dwarf all of the other words. If you first drop the tweets into Word, you can then do a series of search and replaces (which you can record as a macro if you’re going to repeat the analysis over time) — replace the search terms, “RT,” and any other terms that you know will be dominant-but-not-interesting with blanks.

Not Exactly the Holy Grail…

Do all of these techniques, when appropriately combined, provide near-perfect measurement of Twitter? Absolutely not. Not even close. But, they’re cheap, they do have meaning, and they beat the tar out of not measuring at all. If I had to pick one tool that I was going to bet on that I’d be using inside of six months for more comprehensive performance measurement of Twitter, it would be Twitalyzer. It sure looks like it’s come a long way in the 6-9 months since I last gave it a look. What it does now that it didn’t do initially:

• Offers a much larger set of measures — you can pick and choose which measures make sense for your Twitter strategy
• Provides clear definitions of how each metric is calculated (less obfuscated than the definitions used by Klout)
• Allows trending of the metrics (including Lists and Followers).

Twitalyzer, like Klout, and Twitter Counter and countless other tools, is centered on the Twitter account itself. As I’ve described here, there is more going on in Twitter that matters to your brand than just direct engagement with your Twitter account and the social graph of your followers. Online listening tools such as Nielsen Buzzmetrics can provide keyword-based monitoring of Twitter for brand mentions and sentiment — this is not online listening per se, really, but it is using online listening tools for measurement.

For the foreseeable future, “measuring Twitter” is going to require a mix of tools. As long as the mix and metrics are grounded in clear objectives and meaningful measures, that’s okay. Isn’t it?Similar Posts:

• Twitter Influence — Still Searching for the Perfect Answer

© Tim for Gilligan on Data by Tim Wilson, 2010. | Permalink | 3 comments | Add to del.icio.us
Post tags: Klout, Nielsen Buzzmetrics, Twapper Keeper, Twitter Counter

Feed enhanced by Better Feed from Ozh