Gilligan on Data by Tim Wilson

Last week’s Columbus set a new record for the meetup — we had exactly FIFTY attendees, which was a great showing. Part of the large draw was undoubtedly the event sponsor, CRM Metrix (@crm_metrix on Twitter).

Pre-Meal Networking (and a Friendly Wave from Jonghee!)

Hemen Patel, CRM Metrix CTO, facilitated a lively discussion about incorporating the voice of the customer in web site measurement and optimization.

Hemen Patel Presents

Hemen walked through a brief deck (below) that sparked some great back-and-forth with the crowd.

A Rapt Audience

Monish Datta Asks a Question

With a crowd of fifty people, not only did I not get to meet the first-time attendees, but I barely had a chance to say, “Hi” to some of the long-time regulars. I guess we’ll just have to have another one in February (I’m working on it!) so I’ll get that chance!

In my last post, one of the challenges I described was that it was impossible to get a good read on the number of impressions a brand was garnering from their fan page status updates — a status update on a fan page appears in the live feeds of the page’s fan…assuming the fan hasn’t hidden updates from that page and the fan logs in to Facebook and views his/her live feed before there are so many new updates from his/her other friends that the status update has slid down into oblivion.

A lot has changed since that post! A few days after that post, Nick O’Neill reported that a Facebook staffer had let the cat out of the bag during a presentation in Poland and announced that impression measurement was on the way. And, last Thursday, it became official. IF you have an authenticated Facebook page (at least 10,000 fans and you’ve authenticated the page when prompted), you now get (with some delay), something like this underneath each of your status updates:

Pretty slick, huh?

First, Impressions

I’ll be the first to say that “impressions” is a pretty loose measure — it’s a standard in online advertising, and it became the go-to measure there because print and TV have historically been so eyeball-oriented. It’s not a great measure, but it does have some merit. I’ll even go so far as to claim that a Facebook impression is “heavier” than a typical online display ad (be it on Facebook or some other site), because many online display ads are positioned somewhere on the periphery of the page where we’ve trained ourselves to tune them out. A Facebook impression is in the fan’s feed.

Of course, the other way to look at it is that it’s only showing up for people who are already fans of your page, which, presumably, are people who already have an affinity for your brand (although, considering that “fan” is short for “fanatic”…methinks the meaning of the term has evolved to be a much lesser state of enthusiasm than it was 20 or 30 years ago). So, it’s not much of a “brand awareness”-driving impression.

Facebook’s note on the subject gives a pretty clear definition of how impressions are counted:

…the number of impressions measures the number of times the post has been rendered on user’s [sic] browsers. These impressions can come from a user’s news feed, live feed, directly from the Page, or through the Fan Box widget. This includes instances of the post showing up below the fold.

Clear enough. This will be really useful information for sifting through past status updates and seeing which ones garner the highest impressions per fan to determine what day (and time of day) is optimal for getting the broadest reach for the update (remember that impressions have nothing to do with the quality of the content — it’s just a measure of how many people had that post rendered in their browser). Juicy stuff. The impression count will (or should…Facebook metrics have a record of being a little shifty) only go up over time. So, to get a good handle on total impressions, you’ll have to let the update be out there for a few days or a week before it really closes in on its top end.

% Feedback

So, what about that “% Feedback” measure? This is a good one, too — it’s actually a tighter measure of “post quality” than the Post Quality measure provided through Facebook Insight (Post Quality is vaguely defined by Facebook as being “calculated with an algorithm that takes into account your number of posts, total fan interactions received, number of fans, as well as other factors.” Yeesh!). It’s simple math:

(Likes + Comments) / Impressions

What percent of people not only had the status update presented to them, but also reacted to it strongly enough to take an action in response to the post? In the screen cap above: (11 likes + 9 comments) / 31,895 impressions = 0.06% Feedback. Is that good or bad? It’s too early to tell (the same page that I pulled the above from had another status update with a 1.62% Feedback value), but I like the measure as a general idea. And, it’s easy to understand and recreate, so all the better. It is a measure of the quality of the content (although, in theory, a status update could go out that really upset a lot of people, which could drive a high % Feedback score by attracting a lot of negative comments).

I’m a little bothered by combining Likes and Comments. To me, a Like is a much lower-weighted interaction than a Comment — a like is a “I read it and agree enough to click a link while I move along” reaction, whereas a comment is a “I read it and had a sufficiently strong reaction to form a set of words and take the time to type them in” reaction. But, for the sake of simplicity, I’m good with combining them. And, the calculation is so simple that it would be easy enough to manually calculate a different measure.

As far as I can tell (so far), Facebook isn’t providing a way to get “overall impressions and % Feedback” measures by day through Facebook Insights. The data is available on a “by update, manually gathered” basis only. But, I don’t want to be difficult — I love the progress!

If you are a marketer, Facebook is important — the number of active users of the site exceeds the population of the United States, and it’s growth is going to do nothing but increase. Check out the Facebook statistics page for a slew of numbers that are all…big. Because of the growth of Facebook as a critical marketing channel, a hot topic around the office right now is “Facebook measurement.” It’s a tricky topic that reminds me of the early days of web analytics: there’s some basic stuff that’s easy to measure, and it’s basically helpful, but there’s a lot more that can’t be measured or can’t be measured well, and that’s where the real value is.

There are (at least) five different aspects of Facebook that can be measured:

• Facebook display ads — I’m not going to cover that at all; I haven’t spent a whole lot of time digging into it with our clients, so I’m not going to write about it. Check out What about advertising on Facebook? over on the Site Pro Specialties site for a quick overview of the ins-and-outs and their experience with Facebook display ads
• Facebook applications — I’m not going to cover this, either, largely because my experience on the subject is pretty limited, but also because Facebook apps annoy the bejeezus out of me, and I don’t want to have to stifle a gag reflex by writing about measuring them
• Facebook groups — Holy cow! Another topic I’m not going to cover! Since Facebook pages came on the scene, that’s where brands tend to be living more, and Facebook provides more measurement support for pages, so that’s where I’m going to focus the most
• Facebook pages — I’ll focus on these quite a bit, as this is an area that brands are really starting to settle into as a formal presence on Facebook
• General Facebook activity – this is an area where measurement is highly limited, but I’ll lay out what is there and what I hope comes sooner rather than later

We’re in a bit of an ugly period from an analyst’s perspective, in that Facebook hasn’t made supporting marketers a high priority beyond paid advertising. And, the company is being very cautious on the privacy front (which, from a consumer’s perspective, is a good thing!). The easiest way to reduce the risk of a PR blowup from misuse of Facebook data is to limit the availability of that data to marketers. I can’t blame them, but it doesn’t mean they’ve made my life easy on that front.

Ready? Let’s go! Here’s a quick set of links to use to jump down to specific topics:

• Facebook Pages — Fan Count
• Facebook Pages — Facebook Insights Data
• Facebook Pages — Web Analytics Measurement, Part I (The Ugly Part)
• Facebook Pages — Web Analytics Measurement, Part II (The Pretty…but Short…Part)
• General Facebook Activity — Web Analytics Measurement
• General Facebook Activity — LOTS is Missing

Facebook Pages — Fan Count

Facebook pages are a way for brands to establish a formal, managed presence on the site. They’re easy to set up, and they can range from the very simple and unused (see the Smuirfield Golf Club page) to the very elaborate and active (see the Victoria’s Secret PINK page). For any page, regardless of whether you are an admin for it or not, you can see the total number of fans at a point in time — the example below is from the Slate Political Gabfest page:

That can be useful for a couple of reasons:

• Organically grown pages — it’s fairly common for major brands to have their fans set up pages and grow a decent following; being able to tell the reach of those pages can help identify when outreach or integration might be in order
• Competitive research — it can be tedious, but assessing the size and growth of competitor fan pages over time can provide insight (albeit limited insight) into their overall social media strategy and their ability to execute

There is no way to measure the change in any page’s fan count over time other than periodically going and checking and recording it. And, what does total fans tell you? It tells you something…but not as much as you might like. More on that later.

Facebook Pages — Facebooks Insights Data

Now, if you have admin access to a Facebook page, you can get much richer data, and you can get a historical view of some of that data. On the page itself, above the Fans box, is the basic Facebook Insights box:

While this looks encouraging, it’s not particularly useful. “Post Quality” sounds like a good idea (pick any measure of activity volume, and you can say, “It’s not just about quantity — it’s about quality!” and sound smart), exactly how Facebook determines quality is a bit of a mystery. From the Facebook Help Center:

The Post Quality score measures how engaging your Posts have been to Facebook users over a rolling seven-day window.

Post Quality is an important indicator for how fans gauge your posts. This score is calculated with an algorithm that takes into account your number of posts, total fan interactions received, number of fans, as well as other factors.

It’s a measure that’s almost too vague to be useful. And, in practice, the historical trending of Post Quality shows that something about the way it is measured makes it pretty non-actionable — even for pages that have a high level of fan engagement consistently, a trendline of Post Quality goes all over the place.

So, now we dive into the real meat of Facebook Insights, which initially looks like a nice, juicy T-bone, but which turns out to be more like a pretty lean cut of venison. The See All link in the Insights box brings up the main Facebook Insights page (click on the image below to view a larger version):

This page has the second not-nearly-as-useful-as-you’d-like measure: Active Fans. Facebook is even more fuzzy about how this is calculated than it is about Post Quality. And, historical data is not available. In my experience, Active Fans is a pretty big crap shoot — it varies widely from day to day and, since it’s not easy to get historical data, it’s a mess to try to analyze what is going on and how it is really changing over time with any granularity. Conceptually, active fans are high-quality fans. In my experience, the number of active fans in any given period is a tiny fraction of the overall fans. So, the million-dollar question — “What is the value of a Facebook fan?” — should probably include a separate calculation for an “active fan.” But, “active fan” is such a messy measure with such limited availability, that it’s barely worth pursuing until it’s more accessible and explainable.

Most of the other measures, though, have historical data available via the graphs shown on the page. Some underlying data can be exported as a CSV or Excel file with granularity at the individual day level. Two wrinkles with that data, though:

• The timing of the data updates is inconsistent, and it doesn’t seem like “if data is there, it’s good data” — a note in the bottom of the Insights window states: “Please allow 48 hours for data to be available for a daily report;” it’s common to see some data for a given day populated while other data for the same day isn’t; while I don’t feel like “real-time” data is generally warranted, the 48-hour lag can put a real crimp in effectively weekly reports, as well as in getting a good, timely view into the results of a new Facebook campaign
• The data doesn’t appear to be kept forever; it used to seem like data dropped off once it was ~3 months old, but the actual range of available data seems to vary, and Facebook doesn’t provide information on the subject; we’re in the practice of exporting all available data monthly so that we’ve got it retained offline for our clients

The main export option is the Fans and Interactions export. The other two exports that are available are Demographics and Country. The demographics export simply shows, by day, the number of fans of a given age range/gender. The demographics of active fans over time is not available, unfortunately. The Country export simply shows the number of fans from each country over time.

Now, Fans and Interactions is where the most useful information is. You can get a great look into how fan growth has been growing over time — new fans, total fans, unsubscribes, etc. This provides a way to do a classic “leaky bucket” report — how many fans you are  losing compared to how many new fans you are acquiring. Unsubscribes are interesting, because that means fans have explicitly removed themselves as fans rather than simply choosing to remove the page’s updates from their feeds. Which…alludes to the Big Wrinkle when it comes to fans — just because someone is a fan of your page doesn’t mean they’re seeing anything that happens on the page — it’s very easy for users to hide all updates from a page from their feeds. And Facebook doesn’t provide data as to how many people have done that!

Fans and Interactions also provides data on the number of “interactions” which is the sum of all of the likes, posts, and comments that occur each day. In my mind, a “like” is a pretty light interaction, while a post or a comment is a more significant interaction, because a fan actually had to put together words to express an idea. Facebook Insights provides details for each type of interaction, too, though, so you can measure the different types of interactions. This export provides four types of interactions: Likes, Comments, Wall Posts, and Discussion Posts.  It can get a little confusing as to which type of user activity occurs where, so be prepared to click back and forth between your page and the data for a while to get the hang of it (I’d write it out here, but this post is already getting pretty long and unwieldy!). The data also includes “Posts” — these are your posts rather than fan posts.

Finally, Fans and Interactions provides basic web analytics data. VERY basic. Page views, unique page views, audio plays, video plays, and photo views. At a very high level, this is useful information, as it’s a measure of whether the page is sufficiently engaging to drive people to visit (note that someone by no means has to be a fan to visit the page, view content, and comment on it — if a page has a lot of page views but a small number of fans, then it may be an indication that users would like to engage with the brand in Facebook, but the actual content/activity occurring on the page is not strong enough to get them to become a fan once they actually visit). Data that is not provided includes: which tabs of the fan page were visited, which videos were played (and how much of the video was viewed), and which photos were viewed.

Facebook Insights also doesn’t provide data on:

• Suggest to Friends usage
• Subscribe via SMS usage
• Add to My Favorites usage
• The ability to export wall posts, discussion posts, and comments (more on this in the last section of this post)
• Page visit frequency

The lack of Suggest to Friends data is particularly painful — this would be a powerful measure of how engaging the content on the page is, and there is zip when it comes to any visibility into that.

I expect that Facebook Insights will evolve over time to provide more content-level detail, as well as usage of other “page” features. It’s less likely that Insights will evolve to include user-level detail due to privacy concerns, although it’s not inconceivable — this would be the equivalent of having access to detailed behavioral data for users who have registered with your web site and are making subsequent visits.

Facebook Pages — Web Analytics Measurement, Part I (The Ugly Part)

Depending on how you squint when you look at it, a Facebook fan page for your brand is just an off-site extension of your web site — just like any content you host on a third-party site (job postings that are hosted by a recruiting site, events that get managed through a third-party event management site, etc.). For third-party sites whose bread and butter is extending the content offerings from web sites, it’s common to deploy the main site’s web analytics page tag on the third-party content pages. There are myriad ways to set up the reporting for that in any web analytics tool — Google Analytics, SiteCatalyst, Webtrends, Coremetrics, etc. In theory, Facebook pages would be the same way — just as you can embed all sorts of rich content on custom tabs, it seems like you would be able to insert your web analytics page tag on the pages where you have heavy control over content.

But, Facebook currently has an industrial-sized monkey wrench inserted into that approach by not allowing Javascript to execute on its pages. Presumably, this gets back to privacy — concern that opening up the site to allow scripts to execute would open up the potential for some page admins to figure out a way to capture too much personal information from visitors/fans of their pages.

So, what options are there? There are several, but they’re all clunky.

[UPDATE: The next little section is not entirely accurate, as I've been discovering over the past couple of weeks -- iFrames really aren't an option, for instance. I'll get this section more thoroughly cleaned up once I have a clearer picture. I've italicized the area that needs cleanup.]
Generally speaking:

[End of section that is a bit dicey]

In short, though, this is pretty messy.

Facebook Pages — Web Analytics Measurement, Part II (The Pretty…but Short…Part)

If you link back to your main site from your Facebook page (which, presumably, you do in multiple places), then standard parameter-based campaign tracking works. Use it. ‘nuf said.

General Facebook Activity — Web Analytics Measurement

In addition to tracking links that you control on Facebook with campaign tracking (the previous section), you can and should look at Facebook as a broader source of traffic to your site. If you are posting content on your site that is share-worthy, then Facebook users can pick it up and share it through Facebook, which will drive referrals to your site. If you’ve actually enabled content-sharing capabilities on your site, and those capabilities include Facebook, then you can add campaign tracking parameters to content as it gets shared, which will give you better visibility into what specific content is most compelling and passed along. Beyond just the traffic to the site, the bounce rate and conversions from that traffic are useful — is the sharing of your content bringing visitors to your site who are finding value and doing valuable things?

The caution here is to not get overly obsessed with Facebook as a source of traffic to your site. It certainly can (and probably should) be a source of traffic, but your site isn’t necessarily the best destination point for all of your customers. Just because this is easy data to get to doesn’t mean that it is the best data to use to measure the performance of your site.

General Facebook Activity — LOTS is Missing

Overall, Facebook measurement — measurement of what really matters — is still very immature. We’re largely stuck with measuring basic counts of things that are easy to measure: total fans, unique pageviews, etc. But, when it comes to both measuring the impact of a Facebook investment as well as being able to analyze what is and is not working, we’re missing a lot:

• Impressions – how many people are actually being presented with content related to your brand? Besides Facebook display ads, this is total guesswork; just because a page posts a status update doesn’t mean it ever shows up on the screen of a fan (the update may slip well into the “More” area before the fan logs on again, the fan may have those updates hidden); “impressions” is far from being an end-all/be-all measure, but it’s a pretty good indicator of reach, and it’s really not available in the Facebook world
  [UPDATE: Since I originally wrote this post, I've found out that Facebook has something in the works for this -- the one referenceable source is Facebook Presentation Reveals "Post Analytics" And Real-Time Ad Targeting. It's a total crapshoot as to when this functionality will be available and to whom it will be available.]
• Social Graph and Impact — all Facebook users (and, thus, all Facebook page fans) are not equal; all of the major online listening platforms attempt to measure the influence of the “speaker,” and, conceptually, this construct applies in the Facebook world, driven by various aspects of the user: how many friends they have, how often they update their status, and, most importantly, how often the content they share gets liked/commented on/re-shared; it is currently not possible to get any visibility into and segment users who are interacting with your brand on Facebook based on their influence in the medium
• Sentiment – Facebook has the “Like” feature, but no comparable “Dislike” option; this is grade school manners enforcement: “If you can’t give it a thumbs-up, don’t give it any thumb at all…” From a brand perspective, though, it would be nice to be able to track what sorts of posts raise users’ ire (especially for user-generated content) without having to sift through individual posts and comments by hand, which leads me to…
• Sentiment…continued — sentiment is a tough nut to crack, but it’s something that everyone who deals with social media recognizes as being important; while I don’t necessarily expect Facebook to develop sentiment measurement tools inherently, if Facebook Insights was enhanced to enable the export of all user interactions for a fan page, then third-party tools could be used to conduct a sentiment analysis, and that would be useful
  [UPDATE: While it's not necessarily a business/analyst-friendly option, the Facebook API does allow the retrieval of comments and posts. If you have the chops to tackle it, you can read about the options at http://wiki.developers.facebook.com/index.php/API#Data_Retrieval_Methods. One company that is using the API for that purpose (among others) is Vitrue -- comments and posts get pulled into their Vitrue SRM product in a pretty slick way.]
• Online Listening…to Facebook — Google announced late last year that they were going to start crawling publicly available content in Facebook, and, presumably, online listening platforms will not be far behind (maybe some of them already do?). But, this listening is inherently limited to public content in Facebook (fan pages are public, so they would be included, presumably, which is a good thing). There would be a major backlash if Facebook enabled third-party tools to crawl and index “private” content. Does that mean that Facebook should enable it’s own intra-Facebook online listening capability? Marketers would certainly love to have the information, even if it is only available in a way that maintains users’ anonymity, but any move in this direction would be a dicey proposition for Facebook (even if they hid user information, it would be conceivable that users would provide enough information in what they post that a company would still be able to identify a specific individual — even if that was only going to be possible 1 time in 100,000, privacy advocates would jump all over Facebook for allowing the theoretical possibility)

It will be interesting to see where Facebook goes over the next 1-2 years when it comes to empowering marketers to measure and analyze their Facebook-based tactics. It should be a fun ride.

What am I missing here?

I don’t think I will ever forget the first time that I made a presentation at work. It was just over a decade ago, I was just a few months into my employment at a company where I would work for the next eight years, and I was on the hook to present a new process to a room of 20 engineers. I diligently prepared my transparencies (I’m old enough to have used an overhead projector, but not old enough to refer to the medium they supported as “foils”). I rehearsed the material again and again.

And I bombed.

The material was dry as it was, but it wasn’t, by any means, unmanageable content. I just didn’t do a good job of managing it!

Fast forward 10 years, and I found myself giving a presentation to a room of 50-60 people, and the material was set up to be just as naturally engaging — presenting on an approach to measurement and analytics to…a bunch of marketers.

The presentation went much better, judging both from the engagement level of the audience and discussions that it has prompted weeks later. I’m no Steve Jobs, but I’ve paid attention to what seems to work and what doesn’t (both in my presentations and others), read some articles here and there, and, I realized, read a few books along the way that have really helped.

So, with that — four books that all have a heavy component of “how the brain works” and that, collectively, have taught me a lot about how to present information, be it a dashboard, a report, or a presentation.

Gladwell and Gilbert

The first two books are books that I read within a few months of each other. To this day, I recall specific anecdotes with no idea which book they came from. Blink: The Power of Thinking Without Thinking made the rounds when it first came out as “another great book by Malcolm Gladwell” (following The Tipping Point: How Little Things Can Make a Big Difference). The fundamental anecdote of Blink has to do with our “adaptive unconscious” — our intuition and ability to “know” things without fully needing to process them. As he dives into example after example, Gladwell touched on various aspects of how the brain works.

Daniel Gilbert’s Stumbling on Happiness takes a more directly psychological angle, but it covers some of the same territory. One of Gilbert’s main points is that the human brain does not remember things like we think it does — pointing out that a vividly remembered, down-to-the-color-of-the-shirt-you-were-wearing memory is not really an as-recorded memory at all. Rather, the brain remembers a few specific details and then makes up / fills in the rest when the memory gets called up. It’s so good at filling in these blanks that it fools itself into not being able to tell fact from interpolation!

Both of these books made an impact on me, because they pointed out that how we take in, process, and store information doesn’t work at all like we intuitively think it does. And, both books set up the next two books by shaking the assumptional foundations I had of how we, as humans, think.

Straight-Up Business Reading

Chip and Dan Heath’s Made to Stick: Why Some Ideas Survive and Others Die is a practical manual for communicating information that you want your audience to pay attention to and retain. They boil the components into a five-letter acronym — S.U.C.C.E.S. — and go into each component in detail.

The elements are Simple, Unexpected, Concrete, Credible, Emotional, and Stories, and they provide a nice framework for critiquing how we communicate any idea. Irecognize that I regularly struggle with Simple, Concrete, and Stories as elements in my blog posts. But, every element is one that can be injected using some discipline and time to do so. I nailed all three of these elements a number of years ago when I found myself on an internal lecture circuit trying to drum up large donors for my company’s annual United Way campaign — I was heavily vested in conveying a strong message, and I wound up using an example of my grandfather’s battle with Alzheimer’s as a way to pull the audience in and ask them to find something they were passionate about and support it. I also wove in various quirky takes on how $10/week would really add up — think the sort of thing you hear again and again from your local NPR station during fundraising drives. In the case of that campaign, we blew our numbers out of the water — had a 500% increase in the number of people who gave at the “leadership level” that year. Now, a lot of things had to come together to make that happen, but, to this day, I’m sure my well-crafted, well-rehearsed, and sincere speech made to at least a dozen different groups of employees (and the fact that I was a fairly low-level employee making the case — I was asking people who were making a lot more money than I was to give at least as much as I was), played a non-trivial role.

And that was years before I read Made to Stick. But, the book helped me reflect on any number of presentations — ones that worked and ones that didn’t.

And, Finally, Wisdom from a Neuroscientist

The last book in this tetralogy is one that I just finished reading — Brain Rules: 12 Principles for Surviving and Thriving at Work, Home, and School, by John Medina. I stumbled across the book as a recommendation from Garr Reynolds of Presentation Zen, so I wasn’t surprised that it had some very practical tips, as well as the “why?” behind them, for communicating effectively. Medina’s premise is that there’s a ton of stuff we don’t yet understand about the brain. BUT, there are also a lot of things we do know about the brain, and many of those lay out pretty clearly that the way we work in business and the way our education system is set up both run counter to how the brain naturally functions.

These “things we do know” are broken down into 12 “rules” — exercise (good for the brain), survival (why and how the brain evolved…and implications), wiring (how the brain works at a highly micro level), attention (there’s NO SUCH THING as multitasking…and other goodies), short-term memory (what makes it there and how), long-term memory (what makes it there, how, and how long it takes to get there), sleep (good for the brain), stress (some kinds are good, some kinds are bad), sensory integration (the more senses involved, the better the memory), vision (the #1 sense), gender (men are from Mars…), exploration (age doesn’t really degrade our ability to learn). Medina ends each chapter (one rule per chapter) with “Ideas” — implications for the real world based on the information presented.

The book goes into very technical detail about how, when, and where electrical charges zip around in our skulls to accomplish different tasks. While that information is not directly applicable, each time he goes there it’s as a setup to more directly useful information. Throughout the book, Medina provides practical thoughts for how to communicate more effectively — helping people pay attention (getting the information you are communicating into working memory) and retain the information over both the short and the long term. Two of my absolute favorite nuggets from the book were:

• p. 130 (in the chapter on long-term memory) — Medina has the reader do a little memory exercise with the following characters: “3 $ 8 ? A % 9.” The fact he drops after the exercise is interesting: “The human brain can hold about seven pieces of information for less than 30 seconds! If something does not happen in that short stretch of time, the information becomes lost.” This is about getting information on its way from working memory to long-term memory and how repetition, thinking about the information, and talking about the information all helps it on its way. As a communicator (be it through a presentation or through a dashboard of data), this seems like powerful stuff — how often have we all seen someone cut loose with slide after slide of mind-numbing information? The human brain simply cannot take all of that in and retain it without some help!
• p. 239 (in the chapter on vision) — Medina has a section titled “Toss your PowerPoint presentations.” I groaned. While I get highly annoyed by the rampant misuse of PowerPoint, I’m not a Tufte acolyte to the point that I see the tool itself as evil. In the second paragraph, though, Medina clarifies by providing a two-step prescription: 1) burn your current presentations, and 2) make new ones. Medina’s beef with PowerPoint is that the default slide template is text-based with a six-level hierarchy. This entire chapter is about how a picture really is worth 1,000 words, and Medina pleads with the reader to cut wayyy back on the text in his/her presentations (he has a fascinating explanation of how, when we read, we’re really interpreting each letter as a small picture…and that’s actually not a good thing for retention of information).

There are oodles of other good information in the book, but these are two of the snippets that really resonated with me.

Better to Be Steve Jobs than Bill Gates

I do believe that some people have better communication instincts than others. I’ll never be Steve Jobs when it comes to holding an auditorium in the palm of my hand. But, between reading these books and thinking through my own evolution as a communicator (this blog notwithstanding…but I’ve always said that I write this blog to keep my e-mails shorter and to try out ideas that occur to me during the day — sorry folks…both of you…but this blog is mostly for me!), I’m convinced that effective communication is a trainable skill.

I’ve also noticed that, the more I have to communicate, and the more I work to do so effectively, the easier it seems to be getting. In another 20 years, I might just have it nailed!

I’ve been using an anecdote of late that Malcolm Gladwell supposedly related at a SAS user conference earlier this year: over the last 30 years, the challenge we face when it comes to using data to drive actions has fundamentally shifted from a challenge of “getting the right data” to “looking at an overwhelming array of data in the right way.” To illustrate, he compared Watergate to Enron — in the former case, the challenge for Woodward and Bernstein was uncovering a relatively small bit of information that, once revealed, led to immediate insight and swift action. In the latter case, the data to show that Enron had built a house of cards was publicly available, but there was so much data that actually figuring out how to extract the underlying chicanery without knowing exactly where to look for it was next to impossible.

With that in mind, I started thinking about all of the sources of data that marketers now have available to them to drive their decisions. The challenge is that almost all of the data sources out there are good tools — while they all claim competitive advantage and differentiation from other options…I believe in the free markets to the extent that truly bad tools don’t survive (do a Google search for “SPSS Netgenesis” and the first link returned is a 404 page — the prosecution rests!). To avoid getting caught up in the shiny baubles of any given tool, it seems worth organizing the range of available data some way — put every source into a discrete bucket.  It turns out that that’s a pretty tricky thing to do, but one approach would be to put each data source available to us somewhere on a broad spectrum. At one end of the spectrum is data from secondary research — data that someone else has gone out and gathered about an industry, a set of consumers, a trend, or something else. At the other end of the spectrum is the data we collect on our customers in the course of conducting some sort of transaction with them — when someone buys a widget from our web site, we know their name, how they paid, what they bought, and when they bought it!

For poops and giggles, why not try to fill in that spectrum? Starting from the secondary research end, here we go…!

Secondary Research (and Journalism…even Journalism 2.0)

This category has an unlistable number of examples. From analyst firms like Forrester Research and Gartner Group, to trade associations like the AMA or The ARF, to straight-up journalists and trade publications, and even to bloggers. Specialty news aggregators like alltop.com fall into this category as well (even if, technically, they would fit better into a “tertiary research” category, I’m going to just leave them here!).

I stumbled across iconoculture last week as one interesting company that falls in this category…although things immediately start to get a little messy, because they’ve got some level of primary research as well as some tracking/listening aspects of their offer.

Listening/Collecting

Moving along our spectrum of data sources, we get to an area that is positively exploding. These are tools that are almost always built on top of a robust database, because what they do is try to gather and organize what people — consumers — are doing/saying online. As a data source, these are still inherently “secondary” — they’re “what’s happening” and “what’s out there.” But, as our world becomes increasingly digital, this is a powerful source of information.

One group of tools here are sites like compete.com, Alexa, and even Google’s various “insights” tools: Google Trends, Google Trends for Websites, and Google Insights for Search. These tools tend to not be so much consumer-focussed as site-focussed, but they’re getting their data by collecting what consumers are doing. And they are darn handy.

“Online listening platforms” are a newer beast, and there seems to be a new player in the space every day. The Forrester Wave report by Suresh Vittal in Q1 2009 seems like it is at least five years old. An incomplete list of companies/tools offering such platforms includes (in no particular order…except Nielsen is first because they’re the source of the registration-free PDF of the Forrester Wave report I just mentioned):

• Nielsen Buzzmetrics
• BuzzLogic
• Radian6
• Alterian/Techrigy SM2
• Filtrbox
• Crimson Hexagon
• Collective Intellect
• Spiderfly

And the list goes on and on and on… (see Marshall Sponder’s post: 26 Tools for Social Media Monitoring). Each of these tools differentiates itself from their competition in some way, but none of them have truly emerged as a  sustained frontrunner.

Web Analytics

I put web analytics next on the spectrum, but recognize that these tools have an internal spectrum all their own. From the “listening/collecting” side of the spectrum, web analytics tools simply “watch” activity on your web site — how many people went where and what they did when they got there. Moving towards the “1:1 transactions” end of the spectrum, web analytics tools collect data on specifically identifiable visitors to your site and provide that user-level specificity for analysis and action.

Google Analytics pretty much resides at the “watching” end of this list, as does Yahoo! Web Analytics (formerly IndexTools). But, then again, they’re free, and there’s a lot of power in effectively watching activity on your site, so that’s not a knock against them. The other major players — Omniture Sitecatalyst, Webtrends, Coremetrics, and the like — have more robust capabilities and can cover the full range of this mini-spectrum. They all are becoming increasingly open and more able to be integrated with other systems, be that with back-end CRM or marketing automation systems, or be that with the listening/collecting tools described in the prior section.

The list above covered “traditional web analytics,” but that field is expanding. A/B and multivariate testing tools fall into this category, as they “watch” with a very specific set of options for optimizing a specific aspect of the site. Optimost, Omniture Test&Target, and Google Website Optimizer all fall into this subcategory.

And, entire companies have popped up to fill specific niches with which traditional web analytics tools have struggled. My favorite example there is Clearsaleing, which uses technology very similar to all of the web analytics tools to capture data, but whose tools are built specifically to provide a meaningful view into campaign performance across multiple touchpoints and multiple channels. The niche their tool fills is improved “attribution management” — there’s even been a Forrester Wave devoted entirely to tools that try to do that (registration required to download the report from Clearsaleing’s site).

Primary Research

At this point on the spectrum, we’re talking about tools and techniques for collecting very specific data from consumers — going in with a set of questions that you are trying to get answered. Focus groups, phone surveys, and usability testing all fall in this area, as well as a plethora of online survey tools. Specifically, there are online survey tools designed to work with your web site — Foresee Results and iPerceptions 4Q are two that are solid for different reasons, but the list of tools in that space outnumbers even the list of online listening platforms.

The challenge with primary research is that you have to make the user aware that you are collecting information for the purpose of research and analysis. That drops a fly in the data ointment, because it is very easy to bias that data by not constructing the questions and the environment correctly. Even with a poorly designed survey, you will collect some powerful data — the problem is that the data may be misleading!

Transaction Data

Beyond even primary research is the terminus of the spectrum — it’s customer data that you collect every day as a byproduct of running your business and interacting with customers. Whenever a customer interacts with your call center or makes a purchase on your web site, they are generating data as an artifact. When you send an e-mail to your database, you’ve generated data as to whom you sent the message…and many e-mail tools also track who opened and clicked through on the e-mail. This data can be very useful, but, to be useful, it needs to be captured, cleansed, and stored in a way that sets it up for useful analysis. There’s an entire industry built around customer data management, and most of what the tools and processes in that industry focus on is transaction data.

What’s Missing?

As much as I would like to wrap up this post by congratulating myself on providing an all-encompassing framework…I can’t. While there are a lot of specific tools/niches that I haven’t listed here that I could fit somewhere on the spectrum of tools as I’ve described it, there are also sources of valuable data that don’t fit in this framework. One type that jumps out to me is marketing mix-type data and tools (think Analytic Partners, ThinkVine, or MarketShare Partners). I’m sure there are many other types. Nevertheless, it seems like a worthwhile framework to have when it comes to building up a portfolio of data sources. Are you getting data from across the entire spectrum (there are free or near-free tools at every point on the spectrum)? Are you getting redundant data?

What do you think? Is it possible to organize “all data sources for marketers” in a meaningful way? Is there value in doing so?

I’m right at three months into my new gig, and, around the office, probably the most commonly known fact is, “He hates pie charts.” It’s not that I’ve exactly been standing at the elevator handing out leaflets explaining why pie charts are evil, but I have, apparently, chosen a couple of particularly public venues to make a mild statement or two. And, the quasi-preplanned visceral groan when some co-workers put up a pie chart might’ve contributed just a teensy bit.

I’ve been put on the spot since then a couple of times to do one of two things:

• Explain why pie charts are evil, or
• Agree that one or another particular usage of a pie chart is appropriate

After catching up on some blog reading yesterday morning and seeing an excellent example of pie chart alternatives from Jon Peltier, and then watching seven presentations yesterday, six of which used the same basic presentation template, and five of which stuck with a pie chart for the sole non-text slide in the presentation, how could I not write another post?! Let’s see how succinct I can make it (don’t hold your breath that you could read the whole thing before exhaling!).

Yes, There is ONE Thing That a Pie Chart Does Well

This kills me, because there’s one way, in a a very narrow set of circumstances, that pie charts do marginally better than alternatives. All THREE of the following criteria have to be met for this to be the case:

• Exactly 2 or 3 categories that make up the “whole”
• A fairly significant difference in % makeup for each of the categories
• Plenty of space available to present the information

99 times out of 100 when pie charts get used, all of these criteria are not met. But, there, I’ve admitted that there is a situation where pie charts are appropriate.

Of course, mullets are an appropriate hairstyle if you are prone to both warm ears and spontaneous hair donations…but that doesn’t mean I’m going to sport one!

Of Course, We Must Start with a Before/After Example

With only the category names changed, below is one of the pie charts I saw yesterday:

In my experience, a simple horizontal bar chart is a better option (among a variety of better options):

Why is this a better option? Oh, let me count the ways…

1. Rainbows Are Good in Princess Tales — Not in Data Visualization

When it comes to data visualization, a chart that doesn’t rely on multiple colors always trumps a chart that does. Four reasons:

• If you use subtle/muted colors, you can’t get past 4 or 5 categories before you are asking the person reading the chart to work hard to distinguish between subtle shading differences
• If you use bright/high-contrast colors, you’re asking your user to put on sunglasses to keep from wincing at the visual overkill
• Roughly 10% of men suffer from some form of color-blindness — it’s darn tricky to nail a palette with more than a small handful of colors that works across the various types of the condition (of course, if you’ve got a secret agenda to have women take over the world, this is one way to contribute, as color blindness is exceedingly rare in women)
• Maybe you’re presenting your chart in glorious, projected color…but are you sure no one is going to try to print it in black-and-white?

These are all issues with any pie chart that has more than 3 categories. None of these are an issue with a horizontal bar chart.

2. Labels, Labels, Labels

If you’ve every constructed a pie chart in Excel, you’ve run into the challenge of trying to get all of the wedges labeled right there on the chart. Excel continues to make odd choices as to where to wrap text in pie charts, and the circular nature of the whole layout means some wedges have plenty of horizontal labeling room, while others have almost none. You’ve tried some (or all) of the following:

• Using leader lines for some of the wedges so you can label the most troubling wedges somewhere more spacious
• Abbreviating the category names
• Strategically rotating the chart so that the labeling all happens to work (it never does)
• Rearranging the underlying data so that the pie wedges occur in a different order (which also never works)

After fiddling with the above, you finally break down and yank the labels from the chart and just use a legend. This is bad, bad, BAD! Scroll back up to the pie chart example above and pretend you’re actually trying to interpret the data, but pay attention to how many times you look back and forth between the legend and the pie. This is putting a totally unnecessary strain on your brain! Take a look at the horizontal bar chart — no jumping back and forth needed!

With a horizontal bar chart, the label sits right next to the data, and it doesn’t need to be abbreviated to do so (this is one reason that I find horizontal bar charts to be better than vertical column charts in many cases — with a horizontal orientation, the labels have more width with which to work).

3. Those Pesky Near-Zero Values

Pie charts suck at the small percentages. Small percentage categories wreak havoc on the labeling issue, for sure, but they’re also nearly impossible to compare to each other. In the example above, the smallest percentage is 3%, and that’s almost manageable. But, heaven forbid you have a couple of pesky sub-one-percent categories, and you’re looking at wedges that look suspiciously like the lines between wedges.

4. Seeing Small Differences

Fundoogles & Flibbers came in at 3%, while Dracula’s Mickety Micks came in at 5%. Do the wedge sizes really look different? That’s a fundamental challenge with pie charts — we don’t do a very good job of comparing the areas of these odd sorta-triangular-but-with-one-curved-side shapes. In the case of the bar chart, all you have to compare is lengths — much easier.

5. Economy (of Space) Is a Virtue

Check out the overall size of the charts. While they have the same font size, the same text displayed, and the same width, the bar chart is 20% shorter…and it could have been shorter still! Bar charts are more efficient space-wise. With pie charts, and largely because of the other issues listed above, it’s often necessary to make the chart larger and larger to make it readable.

Of Course, This Exampel Was At Least Flat

This post would be twice as long if I went into the additional issues of using the “3D effect” version of the pie chart.

[Update] Always Room for Improvement

Of course, the danger of posting a “here’s a better way” is that you leave yourself open for suggestions as to how the better way can be improved! See Naomi’s comment below. She raises a good point — basically, that I didn’t do a great job of heeding the data-pixel ratio with my bar chart! So, below is a revised version.

In a subsequent email exchange, Naomi made the case for keeping the x-axis and the numbers, but simply removing the “%” signs entirely and putting the word “Percent” in the axis label:

Her main point is that numbers can be read more easily if they are not cluttered with symbols like dollar signs and percent signs. And, her case for keeping the gridlines and labeled axis is that it helps show that the bars are drawn to scale — there hasn’t been any incorrect or misleading scaling (intentional or not — in the same spate of presentations that spurred this post, there was a bar chart with an accompanying table of data…and one of the bars was clearly not accurate).

I’m partial to the version with all of the lines removed, but, at this point, the debate is at a much healthier level than “pie vs. bar,” so I’m happy!

Last week was our monthly Web Analytics Wednesday in Columbus. Foresee Results sponsored the event and provided a highly engaging speaker: Kevin Ertell, Foresee’s VP of Retail Strategy and the blogger behind Retail: Shaken Not Stirred.

We had a good crowd — just under 30 people — and we did our usual half-hour of networking before sitting down to order food and cover the evening’s topic.

We had attendees from a wide range of companies: Nationwide Insurance, Resource Interactive, Victoria’s Secret (including Monish Datta…which I mention here solely for quasi-inside SEO joke purposes), DSW, Diaz & Kotsev Business (Web) Consulting, WebTech Analytics, Quest Software (makers of Foglight, actually, which I didn’t realize until I was writing the rest of this post), QStart Labs, Submerged Solutions, Bizresearch, Lightbulb Interactive, JoeMetric, Express, Cardinal Solutions, and various independent consultants. By my count, 30% of the attendees were first-timers, and the remaining attendees were a pretty even split between hard-core regulars and every-few-months dabblers.

Kevin is a great speaker — one of those guys whose use of PowerPoint is primarily to provide images that back up the stories he weaves.

One of the stories was the “tree stump on the conference room table” story, which was about how we get used to having odd, not-particularly-helpful aspects of our web sites that are jarring to first-time and infrequent visitors, but that we never think to address.

You can ping Kevin on Twitter directly for a more complete explanation on that analogy, if you want. If I try to recreate it entirely, I’ll butcher it for sure! I will take a shot at summarizing the four-step process Kevin laid out for going beyond web analytics data to drive site improvement, though, which was the meat of the presentation.

Step One: Ask Your Visitors for Feedback

On-site surveys provide valuable information, because they let you ask your visitors questions directly rather than simply trying to infer what it was they are trying to do, how successful they were at doing it, and how smooth the process was based strictly on behavioral data. Web analytics = bahavioral data. Survey data = attitudinal data. Got it?

Some of the highlights on this step:

• Incentives aren’t needed to get people to take a 15-30 question survey — I think Kevin said they see something like 6-10% of the people who are offered a survey actually accept the offer (not all visitors to a site get offered the survey) and they’re able to build up an adequate sample fairly quickly in most cases
• The way Foresee Results offers surveys, typically, is that they offer the survey when visitors arrive on the site, but then conduct the survey on exit
• The wording of the survey questions matters — there are good/valid ways to word questions and there are bad/invalid ways to word questions; there are oodles of research and expertise on that subject, and it’s worth partnering with someone (a consultant, a company) who really knows the ins and outs on that front to make sure that the data you collect is valid
• The Foresee Results secret sauce is that they ask questions that fall into three broad categories: 1) questions about different aspects of the site (content, functionality, navigation, search, etc.), 2) questions to gauge customer satisfaction (very precisely worded questions that are backed up by the research behind The American Customer Satisfaction Index — ACSI), and 3) questions to gauge likely future behavior (likelihood to purchase online, likelihood to purchase offline, likelihood of returning to the site, etc.). Foresee Results then uses an analytic model to link these three elements together: the first category as a dependent variable affecting customer satisfaction, and customer satisfaction, in turn, being a dependent variable affecting the various future behaviors. It’s a pretty nifty tool that I’ve been learning more about over the past few months. Powerful stuff.

This step, done right, gives you the basic diagnostics: where the most significant opportunities for driving improvements exist with your site.

Step Two: Augment Quantitative with Qualitative

This step is to augment the quantitative survey data with more qualitative information. The quantitative data can help you slice/segment the data so that you can review the responses to open-ended questions in a more meaningful way.

Presumably, these qualitative questions are ones that you update over time as you are identifying specific areas on which you want to focus. If for instance, you found out in Step One that the navigation was an area where your site scores low and also has a significant impact on customer satisfaction, then you might want to gather some qualitative data specifically regarding navigation, and you might want to break that out between people who came to the site expecting to make a purchase, as opposed to people who came to the site simply to do comparison shopping.

This sort of analysis will give you insight into the specific friction points on the site — what types of visitors hit them and what sorts of tasks they’re trying to accomplish when they do.

Step Three: Watch Customers (in a Focussed Manner)

This is a step that Kevin pointed out companies sometimes try to put first, which makes it unnecessarily expensive and time-consuming. The key here is to use the information from the first two steps to focus what you are going to observe and how. Various options for watching customers:

• Session replay — what exactly did visitors on the site do and how; in the case of Foresee Results, these replays can be tied directly to specific survey respondents (pretty slick), but Tealeaf and Foglight are tools that provide replay functionality, too
• Eye-tracking — this requires getting people into a lab of some sort, so, obviously, the more focussed you can get, the better
• Usability testing — this may include eye-tracking, but it certainly doesn’t have to; obviously, there are benefits of being able to focus the usability testing, whether it’s conducted in a usability lab or even in-store

Now, you should really have a good handle on specifically what’s not working. But, what if you don’t really have any good ideas as to what to do about it? Then…

Step Four: Usability Audit

Work with usability experts to assess the aspects of your site that are underperforming. Arm them with what you have learned in the first three steps!

To me, it seems like you could swap steps three and four in some cases — let a usability expert audit your site and identify likely opportunities to improve the trouble spots.

Driving Continuous Incremental Improvement

By keeping the survey running on an on-going basis — adjusting questions as needed, but keeping the core questions constant — you can monitor the results of changes to the site as you roll them out. And, of course, your web analytics data — especially on-site conversion data — is one tool for monitoring if you are driving outcomes that matter.

One point on the incremental changes front: during the Q&A, Kevin talked about how sites that roll out major redesigns invariably see a temporary dip in results while visitors get used to the new site. Incremental changes, on the other hand, can occur without that temporary drop in performance.

Interesting stuff!

As I wrote in my last post, I’ve been spending a lot of time building out Excel-based dashboard structures and processes of late. I also wrote a few weeks ago about calculating trend indicators. A natural follow-on to both of those posts is a look at the “metric widget” that I use as a basis for much of the information that goes on a dashboard. Below is an example of part of a web site dashboard (not with real data):

I’ll walk through some of the components here in detail, but, first, a handful of key points:

• There is no redundant information — it’s not uncommon to see dashboards (or reports in general) where there is a table of data, and that table of data gets charted, and the values for each point on the chart then get included as data labels. This is wasteful and unnecessary.
• Hopefully, your eyes are drawn to the bold red elements (and these highlights should still pop out for users with the most common forms of colorblindness — I haven’t formally tested that yet, though) — this is really the practical application of the vision I laid out in my Perfect Dashboard post.
• I have yet to produce a dashboard solely comprised of these widgets — there are always a few KPIs that needs to be given more prominent treatment, and there are other metrics that don’t make sense in this sparkline/trend/current format
• I do mix up the specific measures on a dashboard-by-dashboard basis. In the example above, showing the past two years of trends by month, and then providing quarterly totals and comparisons, makes the most sense based on the planning cycle for the client. But, that certainly is not a structure that makes sense in all situations.

And now onto the explanation of the what and why of each element, working our way from left to right.

Metric Name

This one hardly warrants an explanation, but I’ll point out that I didn’t label that column. That was a conscious decision — the fact that these are the names of the metric is totally obvious, and Edward Tufte’s data-ink ratio dictates that, if it doesn’t add value, don’t include it!

Past 12 Months Sparkline

The sparkline is another Tufte invention, and it’s one that has really taken off in the data visualization space. That’s good, because sparklines are darn handy, and the more people get used to seeing them, the less there will need to be any “training” of dashboard users to interpret them. Google Analytics has been using sparklines for a while, even, so we’re well on our way to mass adoption!

One tweak on the sparkline front that I came up with (although I’m sure others have done something similar): I add a second, gray sparkline for either the target or the prior reporting period. I like that this gives a quick, easily interpretable view of the metric’s history over a longer period — has it been tracking to target consistently, consistently above or below the target, or bouncing back and forth? Is there inherent seasonality in the metric (signified by both the black and gray sparklines having similar spike/dip periods)?

One limitation of sparklines is that they don’t represent magnitude very well. If, for instance, a particular metric is barely fluctuating over time, then, depending on how the y-axis is set up, the sparkline can still show what looks like a wildly varying value. It’s a minor limitation, though, so I’ll live with it.

4-Month Trend Arrow

The 4-month trend is the single icon that results from a conceptually simple (but a little hairy to calculate) assessment of the most recent four data points. That was the punchline of an earlier post on calculating trend indicators. Whether the basis of the trend is months, weeks, or days can vary (not within one dashboard, generally, but as a standard for the dashboard overall), as well as whether it’s 4, 5, 6, or more data points. It’s a judgment call for both driven by the underlying business need that the dashboard supports.

I promise, promise, promise to make a simplified example of this arrow calculation and post it in a future post — check the Comments section for this post to see if a linkback exists (I’ll come back and update this entry as well once it’s done)

Current

Typically, when sparklines are used, the exact value of the last point in the sparkline is included. In the example above, I’ve done something a little different, in that I actually provide the sum of the last three data points. This is a quarterly dashboard, but the sparkline has a monthly basis to it to show intra-quarter trends. If the current value is sufficiently below the target threshold, then the value is automatically displayed as bold and red.

There are certainly situations where “Current” would actually be the last point on the sparkline. Like the trend arrow calculations, it’s a judgment call based on the business need that the dashboard supports.

YOY

In the example above, there is a comparison to the prior year. But, this could be a comparison to the target instead. Target-based comparison is even better — straight period-over-period comparisons tend to feel like something of a cop out, as prior periods really are more “benchmarks” than true “targets.” Now, setting a target as something like “15% growth over the prior year” has some validity! That would then impact both the gray sparkline, the “when does Current go bold red,” and this %-based calculation.

28 Data Points

In the version of the widget above, there are 28 unique pieces of data presented for each metric: the metric name (1), the black sparkline (12), the gray sparkline (12), the trend indicator (1), the current value (1), and the year-over-year growth percentage (1). And that’s not counting the conditional formatting that highlights values as bold and red when certain criteria are met. That’s a key aspect of the widget design. 28 sounds like a lot of data to represent for a single metric. Yet, they seem pretty digestible in this format, don’t they?

Let me know what you think. Does this work? What doesn’t work?

I’ve been spending a lot of time lately working on dashboards — different dashboards for different purposes for different clients, with a heavy emphasis on making dashboards that can be efficiently updated. I’m finding that I keep coming back to two key principles:

• A dashboard, by definition, fits on a single page — this is straight out of Stephen Few’s book Information Dashboard Design: The Effective Visual Communication of Data; I was skeptical that this was really possible when I first read it, but I’ve increasingly become a believer…with the caveat that there is ancillary data that can be provided with a dashboard as backup/easy drilldowns
• The dashboard must include logic to dynamically highlight the areas that require the most attention.

The second principle is the focus of this post.

Actionable Metrics

It’s become cliché to observe that data must be converted to information that drives action. I’ve got no argument with that, but, all too often, the people who make this statement would also see this statement as blasphemy:

Most metrics should drive no action most of the time

Any good performance measurement system is based on a structured set of meaningful metrics. Each of those metrics has a target set, either as a hard number, as a comparison to a prior period, as a comparison to some industry measure, or something else.

Here’s the key, though: most of those metrics will likely come in within their target range most of the time! That’s a good thing, because it is rare that a business is equipped to chase more than a handful of issues at once.

A Conceptual (If Not Realistic) Dashboard

At the end of the day, when your user looks at a dashboard, here’s what they really are hoping to get:

This is as actionable as it gets:

• Only the areas that are performing well outside of expectations are shown
• What’s actually happening is stated in plain English
• The person viewing the dashboard has a concise list of what he/she needs to start looking into immediately

Will your users ever tell you this is what they’re looking for? No! And, if asked, the reasons why not would include:

• “I need to see everything that is going on — not just the stuff that is performing outside targets (…because I’m just not comfortable trusting that we designed a dashboard that is good enough to catch all the things that really matter).”
• “My boss is likely to ask me about her specific pet metric…so I need to have that information at my fingertips, even if it’s not going to drive me to take new action.”
• “I need to see all of the data so that I can identify patterns and correlations across different aspects of the marketing program.”

Arguing any of these points is an exercise in futility. Between the explosion of data that is available, the fact that not a day passes without a Major Marketing Mind talking about how important it is for us to leverage the wealth of data at our fingertips, and the fact that humans are inherently distrustful of automation until they have seen it working successfully for an extended period of time, all mean that a dashboard, in practice, has to include a decent chunk of information that will not drive any new action.

But the Concept Is Still Useful

I believe the conceptual dashboard above is a useful guiding vision. At the end of the day, we want to provide, and our users want to receive, information that is clear and concise, which the dashboard above certainly is. if we morph the concept above just a little bit, though, we get a dashboard that is only slightly less impactful but heads off all of the concerns listed earlier:

Get the idea? The same highlights pop, but additional data is included, and it all still fits on a single page. Obviously, the real dashboard would be one step further diluted from this by presenting actual metrics — numbers, sparklines, etc. But, by working hard to keep all of the on-target data as muted as possible, some clever use of bold and color through conditional formatting can still make what’s important pop.

Parting Thoughts and Clarifications

Any dashboard, whether it’s managed through an enterprise BI tool, through Microsoft Excel, or even through PowerPoint, should be designed so that the structure of the dashboard does not change from one reporting period to the next — the same metrics appear in the same place week in and week out. BUT, within that structure, there should be a concerted effort to make sure that the metrics that are the farthest off target (usually the ones that are the farthest off target in a bad way, but if something is off the charts in a good way, that needs to be highlighted as well) are what the user’s eye is drawn to. And, furthermore, those are the metrics that warrant the first pass of drilling down to look for root causes.

I’m finding myself in a lot of conversations where I’m explaining the difference between “outputs” and “outcomes.” It’s a distinction that can go a long way when it comes to laying out a measurement strategy. It’s also a distinction that can seem incredibly academic and incredibly boring. To the unenlightened!

Outputs are simply things that happened as the result of some sort of tactic. For instance, the number of impressions for a banner ad campaign is an output of the campaign. Even the number of clickthroughs is an output — in and of itself, there is no business value of a clickthrough, but it is something that is a direct result of the campaign.

An outcome is direct business impact. “Revenue” is a classic outcome measure (as is ROI, but this post isn’t going to reiterate my views on that topic), but outcomes don’t have to be directly tied to financial results. Growing brand awareness is an outcome measure, as is growing your database of marketable contacts. Increasing the number of people who are talking about your brand in a positive manner in the blogosphere is an outcome. Visits to your web site is an outcome, although if you wanted to argue with me that it is really just an aggregated output measure — the sum of outputs of all of the tactics that drive traffic to your site — I wouldn’t put up much of a fight.

Why Does the Distinction Matter?

The distinction between outputs and outcomes matters for two reasons:

• At the end of the day, what really matters to a business are outcomes — if you’re only measuring outputs, then you are doing yourself a disservice
• Measuring outputs and outcomes can help you determine whether your best opportunities for improvement lie with adjusting your strategy or with improving your tactics

Your CEO, CFO, CMO, COO, and even C-3PO (kidding!) — the people whose tushes are most visibly on the line when it comes to overall company performance — care that their Marketing department is delivering results (outcomes) and is doing so efficiently through the effective execution of tactics (outputs).

Campaign Success vs. Brand Success

Avinash Kaushik wrote a post a couple of weeks ago about the myriad ways to measure the results of a “brand campaign.” Avinash’s main point is that “this is a brand campaign, so it can’t be measured” is a cop-out. If you read the post through an “outcomes vs. outputs” lens, you’ll see that measuring “brand” tends to be more outcome-weighted than output-weighted. And (I didn’t realize this until I went back to look at the post as I was writing this one), the entire structure of the post is based on the outcomes you want for your brand — attracting new prospects, sharing your business value proposition more broadly, impressing people about your greatness, driving offline action, etc.

Avinash’s post focuses on “brand campaigns.” I would argue that all campaigns are brand campaigns — while they may have short-term, tactical goals, they’re ultimately intended to strengthen your overall brand in some fashion. You have a strategy for your brand, and that strategy is put into action through a variety of tactics — direct marketing campaigns, your web site, a Facebook page, press releases, search engine marketing, banner ads, TV advertising, and the like. Many tactics are in play at once, and they all act on your brand in varying degrees:

And, of course, you also have happenstance working on your brand — a super-celebrity makes a passing comment about how much he/she  likes your product (or, on the other hand, a celebrity who endorses your product checks into rehab), you have to issue a product recall, the economy goes in the tank, or any of these happen to one of your competitors. You get the idea. The picture above doesn’t illustrate the true messiness of managing your brand and all of the other arrows that are acting on it.

Oh, and did I mention that those arrows are actually fuzzy and squiggly? It’s a messy and fickle world we marketers live in! But, here’s where outcomes and outputs actually come in handy:

1. In a perfect world, you would measure only outcomes for your tactics…which would mostly mean you would actually measure at some point after the arrows enter the brand box above, but…
2. You don’t live in a perfect world, so, instead, you find the places where you can measure the brand outcomes of your tactics, but, more often than not, you measure the outputs of your tactics (measuring closer to the left side of the arrows above), which means…
3. You actually measure a mix of outcomes and outputs, which is okay!

Tactics are what’s going on on the front lines. Their outputs tend to be easily measurable. For instance, you send an e-mail to 25,000 people in your database. You can measure how many people never received it (output — bouncebacks), how many people opened it (output), how many people clicked through on it (output), and how many people ultimately made a purchase (outcome). Except the outcome…is probably something you wildly under count, because it can be darn tough to actually track all of the people for whom the e-mail played some role in influencing their ultimate decision to buy from your company. The outputs  can also be measured very soon after the tactic is executed (open rate is a highly noisy metric, I realize, but it is still useful, especially if you measure it over time for all of your outbound e-mail marketing), whereas outcomes often take a while to play out.

At the same time, if you ignored measuring the tactics and, instead, focussed solely on measuring your brand, you would find that you were measuring almost exclusively outcomes (see Avinash’s post and think of typical corporate KPIs like revenue, profitability, customer satisfaction, etc.)…but you would also find that your measurements have limited actionability, because they reflect a complex amalgamation of tactics.

So, What’s the Point?

Measure your brand. Measure each of your tactics. Accept that measurement of the tactics is heavily output-biased and measurable on a short cycle, while measurement of your brand is heavily outcome-biased and is a much messier and sluggish beast to affect.

Watch what happens:

• If your brand is performing poorly (outcomes), but your tactics are all performing great (outputs), then reconsider your strategy — you chose tactics that are not effective
• If your brand is performing poorly (outcomes) and your tactics are performing poorly (outputs), then scrutinize your execution
• If your brand is performing well…cut out early and play some golf! Really, though, if your tactics are performing poorly, then you may still want to scrutinize your strategy, as you’re succeeding in spite of yourself!

The key is that tactics are short-term, and driving improvement in how they are executed — through process improvements, innovative execution, or just sheer opportunism — is an entirely different exercise (operating on a different — shorter — time horizon) than your strategy for your brand. Measure them both!