Gilligan on Data by Tim Wilson

Not a week goes by that I don’t see, hear, read, or preach on the topic of measuring marketing results. From equating Marketing ROI to The Holy Grail, to sticking my tongue in my cheek to the point of meanness when it comes to a “simple” process for establishing corporate metrics, to mulling over Marketing ROI vs. Marketing Accountability, there really is no end to the real-world examples that warrant commentary. The reason? Because it’s hard to figure out how to measure marketing’s impact in a meaningful way. It can be done, and it needs to be done, but it requires having a very clearly defined strategy and objectives to do it well, and, even then, the measurement is not as perfect and precise as we would like it to be.

So…it’s hard. I agree.

Try being a non-profit.

I do some volunteer work with the United Way of Central Ohio. Specifically, I sit on the Meeting Emergency and Short Term Basic Needs Impact Council, as well as the Emergency Food, Shelter, and Financial Assistance Results Committee that reports into that impact council, as well as the Emergency Food, Shelter, and Financial Assistance Performance Measures Ad Hoc Committee, which reports into the results committee. Yeah. A mouthful, to say the least. But, it’s the ad hoc committee that has been doing the most tangible work of late and, lookie there!, it’s a committee geared towards performance measurement. Some of the work of that committee inspired an Outputs vs. Outcomes post earlier this year. I find a lot of parallels between measurement in the non-profit world and measurement in the Marketing world.

One difference is that, while Marketers (broad generalization alert!) typically view measurement as a necessary evil — they do want to be data-driven, and they understand the conceptual value of doing measurement…but it’s simply not baked into their DNA to truly want to do it — nonprofits increasingly view measurement as a necessity. (At least) two reasons for this:

• In the nonprofit world, resources are pretty much infinitely scarce — no agency has a real surplus of the services they supply; if they actually get to a point where they’ve got one area reasonably well covered…they expand their offering to meet other needs of their clients
• Donors want to know that their investment is making a difference — on the surface, this may seem similar to investors in a publicly held company; but, investors look at revenue, profitability and growth — financial measures — much more than they scrutinize “Marketing” results (although the “average tenure of a CMO is 27 months” is a stat that gets bandied around quite a bit, so there is some flow down the chain of command to Marketing for accountability); donors to nonprofits are scrutinizing “results” that need to be tied to the agency’s efforts (their investment) and meaningful in an oftentimes relatively soft context

As more and more nonprofits are being driven to collaborate to gain efficiency, more of them are working with foundations or some sort of umbrella organizing/coordinating entity. The Community Shelter Board in Columbus is a really good example of this. It’s an organization that, on its own, does not provide any direct services…but most of the homeless shelters in the area receive funding and some level of direction from the organization. And they do some pretty nice quarterly reports — using plain ol’ Excel. They do it right by: 1) choosing metrics that matter and balance each other, 2) setting targets for those metrics and assessing each metric against its target, and 3) providing a contextual analysis of the results for each set of metrics.  Two thumbs up there.

Right now, the United Way of Central Ohio is trying to do something similar — narrowing its focus, establishing clear strategies in each area, and then honing in on meaningful performance measures for each strategy. It’s a fairly grueling exercise, but well worth undertaking. We constantly find ourselves battling the tendency to broaden the scope of a strategy — it’s hard to find any nonprofit that isn’t doing good work, but trying to support “everything that is good” means not really moving any of the needles in a meaningful way.

One similarity I’ve seen between the non-profit world and Marketing in the for-profit world has to do with capturing data. I touched on this in my post on being data-oriented vs. process-oriented. When trying to establish good, meaningful metrics, it can be very tempting to envision ways the data you want would be captured through a minor process change: “When the inside sales representative answers the phone, we will have him/her ask the caller where they heard about the company and get that recorded in the system so we’ll be able to tie the caller back to specific (or at least general) Marketing activity” or “In order to verify that our agency referral program is working, we’ll call the client we referred 1-2 weeks after the referral to find out if the referral was appropriate and got them the services they needed.” This is dangerous territory. The reason? In both cases, you’re inserting overhead in a process that is not inherently and immediately valuable to person using the process. Sure, it’s valuable in that you can sit back and assess the data later and determine what is/is not working about the process and use that information to come back and make improvements…but that’s an awfully abstract concept to the person who is answering the telephone day in and day out (in both of the above examples). I’ll take an imperfect proxy metric that adds zero overhead to the process that generates it any day over a more perfect metric that requires adding “jus’ a li’l” complexity to the process. And, you know what? My metric will be more accurate!

Photo by batega

In Part I, I took a run at assessing a couple of the most popular baseball statics to see how they measured up as well-formed performance metrics. The other thought that has been running through my mind as I’ve been scoring my son’s baseball games has to do with data management and data quality.

Scoring a baseball game requires a couple of things:

• Making judgment calls as to what actually happened
• Capturing the right information on screwy plays where a lot of stuff happens (this happens a lot more in 9-year-old baseball games than it does in college or professional games)

The first item is one of the reasons why college and professional games have an “official scorekeeper.” There are some plays that are clearly fielding errors…but there are some that require a subjective assessment. And, even if there is clearly an error, it’s sometimes subjective as to whether it was a bad throw or a bad catch.

And, things can get a little complicated. For instance, if you look at this picture closely, you’ll be able to tell that my son is churning his 9-year-old legs as fast as he can (admittedly in pants that would fit most 12-year-olds) as he runs towards first base. And, yet, the catcher is standing right at home plate with the baseball, looking like he’s about to make a throw. What’s going on is either totally obvious to you — meaning you played baseball or have followed it with a decent level of interest — or it seems very bizarre. My son had just struck out. The rule in baseball is that, if a player strikes out AND the catcher drops the ball AND EITHER first base is unoccupied OR there are already two outs in the inning, the catcher needs to retrieve the ball and either tag the batter or throw the ball down to first base so the first baseman can tag first base. This is what’s called a “strike him out, throw him out.” You don’t see it very often in the major leagues or college, because catchers don’t drop that many balls. You see it quite a bit when the players are nine and ten years old.

Either way, my son had an official at bat with a strikeout, even if he made it to first base safely (if, for instance, the catcher overthrew first base). If that had happened (in this case, it didn’t), I would have needed to record a strikeout as well as an error on the catcher.

Sound complicated? It is, and it isn’t. Baseball has other semi-obscure rules — if a baserunner passes another baserunner, he is out. I didn’t learn that rule until I saw it happen to Baylor in the College World Series several years ago. So, scoring a baseball game correctly requires:

• Paying close attention to every play throughout the game
• Knowing the rules well
• Knowing how to quickly and accurately record both “normal” plays and oddball plays
• Being able to make the subjective calls quickly and effectively

I’ve never actually tried to verify this, but I am fairly certain that, if you take three run-of-the-mill scorekeepers and have them score the same game and then compare their results, you will get three slightly different versions of what happened. Yet, we view baseball stats and box scores as being completely black-and-white.

I worked with a data management guru at National Instruments who had a Mark Twain quote in her e-mail signature that said something to the tune of: “A man with one watch always knows what time it is. A man with two watches is never sure.” (I’ve tried to look up the exact wording and confirm that this indeed originated with Mark Twain in the past, and I didn’t have much luck.) This is an excellent point, and it applies to both baseball and business.

If we see a number that appears to be precise — 73 pitches, 10,327 visits to a web site, 2,342 leads — we equate precision with accuracy. It doesn’t cross our mind that a scorekeeper might have inadvertently clicked his pitch counter when the pitcher actually made a throw over to first base to try to pick off a runner. We ignore the fact that all data capture methods when it comes to web analytics are inherently noisy. We forget that sometimes our lead management processes break down and load a duplicate lead or miss a lead. We assume that the data that gets entered into our systems by humans gets entered by a robot rather than by a human — no judgment calls, no mental lapses. And that is simply not reality.

None of this is to say that we should throw out the data. At the end of the day, the ERAs that I calculate for the pitchers on my son’s team are going to be pretty close to the ERAs that another scorekeeper would calculate. Close enough. But, it’s easy to get caught up first in assuming that precise numbers are perfectly accurate, and, then, when something happens where you see a discrepancy, focussing on trying to get the “right” number rather than asking, “Is the difference material?”

The moral? Well…baseball is a great sport!

Oh, wait. There’s more. Don’t rely too much on your data. Don’t expect it to be perfect. Don’t focus on making it perfect. Make sure it’s “good enough” and go from there.

It’s late spring, and my 9-year-old’s baseball season is getting rolling. Due to my gross lack of eye-hand coordination, I volunteered to do the scoring for the team.

There are two basic reasons to score a baseball game:

• Capture enough information on a single page (two pages, actually) that would allow you to entirely recreate the game, play by play, after the fact
• Capture information required to compile game/season statistics for individual players — things like batting average on offense, fielding effectiveness on defense, and ERA for pitchers (also technically a defensive thing)

This means you need to capture a lot of information. Every pitch typically gets recorded in some fashion, and any time a batter finishes at the plate (through a hit, a walk, hit by a pitch, etc.) requires recording additional information. The more detailed the information, the more fun statistics you can pull from the data. But, generally, it’s good to capture a bit more data than you expect to use. For instance, with the system I’m using now, I actually catch the sequence of pitches for any batter: ball, then strike, then strike, then ball, then hit, for instance. That detail, in theory, would allow me to report how a batter fares when he is “behind in the count” (more strikes than balls) vs. “ahead in the count” (more balls than strikes). I’m not going there at all at this point.

At my son’s age, we really just want to make sure we get the final score right. But, the statistics are awfully alluring, so I’ve been logging the information in a spreadsheet so I can do some crunching and see what it tells me. We’re only four games in, and I’m no baseball sophisticate, so I started with the two most popular stats in baseball: earned run average (ERA) and batting average. I regularly mount my “a metric that isn’t tied to a clear objective is not a good metric” soapbox, and it turns out ERA is a pretty great metric. A pitcher’s objective is pretty clear: allow as few runs to score as possible. But, you can’t simply look at the total runs scored on a pitcher for two reasons:

1. A great pitcher who has an infield that regularly flubs plays is going to have more runs scored on him than a similar pitcher who has Derek Jeter and Alex Rodriguez shagging grounders
2. The more innings a pitcher pitches, the more runs he’s going to have scored on him

The “earned run” part of the ERA addresses the first issue by trying to isolate how many runs would have been scored if the other 8 players on the field played perfectly. The “average” part of ERA addresses the second issue by normalizing the metric to a 9-inning average (or a 6-inning average in my son’s case, as their games are only 6 innings long).

What about setting a target? The Gospel According to Gilligan clearly states “Thou shalt not consider a metric worthy if it does not have a preset target.” In the majors, an ERA below 3.00 is considered to be pretty darn good. It’s a “benchmark” of sorts. Or, the other way to look at the metric is to say the target is a 0.00, which is unattainable, but a worthy stretch goal.

So, what about batting average? This seems pretty simple. The batting average is the percent of a player’s at bats where he gets a hit. It’s actually represented as a 3-place fraction rather than a percentage (a .347 batting average means the player gets a hit on 34.7% of his at bats). The stat has been around as long as ERA and has long been considered the metric that is the single best measure of a player’s offensive output. There are a couple of problems with the metric, though. First off, what is a batter’s primary objective? Ultimately, it’s to score runs…but there are too many other factors at play to use that as metric. And, as it turns out, it’s not to get hits as much as it is to get on base. And hits are only one way of doing that. When you peel back the batting average calculation a bit, you find that a walk is not considered an official at bat, so it doesn’t go into the numerator or the denominator of the equation. The reasoning is that the batter got on base because the pitcher screwed up. That’s giving the pitcher a bit too much credit, as a batter who has “plate discipline” is a batter that doesn’t swing at balls — he gets more walks, and when he swings, he’s more likely to be swinging at a hittable ball. (Sacrifices also don’t count as an at bat, but I’m okay with that, as the batter’s objective in that case is to move the baserunner(s) up, so he’s not really trying to get on base himself. A fielder’s choice where the hitter winds up on base doesn’t count as a hit, which makes sense. And, if a batter puts a ball in play and then reaches base on an error, that’s still not considered a hit, because that was more a defensive goof than an offensive success, so it goes into the denominator as an at bat but not in the numerator as a hit. Oh…MAN…can I digress on this subject…!)

Whether it’s true or not, or whether it’s a gross oversimplification, Billy Beane, the general manager of the Oakland A’s, gets credited with this epiphany. The story of how Billy used data to go against baseball’s conventional wisdom to make the Oakland A’s a consistent contender despite their minuscule payroll (by MLB standards) is the basic premise of Moneyball: The Art of Winning an Unfair Game. One of the metrics that Billy and his number crunching assistant started focussing on was on-base percentage (OBP), which includes walks in the numerator and denominator of the calculation. OBP gets a lot closer to a batter’s objective than batting average does. And, Beane started picking up college players who walked a lot but didn’t have a great batting average. And it worked.

Theo Epstein, the general manager of the Boston Red Sox, followed in Beane’s footsteps (he actually worked for Beane for all of 12 hours during Beane’s one-day stint as GM of the Red Sox). And the Red Sox finally won another World Series.

So, as I’ve started tallying the stats for my son’s team, I’ve calculated both batting average and OBP, and, lo’ and behold, we’ve got a couple of kids who are in the lowest third of the team based on batting average…but move up considerably when it comes to OBP. None of this is to be shared with the kids — at this point, they’re having a good time, they’re trying hard, and they’re learning to support each other, so introducing a hierarchy of “who’s better” is wildly counter-productive.

In the end, I’ve violated my core tenet — I’m looking at metrics that are not, in the end, actionable at all! But I’m having fun, and it’s got me thinking about data in some new ways. This post was about metrics. I’ll explore data quality in the next post. Stay tuned!

Awareness (the company) has a Measuring ROI Around Web 2.0 webinar this Thursday, May 22, at 2:00 PM EDT. That’s heavy on the buzzwords, but it sounds like it might have some interesting information. And, I found out about it thanks to a mention on Twitter from Connie Bensen, who will be leaving her new kayak behind and heading to London and Paris for some R&R, so will be missing the live event herself.

Unfortunately, it partially conflicts with Kalido’s What’s Behind Your BI? webinar, which starts at 2:30 PM EDT, and it conflicts with Fusing Field Marketing and Sales, which Hoover’s and Bulldog Solutions are putting on at 2:00 PM EDT on Thursday as well.

It looks like I’ll be doing some on-demand catch-up after the fact.

Connie Bensen has a Social Media Measurement post that is worth a read. While the post is focussed on measuring social media specifically, she hits on some areas that, all too often, are overlooked when it comes to developing metrics and then reporting on them over time.

The post includes a lot of resources for measuring social media — going well beyond simply web analytics data — as well as a list of examples of things that can be measured. What really struck me, though, was the list at the end of the post of what a community manager’s monthly report should include. First, the fact that it is a monthly report is somewhat refreshing — real-time on-demand reports are way overrated, and really are not practical when it comes to providing the sort of context that Connie describes.

On to Connie’s list of report elements — the bold text is from her list, and the non-bold description is my own take on the item:

• Ongoing definition of objectives — the framework of any recurring report should be the objectives that it is attempting to measure, so I love that this is the first bullet on the list. I would qualify it just a bit — it does not seem right to be making the defining of objectives an ongoing exercise; rather, objectives should be established, reiterated on an ongoing basis (so that everyone remembers why we’re tackling this initiative in the first place), and revisited periodically (objectives can and should change).
• Web analytics — this is the “easy” data to provide on a recurring basis, it’s data that most people are getting comfortable with, and, even though there is a lot of noise in the data, it is still reasonably objective; the key here is to focus on the web analytics data that actually matters, rather than including everything.
• Interaction - Trends in members, topics, discovery of new communities – this is a somewhat community-specific component, but it’s a good one; the “discovery of new communities” actually implies an objective regarding the role of a community manager; what a great metric, though, to drive behavior within the role.
  
• Qualitative Quotes - helpful for feedback & marketing – to broaden this list to beyond reporting for social media, let’s change “Quotes” to “Data;” make the report real by providing tangible, but qualitative, examples of what is going well (or not); reporting on lead generation activity, for instance, can include selected comments that were made by attendees at a webinar — highlighting what resonated with the audience (and what did not).
• Recommendations - Based on interactions with the customers – recommendations, recommendations, recommendations! What is the point of pulling all of this information together if nothing gets done with it? I sometimes like to include recommendations at the beginning of a report — they’re a great way to engage the report consumer by making statements about a course of action right up front.
• Benchmark based on previous report – my preference is to use stated targets (where it makes sense) as the benchmark, rather than simply looking for the delta of the data over a prior reporting period. But, sometimes, that is simply not feasible. Including “here’s the measurement…and here’s the direction it is heading” is definitely a good thing. But, it’s also important to not look at a 2-month span and jump to “we have a trend!”

Having recently relaunched the Bulldog Solutions blog, I’ve got a good opportunity to put Connie’s post into practice. Oh, dear…that’s going to require re-opening the, “What are our objectives for this thing…clearly stated, please?!” Stay tuned…

Avinash Kaushik has a great post today titled The “Action Dashboard” (An Alternative to Crappy Dashboards. As usual, Avinash is spot-on with his observations about how to make data truly useful. He provides a pretty interesting 4-quadrant dashboard framework (as a transitional step to an even more powerful dashboard). I’ve gotten red in the face more times than I care to count when it comes to trying to get some of the concepts he presents across. It’s a slow process that requires quite a bit of patience. For a more complete take on my thoughts check out my post over on the Bulldog Solutions blog.

And, yes, I’m posting here and pointing to another post that I wrote on a completely different blog. We’ve recently re-launched the Bulldog Solutions blog — new platform, and, we hope, with a more focussed purpose and strategy. What I haven’t fully worked out yet is how to determine when to post here and when to post there…and when to post here AND there (like this post).

It may be that we find out that we’re not quite as ready to be as transparent as we ought to be over on the corporate blog, in which case this blog may get some posts that are more “my fringe opinion” than will fly on the corporate blog. I don’t know. We’ll see. I know I’m not the first person to face the challenge of contributing to multiple blogs (I’ve also got my wife’s and my personal blog…but that one’s pretty easy to carve off).

I realized that my Data Posts from Non-Data Blogs Yahoo! pipe wasn’t working correctly, and when I fixed it, a recent post from Debbie Weil at BlogWrite for CEOs popped up: More on the ROI of Social Media: Return on Influence. Ordinarily, I’m a big fan of Weil’s thoughts, but this one had me wondering if I ought to try to track down some blood pressure medication. Weil by no means invented the phrase (and does not claim to have), “When it comes to social media, ROI really means ‘return on influence,’” but, sadly, she has jumped right on that misguided bandwagon.

Maybe it’s that I was raised in a house where one parent was an engineer and the other was an English major. Maybe it’s because I’ve got a contrarian bent — a slight one (I like “alternative” music but not “experimental” music). For whatever reason, “ROI is return on influence” has stuck in my craw from the first time I heard it. And it still makes me twitch whenever I stumble across a post where someone waxes eloquently about the genius of the phrase.

Weil has a couple of “short answers” for why return on influence makes sense. Her first is that it makse sense “because the return is soft. The benefits of incorporating social media strategies into your marketing are real (and can no longer be ignored) but they’re not normally measured in dollars.” I have no argument with any part of that assertion after the word “because.” Weil points out that the return is soft. So, why isn’t the “return” being replaced in this platitude? “Influence from (social media) investment” I get. And that is something that you should try to measure.

Are you still with me? No one who has picked up this phrase has stopped to think that it doesn’t make sense! If you develop influence in your market, then you will get a return, which may or may not be soft. But, are you trying to measure the return on that influence, or are you trying to measure the influence that you garnered by engaging in social media?

Marketers really are freaked out by the increasing focus on Marketing ROI. That focus is driven by CEOs and CFOs. In my experience, CFOs are pretty sharp people. They get that Marketing is important. What they want is accountability, efficiency, and effectiveness from Marketing. They want to know that the chunk of the company’s budget that is being invested in Marketing is being well-used. Unfortunately, they communicate that imperative in financial terms: “What’s the ROI?” They’re Finance people, folks! What would you expect?

Marketers, rather than getting to the heart of delivering business value — driving improvements in efficiency and effectiveness, and demonstrating results — have instead gone nutso with, “I have to show ROI!” Return on Influence is a headless-chicken response to this belief. And, almost comically, it has resulted in a classic marketing response: “Let’s spin and message it! Let’s talk about how, for Marketing in the social media world, ROI really stands for ‘Return On Influence.’”

Oh, man oh man, what I would pay to sit in the room when a Fortune 1000 CMO proudly rolls out that explanation to the CFO. It completely, utterly, totally, and ridiculously misses the point.

Accountability and continuous improvement, people: the executives in your company are not stupid (if you think they are, then they either are, or they aren’t but you think they are: in either case, find a new company). Understand what you are trying to accomplish with your social media strategy. Is it to build your brand? Is it to engage with your most avid customers? Is it to position your company as being full of cutting-edge thought leaders? Articulate that. Measure whether you are making headway with your efforts.

Am I right?

The topic of “Marketing ROI” has crossed my inbox and feed reeder on several different fronts over the past few weeks. I don’t know if the subject actually has peaks and valleys, or if it’s just that my biorhythms periodically hit a point where the subject seems to bubble up in my consciousness.

The good news is that the recent material I’ve seen has had a good solid theme of, “Don’t focus too much on truly calculating ROI.” The bad news is that that message has been in response — directly or indirectly — to someone who is trying to do just that.

One really in-depth post came from — no surprise — My Hero Avinash Kaushik. He did a lengthy post, including five embedded videos, each 4-9 minutes long: Standard Metrics #5: Conversion / ROI Attribution.  What the post does is walk through a series of scenarios  where a Marketer might be trying to calculate the ROI for their search engine marketing (SEM) spend. He starts with the “ideal” scenario: a visitor does a search, clicks on a sponsored link, comes to the site, moves through and makes a purchase. In that case, calculating/attributing ROI is very simple. But, that’s just a setup for the other scenarios…which are wayyyyyy closer to reality. The challenge is that, as Marketers, it’s we all too often ignore our own typical behavior and common sense so that we can assume that most of our potential customers behave in an overly simplistic way. When was the last time you did a search, clicked on a sponsored link, and then, during that visit, made a purchase?

Unfortunately, very, very, very few Marketing executives would ever actually spend the 45 minutes it would take to truly consume all of Avinash’s post.  And, honestly, that’s not really “the solution.” The smart Marketing executive will find the Avinashes of the world and will hire them and trust them. Avinash (and John Marshall) really make the case that “time on site” is a more useful metric for assessing the effectiveness of your SEM spend — ROI just brings in too many variables and too much complexity.

In short: Don’t treat ROI as the Holy Grail and try to tie every one of your marketing tactics to “revenue generated.” For one thing, you will head down so many rat holes that you’ll start drooling whenever someone says, “cheese.” For another thing, you will find yourself facing decisions that seem right based on your ROI calculation…but that you just know are wrong.

Another place where this topic came up was in a thread titled ROI Models - High Level Thinking on the webanalytics Yahoo! group. I responded, but others chimed in as well. Some of those responses, in my mind, are still a bit too accepting of the premise that “I need to calculate a hard ROI.” But, other responses go more to a “back up and don’t look at ROI as the be-all/end-all.”

And, finally, ROI crossed my inbox last week by way of a CMO Council press release from back in January. I saw this when it came out, but a colleague forwarded it along last week, which prompted me to re-read it. The press released emphasized how much marketers are focussing on accountability when it comes to their marketing investments. One data point that jumped out was “34 percent [of marketers] said they were planning to introduce a formal ROI tracking system.” This is an alarming statistic. Marketers absolutely should be focusing on accountability – finding ways that they can measure and analyze the results of their efforts. But, if they truly are framing this as the need for “a formal ROI tracking system,” then that means 34 percent of marketers are going to be largely chasing their tails rather than driving business value.

For anyone who knows me or follows this blog, it will be no surprise that I can get a bit…er…animated when it comes to data visualization. Partly, this may be from my background in Art and Design. I got out of that world as quickly as possible, when I realized that I lacked the underlying wiring to really do visual design well.

As a professional data practitioner, I also see effective data visualization as being a way to manage the paradox of business data: the world of business is increasingly complex, yet the human brain is only able to comprehend a finite level of complexity. And, while I love to bury myself up to my elbows in complex systems and processes, I’m the first person to admit that my eyes glaze over when I’m presented with a detailed balance sheet (sorry, Andy). A picture is worth a thousand words. A chart is worth a thousand data points. That’s how we interpret data most effectively — by aggregating and summarizing it in a picture.

So, it’s pretty important that the picture be “drawn” effectively. I had a boss for a year or two who flat-out was much closer to Stephen Hawking-ish than he was to Homer Simpson when it came to raw brainpower. He took over the management of a 50-person group, and promptly called the whole group together and presented slide after slide of data that “clearly showed”…something or other. The presentation has become semi-legendary for those of us who witnessed it. The fellow was facing a room of blank-confused-bored-bewildered gazes by the time he hit his third slide. Now, to his credit, he learned from the experience. He still looks at fairly raw data…but he’s careful as to how and where he shares it.

All that is a lengthy preamble to a Presentation Zen post I read this evening about Depth vs. Breadth of presentations. It’s a simple concept (meaning I can understand it), with some pretty good, rich examples to back it up. The fundamental point is that none of us spend very much time thinking about what to cut from our presentations. I would extend that to say we don’t spend very much time thinking about what data not to share or show. It’s easy to see this as a case for “make the data support what you want it to,” which it is not. At all! Really, it’s more about focussing on showing the data — and only the data — that directly relates to the objectives you are measuring or the hypotheses that you are testing.

Then, focus on presenting that data in a way that makes it clear as to what story it is telling. You do the hard work of interpreting the data. Then, highlight what is coming out of that intepretation. If there is ambiguity, highlight that, too. If there is a clear story, and your audience gets it, and you then introduce an anomaly, you’re much more likely to have a fruitful, engaging discussion about it. You will learn, and your audience will retain!

In the end, this is a riff on a bit of a tangent, I realize. Robert Frank presents some fairly alarming evidence of college professors aiming for broad and deep…and not gaining any better retention than the slide-happy, chart-crazy PowerPoint users provide in the business setting. He goes on to talk about how, in his teaching, he makes a point, repeats it, comes at it from a different angle, makes the students think about it, and then repeats it again. He goes for deep. His students, I’m sure, leave his introductory economics class with a thoroughly embedded (and accurate) understanding of “opportunity cost” (having seen the term mis-applied more than once in my day…and still having to struggle to get to the correct answer…and barely…and barely in time…in his presentation…I applaud that!).

I’m not arguing for simplicity for simplicity’s sake. I’m arguing for going deep, understanding the complexity, and then distilling it down to a narrative, cleanly presented, that leaves your audience with takeaways that are accurate and absorbed.

And…on that note, have any of you read The Economic Naturalist? It sounds like it would be right up my alley. It’s just a bonus that, if I ever actually attended something that could be labeled a “cocktail party,” I could talk about how I’d “read some of Bernanke’s work!”

I’ve been involved with United Way for the past seven or eight years in Austin and, now, in Columbus. One of the attractions to spending my volunteer energy with United Way is that they are very accountability-focussed. That means that, in their agency funding cycle, they require agencies that are requesting funding to specify measures and targets for the specific programs they describe in their funding requests.

For the last few months, I’ve been getting involved with the United Way of Central Ohio (side note: if you’ve thought about doing volunteer work and just can’t figure out how to get started, it’s insanely easy; one phone call to any nonprofit organization that piques your interest, and you WILL have the opportunity to get involved). I’m on a couple of standing committees that are focussed on emergency food, shelter, and financial assistance. And, I’m on an ad hoc committee focused on developing performance measures for that overall “impact area.”

One common distinction I learned when working on agency funding committees with two different United Ways is the distinction between an “outcome” and an “output.” An output is something like “provided 1,000 families in a housing crisis with one-time emergency financial assistance.” An outcome is more like “reduced the number of families who became homeless due to a financial crisis by 15% over the previous reporting period.” Does the distinction make sense? The output is what the nonprofit agency did, whereas the outcome is why they did it — what result they were really trying to achieve at the end of the day.

In the business world — specifically, in marketing — examples of outputs would be “deployed 20 new pages,” “conducted 3 webinars,” “published 2 white papers.” And, really, some highly tactical measures such as “achieved an open rate of 54%,” “achieved a clickthrough rate of 12%,” and even “drove 450 registrations” are all much more outputs than outcomes.

The marketing outcome that is wildly in vogue right now is ROI — how much revenue did all of this marketing activity drive? In this sense, Marketing in the for profit world is paralleling the nonprofit world (it’s becoming a cliche in the nonproft arena that nonprofits need to be “run more like for profit businesses”) — both are starting to accept as gospel that measuring outputs is bad, and the only measures that matter are outcome-based.

This, I fear, is another case of a perfectly valid concept being oversimplified to the point that it is presented as an absolute rule. And it really shouldn’t be. Here’s the problem with throwing out all output measures: the larger the organization and the more complex the business, the more factors there are that influence the ultimate outcome!

Take the case of a brilliantly executed Marketing campaign — just accept that it was perfect in all possible ways. BUT, during that same measurement period, the Sales organization was in total upheaval: senior leadership turnover, processes in flux, and a grossly understaffed inside sales organization. Marketing — in an effort to be outcome-based — assesses their efforts solely based on the conversion to revenue of the leads they generated and nurtured. The results were abysmal. The CMO loses his job. The CEO steps in temporarily and demands that, whatever Marketing did for the last six months…they need to do the opposite…

This example is only slightly dramatized. The same potential folly exists for nonprofits. If an agency is focussed on addressing short-term food and shelter crises, their outputs may actually be the best thing for them to measure — are they managing their resources to meet the demands for assistance that they get every day of the year? If they start focussing on longer-term, root causes of the crises, in order to get to the true outcome of food/housing crisis prevention and food/housing stability, then there will be a gap in short-term services. Better, in my book, to allow (and encourage) a focus on outputs when it makes sense. Still with a bias to outcomes, but not to the black-and-white exclusion of outputs.
�
I like the “outputs vs. outcomes” distinction. It’s a distinction that Marketers could benefit from making. I don’t like blanket beliefs that one is good and one is bad, or one is right and one is wrong. The world, folks, is just too complicated for that.