Gilligan on Data by Tim Wilson

I was working with a client last week who was looking to update their lead scoring. This wasn’t any fancy-schmancy multidimensional lead scoring — it was plain ol’ pick-a-few-fields-and-assign-’em-some-values lead scoring. Which is a great place to start.

In this case, the company was in the process of streamlining their registration form on one area of their web site. This was an experiment to see if we could improve their registration form conversion rates by reducing the number and complexity of the fields they required visitors to fill out. We took a good hard look at the fields and asked two things: 1) Do we really need to know this information up front? 2) Is the information “easy” to provide (an “Industry” list with 25 fields was deemed “hard,” because the visitor had to scan through the whole list and then make a judgment call as to which industry most fit his situation).

The result was that we combined a couple of fields, removed a couple of fields, and reworded one question and the possible answers. So far, so good. The kicker was that these changes, while still giving us all of the same underlying information that the company was using to assess the quality of their leads, required changing the lead scoring formula. The formula was going from having three variables to two, because two of the scored variables had been merged into a single, much shorter, much clearer field.

We interrupt this blog entry to provide an aside on cognitive dissonance

The company’s existing three-variable lead score was fairly problematic. When qualitatively assessing a batch of leads, the Sales organization could always pick out a number of high-scoring leads whom they were not interested in calling, and they could pick out a number of low-scoring leads who they absolutely wanted to reach out to. “Our lead score is pretty awful,” was the general consensus.

At the same time, the lead score was used at an aggregate level — by the same people — to assess the results from various lead generating activities. “We had 35 leads that scored over 1,000! This event was great!”

We’ll go with the wiktionary defintion of cognitive dissonance: “a conflict or anxiety resulting from inconsistencies between one’s beliefs and one’s actions or other beliefs.” In this case, a strongly held belief that the lead scoring was fatally flawed, and an equally strongly held belief that the lead score was a great way to assess the results of lead gen efforts.

Initially, we (I) actually let the latter belief prevail, and I struggled to come up with a new lead scoring formula and value weighting that would provide as similar as possible an assessment of each lead between the old scoring system and the new.

And I kept hitting dead ends.

In then occurred to me that, by going through the exercise to streamline the fields, we had actually gained some valuable insight into what the Sales organization did/did not see as important qualification criteria for the leads that were sent to them.

So, I started over.

The two scored fields that we were planning to continue to capture were “Job Role” and “Annual Revenue.” Job role was a hybrid of job title and department — a short list that really honed in on the types of people who were most likely to be influencers or decision-makers when it came to the company’s services. We’d discovered, while getting to those fields on the registration form, that if a company had greater than $25 million in revenue in any year, the Sales organization wanted to talk to them regardless of their role in the company. Likewise, there were a handful of job roles that, regardless of the (reported) annual revenue, Sales wanted to talk to them as well. So, we started by making sure that those “trump” values would put the lead over the qualification threshhold regardless of the other field’s value. We then worked backwards from there to the mid-tier fields — fields that, if the other field was promising, then Sales would want to talk to the lead. And so on from there. This was much more an exercise in logic than an exercise in analysis. But, it made more sense than the lead score it was replacing.

As a check, we compared a sample of leads using both the old and new scoring methods. We highlighted a random set of leads that would have moved from below the qualification threshhold in the old scoring system to above it in the new, and vice versa. The majority of these shifts made sense. And, overall, we were looking like we would be qualifying a slightly higher percentage of leads under the new scoring system. We patted ourselves on the back, summarized the changes, the logic, and the before-vs-after results…and headed down to Sales to make sure they were looped in and could identify any gaping holes in our logic.

Instead…they honed in on two things:

• The slight increase in leads that would be qualified using the new system
• One lead who had a very low level job title…at a >$1 billion company — she was not qualified under the old system but became qualified under the new

Things then got a bit ugly, which was unfortunate. Cognitive dissonance again. The old system let plenty of not-good leads through to Sales and kept just as many good leads out. And it was not really fixable by simply tweaking the formula. It was broken.

The new system took input directly from the Sales organization and, using the two attributes they cared about the most, applied a logical approach. But, lead scoring is not perfect. The only way to have a “perfect” lead score is to ask your leads 50 questions, check the veracity of all of their answers, and build up a very complex system for taking all of those variables into account. In a way, multidimensional lead scoring is a step in that direction…without putting an undue burden on the lead to answer so many questions, and without requiring a PhD and a Cray supercomputer to develop the right formula.

But, lead scoring is really simply intended to identify the “best” leads, to disqualify the clearly bad leads, and to leave a pretty big gray area where the quality of the lead simply isn’t known. It’s then up to the individual situation to determine where in that gray area to put the qualification threshold. The higher the threshhold, the fewer false positives and more false negatives there will be. The lower the threshold, the fewer false negatives, but the more false positives.

“Analysis paralysis” is a cliché, but it’s a well-warranted one. Looking for perfection when you shouldn’t expect it to exist can be crippling.

Avinash Kaushik has a great post today titled The “Action Dashboard” (An Alternative to Crappy Dashboards. As usual, Avinash is spot-on with his observations about how to make data truly useful. He provides a pretty interesting 4-quadrant dashboard framework (as a transitional step to an even more powerful dashboard). I’ve gotten red in the face more times than I care to count when it comes to trying to get some of the concepts he presents across. It’s a slow process that requires quite a bit of patience. For a more complete take on my thoughts check out my post over on the Bulldog Solutions blog.

And, yes, I’m posting here and pointing to another post that I wrote on a completely different blog. We’ve recently re-launched the Bulldog Solutions blog — new platform, and, we hope, with a more focussed purpose and strategy. What I haven’t fully worked out yet is how to determine when to post here and when to post there…and when to post here AND there (like this post).

It may be that we find out that we’re not quite as ready to be as transparent as we ought to be over on the corporate blog, in which case this blog may get some posts that are more “my fringe opinion” than will fly on the corporate blog. I don’t know. We’ll see. I know I’m not the first person to face the challenge of contributing to multiple blogs (I’ve also got my wife’s and my personal blog…but that one’s pretty easy to carve off).

Stéphane Hamel, it seems, is a bit peeved with Eric Peterson. These are two pretty big names in web analytics — Eric as one of the fathers of web analytics, and Stéphane as both a thought leader in the space as well as the creator of one of the most practical, useful web analytics supplemental tools out there — WASP: The Web Analytics Solution Profiler plugin for Firefox. With the plugin, you visit any site, and a sidebar will tell you what web analytics solutions it looks like it’s running. It’s pretty cool.

I don’t know the full background of the current back-and-forth between these two guys, but I’m a huge fan of Stéphane, and my ears perked up when I read this observation in the post:

Business Process Analysis implies understanding & improving a collection of interrelated tasks which solve a particular issue. Nothing new here… Most businesses face complex and “hard” processes, and the way to make them “easy” is by decomposing them into smaller sub-processes until they are manageable.

For one thing, for a period of ~8 months, my job title was “Director of Business Process Analytics.” And, frankly, I was never sure what that meant. In hindsight, if I’d had these two sentences from Stéphane and if I’d replaced “Analytics” with “Analysis,” I would have seen a much clearer mapping from my label to what I was actually doing in the role.

More important, though, is the concept of “decomposition.” I find myself preaching the Decomposition Doctrine regularly. And I believe in it strongly.

As an example, when it comes to the Holy Grail of Marketing Analysis — calculating the ROI of your marketing spend — many, many B2B marketers start out looking for the correlation between leads generated and revenue. I have yet to see a case in B2B where this can be found with a sufficiently tight, sustained correlation to be meaningful. That actually makes sense. It’s like looking for a correlation between the state someone is born in and the achievement of a PhD. There’s a lot going on over time between Point A and Point Z.

In the case of B2B marketing, decomposition makes sense. Decompose the process:

• The lead-to-qualified lead sub-process
• The qualified lead to sales accepted lead sub-process
• The sales accepted lead to sales qualified lead sub-process
• The sales qualified lead to close sub-process

Each of these sub-processes have people who proceed to the next sub-process as well as people who do not — put simplistically: people who “fall out of the funnel.” But, you can further decompose — of the people who fell out, where did they fall out and why? And does that mean they are gone forever, or are there processes/subprocesses that can be used to reengage them in the future?

The key here is that, from a theoretical perspective, if you link together all of the simpler sub-processes, then you’ve got an accurate representation of the more complex master process. The problem is that this is mostly true. There are probably other sub-processes that are unknown — those pesky “corner cases” that the real world insists on throwing at us. And, each sub-process likely experiences various anomalies over time. Add those together, and you’ve got a complex process that verges on the unanalyzable.

On the other hand, if you focus on a sub-process, you can analyze what is going on, including accounting for the anomalies. “But, isn’t there a risk that you’ll be missing the forest for the trees?” you ask. Absolutely. That’s why it’s important to start with a high-level view of the whole process, with a clear picture of the components that go into it. If you simply pick a “simple sub-process” to focus on, without understanding how and where that fits into the big picture, you run the risk of rearranging deck chairs on the Titanic. On the other hand, if you simply try to “analyze the Titanic,” without some level of decomposition, you’re equally doomed.

At the top list of blogs I follow is Jeremiah Owyang’s Web Strategist blog. He posts frequently, with depth, and with insight. However, I was in the midst of a hectic week in Austin when he posted his Analysis of the Zuckerberg Lacy Interview, and, frankly, while the title persisted in a couple of places (not only in his feed, but in my Yahoo! Pipe feed on data from non-data blogs because of the word “analysis” in the title), I’d been pretty much Zuckerberg-Lacy’d out, and I thought this was going to be a “my take on what happened” type of “analysis.” I was wrong.

The one-sentence recap of the actual incident: Sarah Lacy, of Business Week, interviewed Mark Zuckerberg, founder and CEO of Facebook, at SXSW Interactive in Austin, and the crowd, which was none too happy with the way the interview was going, pretty much took over using Twitter as a backchannel of communication. Google it and you can read scads more as well as see video of the event.

Well, I finally got around to looking at Jeremiah’s post, and I’ll be damned if it wasn’t a brief post linking to a really interesting TechnoSocial post: Anatomy of a Mob: The Lacy/Zuckerberg Interview. Kee Hinckley sifted through a bunch of Twitter data to try to get some insight into what really went on through Twitter during the keynote. It’s a fascinating read.

What I want to point out, though, is not so much the results of the analysis, but some pretty darn noteworthy aspects of what went into it.

First, I immediately started wondering how on earth Hinckley figured out which Twitter users were at the keynote. As it turns out, he explains it — recognizing that it’s imperfect, but, by golly, still pretty clever! And, it took a mix of tools, some level of clunky automation (no one likes to do screen scraping), and quite a bit of flat-out manual effort. He revised what he included/excluded as he got into the manual part of the exercise. What’s Noteworthy: the data Hinckley wanted was not easily accessible (the data always requires more prep work than most people realize), and it required some judgment when it came to getting it. That’s stepping out of a formulaic approach to analysis of “pull the data that’s available and present it.”

Second, the visualization. I am almost always opposed to 3D representations of data. Categorically when it’s two dimensions of data presented with “depth” — that’s just silly. But, even when it’s three dimensions presented in three dimensions, more often than not, the result is uninterpretable. Not the case here! Hincklely steps outside of the box to think about ways to effectively visualize the data — much, much more thought than simply “How do I get all of the data displayed?” He even includes two different charts, one with bubbles and one with a color spectrum, of the same data — clearly grappling with how best to show the information clearly (both work, IMHO). What’s Noteworthy: All too often, I see analysts go through all of the hurdles of prepping the data and “running the numbers” only to take shortcuts when it comes to the visual representation of the results. That’s the equivalent of running 25.5 miles of a marathon really hard…and then going home.

Finally, Hinckley puts a lot of text-based interpretation behind his analysis. In this case, he clearly had the question, ran with trying to find the answer, and took responsibility for explaining the whole process and the results. And, he did all three swimmingly! What’s Noteworthy: In many situations, one person is asking the question, while an analyst tries to find the answer. It’s that third area — explaining the process and results — where many analysts decide not to tread. Rather, they “do the analysis” and turn the “results” (mostly the data, including charts) over to the original requestor to interpret and explain. This bothers me. I much prefer to see an analyst actually draw conclusions and provide real context and interpretation. Whether they are expected to or not! It’s up to the original requestor to decide whether to use that information and how. More often than not, it gets used. To good results.

Overall, it’s a fascinating read. Top. Notch. Work!

I’ll admit right up front that this is the least value-add post on this blog to date. Part of me sincerely hopes that it holds that distinction indefinitely. But, I know me better than that, so no promises.

We all have them. Those moments where someone says something — in person, in an e-mail, in an instant message — that triggers a completely random, but oddly inspired, response.

What happened: One of my pet peeves is the cliche, “If you can’t measure it, don’t do it.” It sounds good, but I challenge any company to fully apply this overly simplistic maxim and survive. I’m all for having a bias towards measurement, but I get nervous when people speak in absolutes like this.

Earlier this week, I fired off an internal e-mail proposing an initiative that was extremely low cost that seemed like a good idea to me. It really wasn’t an initiative where it made sense to try to quantify the benefits, though. I made a comment as such in the e-mail — that, despite it not being practical to measure the results, I still thought it was a good idea. (I was having one of the 15-20 snarky moments I have throughout any given day.) Two of the five people on the distribution list immediately responded with demands for an ROI estimate.

FLASH!

10 minutes later, and I’d fashioned the following chart in Excel and responded to the group with my analysis:

Everyone had a good chuckle. 

Here’s the spreadsheet file itself. It’s as clean as clean can be, so feel free to snag it and put it to your own use. If you put it to use with entertaining results, I’d appreciate a quick comment with the tale. Or, if you make modifications to enhance the end result, I’d love to get a copy.

Enjoy.

I had an experience over the past week that, in hindsight, I really should have been able to avoid. The situation was basically this: several different people had made comments in passing about how we were probably “overcommunicating” to our database. “Overcommunication” being the tactful way to say “spamming.” In this case, I can actually trace the perception back to at least two different highly anecdotal events, which then spawned comments that led to assumptions, and so on.

Now, I am all for diligent database management, especially when it comes to how often and with what content we communicate with our contacts. My general sense was that we could be doing better, but we were far from reaching a crisis point (I lived through a situation at another company where we did reach that crisis point, and there were plenty of telltale signs leading up to that). “I can pull some quick data on that to at least get some basic facts circulated,” I innocently thought. And, that’s what I did.

I knew going in that, while the data was one thing, the definition of “good” vs. “bad” was likely all over the map, so I wasn’t likely to change many people’s opinions as to the situation by simply sharing the data. So, I shot an e-mail out to a group of interested parties and told them I had the data, and I’d be happy to share it, if they shared with me their opinions as to what an acceptable maximum of communications per week and per month would be.

As I suspected, I got a wide range of responses, and most of the responses had some form of qualifier — well-founded qualifiers regarding the type of communication, actually. So far, so good.

I then shared the data that I had spent 15 minutes compiling in a way to make for easy analysis, still knowing that there was no clear good/bad definition, and there was no clear hypothesis being tested or action being planned that this analysis would influence. The data did show a few things unequivocably — really just highlighting that the concerns were somewhat well-founded and that discussions should continue amongst the people who already tacitly owned the situation. But, it also spawned requests for additional data that was more curiousity-driven than actionability-driven. Several people asked that the data be pulled with their particular qualifiers addressed. Most of these people were in no position to actually take any action based on the results. And, unfortunately, as reporting and analysis systems can sometimes be — applying the qualifiers would have turned the analysis into a highly manual, multiple man-hours exercise, whereas the high-level, basic pull was a 15-minute task.

On the one hand, I could ding our data storage system. By golly, Tenet No. 1 of good BI/DW design is to design for flexibility, right? In this case, the system limitations are actually a boon — they give me an out for simply saying, “No,” rather than the much more involved discussion that begins, “Why?”

It’s a punt, I realize. And not an out I would take if it was throwing anyone in IT under a bus.

My point is that “interesting” can be a Siren Song that dwarfs the pragmatism of “actionability.”

For anyone who knows me or follows this blog, it will be no surprise that I can get a bit…er…animated when it comes to data visualization. Partly, this may be from my background in Art and Design. I got out of that world as quickly as possible, when I realized that I lacked the underlying wiring to really do visual design well.

As a professional data practitioner, I also see effective data visualization as being a way to manage the paradox of business data: the world of business is increasingly complex, yet the human brain is only able to comprehend a finite level of complexity. And, while I love to bury myself up to my elbows in complex systems and processes, I’m the first person to admit that my eyes glaze over when I’m presented with a detailed balance sheet (sorry, Andy). A picture is worth a thousand words. A chart is worth a thousand data points. That’s how we interpret data most effectively — by aggregating and summarizing it in a picture.

So, it’s pretty important that the picture be “drawn” effectively. I had a boss for a year or two who flat-out was much closer to Stephen Hawking-ish than he was to Homer Simpson when it came to raw brainpower. He took over the management of a 50-person group, and promptly called the whole group together and presented slide after slide of data that “clearly showed”…something or other. The presentation has become semi-legendary for those of us who witnessed it. The fellow was facing a room of blank-confused-bored-bewildered gazes by the time he hit his third slide. Now, to his credit, he learned from the experience. He still looks at fairly raw data…but he’s careful as to how and where he shares it.

All that is a lengthy preamble to a Presentation Zen post I read this evening about Depth vs. Breadth of presentations. It’s a simple concept (meaning I can understand it), with some pretty good, rich examples to back it up. The fundamental point is that none of us spend very much time thinking about what to cut from our presentations. I would extend that to say we don’t spend very much time thinking about what data not to share or show. It’s easy to see this as a case for “make the data support what you want it to,” which it is not. At all! Really, it’s more about focussing on showing the data — and only the data — that directly relates to the objectives you are measuring or the hypotheses that you are testing.

Then, focus on presenting that data in a way that makes it clear as to what story it is telling. You do the hard work of interpreting the data. Then, highlight what is coming out of that intepretation. If there is ambiguity, highlight that, too. If there is a clear story, and your audience gets it, and you then introduce an anomaly, you’re much more likely to have a fruitful, engaging discussion about it. You will learn, and your audience will retain!

In the end, this is a riff on a bit of a tangent, I realize. Robert Frank presents some fairly alarming evidence of college professors aiming for broad and deep…and not gaining any better retention than the slide-happy, chart-crazy PowerPoint users provide in the business setting. He goes on to talk about how, in his teaching, he makes a point, repeats it, comes at it from a different angle, makes the students think about it, and then repeats it again. He goes for deep. His students, I’m sure, leave his introductory economics class with a thoroughly embedded (and accurate) understanding of “opportunity cost” (having seen the term mis-applied more than once in my day…and still having to struggle to get to the correct answer…and barely…and barely in time…in his presentation…I applaud that!).

I’m not arguing for simplicity for simplicity’s sake. I’m arguing for going deep, understanding the complexity, and then distilling it down to a narrative, cleanly presented, that leaves your audience with takeaways that are accurate and absorbed.

And…on that note, have any of you read The Economic Naturalist? It sounds like it would be right up my alley. It’s just a bonus that, if I ever actually attended something that could be labeled a “cocktail party,” I could talk about how I’d “read some of Bernanke’s work!”

Avinash Kaushik just published a post titled History Is Overrated (Atleast For Us, Atleast ForNow). The point of that post is that, in the world of web analytics, it can be tempting to try to keep years of historical data…usually “for trending purposes.” Unfortunately, this can get costly, as even a moderately trafficked site can generate a lot of web traffic data. And, even with a cost-per-MB for storage of a fraction of a penny, the infrastructure to retain this data in an accessible format can get expensive. Avinash makes a number of good points as to why this really isn’t necessary. I’m not going to reiterate those here.

The post sparked a related thought in my head, which is the title of this post: bounce rate is not revenue. Obviously, bounce rate (the % of traffic to your site that exits the site before viewing a second page) is not revenue. And, bounce rate doesn’t necessarily correlate to revenue. It might correlate in a parallel universe where there is a natural law that no dependent variable can have more than 2 independent variables. But, here on planet Earth, there are simply too many moving parts between the bounce rate and revenue for this to actually happen.

But.

That’s not really my point.

What jumped out at me from Avinash’s post, as well as some of the follow-up comments, was that, at the end of the day, most companies measure their success on some form of revenue and profitability. Realizing that there is incredible complexity in calculating both of these when it comes to GAAP and financial accounting, what these two measures are trying to get at, and what they mean, are fairly clear intuitively. And, it’s safe to say that these are going to be key measures for most companies 10, 20, or 50 years from now, just as they were key measures for most companies 50 years ago.

Sales organizations are typically driven by revenue — broken down as sales quotas and results. Manufacturing departments are more focussed on profitability-related measures: COGS, inventory turns, first pass yields, etc.  Over the past 5-10 years, there has been a push to take measurement / data-driven decision-making into Marketing. And, understandably, Marketing departments have balked. Partly, this is a fear of “accountability” (although Marketing ROI is not the same as accountability, it certainly gets treated that way) Partly, this is a fear of figuring out something that can be very, very, very difficult.

But, many companies are giving this a go. Cost Per Lead (CPL) is a typical “profitability” measure. Lead Conversion is a typical “revenue” measure. That is all well and good, but the internet is adding complexity at a rapid pace. Pockets of the organization are embracing and driving success with new web technologies, as well as new ways to analyze and improve content and processes through web analytics. No one was talking about “bounce rate” 5 years ago and, I’d be shocked if anyone is talking about bounce rate 5 years from now.

Social media, new media, Web 2.0 — call it what you like. It’s changing. It’s changing fast. Marketing departments are scrambling to keep up. In the end, customers are going to win…and Marketing is going to be a lot more fun. But we’ve got a lonnnnnnnnng period of rapidly changing definitions of “the right metrics to look at” for Marketing.

While it is easy to get into a mode of too constantly reevaluating what your Marketing KPIs are, it is equally foolish to think that this is a one-time exercise that will not need to revisited for several years.

Oh, what exciting times we live in!

Kim Haynes, the HR Manager at Bulldog Solutions, forwarded a pretty interesting post by Lorri Randle about Web 2.0 usage last week. It’s an intriguing post. For one thing, she quotes some interesting statistics: 1.2 million Internet users in the world today, 500,000 users of Twitter as of October 2007, almost 10 million users of Second Life (although that probably includes a lot of people like me who set up an account, got in once, and haven’t ever been back), and so on.

The point that Randle is making, though, is that, in many ways, these numbers are misleading. They’re large numbers in absolute terms, but what does an absolute number tell you? While there are 1.2 million Internet users, there are over 6.5 million people. So…less than 20% of people on Planet Earth are Internet users. Randle goes on to look at the numbers for podcasting, blogging (both writing and reading), and other social media. Her point: for those of us who participate in these sorts of things, we sometimes get caught up in the growth of adoption, as well as the raw numbers. We tend to assume near-ubiquity, or at least impending near-ubiquity, when that is not accurate.

I like the post — not just because of the interesting statistics, but because of the point that all numbers are relative, and context is important!

It’s amazing what an airplane flight can do when it comes to finishing up books that have been lingering. Especially when you pack the “fun” book in your checked baggage so you can’t cheat. Sucks when you arrive at the hotel to find that your “fun” book is, apparently, still sitting in the bedroom at home! Argh!

Part 2 will be a bit less harsh, as Ayres does eventually touch on some of my bigger issues. Sort of. I’m still not satisfied with his treatment, though.

Ayres does, at least once, throw in a pretty critical caveat: “As long as you have a large enough dataset, almost any decision can be crunched.” That’s a HUGE caveat, even in our increasingly data-capturing world. As a matter of fact, I like to talk about the data explosion. But, when I discuss it, it’s as a warning that you can’t just see “getting the raw data” as the biggest challenge in making data-driven decisions. My claim is that the bigger challenge is developing discipline about how you approach that data. While Ayres does periodically speak to the fact that there is real skill and creativity required to develop the hypotheses that you want to test, he does not put nearly enough emphasis on this point. And, not only that there needs to be diligence in developing the hypothesis, but also considerable rigor in determining how that hypothesis will be tested.

Ayres walks through a pretty fascinating example of super crunching gone awry in the case of John Lott, who did some super crunching that demonstrated that concealed handgun laws brought down the crime rate. According to Ayres, Lott made an error in the data prep for his analysis that, when corrected, did not show this at all. Ayres uses the example more to preach that even data-oriented people can still get caught up emotionally and refuse to face hard facts. While that is undoubtedly true, Ayres also misses the opportunity to speak in any real depth as to the amount of data prep work that needs to be done to normalize and cleanse data before actually running a regression. He does mention this…but it is very brief.

More good stuff: Ayres devotes a good chunk of a chapter to explaining (and illustrating) just how bad humans are at gauging the quality of their own intuition. Many of the points he makes here echo Daniel Gilbert’s points in Stumbling on Happiness. Points very well taken. But — and I’m sure Ayres would say that my next statements prove his point and that I’m just one more person in denial — things get stretched pretty far at times here. For instance, Ayres claims that the safety procedures that flight attendants follow as a hard script almost all of the time make them more effective than a less structured approach. Seeing as I was sitting on a flight, where I had easily tuned out the flight attendant’s script, I had a “Gimme a break!” moment. Everyone who travels at all has occasionally stumbled across a Southwest (and it even happened to me on United once) flight where the flight attendant has a little more fun. Everyone listens! And, they are clearly sticking to the content of the script, if not the specific language.

Oops. Language. I’ll nitpick just a little bit. Ayres uses “digitalize” a lot. Maybe he can’t help it — he’s a lawyer AND an academic, so why go with the shorter, common synomym — digitize — when a longer word will suffice. He also defines CDI as “consumer data integration” in the context of Acxiom Corporation’s services. While Google indicates this is one possible explanation of the acronym, even Acxiom seems to use the breakdown that I’m much more familiar with: customer data integration. Splitting hairs a little bit, I realize. But, it’s an indication that Ayres really isn’t all that familiar with CDI, which is a descriptor of a host of technologies that are trying to actually solve all of the complexity of normalizing data from disparate data source that Ayres barely acknowledges. Ayres also uses “commodification” a lot, and I was prepared to zing him there, too…but Google shows this as a more common word than it’s synomym, commodization. So, I learned something there!

Two more specific beef examples and I’ll wrap up.

Ayres devotes a good chunk of writing to various plans that use super crunching to predict the likelihood of recidivism for inmates who are being paroled. The statement that really got me was that, according to the data, an inmate who received a score of four or higher when his history was plugged into a certain model would have a 55% chance of committing another sex offense within 10 years. Ayres then jumps into example of one such inmate who received a score of four, was paroled anyway, and promptly disappeared. The 55% is what gets me, though. Ayres ignores that this is far from an overwhelming indication. It may very well be my liberal bias that I don’t think that it’s a slam dunk to keep everyone locked up knowing that 45% of these people would NOT commit another sex offense within ten years. Ayres completely glosses over this question, which seems to be an ethical one worth addressing. I actually made a note — Minority Report — when I read this. I was thinking of the Philip K. Dick-inspired Steven Spielberg movie that starred Tom Cruise. In the movie, citizens are arrested and based on “foreknowledge” — a vision by one of three “pre-cogs” (it’s a very small leap to turn these specially-endowed humans into a super crunching computer) that the person is going to commit a murder in the future. The movie is chilling in a 1984 kind of way. And, ultimately, condemns persecuting someone for something they have not yet done. Ayres sounds just a little too much like the films main antagonist — Director Lamar Burgess (played by Max von Sydow) — for my comfort. Interestingly, later in the book, Ayres refers to the same film…but not at all for the same reasons. Rather, he references the personalized ads that Tom Cruise is bombarded with at all times while walking down a street in the movie (which really isn’t that far of a stretch from today, given the proliferation of RFID technology).

Second specific beef has to do with NASA history. Ayres points to Gus Grissom as one of the astronauts who balked at the idea that the Mercury splashdown capsules would be designed so that that would not be able to be opened from the inside. Ayres then points out that, because the astronauts wan out, Gus Grissom was able to panic upon splashdown and blow the hatch open prematurely. Which he did. And almost drowned. First off, Ayres gives no acknowledgement to the fact that there are a lot of reasons to believe (as NASA does) that the hatch blew open due to an equipment malfunction — not because Grissom did anything inappropriate. Second, Gus Grissom was killed in a fire inside a training capsule for the Apollo 1 mission. I honestly don’t know enough of my history here to know if the training capsule could be opened from the inside, or, if not, if it would have made a difference. My guess is that the fire was so fast that it wouldn’t have mattered. But, on both accounts, Grissom seems like a spectacularly lousy example of Ayres’s point.

So, I’ll wrap up. The book has left a dirty taste in my mouth. Ayres delivered a book that he hopes will sell with the gee-whiz factor. My guess is that Bantam Books smelled “Jump on the Freakonomics Bandwagon” money, and, as bandwagon books, movies, and TV shows are prone to do, it underdelivers. The book dramatically downplays the challenges involved in super crunching — not just challenges in the “it’s hard” sense, but challenges in the “large, clean, relevant data sets” are not the same as “large amounts of raw data.” He actually includes examples of super crunching where, when the results came out, they were applied on a limited scale because the basic approach to the analyses were called into question. So, how about a chapter on “design of experiments?” He does have good points, and his assertion that companies could benefit from having a designated “devil’s advocate” to question — hard — any analysis is a great idea. There is definitely a shock of wheat here and there in the book. Unfortunately, Ayres spends most of his writing about the chaff.