Welcome to Gilligan on Data, where you will find thoughts, musings, and, hopefully, not too many redundancies on the world of business data. If you missed the irony in the previous sentence, you may struggle with my writing style.

Four Dimensions of Value from Measurement and Analytics

By on in , , , with No Comments.

When I describe to someone how and where analytics delivers value, I break it down into four different areas. They’re each distinct, but they are also interrelated. A Venn diagram isn’t the perfect representation, but it’s as close as I can get: Earlier this year, I wrote about the three-legged stool of effective analytics: Plan, Measure, Analyze. The value areas covered in this post can be linked to that process, but this post is about the why, while that post was about the how.

Alignment

Properly conducted measurement adds value long before a single data point is captured. The process of identifying KPIs and targets is a fantastic tool for identifying when the appearance of alignment among the stakeholders hides an actual misalignment beneath the surface. “We are all in agreement that we should be investing in social media,” may be a true statement, but it lacks the specificity and clarity to ensure that the “all” who are in agreement are truly on the same page as to the goals and objectives for that investment. Collaboratively establishing KPIs and targets may require some uncomfortable and difficult discussions, but it’s a worthwhile exercise, because it forces the stakeholders to articulate and agree on quantifiable measures of success. For any of our client engagements, we spend time up front really nailing down what success looks like from a hard data perspective for this very reason. As a team begins to execute an initiative, being able to hold up a concise set of measures and targets helps everyone, regardless of their role, focus their efforts. And, of course, Alignment is a foundation for Performance Measurement.

Performance Measurement

The value of performance measurement is twofold:

  • During the execution of an initiative, it clearly identifies whether the initiative is delivering the intended results or not. It separates the metrics that matter from the metrics that do not (or the metrics that may be needed for deeper analysis, but which are not direct measures of performance). It signifies both when changes must be made to fix a problem, and it complements Optimization efforts by being the judge as to whether a change is delivering improved results.
  • Performance Measurement also quantifies the results and the degree to which an initiative added value to the business. It is a key tool in driving Internal Learning by answering the questions: “Did this work? Should we do something like this again? How well were we able to project the final results before we started the work?”

Performance Measurement is a foundational component of a solid analytics process, but it’s Optimization and Learning that really start to deliver incremental business value.

Optimization

Optimization is all about continuous improvement (when things are going well) and addressing identified issues (when KPIs are not hitting their targets). Obviously, it is linked to Performance Measurement, as described above, but it’s an analytics value area unto itself. Optimization includes A/B and multivariate testing, certainly, but it also includes straight-up analysis of historical data. In the case of social media, where A/B testing is often not possible and historical data may not be sufficiently available, optimization can be driven by focused experimentation. This is a broad area indeed! But, while reporting squirrels can operate with at least some success when it comes to Performance Measurement, they will fail miserably when it comes to delivering Optimization value, as this is an area that requires curiousity, creativity, and rigor rather than rote report repetition. Optimization is a “during the on-going execution of the initiative” value area, which is quite different (but, again, related) to Internal Learning.

Learning

While Optimization is focused on tuning the current process, Internal Learning is about identifying truths (which may change over time), best practices, and, “For the love of Pete, let’s not make the mistake of doing that again!” tactics. It pulls together the value from all three of the other analytics value areas in a more deliberative, forward-looking fashion. This is why it sits at the nexxus of the other three areas in the diagram at the beginning of this post. While, on the one hand, Learning seems like a, “No, duh!” thing to do, it actually can be challenging to do effectively:

  • Every initiative is different, so it can be tricky to tease out information that can be applied going forward from information that would only be useful if Doc Brown appeared with his Delorean
  • Capturing this sort of information is, ideally, managed through some sort of formal knowledge management process or program, and such programs are quite rare (consultancies excluded)
  • Even with a beautifully executed Performance Management process that demonstrates that an initiative had suboptimal results, it is still very tempting to start a subsequent initiative based on the skeleton of a previous one. Meaning, it can be very difficult to break the, “that’s how we’ve always done it” barrier to change (remember how long it took to get us to stop putting insanely long registration forms on our sites?)

Despite these challenges, it is absolutely worth finding ways to ensure that ongoing learning is part of the analytics program:

  • As part of the Performance Measurement post mortem for a project, formally ask (and document), what aspects, specifically, of the initiative’s results contain broader truths that can be carried forward.
  • As part of the Alignment exercise for any new initiative, consciously ask, “What have we done in the past that is relevant, and what did we learn that should be applied here?” (Ideally, this occurs simply by tapping into an exquisite knowledge management platform, but, in the real world, it requires reviewing the results of past projects and even reaching out and talking to people who were involved with those projects)
  • When Optimization work is successfully performed, do more than simply make the appropriate change for the current initiative — capture what change was made and why in a format that can be easily referenced in the future

This is a tough area that is often assumed to be something that just automatically occurs. To a certain extent, it does, but only at an individual level: I’m going to learn from every project I work on, and I will apply that learning to subsequent projects that I work on. But, the experience of “I” has no value to the guy who sits 10′ away if he is currently working on a project where my past experiences could be of use if he doesn’t: 1) know I’ve had those experiences, or 2) have a centralized mechanism or process for leveraging that knowledge.

What Else?

What do you say when someone asks you, “How does analytics add value?” Do you focus on one or more of the areas above, or do you approach the question from an entirely different perspective? I’d love to hear!

10 Tips for Web Analytics Wednesday Awesomeness (and a Downloadable Checklist)

By on in with 3 Comments.

I’ve become enamored with the “10 tips” format for organizing information (thank you, ACCELERATE), and I’ve had a couple of recent situations where people I know have asked for my advice on getting rolling with or successfully sustaining Web Analytics Wednesdays. A couple of years ago, someone actually tried to get a group of WAW organizers around the world together to come up with a handy guide for WAW organizers, but, due to scheduling issues, that never came together. After a successful Columbus WAW last week (shown below), it seemed worthwhile to write up what I’ve learned about planning and running WAWs over the last four years.

Columbus Web Analytics Wednesday - April 2012

Some of these tips overlap with the FAQ posted on the WAW site, and I’ve also created a one-page Excel checklist that covers the various details that go into our events to supplement this post.

And now, onto the tips!

Tip No. 1: Start Small

In Columbus, we now have a WAW almost every month, and we have between 40 and 60 attendees at each on . It took us several years to get to that level of consistent turnout, and that, in my mind, was a good thing. The core group that met over the first year or so got to know each other really well, as there were only 8-15 us at each event, and we could actually have group discussions in which everyone participated. Those early participants are still regularly attendees. People came consistently because they enjoyed the people, and they were patient with logistical hiccups and not-so-great venues. They provided feedback and made suggestions that helped us refine the what, the how, and the where of future events.

The other benefit of starting small is that you don’t have to worry about paying for the event – the Web Analytics Wednesday Global Sponsors are insanely easy to tap into to cover the cost (more on that in Tip No. 9).

Tip No. 2: Location, Location, Location

Location matters. In Columbus, this was something that took us over a year to really nail down, and I wasn’t much help, as I had only recently moved to the area. Some things to look for in a venue:

  • Centrally located – most cities have some degree of sprawl, so there is no location that is perfect for everyone; but, what we’ve found is that, the closer we can get the venue to the main business district, the better
  • Separate meeting room – lots of restaurants have rooms that can be reserved for private parties; sometimes, they require a separate fee, but sometimes they just require a minimum total spend. All things are negotiable – you’re bringing business to them on a Wednesday night, so they are generally flexible.
  • Low-to-moderate noise level – if the venue has a separate room, this is less of an issue; if it doesn’t, the noise level is key. WAWs are, first and foremost, about people meeting and talking to other people, and no one wants to be hoarse on Thursday morning. Live music and happenin’ bar scenes are cool…but they don’t make for great WAWs
  • Presentation-friendly – at a minimum, having a room that has a layout that is conducive to a projector and screen is important if there will be any presenting (see Tip No. 7); some venues have screens, and some actually have projectors. But, if the room layout isn’t such that it will support a projector and screen, then make sure you’ve thought through how visual information will be shared in the absence (tip: large companies typically have projectors that employees can check out for meetings – we regularly tap into attendees who work at such companies to actually provide the projectors). Handouts work, too.

Nailing down a single good location is hard enough, but we actually now have 2-3 good locations. This allows us to mix things up so that the event doesn’t start to seem like it has fallen into a rut. And, it gives us options – if one venue is booked for the preferred WAW date, another one is likely to be open.

Tip No. 3: Be Consistent

The cadence of WAWs seems to matter. We aim for an event once per month and know that, occasionally, we won’t manage to have one. Having the events on a regular schedule adds credibility to the event overall (which helps with sponsors and attendees alike), and it really helps convert “networking acquaintances” into “professional friends.”

There is definitely a commitment required in order to follow this tip. From the get-go in Columbus, we had multiple co-organizers, and that group of organizers has grown. We split up the effort — one secured a venue each month, one person handled the emails to past attendees, another person handled finding new ways to promote the event — and have built a pretty solid and repeatable process.

It’s difficult to build momentum without a consistent and recurring schedule, so getting organized and making it a group effort is key (see Tip No. 10).

Tip No. 4: Build a WAW Database

From our first event onward, I started entering the name and email address of each person who registered for a Columbus WAW into a Google Spreadsheet (I now use ExactTarget for this). This requires a little bit of sleuthing, as the WAW registration form only collects an email address. But, 9 times out of 10, it’s pretty easy to figure out the person’s name (the internet being scary that way and all…) and company. This is a bit tedious, but it’s worth it, as it gives us an ever-growing “house list” to whom we can promote upcoming events.

We now have a sign-in sheet at every event to collect the name and email address of each attendee. To reduce the level of data entry and handwriting-deciphering required, I pre-print a list of all registrants for the sign-in sheet and just ask people to check a box next to their name to indicate they’ve arrived. That sheet has blank rows for people who registered late or didn’t register to write in their information.

Tip No. 5: Invite and Remind

Obviously, it’s not enough to just build and maintain a house list if it doesn’t get used. For every WAW, each person on that list gets sent at least two emails (but no more than three):

  • Notification / invitation – a couple of weeks out, we send an email to the entire list letting them know of the upcoming event
  • Second invitation – for anyone who has not registered a week out, we send a second invitation; the content is very similar to the first one, but we generally mix up the subject line and the body copy a bit
  • Reminder – for anyone who has registered, we send a reminder email 2-3 days before the event

We try to consistently hit some key information with each email:

  • The date and location for the event
  • Information as to the topic that will be presented (if we have a presentation)
  • A reminder that the event is free
  • A link to the event registration page on the WAW site

We’ve even done some A/B testing on the subject lines, but, with a list that is only several hundred people, that’s more because it’s a good way to experiment with the process for A/B testing in ExactTarget than because we’ve been able to learn anything of note about effective subject lines for WAW emails.

And, while we haven’t always been 100% CAN-SPAM compliant, we’ve always been clear in all communications as to how the recipient could opt out of future emails, and we honor any opt out requests we receive.

Tip No. 6: Multi-Channel Promotion

In addition to email, we consistently push out notifications through as many channels as possible:

We don’t actively maintain any of these channels for any purpose other than notifications of upcoming events. That may not be a social media best practice, but it works, in that participants can opt in to non-email communication through whatever channel they prefer.

One thing we did learn was that we shouldn’t just sit down on one night and send out the email and simultaneously update every social media channel. This just meant that users who were connected through multiple means got spammed with the same information all at one point in time, which reduced its effectiveness (and was a little annoying). We now spread out the updates over the course of several days.

Tip No. 7: Limited Formal Presentations / Plenty of Time for Networking

We tell our presenters to aim for 15-20 minutes and to avoid presentations that are simply sales pitches for their companies. With brief presentations on relevant topics (sometimes the sponsor presents, sometimes it’s simply one of the organizers or an attendee who has volunteered a topic), we tend to spend another 15-30 minutes in Q&A and discussion. The feedback we’ve consistently gotten is that attendees enjoy both the networking and having some formally presented content. So, we strive to keep a balance between the two. Two keys to that:

  • Very clear (polite, but firm) communication to the presenters ahead of time as to expectations regarding presentation length
  • Having one of the organizers prepared to manage the clock — be it signaling the presenter to wrap up or announcing “let’s do one more question” if things run long and the crowd starts to squirm (some day, I’ll live down cutting off Chris Grant after she traveled all the way down from Michigan for our WAW…)

The schedule we’ve followed for the past few years is:

  • 6:30 – 7:00 — sign-in and networking
  • 7:00 – 7:10-ish — find seats, welcome and announcements
  • 7:10 – 7:45-ish — presentation and Q&A
  • 7:45-ish – 8:30/9:00 — more networking

I’ve got the word “networking” in the title of this tip and a couple of times in the listed schedule above, but, honestly, “hanging out” is probably a better description. Like-minded people with food and beer… it’s fun!

Tip No. 8: Encourage Tweeting

We encourage tweeting at our WAWs for all of the same reasons tweeting is encouraged at conferences:

  • It publicizes the event and content out to the followers of the attendees
  • It fosters networking as people engage with each other during the presentation
  • It provides a nice way to have crowdsourced “notes” from the presentation

To promote tweeting, we have started printing out little cards that we put at all of the tables that include:

  • The Twitter usernames of the presenter(s)
  • The hashtag for the event (we use #cbuswaw)
  • The logos of our sponsors (nothing should get printed or emailed that doesn’t include a thank you to the sponsors)

Even if there are only a small number of attendees, and even if there is no formal presentation, tweets can help spread the word.

Tip No. 9: Free Drinks (and Food, if Possible)

We’re reaching the end of this list, but that doesn’t mean these tips are any less important! Free drinks are a must! While no one attends a WAW simply because they are burdened with an empty bank account and a drinking problem, by offering booze, the overall vibe and purpose gets communicated as a “fun event” more than a “professional obligation.”

Providing free drinks can get expensive…but it’s worth the effort to make sure it happens. Sub-tips on that front:

  • If you’re just getting started, and it’s a small event, tap into the Web Analytics Wednesday Global Sponsors. That’s what their sponsorship is there for!
  • Use drink tickets to manage the total outlay. I have yet to host an event at a bar or restaurant that doesn’t have drink tickets on hand for our use, and, by handing out 1-2 tickets (we usually do 2), you can ensure that your sponsors aren’t inadvertently funding a fraternity party
  • Seek out sponsors — the smaller the event, the smaller the ask; the larger the event, the more worthwhile it is for the sponsor. Use your and other attendees connections to the analytics vendors and services they use. Many of them have marketing funds available, and it’s a great way for them to make connections with prospective customers in their territory.

We almost always provide food at our events as well. To manage costs on that front, we typically go with a “heavy appetizer buffet” rather than a full-on meal. We typically order food to cover 15-20% fewer people than we actually expect to attend. Otherwise, we wind up with crazy amounts of leftovers

Tip No. 10: Ask for Help

As I put together the checklist to accompany this post, and as I wrote the post itself, I realized how many moving parts there are in our process. No single event will ever be perfect, and it doesn’t have to be. But, the more details that get consistently covered, the more likely the WAWs are to flourish and grow. The best way to cover those details is through organization and teamwork: ask for volunteers to help with future events at each of your events; pay attention to who seems to be most engaged and has useful ideas and suggestions for future events. Recruit!

What’s Missing?

The downloadable checklist is intended as a companion to these tips, and it’s organized based on the different aspects of managing a WAW. I hope you find it useful.

What else have you seen — either when organizing or attending a WAW — that works particularly well? I’d love to get some comments that give us some ideas for continuing to improve our events!

Digital Analytics: From Data to Stories and Communication

By on in , , with 6 Comments.

This will be a quick little post as I try to pull together what seems to be an emerging theme in the digital analytics space. In a post late last year, I wrote:

I haven’t attended a single conference in the last 18 months where one of the sub-themes of the conference wasn’t, “As analysts, we’ve got to get better at telling stories rather than simply presenting data.”

Lately, though, it seems that the emphasis on “stories” has shifted to a more fundamental focus on “communication.” As evidence, I present the following:

A 4-Part Blog Series

Michele Kiss published a 4-part blog series over the course of last week titled “The Most Undervalued Analytics Tool: Communication.” The series covered communication within your analytics team, communication across departments, communication with executives and stakeholders, and communication with partners. Whether intentionally or not, the series highlighted how varied and intricate the many facets of “communication” really are (and she makes some excellent tips for addressing those different facets!).

A Data Scientist’s “Day to Day” Advice

Christopher Berry, VP of Marketing Science at Syncapse, also published a post last week that touched on the importance of communication. Paraphrasing (a bit), he advised:

  • Recognize that you’re going to have to repeat yourself — not because the people your communicating with are stupid, but because they’re not as wired to the world of data as you are
  • Communicate to both the visual and auditory senses — different people learn better through different channels (and neuroscience has shown that ideas stick better when they’re received through multiple sensory registers)
  • Use bullet points (be concise)

Christopher is one of those guys who could talk about the intricacies of shoe leather and have an audience spellbound…so his credibility on the communication front comes more from the fact that he’s a great communicator than from his position as a top brain in the world of data scientistry.

Repetition at ACCELERATE

During last Wednesday’s ACCELERATE conference in Chicago, I tweeted the following:

The tweet was mid-afternoon, and it was after a run of sessions — all very good — where the presenters directly spoke to the importance of communication when it come to a range of analytics responsibilities and challenges.

A Chat with Jim Sterne

At the Web Analytics Wednesday that followed the conference, I got my first chance (ever!) to have more than a 2-sentence conversation with Jim Sterne (I’m pretty sure the smile on his face all day was the smile of a man who was attending a conference as a mere attendee than as a host and organizer, and the plethora of attendant stresses of that role!).

During that discussion, Jim asked me the question, “What is it that you are doing now that is moving towards [where you want to be with your career].” We’ll leave the details of the bracketed part of my quote aside and focus on my answer, which I’d never really thought of in such explicit terms. My answer was that, being a digital analyst at an agency that was built over the course of 3 decades on a foundation of great design work and outstanding consumer research (as in: NOT on measurement and analytics), I have to keep honing my communication skills. In many, many ways I have a conversation every day where I am trying to communicate the same basics about digital analytics that I’ve been communicating for the past decade in different environments. But, I’m not just repeating myself. If I look back over my 2.5 years at the agency, I’ve added a new “tool” to my analytics communication toolbox every 2-3 months, be it a new diagram, a new analogy, a new picture, or a new anecdote. I’ve been working really hard (albeit not explicitly or even consciously) to become the most effective communicator I can be on the subject of digital analytics. Not every new tool sticks, and I try to discard them readily when I realize they’re not resonating.

It’s a work in progress. Are you consciously working on how you communicate as an analyst? What’s your best tip?

The Many Dimensions of Social Media Data

By on in , , with No Comments.

I’ve been thinking a bit of late about the different aspects of social media data. This was triggered by a few different things:

  • Paul Phillips of Causata spoke at eMetrics in San Francisco, and his talk was about leveraging data from customer touchpoints across multiple channels to provide better customer relationship management
  • I’ve been re-reading John Lovett’s Social Media Metrics Secrets book as part of an internal book group at Resource Interactive
  • We’ve had clients approaching us with some new and unique questions related to their social media efforts

What’s become clear is that “social media analytics” is a broad and deep topic, and discussions quickly run amok when there isn’t some clarity as to which aspect of social media analytics is being explored.

As I see it, there are four broad buckets of ways that social media data can be put to use by companies:

No company that is remotely serious about social media in 2012 can afford to ignore the top two boxes. The bottom two are much more complex and, therefore, require a substantial investment, both in people and technology.

Now, I could stop here and actually have a succinct post. But, why break a near-perfect (or consistently imperfect) streak? Let’s take a slightly deeper look at each bucket.

Operational Execution

(I almost labeled this bucket “Community Management,” but the variety of viewpoints in the industry on the scope of that role convinced me to leave that can of worms happily sealed for the purposes of this post.)

Social media requires a much more constant intake and rapid response/action based on data than web sites typically do. Having the appropriate tools, processes, and people in place to respond to conversations with appropriate (minimal) latency is key.

Key challenges to effectively managing this aspect of social media data include: determining a reasonable scope, being realistic about the available on-going people who will manage the process, and, to a lesser extent, selecting the appropriate set of tools. Tool selection is challenging because this is the area where the majority of social media platforms are choosing to play — from online listening platforms like Radian6, Sysomos, Alterian, and Syncapse; to “social relationship management” platforms like Vitrue, Buddy Media, Wildfire, (Adobe) Context Optional, and Shoutlet; and even to the low-cost platforms such as Hootsuite and TweetDeck. These platforms have a range of capabilities, and their pricing models vary dramatically.

Performance Measurement

Ahhh, performance measurement. When it comes to social media, it definitely falls in the “simple, but not easy” bucket. And, it’s an area where marketers are perpetually dissatisfied when they discover that there is no “value of a fan” formula, nor is there “the ROI of a tweet.” But, any marketer who has the patience to step back and consider where social media plays in his/her business can absolutely do effect performance measurement and report on meaningful business results!

Chapters 4 and 5 of John Lovett’s book, Social Media Metrics Secrets, get to the heart of social media performance measurement by laying out possible social media objectives and appropriate KPIs therein. High on my list is to make it through Olivier Blanchard’s Social Media ROI: Managing and Measuring Social Media Efforts in Your Organization, as I’m confident that his book is equally full of usable gems when it comes to quantifying the business value delivered from social media initiatives.

When it comes to technologies for social media performance measurement, we generally find ourselves stuck trying to make use of the Operational Execution platforms. They all tout their “powerful analytics,” but their product roadmaps have typically been driven more by “listenting” and “publishing” features than they have been driven by “metrics” capabilities. With Google’s recent announcement ofGoogle Analytics Social Reports, and with Adobe’s recent announcement of Adobe Social, this may be starting to change.

(Social-Enhanced) CRM

Leveraging social media data to improve customer relationship management is something that there has been lots of talk about…but that very few companies have successfully implemented. At its most intriguing, this means companies identifying — through explicit user permission or through mining the social web — which Twitter users, Facebook fans, Pinterest users, Google+ users, and so on can be linked to their internal systems. Then, by listening to the public conversations of those users and combining that information with internally-captured transactional data (online purchases, in-store purchases, loyalty program membership, email clickthroughs, etc.), getting a much more comprehensive view of their customers and prospects. That “more comprehensive view,” in theory, can be used to build much more robust predictive models that can let the brand know how, when, and with what content to engage individual customers to maximize the value of that relationship for the brand.

The challenges are twofold:

  • Consumer privacy concerns — even if a brand doesn’t do anything illegal, consumers and the press have a tendency to get alarmed when they realize how non-anonymous their relationship with the brand is (as Target learned…and they weren’t even using social media data!)
  • Complexity and cost — there is a grave tendency for marketers to confuse “freely available data” with “data that costs very little to gather and put to good use.” Companies’ customer data is data they have collected through controllable interactions with consumers — through a form they filled out on the web, through a credit card being run as part of a purchase, through a call into the service center, etc. Data that is pulled from social media platforms is at the whim of the platforms and the whim of the consumer who set up the account. No company (except Twitter) can go out to a Twitter account and, in an automated fashion, bring back the user’s email address, real name, gender, or even country of residence. It takes much more sophisticated data crawling, combined with probabilistic matching engines, to get this data.

Despite these challenges, this is an exciting opportunity for brands. And, the technology platforms are starting to emerge, with the three that spring the most quickly to my mind being Causata, iJento, and Quantivo.

Trend / Opportunity Prediction

This is another area that is really tough to pull off, but it’s an area that, admittedly, has great potential. It’s a “Big Data” play if ever there was one — along the lines of how the Department of Homeland Security supposedly harnesses the data in millions of communications streams to identify terrorist hot spots. It’s sifting through a haystack and not knowing whether your’re looking for a needle, a twig, a small piece of wire, or a paperclip, but knowing that, if you find any of them, you’ll be able to put it to good use.

The wistfully optimistic marketing strategist describes this area something like this: “I want to pick up on patterns and trends in the psychographic and attitudinal profile of my target consumers that emerge in a way that I can reasonably shift my activities. I want an ‘alert’ that tells me, ‘There’s something of interest here!’”

It’s a damn vague dream…but that doesn’t mean it’s unrealistic. It’s a multi-faceted challenge, though, because it requires the convergence of some rather sticky wickets:

  • Identifying conversations that are occurring amongst people who meet the profile of a brand’s target consumers (demographic, psychographic, or otherwise) — yet, social media profiles don’t come with a publicly available list of the user’s attitudes, beliefs, purchasing behavior, age, family income, educational level, etc.
  • Identifying topics within those conversations that might be relevant for the brand — we’re talking well beyond “they’re talking about what the brand sells” and are looking for content with a much, much fuzzier topical definition
  • Identifying a change in these topics — generally, what marketers want most is to pick up on an emerging trend rather than simply a long-held truism

To pull this off will require a significant investment in technology and infrastructure, a significant investment in a team of people with specialized skills, and a significant amount of patience. I chuckle every time I hear an anecdote about how a brand managed to pick up on some unexpected opportunity in real time and then quickly respond…without a recognition that the brand was spending an awful lot of time listening in real-time and picking up nothing of note!

This area, I think, is what a lot of the current buzz around Big Data is focused on. I’m hoping there are enough companies investing in trying to pull it off that we get there in the next few years, because it will be pretty damn cool. Maybe IBM can set Watson up with a Digital Marketing Optimization Suite login and see what he can do!

The Three-Legged Stool of Effective Analytics: Plan, Measure, Analyze

By on in , with One Comment.

Several weeks ago, Stéphane Hamel wrote a post that got me all re-smitten with his thought process. In the post, he postulated that there are three heads of online analytics. He covered three different skillsets needed to effectively conduct online analytics: business acumen, technical (tools) knowledge, and analysis. And, he made the claim that no one person will ever excel at all three, which led to his case for building out teams of “analysts” who have complementary strengths.

I’ve had several unrelated experiences with different clients and internal teams of late that have led me to try to capture, in a similar fashion, the three-legged stool of an online analytics program. Just as others have started tacking on additional components to Stéphane’s three skillsets, I’m sure my three-legged stool will quickly become a traditional chair…then some sort of six-legged oddity. But, I’d be thrilled if I could consistently communicate the basics to my non-analyst co-workers and clients:

I hold to a pretty strict distinction between “measurement and reporting” and “analysis,” and I firmly believe there is value in “reporting,” as long as that reporting is appropriately set up and applied.

Just as I believe that reporting should generally occur either as a one-time event (campaign wrap-up, for instance) or at regular intervals, I firmly believe that testing and analysis should not be forced into a recurring schedule. It’s fine (desirable) to be always conducting analysis, but the world of “present the results of your analysis — and your insights and recommendations therein — once/month on the first Wednesday of the month” is utterly asinine. Yet…it’s a mindset with which a depressing majority of companies operate.

Reporting Done Poorly…Which Is an Unfortunately Ubiquitous Habit

I’ve been client side. I’ve been agency side. I’ve done a decent amount of reading on human nature as it relates to organizational change. My sad conclusion:

The business world has conditioned itself to confuse “cumbersome decks of data” with “reporting done well.”

It happens again and again. And again. And…again! It goes like this:

  1. Someone asks for some data in a report
  2. Someone else pulls the data
  3. The data raises some additional questions, so the first person asks for more data.
  4. The analyst pulls more data
  5. The initial requestor finds this data useful, so he/she requests that the same data be pulled on a recurring schedule
  6. The analyst starts pulling and compiling the data on a regular schedule
  7. The requestor starts sharing the report with colleagues. The colleagues see that the report certainly should be useful, but they’re not quite sure that it’s telling them anything they can act on. They assume that it’s because there is not enough data, so they ask the analyst to add in yet more data to the report
  8. The report begins to grow.
  9. The recipients now have a very large report to flip through, and, frankly, they don’t have time month in and month out to go through it. They assume their colleagues are, though, so they keep their mouths shut so as to not advertise that the report isn’t actually helping them make decisions. Occasionally, they leaf through it until they see something that spikes or dips, and they casually comment on it. It shows that they’re reading the report!
  10. No one tells the analyst that the report has grown too cumbersome, because they all assume that the report must be driving action somewhere. After all, it takes two weeks of every month to produce, and no one else is speaking up that it is too much to manage or act on!
  11. The analyst (now a team of analysts) and the recipients gradually move on to other jobs at other companies. At this point, they’re conditioned that part of their job is to produce or receive cumbersome piles of data on a regular basis. Over time, it actually seems odd to not be receiving a large report. So, if someone steps up and asks the naked emperor question: “How are you using this report to actually make decisions and drive the business?”…well…that’s a threatening question indeed!

In the services industry, there is the concept of a “facilitated good.” If you’re selling brainpower and thought, the theory goes, and you’re billing out smart people at a hefty rate, then you damn well better leave behind a thick binder of something to demonstrate that all of that knowledge and consultation was more than mere ephemera!

And, on the client side, if the last 6 consultancies and agencies that you worked with all diligently delivered 40-slide PowerPoint decks or 80-page reports, then, by golly, you’re going to look askance at the consultant who shows up and aims for actionable concision!

Nonetheless, I will continue my quixotic quest to bring sanity to the world. So, onto the three legs of my analytics stool…

First, Plan (Dammit!!!)

Get a room full of experienced analysts together and ask them where any good analytics program or initiative starts, and you’ll get a unanimous response that it starts: 1) at the beginning of the initiative, and 2) with some form of rigorous planning.

The most critical question to answer during analytics planning is: “How are we going to know if we’re successful?” Of course, you can’t answer that question if you haven’t also answered the question: “What are we trying to accomplish?” Those are the two questions that I wrote about in this Getting to Great KPIs post.

Of course, there are other components of analytics planning:

  • Where will the data come from that we’ll use?
  • What other metrics — beyond the KPIs — will we need to capture?
  • What additional data considerations need to be factored into the effort to ensure that we are positioned for effective analysis and optimization down the road?
  • What (if any) special tagging, tracking, or monitoring do we need to put into place (and who/how will that happen)?
  • What are the known limitations of the data?
  • What are our assumptions about the effort?
  • …and more

In my experience both agency-side and client-side, this step regularly gets skipped like it’s a smooth round, rock in the hand of an adolescent male standing on the shore of a lake on a windless day.

An offshoot of the planning is the actual tagging/tracking/monitoring configuration…but I consider that an extension of the planning, as it may or may not be required, depending on the nature of the initiative.

Next, Measure and Report

Yup. Measurement’s important. That’s how you know if you’re performing at, above, or below your KPIs:

Here’s where I start to get into debates, both inside the analytics industry and outside. I strongly believe that it is perfectly acceptable to deliver reports without accompanying insights and analysis. Ideally, reports are automated. If they’re not automated, they’re produced damn quickly and efficiently.

Dashboards — the most popular form of reports — have a pretty simple purpose: provide an at-a-glance view of what has happened since the last update, and ensure that, at a glance, any anomalies jump out. More often than not, there won’t be anomalies, so there is nothing that needs to be analyzed based on the report! That’s okay!

I was discussing this concept with a co-worker recently, and, in response to my claim that reports should simply get delivered with minimal latency and, at best, a note that says, “Hey, I noticed this apparent anomaly that might be important. I’m going to look into it, but if you (recipient) have any ideas as to what might be going on, I’d love to get your thoughts,” she responded:

I think this makes sense, but wouldn’t we provide some analysis as to the “why” on the monthly reports?

I immediately went to the “dashboard in your car” analogy (I know — it breaks down on a lot of fronts, but it works here) with my response:

You don’t look at your fuel gauge when you get in the car every day and ask, “Why is the needle pointing where it is?” You take a quick look, make sure it’s not pegged on empty, and then go about your day.

That’s measurement. It may spawn analysis, but, often, it does not. And that’s to be expected!

Which Brings Us to Testing and Analysis

Analysis requires (or, at least, is much more likely to yield value in an efficient manner) having conducted some solid planning and having KPI-centric measurement in place. But, the timing of analysis shouldn’t be forced into a fixed schedule.

The bottom part of the figure above gets to the crux of the biscuit when it comes to timing: sometimes, the best way to answer a business question is through analyzing historical data. Sometimes, the best way to answer a question is through go-forward testing. Sometimes, it’s a combination of the two (develop a theory based on the historical data, but then test it by making a change in the future and monitoring the results). Sometimes the analysis can be conducted very quickly. Other times, the analysis requires a large chunk of analyst time and may take days or weeks to complete.

Facilitating the collaboration with the various stakeholders, managing the analysis projects (multiple analyses in flight at once — starting and concluding asynchronously based on each effort’s unique nature), can absolutely fall under the purview of the analyst (again referencing Stéphane’s post, this should be an analyst with a strong “head” for business acumen).

In Conclusion…(I promise!)

There is a fundamental flaw in any approach to using data that attempts to bundle scheduled reporting with analysis. It forces efforts to find “actionable insights” in a context where there may very well be none. And, it perpetuates an assumption that it’s simply a matter of pointing an analyst at data and waiting for him/her to find insights and make recommendations.

I’ve certainly run into business users who flee from any effort to engage directly when it comes to analytics. They hide behind their inboxes lobbing notes like, “You’re the analyst. YOU tell me what my business problem is and make recommendations from your analysis!” I’m sure some of these users had one too many (and one is “too many”) interactions with an analyst who wanted to explain the difference between a page view and a visit, or who wanted to collaboratively sift through a 50-page deck of charts and tables. That’s not good, and that analyst should be flogged (unless he/she is less than two years out of college and can claim to have not known any better). But, using data to effectively inform decisions is a collaborative effort. It needs to start early (planning), it needs to have clear, concise performance measurement (KPI-driven dashboards), and it needs to have flexibility to drive the timing and approach of analyses that deliver meaningful results.

Facebook Status Updates: Exploring Optimal Time of Day/Day of Week

By on in , with 17 Comments.

Although Facebook has unofficially admitted that there seems to be little rhyme or reason these days when it comes to the time of day or day of week when a brand posts content on their page, it’s still worth doing a quick analysis to see if this is, indeed, the case for your page.

The challenge, it turns out, is that there are multiple aspects of what sounds like a pretty straightforward assessment:

  • What metric(s) actually make for a “successful” post?
  • How do you effectively consider time of day and day of week?
  • Have you actually posted on a sufficient variety of dates and times to have the data to do a meaningful analysis?

After scraping together some hasty cuts at this, I thought it would be worthwhile to try to knock out something that was easily shareable and reusable. The result is the downloadable spreadsheet at the end of this post.

What It Looks Like

The spreadsheet takes a simple export of post-level data from Facebook Insights (the .csv format) and generates three basic charts.

The first chart simply shows the number of posts in each time slot and each day of week — this answers the question of, “What spots have I not even really tried posting in?”

In this example, there have not been any posts from 9:00 PM to 6:00 AM, only one post between 6:00 AM and 9:00 AM, and only four posts on a Friday. Don’t worry if you don’t like the time windows — we’ll get to that in a bit.

The next two charts are crude heatmaps of a couple of metrics, but they both use the same grid as above, and they use a pretty simple green-to-red spectrum to show which spots performed best/worst relative to the other slots:

(I know, I know: red/green is not a colorblind-friendly palette selection. I’ll keep working on the visualization technique!)

The first of these charts looks at the average total reach of the updates that were posted in each time slot — the number of unique users of Facebook who were exposed to the post:

In the example above, Wednesdays looked to perform pretty well reach-wise, as did Saturday afternoon. If you have Facebook paid media running, these results may get skewed. It’s easy enough to update this chart to use Organic Reach rather than Total Reach, or, you can simply factor an awareness of what was running and when into your assessment of the results. Also, keep in mind that Facebook continues to fiddle with the EdgeRank/GraphRank algorithm, so there are aspects of a post’s reach that are beyond your control.

The next chart shows the average engagement rate of the posts, defined as the number of users who engaged with the post in some way (clicked on a link, posted a comment, liked the post, viewed a photo, etc.) divided by the total reach of the post. This is a pretty solid measure of the content quality — did the post drive the users who saw it to take some action to engage with the content? Now, arguably, the propensity for a user to engage is less impacted by the time of day and day of week, but, who knows?

In this example, Sundays and Thursdays were the days when posts appeared to get more engagement (although be leery of that Sunday, 6:00 PM to 9:00 PM, block — there was only a single post in the data set).

Timeframe Flexibility

Picking a set of timeframes is the most subjective aspect of this whole analysis, and it may be worth iterating through a few times to get to timeframes that are likely to be meaningful for the page given the target consumer. So, I’ve set up the worksheet to make it easy to customize these timeframes. For, instance, below is the same data set used above, but with only four windows of time:

The change look less than 60 seconds to implement (it’s all about VLOOKUPS, pivot tables, and conditional formatting!).

How to Use This for Your Own Page

If you want to try this out for your page(s), simply download the Excel file (this was created using Excel 2007, so it should work fine in both 2007 and 2010) and follow the instructions embedded in the worksheet. You will need to export post-level Facebook Insights data for your page, which may require several iterations (we’ve found that Facebook Insights is prone to hanging up if you try to export more than a couple of months of data at once):

Then, just follow the instructions in the spreadsheet and drop me a note if you run into any issues!

Some Notes on the Shortcomings

This approach isn’t perfect, and, if you have ideas for improving it, please leave a comment and I’ll be happy to iterate on the tool. Specifically:

  • This approach measures all updates against the other posts for the same page — there is no external benchmarking. This doesn’t bother me, as I’m a proponent of focusing on driving continuous improvement in your performance by starting where you are. Certainly, this analysis should be complemented by performance measurement that tracks the actual values of these metrics over time.
  • The overall visualization could be better. It’s not ideal that you need to jump back and forth between three different visualizations to draw conclusions about what days/times are really “good” or “bad”…including factoring in the sample size. I’ve toyed with making more of a weighted score and then doing the same color grid, but, then, you’d be looking at a true abstraction of the performance, so I didn’t go that route. Suggestions?
  • A red–>yellow–>green scale just isn’t good when it comes to supporting: 1) black-and-white printouts, and 2) certain forms of color blindness. A more iconographic approach might make more sense.

Please do weigh in with how you would change this. I’m happy to rev it based on input!