The Fun of Facebook Measurement
By Tim Wilson on in Analysis, Facebook Insights, Metrics, Web Analytics with 13 Comments
If you are a marketer, Facebook is important — the number of active users of the site exceeds the population of the United States, and it’s growth is going to do nothing but increase. Check out the Facebook statistics page for a slew of numbers that are all…big. Because of the growth of Facebook as a critical marketing channel, a hot topic around the office right now is “Facebook measurement.” It’s a tricky topic that reminds me of the early days of web analytics: there’s some basic stuff that’s easy to measure, and it’s basically helpful, but there’s a lot more that can’t be measured or can’t be measured well, and that’s where the real value is.
There are (at least) five different aspects of Facebook that can be measured:
- Facebook display ads — I’m not going to cover that at all; I haven’t spent a whole lot of time digging into it with our clients, so I’m not going to write about it. Check out What about advertising on Facebook? over on the Site Pro Specialties site for a quick overview of the ins-and-outs and their experience with Facebook display ads
- Facebook applications — I’m not going to cover this, either, largely because my experience on the subject is pretty limited, but also because Facebook apps annoy the bejeezus out of me, and I don’t want to have to stifle a gag reflex by writing about measuring them
- Facebook groups — Holy cow! Another topic I’m not going to cover! Since Facebook pages came on the scene, that’s where brands tend to be living more, and Facebook provides more measurement support for pages, so that’s where I’m going to focus the most
- Facebook pages — I’ll focus on these quite a bit, as this is an area that brands are really starting to settle into as a formal presence on Facebook
- General Facebook activity — this is an area where measurement is highly limited, but I’ll lay out what is there and what I hope comes sooner rather than later
We’re in a bit of an ugly period from an analyst’s perspective, in that Facebook hasn’t made supporting marketers a high priority beyond paid advertising. And, the company is being very cautious on the privacy front (which, from a consumer’s perspective, is a good thing!). The easiest way to reduce the risk of a PR blowup from misuse of Facebook data is to limit the availability of that data to marketers. I can’t blame them, but it doesn’t mean they’ve made my life easy on that front.
Ready? Let’s go! Here’s a quick set of links to use to jump down to specific topics:
- Facebook Pages — Fan Count
- Facebook Pages — Facebook Insights Data
- Facebook Pages — Web Analytics Measurement, Part I (The Ugly Part)
- Facebook Pages — Web Analytics Measurement, Part II (The Pretty…but Short…Part)
- General Facebook Activity — Web Analytics Measurement
- General Facebook Activity — LOTS is Missing
Facebook pages are a way for brands to establish a formal, managed presence on the site. They’re easy to set up, and they can range from the very simple and unused (see the Smuirfield Golf Club page) to the very elaborate and active (see the Victoria’s Secret PINK page). For any page, regardless of whether you are an admin for it or not, you can see the total number of fans at a point in time — the example below is from the Slate Political Gabfest page:
That can be useful for a couple of reasons:
- Organically grown pages — it’s fairly common for major brands to have their fans set up pages and grow a decent following; being able to tell the reach of those pages can help identify when outreach or integration might be in order
- Competitive research — it can be tedious, but assessing the size and growth of competitor fan pages over time can provide insight (albeit limited insight) into their overall social media strategy and their ability to execute
There is no way to measure the change in any page’s fan count over time other than periodically going and checking and recording it. And, what does total fans tell you? It tells you something…but not as much as you might like. More on that later.
Now, if you have admin access to a Facebook page, you can get much richer data, and you can get a historical view of some of that data. On the page itself, above the Fans box, is the basic Facebook Insights box:
While this looks encouraging, it’s not particularly useful. “Post Quality” sounds like a good idea (pick any measure of activity volume, and you can say, “It’s not just about quantity — it’s about quality!” and sound smart), exactly how Facebook determines quality is a bit of a mystery. From the Facebook Help Center:
The Post Quality score measures how engaging your Posts have been to Facebook users over a rolling seven-day window.
Post Quality is an important indicator for how fans gauge your posts. This score is calculated with an algorithm that takes into account your number of posts, total fan interactions received, number of fans, as well as other factors.
It’s a measure that’s almost too vague to be useful. And, in practice, the historical trending of Post Quality shows that something about the way it is measured makes it pretty non-actionable — even for pages that have a high level of fan engagement consistently, a trendline of Post Quality goes all over the place.
So, now we dive into the real meat of Facebook Insights, which initially looks like a nice, juicy T-bone, but which turns out to be more like a pretty lean cut of venison. The See All link in the Insights box brings up the main Facebook Insights page (click on the image below to view a larger version):
This page has the second not-nearly-as-useful-as-you’d-like measure: Active Fans. Facebook is even more fuzzy about how this is calculated than it is about Post Quality. And, historical data is not available. In my experience, Active Fans is a pretty big crap shoot — it varies widely from day to day and, since it’s not easy to get historical data, it’s a mess to try to analyze what is going on and how it is really changing over time with any granularity. Conceptually, active fans are high-quality fans. In my experience, the number of active fans in any given period is a tiny fraction of the overall fans. So, the million-dollar question — “What is the value of a Facebook fan?” — should probably include a separate calculation for an “active fan.” But, “active fan” is such a messy measure with such limited availability, that it’s barely worth pursuing until it’s more accessible and explainable.
Most of the other measures, though, have historical data available via the graphs shown on the page. Some underlying data can be exported as a CSV or Excel file with granularity at the individual day level. Two wrinkles with that data, though:
- The timing of the data updates is inconsistent, and it doesn’t seem like “if data is there, it’s good data” — a note in the bottom of the Insights window states: “Please allow 48 hours for data to be available for a daily report;” it’s common to see some data for a given day populated while other data for the same day isn’t; while I don’t feel like “real-time” data is generally warranted, the 48-hour lag can put a real crimp in effectively weekly reports, as well as in getting a good, timely view into the results of a new Facebook campaign
- The data doesn’t appear to be kept forever; it used to seem like data dropped off once it was ~3 months old, but the actual range of available data seems to vary, and Facebook doesn’t provide information on the subject; we’re in the practice of exporting all available data monthly so that we’ve got it retained offline for our clients
The main export option is the Fans and Interactions export. The other two exports that are available are Demographics and Country. The demographics export simply shows, by day, the number of fans of a given age range/gender. The demographics of active fans over time is not available, unfortunately. The Country export simply shows the number of fans from each country over time.
Now, Fans and Interactions is where the most useful information is. You can get a great look into how fan growth has been growing over time — new fans, total fans, unsubscribes, etc. This provides a way to do a classic “leaky bucket” report — how many fans you are losing compared to how many new fans you are acquiring. Unsubscribes are interesting, because that means fans have explicitly removed themselves as fans rather than simply choosing to remove the page’s updates from their feeds. Which…alludes to the Big Wrinkle when it comes to fans — just because someone is a fan of your page doesn’t mean they’re seeing anything that happens on the page — it’s very easy for users to hide all updates from a page from their feeds. And Facebook doesn’t provide data as to how many people have done that!
Fans and Interactions also provides data on the number of “interactions” which is the sum of all of the likes, posts, and comments that occur each day. In my mind, a “like” is a pretty light interaction, while a post or a comment is a more significant interaction, because a fan actually had to put together words to express an idea. Facebook Insights provides details for each type of interaction, too, though, so you can measure the different types of interactions. This export provides four types of interactions: Likes, Comments, Wall Posts, and Discussion Posts. It can get a little confusing as to which type of user activity occurs where, so be prepared to click back and forth between your page and the data for a while to get the hang of it (I’d write it out here, but this post is already getting pretty long and unwieldy!). The data also includes “Posts” — these are your posts rather than fan posts.
Finally, Fans and Interactions provides basic web analytics data. VERY basic. Page views, unique page views, audio plays, video plays, and photo views. At a very high level, this is useful information, as it’s a measure of whether the page is sufficiently engaging to drive people to visit (note that someone by no means has to be a fan to visit the page, view content, and comment on it — if a page has a lot of page views but a small number of fans, then it may be an indication that users would like to engage with the brand in Facebook, but the actual content/activity occurring on the page is not strong enough to get them to become a fan once they actually visit). Data that is not provided includes: which tabs of the fan page were visited, which videos were played (and how much of the video was viewed), and which photos were viewed. Supposedly, this sort of capability is in the works at Facebook, but no one I’ve talked to is committing to any dates for them to roll out.
Facebook Insights also doesn’t provide data on:
- Suggest to Friends usage
- Subscribe via SMS usage
- Add to My Favorites usage
- The ability to export wall posts, discussion posts, and comments (more on this in the last section of this post)
- Page visit frequency
The lack of Suggest to Friends data is particularly painful — this would be a powerful measure of how engaging the content on the page is, and there is zip when it comes to any visibility into that.
I expect that Facebook Insights will evolve over time to provide more content-level detail, as well as usage of other “page” features. It’s less likely that Insights will evolve to include user-level detail due to privacy concerns, although it’s not inconceivable — this would be the equivalent of having access to detailed behavioral data for users who have registered with your web site and are making subsequent visits.
Depending on how you squint when you look at it, a Facebook fan page for your brand is just an off-site extension of your web site — just like any content you host on a third-party site (job postings that are hosted by a recruiting site, events that get managed through a third-party event management site, etc.). For third-party sites whose bread and butter is extending the content offerings from web sites, it’s common to deploy the main site’s web analytics page tag on the third-party content pages. There are myriad ways to set up the reporting for that in any web analytics tool — Google Analytics, SiteCatalyst, Webtrends, Coremetrics, etc. In theory, Facebook pages would be the same way — just as you can embed all sorts of rich content on custom tabs, it seems like you would be able to insert your web analytics page tag on the pages where you have heavy control over content.
So, what options are there? There are several, but they’re all clunky.
[UPDATE: The next little section is continuing to evolve, as I’ve been doing a lot of digging and experimentation in this area, finding both new roadblocks as well as trying out workarounds]
- Use an iFrame for the content and put your usual page tag in it — the wrinkle here is that you can’t put an iFrame on a custom tab; it has to be a standalone application canvas. Now, you can include within the frame a dummied-up re-rendering of the tabs on your fan page, but that’s really not ideal. There is a mildly helpful thread on the Google Analytis forum on the subject, as well as a thread on the Facebook developers forum with some useful tips
- Either use the <noscript> capability in your web analytics package (if one exists) or hack the actual image call that triggers a page view/action in your web analytics package — this is pretty cumbersome to do, and it has its limitations, as it’s essentially going back to the early days of page beacon/page dot technology for web analytics; it’s better than what you get out of Facebook Insights, though
- Build a custom solution that makes an image (or some other asset) call to a reporting server you manage — you would need a unique call for each activity you want to track — and then sift through the server log file to construct what’s happened; you’re going to run into challenges with caching of images, though, so this will be incomplete data at best
All of these only work on pages where you have a decent level of control over the content, which leaves out the Info, Photos, Videos, and Discussion tabs…and it’s a little dicey as to what’s doable on the Wall. But, presumably, it’s the custom tabs where you’re investing the most resources to develop content, so that’s a pretty good place to get some more granular web analytics data.
We’ve actually managed to get some tracking of interactions occurring on a user’s wall within a Flash-based status update using Google Analytics (using the third approach above), and we’re close to rolling out some pages that will use the second item above with Webtrends (which will track both interactions within a Flash app as well, we expect, as traffic to individual custom tabs).
[End of section that is still evolving]
In short, though, this is pretty messy.
If you link back to your main site from your Facebook page (which, presumably, you do in multiple places), then standard parameter-based campaign tracking works. Use it. ‘nuf said.
In addition to tracking links that you control on Facebook with campaign tracking (the previous section), you can and should look at Facebook as a broader source of traffic to your site. If you are posting content on your site that is share-worthy, then Facebook users can pick it up and share it through Facebook, which will drive referrals to your site. If you’ve actually enabled content-sharing capabilities on your site, and those capabilities include Facebook, then you can add campaign tracking parameters to content as it gets shared, which will give you better visibility into what specific content is most compelling and passed along. Beyond just the traffic to the site, the bounce rate and conversions from that traffic are useful — is the sharing of your content bringing visitors to your site who are finding value and doing valuable things?
The caution here is to not get overly obsessed with Facebook as a source of traffic to your site. It certainly can (and probably should) be a source of traffic, but your site isn’t necessarily the best destination point for all of your customers. Just because this is easy data to get to doesn’t mean that it is the best data to use to measure the performance of your site.
Overall, Facebook measurement — measurement of what really matters — is still very immature. We’re largely stuck with measuring basic counts of things that are easy to measure: total fans, unique pageviews, etc. But, when it comes to both measuring the impact of a Facebook investment as well as being able to analyze what is and is not working, we’re missing a lot:
- Impressions — how many people are actually being presented with content related to your brand? Besides Facebook display ads, this is total guesswork; just because a page posts a status update doesn’t mean it ever shows up on the screen of a fan (the update may slip well into the “More” area before the fan logs on again, the fan may have those updates hidden); “impressions” is far from being an end-all/be-all measure, but it’s a pretty good indicator of reach, and it’s really not available in the Facebook world
[UPDATE: Since I originally wrote this post, I’ve found out that Facebook has something in the works for this — the one referenceable source is Facebook Presentation Reveals "Post Analytics" And Real-Time Ad Targeting. It’s a total crapshoot as to when this functionality will be available and to whom it will be available.][UPDATE No 2: This capability was formally rolled out on January 21, 2010. I posted my take on what that provides.]
- Social Graph and Impact — all Facebook users (and, thus, all Facebook page fans) are not equal; all of the major online listening platforms attempt to measure the influence of the “speaker,” and, conceptually, this construct applies in the Facebook world, driven by various aspects of the user: how many friends they have, how often they update their status, and, most importantly, how often the content they share gets liked/commented on/re-shared; it is currently not possible to get any visibility into and segment users who are interacting with your brand on Facebook based on their influence in the medium
- Sentiment — Facebook has the “Like” feature, but no comparable “Dislike” option; this is grade school manners enforcement: “If you can’t give it a thumbs-up, don’t give it any thumb at all…” From a brand perspective, though, it would be nice to be able to track what sorts of posts raise users’ ire (especially for user-generated content) without having to sift through individual posts and comments by hand, which leads me to…
- Sentiment…continued — sentiment is a tough nut to crack, but it’s something that everyone who deals with social media recognizes as being important; while I don’t necessarily expect Facebook to develop sentiment measurement tools inherently, if Facebook Insights was enhanced to enable the export of all user interactions for a fan page, then third-party tools could be used to conduct a sentiment analysis, and that would be useful
[UPDATE: While it’s not necessarily a business/analyst-friendly option, the Facebook API does allow the retrieval of comments and posts. If you have the chops to tackle it, you can read about the options at http://wiki.developers.facebook.com/index.php/API#Data_Retrieval_Methods. One company that is using the API for that purpose (among others) is Vitrue — comments and posts get pulled into their Vitrue SRM product in a pretty slick way.]
- Online Listening…to Facebook — Google announced late last year that they were going to start crawling publicly available content in Facebook, and, presumably, online listening platforms will not be far behind (maybe some of them already do?). But, this listening is inherently limited to public content in Facebook (fan pages are public, so they would be included, presumably, which is a good thing). There would be a major backlash if Facebook enabled third-party tools to crawl and index “private” content. Does that mean that Facebook should enable it’s own intra-Facebook online listening capability? Marketers would certainly love to have the information, even if it is only available in a way that maintains users’ anonymity, but any move in this direction would be a dicey proposition for Facebook (even if they hid user information, it would be conceivable that users would provide enough information in what they post that a company would still be able to identify a specific individual — even if that was only going to be possible 1 time in 100,000, privacy advocates would jump all over Facebook for allowing the theoretical possibility)
It will be interesting to see where Facebook goes over the next 1-2 years when it comes to empowering marketers to measure and analyze their Facebook-based tactics. It should be a fun ride.
What am I missing here?