Web Analytics Tools Comparison — Columbus WAW Recap Part 2

By on in with 15 Comments

[Update: After getting some feedback from a Coremetrics expert and kicking around the content with a few other people, I rounded out the presentation a bit.]

In my last post, I recapped and posted the content from Bryan Cristina’s 10-minute presentation and discussion of campaign measurement planning at February’s Columbus Web Analytics Wednesday. For my part of the event, I tackled a comparison of the major web analytics platforms: Google Analytics, Adobe/Omniture Sitecatalyst, Webtrends, and, to a certain extent, Coremetrics. I only had five minutes to present, so I focussed in on just the base tools — not the various “warehouse” add-ons, not the A/B and MVT testing tools, etc.

Which Tool Is Best?

This question gets asked all the time. And, anyone who has been in the industry for more than six nanoseconds knows the answer: “It depends.” That’s not a very satisfying answer, but it’s true. Unfortunately, it’s also an easy answer — someone who knows Google Analytics inside and out, has never seen the letters “DCS,” referenced the funkily-spelled “eluminate” tag, or bristled at Microsoft usurping the word “Vista” for use with a crappy OS, can still confidently answer the, “Which tool is best?” question with, “It depends.”

And You’re Different?

The challenge is that very, very few people are truly fluent in more than a couple of web analytics tools. I’ve heard that a sign of fluency in a language is that you actually think in the language. Most of us in web analytics, I suspect, are not able to immediately slip into translated thought when it comes to a tool. So, here’s my self-evaluation of my web analytics tool fluency (with regards to the base tools offered — excluding add-ons for this assessment; since the add-ons bring a lot of power, that’s an important limitation to note):

  • Basic page tag data capture mechanics — 95th percentile — this is actually something pretty important to have a good handle on when it comes to understanding one of the key differences between Sitecatalyst and other tools
  • Google Analytics — 95th percentile — I’m not Brian Clifton or  John Henson, but I’ve crafted some pretty slick implementations in some pretty tricky situations
  • Adobe-iture Sitecatalyst — 80th percentile — I’m more recent to the Sitecatalyst world, but I’ve now gotten some implementations under my belt that leverage props, evars, correlations, subrelations, classifications, and even a crafty usage of the products variable
  • Webtrends — 80th percentile — I cut my teeth on Webtrends and would have put myself in the 95th percentile five years ago, but my use of the tool has been limited of late; I’m actually surprised at how little some of the fundamentals change, but maybe I should
  • Coremetrics — 25th percentile — I can navigate the interface, I’ve dived into the mechanics of the different tags, and I’ve done some basic implementation work; it’s just the nature of the client work I’ve done — my agency has Coremetrics expertise, and I’m hoping to rely on that to refine the presentation over time

So, there’s my full disclosure. I consider myself to be pretty impartial when it comes to tools (I don’t have much patience for people who claim impartiality and then exhibit a clear bias towards “their” tool — the one tool they know really well), but, who knows? It’s a fine line between “lack of bias” and “waffler.”

Any More Caveats Before You Get to the Content?

My goal with this exercise was to sink my teeth in a bit and see what I could clearly capture and explain as the differences. Ideally, this would also get to the, “So what?” question. What I’ve found, though, is that answering that question gets circular in a hurry: “If <something one tool shines as> is important to you, then you really should go with <that tool>.” Two examples:

  • If enabling users to quickly segment traffic and view any number of reports by those segments is important, then you should consider Google Analytics (…or buying the “warehouse” add-on and plenty of seats for whatever other tool you go with)
  • If being able to view clickpaths through content aggregated different ways is important, then you should consider Sitecatalyst

These are more of a “features”-oriented assessment, and they rely on a level of expertise with web analytics in order to assess their importance in a given situation. That makes it tough.

Any tool is only as good as its implementation and the analysts using it (see Avinash’s 10/90 rule!). Some tools are much trickier to implement and maintain than others — that trickiness brings a lot of analytics flexibility, so the implementation challenges have an upside. In the end, I’ll take any tool properly implemented and maintained over a tool I get to choose that is going to be poorly implemented.

Finally! The Comparison

I expect to continue to revisit this subject, but the presentation below is the first cut. You might want to click through to view it on SlideShare and click the “Speaker Notes” tab under the main slide area — I added those in after I presented to try to catch the highlights of what I spoke to on each slide.

Do you see anything I missed or with which you violently disagree? Let me know!




Similar Posts:

15 Comments


  1. Tim,

    Thanks for sharing. Good stuff! Will you be at eMetrics? Would love to give you a spin through Yahoo! Web Analytics to see how you think it compares from a 3rd party perspective.

  2. Thanks for this Tim. It is nice to see even some of the basics differences laid out.

    Even though you were under a time constraint, it might be worth addressing differences in capabilities for conversion attribution.

    My general knowledge is that GA is pretty limited in attribution options out of the box, while the paid tools (SC for sure) are pretty flexible if you’re willing to put the work into them. That seems to be the usual with them. :)

  3. @Ty GA has been trying to improve their attribution capabilities, I think. Maybe that’ll really hit its stride with the rumored paid version? :-)

    I haven’t had much direct experience with attribution. With GA, I’ve dabbled with it enough to know how/when the campaign gets overwritten. With Sitecatalyst, it seems like I have a co-worker who looked into it…and found that it was going to take some development work (and maybe some additional licensing?).

    The challenge with web analytics platforms trying to address attribution, IMHO, is that they’re stuck with too little data — they’re limited to on-site behavior and referral sources, which misses out on off-site (paid media — digital or not, social media, etc.) exposure. Of course, Coremetrics has their impression tag, and that was built for viewthrough data and, presumably, attribution of digital display media.

    You raise a good point, though. One more thing for me to sniff around on and try to understand tool capabilities for!

  4. Pingback Quora

  5. Really great stuff here, although I do feel like some features of Omniture are a little better than pointed out.

    For example:
    Visitor Segmentation – This is very much not just limited to ASI Slots (which I agree, are a huge hassle! Looking forward to them being replaced by Instant Segmentation in SC15). Every eVar is technically a Visitor Segment. It’s not on-the-fly segmentation like Google Analytics (again, yet) but it’s certainly better than Webtrends’ filtered profiles/segments. Every event can be tied back to eVars which have LOTS of powerful expiration and allocation settings. Not to be ignored.

    Integration with 3rd Party Meta Data – Although you have both WT and OMN as smiley faces, I’d say that SAINT files are significantly more powerful than Webtrends’ Translation Files. If you have OnDemand Webtrends you have to email them to Customer Service and have them implement them. SAINT files can be pushed in through HTTP or FTP and are definitely a step above Translation Files. Too bad GA has no solution for this :(

    External data integration – this obviously isn’t a topic you touched on. Possible with Webtrends and Unica, expensive but flexible with Omniture, and impossible with Google Analytics. Not sure of CoreMetrics.

    Ability to track events without substantiating pageviews – Omniture does it very well with Custom links (and even has 500ms pause to make sure the request goes through during onClick events), Webtrends and Google Analytics do it but without the 500ms pause, so these tags may be lost with browsers like Firefox.

    Standard Reports – I must say, Google Analytics’ out-of-the-box reports are the most user-friendly. LOVE how easy it is to set up the Site Search reports (try doing this as quickly with Omniture or Webtrends). Setup for Webtrends’ custom reports is just a complete hassle / No on-the-fly reporting whatsoever.

    Otherwise, there’s some really great information here. As you mentioned you were limited to 5 minutes, so apologies for nitpicking, but I just thought you’d appreciate some additional feedback.

    Thanks and keep up the great work!!

  6. Thanks for adding the detail, Eric!

    Visitor Segmentation — I don’t want to discount the value of eVars, but it seems like we could wind up in quite a semantic debate there. eVars are really built to be “conversion variables,” right? So, when it comes to richly exploring a set of visitors who meet a specific criteria (where they came from, what they visited, ext.), it doesn’t really seem like eVars are up to the task. From what I’ve seen for SC15 (one video touching on the subject), the on-the-fly segmentation is going to be limited to viewing a single segment at a time. Is that correct? The ability in GA to put multiple segments right next to each other is pretty handy.

    On the 3rd Party Meta Data Integration, I’m wondering if you just had a misinformed experience with WTOD? I certainly have had translation files set up such that we were able to FTP updated files into the system. Unless that was an option that has been removed by WT in the past couple of years, I think they’re comparable.

    Track events without impacting pageviews — the 500ms pause is something I didn’t know. I need to do some reading there! That’s a great point.

    Thanks for taking the time to add additional information here. I’d love to develop a standardized structure that can be beaten on and revised over time (and, potentially, even expanded to include the various “warehouse” capabilities, but I’m no where near having the experience across multiple tools to even do a 5-minute comparison).

  7. Pingback The 8 Best Web Analytics Tools of 2011 | Single Grain Blog

  8. Translationfiles are uploaded through FTP in WT.

    Main difference between WT, GA & SC is that WT is based on log files and thus you can reanalyze data back in time with any report you build. Youcan however not segment your data on the fly, sine it is not built on a relational db, like GA and SC. This however also mean that GA and SC sample data when going above 500k visits. Thus looking at a small data slice from a large amount of visits becomes inaccurate.with WT you get accurate data, even if you’re looking at 1 billion visits, and some customers are. Reason for sampling is of course that queries into relational DBs take time, the bigger dataset the more time.

    So it is really down to this: do you prefer
    1. Accurate data that you can reanalyze back in time for new reports but without “live” segmentation. You need to predefine segments. OR
    2. Inaccurate data for large data sets, that you cannot reanalyze for new reports but that you can do live segmentation on.

    I am biased, but I would never recommend WT or SC for s small business. If you run a bank with 40 million monthly visits and you need to hand over your SSL certificate in order to track https pages on your webbank, then the choice is obvious. In other cases it is less obvious.

    It think both SC and WT has strengths and weaknesses. It ISA matter of preferences.

  9. Thanks for the clarification on translation files (I’d missed that slight misstatement in the earlier comment).

    And, you’re raising a good point about the underlying data storage/processing. All page tag solutions are “analyzing log files” (which gets people confused when we talk about “log-based” vs. “tag-based” solutions — even tag-based are ultimately doing log file analysis; they’re just doing logs for the image calls that the page tag generated rather than the logs for the site itself). GA definitely hits the “sampling” point much faster than SC. I’ve never gotten a super-clear answer as to exactly when and where sampling starts occurring for SC, though — my impression is that they’ve been ratcheting up the limits so that that is a less common occurrence.

  10. I would not know about SC and sampling. And yes, I mixed things up bait in my post. They all collect data and store them in a logfile. WT does however provide a software version, re analysis. But no live (almost no) slice and dice.

    In WT if I want to see a report filtered by some segment, I would build the report, do a backup restore a month ago and reanalyze the data. I might not get the answer straight away, but I can have as many custom reports as I like for free and I can build more than 20 levels of drilldowns if I felt like it. I would love the freedom of live segmentation but I am not willing to give up my accuracy and flexibility for it. I am sure an Omniture consultant would argue the other way around. Both are good tools with strengths and weaknesses. Google is simply free and quite limited in flexibility.

  11. Good point. WT does offer the software option. The first full-blown web analytics implementation I ever did was on WTOD, and I’ve only worked with the software in the case of a handful of clients (I worked at a major insurance company that ran the software version, but I rarely crossed paths with that data). While the ability to re-analyze historical data (after updating a profile in some fashion) is feasible, there is a cost to do so. I had it done several times with WTOD, and it took a number of days to reprocess (it was more than a week but less than 3 months of data). Anecdotally, I’ve heard users of the software version complain that, while that’s doable, it gets messy and requires having someone on your side on the IT side of things (another downside of the software version that I’ve heard — tell me if this doesn’t match with your experience — is that, because there are so many things that are “free” with that model, it’s easy to get sloppy and have both profile proliferation and “reprocess request” proliferation). And, we’ve got one client that is running the software…but is two versions behind because they don’t have the internal chops to upgrade (that same client has clearly never tuned their table limits — if they ever had staff that was well-versed in WT, that person/people appears to have moved on to other things!).

    Having now worked extensively with several of the tools, and worked with superusers of all of the major ones in one situation or another, I tend to see them all as having strengths and all having weaknesses. GA is easy to poo-poo, but, as Brian Clifton recently wrote, many (if not most) people who claim that the tool is “limited” haven’t actually maximized its capabilities. That’s not to say it doesn’t have limitations, but the limitations aren’t as extreme as is often claimed…by people who aren’t actually power users of the platform.

  12. Thanks for the fantastic deck! The only thing I would update is slide 10 re: ASI slots, since you can do on-the-fly segmentation with SiteCatalyst Version 15.

  13. Coremetrics allow external data integration, used to analyze & segment from individual level.

Leave your Comment


Notify me of followup comments via e-mail. You can also subscribe without commenting.

« »