Key Takeaways
There are six reasons why attractions should not use review site ratings in lieu of a traditional guest satisfaction survey:
- Annual and monthly sample sizes are too small
- Review site respondents are not randomly selected
- The mix of respondents doesn’t reflect your customer base
- Overall scores don’t reflect your recent performance
- There isn’t enough breadth in the data to make it actionable
- Review site scores can (and sometimes should) be manipulated
Overview
I’ve been asked many times whether it is OK to use TripAdvisor, Google, or Yelp scores in place of traditional guest satisfaction research. After all, these ratings represent unvarnished, unsolicited feedback from consumers and it’s how people are talking about your attraction in the marketplace. How could it be wrong?
I’ve been monitoring customer review site scores for attractions for over a decade. I’ve correlated the results against actual guest satisfaction survey results, and I’ve talked to a number of research professionals. My takeaway is that you should not, under any circumstances, be using review site scores as a proxy for traditional guest satisfaction research.
That isn’t to say review sites like TripAdvisor, Yelp, and Google Reviews don’t serve a purpose. My argument is that these sites are not an appropriate methodology to measure guest satisfaction and that you shouldn’t be using them to measure or contrast your guest satisfaction performance from one period to the next. The results you get will likely be wildly inaccurate.
There are six reasons why review sites such as TripAdvisor, Google Reviews, and Yelp can’t replace a proper guest satisfaction survey.
Annual and Monthly Sample Sizes are Too Small
While it seems like TripAdvisor has a lot of reviews for most attractions, the vast majority of them aren’t current. Broken down by month or even by year, the number of reviews each attraction has is woefully small (from a survey research perspective). Even back in the day when TripAdvisor was more popular (2015-2016) an average theme park received less than a thousand reviews over the course of a year — not terrible, but not nearly enough to measure statistically significant variances over time.
Meanwhile, fewer and fewer people are writing reviews about their visits to theme parks. In 2016, the basket of 35 attractions I monitor received over 34,000 reviews. In 2021, that same group received only 3,100 reviews. That’s less than 100 reviews per venue per year.
Some attractions get more reviews than others — so if you are thinking about using a review site to measure satisfaction, count the number of responses you’ve gotten in the last 12 months. You’ll be surprised.
Review Site Respondents Are Not Randomly Selected
When I conduct a guest satisfaction survey I contact a random, representative sample of recent customers and ask them to rate their experience. I offer them an incentive and I send out multiple reminders. I do whatever it takes to get a random mix of visitors so that my results reflect my visitor base.
Review sites don’t send out invitations or offer incentives. Review writers are self-selected, and I dare say they aren’t “random.”
I’m not suggesting that all review site reviewers have an ax to grind — because many of them don’t. However, compared to regular guest satisfaction research, the ratings among reviewers are significantly more negative. Especially recently. Among the basket of 35 attractions I track, in 2017 only 6% of reviewers rated their experience as “Poor” or “Terrible. In 2021, 32% rated their experience as negative.
What does it mean that a third of theme park reviewers in 2021 say they had a poor or terrible experience? While the theme park experience has perhaps become a little more trying in the wake of the pandemic, I can tell you that, based on feedback from multiple sources, there has not in fact been a five-fold increase in unhappy/dissatisfied guests.
What happened to the scores? Recall that the number of attraction reviewers on TripAdvisor has declined by almost 90% from 2017 to 2021. Sample sizes are small, and respondents are self-selecting.
My theory is that as the number of respondents declined, a higher proportion of those that remain are consumers lashing out through negative word-of-mouth. While these sorts of reviews have always existed, they now reflect a much larger proportion of the total response base.
That isn’t to say that you should ignore the anecdotal feedback reviewers give. Feedback is feedback. But you shouldn’t make the mistake of assuming it is representative of what your customer base as a whole is thinking about your attraction experience.
The Mix of Respondents Doesn’t Reflect Your Customer Base
Very few people understand (or appreciate) how much work goes into ensuring that survey samples are reflective of the populations they are meant to represent. It matters so much…especially when you’re dealing with market segments that have varying levels of engagement (for example, moms in their 30s are much more likely to take surveys than teenage boys — which is a problem if the majority of your guests are teens).
Without careful sample and data preparation, important demographic groups are likely to be under-represented. And measuring the wrong group of people — or allowing a small but vocal group to become overrepresented in your survey results — can lead to very bad decisions.
TripAdvisor, Google Reviews, and Yelp scores do not take into account the demographics of your guest and simply aggregate the scores that are assigned by the reviewers. They couldn’t even if they want to, since they have no way of knowing who your market segments are or how they should be proportioned. The result is that their scores reflect the views of just a portion of your customer base, while other groups — who might even be the majority of your customers — are mostly ignored.
The takeaway here is that the people who fill rate your experience on review websites are probably not representative of your customer base. You run the risk that decisions you make based on the review feedback might not even apply to the majority of your customers.
Overall Scores Don’t Reflect Your Recent Performance
On TripAdvisor, reviewers rate attractions on a scale from 1 (“Terrible”) to 5 (“Excellent”). An attraction’s score is simply the average rating from all of the reviews that have been collected over the last 20 years.
This bears repeating: the TripAdvisor score that appears next to the name of the venue isn’t just based on the last month, or the last year. It’s the average of all review ratings going back to the time TripAdvisor first launched.
This is problematic for several reasons, but mostly because over 75% of your attraction’s rating is based on reviews that were written more than five years ago. Less than five percent of an attraction’s TripAdvisor score is based on reviews written in the last two years.
That means if you’ve made any significant improvements to your venue during the last five years, it is safe to say these changes are not reflected in your attraction’s overall TripAdvisor score.
From an analytical standpoint, there are ways to work around this — you can, for example, capture each reviewer’s score separately and then calculate your own scores for different periods. My own opinion is that it isn’t worth the effort.
There Isn’t Enough Breadth In the Data To Make It Actionable
Good guest satisfaction research has three benefits: First, you get an overall score/rating you can use to measure yourself over time.
Second, you get enough ancillary data to help you understand why your scores are up or down. Overall ratings can usually be traced to specific areas of the business that are struggling. Along these same lines, this ancillary data can be used to determine the key drivers of satisfaction that you need to focus on to be successful.
Third, a good guest satisfaction survey allows you to break down the results by key customer segments so you know who is happy and who isn’t. Some market segments traditionally rate attractions higher than others, and a change in the attendance mix (e.g., more season pass holders, more teens) might be the cause of your overall score going up or down, even though the actual rating by each of these groups remain the same.
Review websites give you scores, but they don’t provide the ancillary data you need to quantitatively monitor which areas of your business are working and which are not. They also don’t provide the demographic data that you can use to determine which segments of your customer base need support.
If you’re serious about measuring guest satisfaction for the sake of improving your guest experience, the data provided by review websites isn’t nearly enough.
Review Site Scores Can (and Sometimes Should) Be Manipulated
There are three different ways attractions can manipulate their TripAdvisor scores. One is OK (approved and supported by TripAdvisor) and the other two are not. But all three make review sites an unreliable source for guest sentiment about your attraction.
The approved method to improve your TripAdvisor score is to respond directly to customers who give you a negative rating. Solve the complaint/problem for the guest, and there is a fair chance they will delete or change their review, thereby improving your average score. TripAdvisor provides you with tools to do this, and frankly, if I put my guest relations/marketing hat on, I would suggest that if you aren’t doing this you should.
The second way is to hire a firm to help you eliminate the bad reviews. I don’t recommend this, but my understanding is that these firms are effective. Many will proactively reach out to reviewers to encourage them to retract their negative reviews, while others will use more underhanded tactics.
The third way to get higher reviewer site scores is to actively push guests who had great experiences to write reviews. This is against TripAdvisor policy, but it’s easy enough to do. Given how few people are organically writing TripAdvisor reviews right now, it can be extremely effective.
My point is that because these scores can be manipulated — and because you should be actively reaching out to unhappy reviewers — you can’t rely on review site scores to reflect how your customers are feeling about your experience.
Review Sites Have Their Place
I’m not in any way, shape, or form recommending that you ignore customer reviews. You should be reading them and you should have someone on your guest relations team responding to them. These scores are public, and people do use them to make decisions about whether or not to visit your attraction.
My argument is simply that review sites are not in any way a good replacement for a solid guest satisfaction survey program. The feedback on review sites is anecdotal, and despite appearance is not representative of the experience you are offering your customers today.
