Why Do Beer Reviews Change So Much on Different Platforms?

The craft beer ecosystem is vast and varied. With an abundance of fascinating and ever-evolving beer styles, over 5,000 breweries and tens of thousands of specific beers at these establishments, craft beer has never been better.

With any burgeoning scene (be it culture, sports, finances, etc.), metrics play an important role, and craft beer is no different. One of the main aspects of the craft beer sphere are the omnipresent and all-important beer reviews.

These reviews help answer what the drinkers really want to know: What beers are the best? How do some beers compare to others? What is the best beer in a certain style category? More broadly, they want to know “What should I be drinking?”

While there are many ways in which to ascertain this information, the most common is unquestionably the beer review. Beer reviews are the easiest way for consumers to quickly decide what beers they should spend their hard-earned cash on, or trade their rarest whales away for. However, with so many different beer competitions, websites and reviewing methods out there, how do we know which to trust?

Many disparate reviewing platforms exist, but the major ones are as follows: Industry beer competitions (and, by extension, homebrew competitions), independent media reviews and website, app and blog reviews. Each of these methods has its merits, but they are all separate entities with differing methods. While that might not negate their efficacy, it certainly seems to muddy the mash tun with a bloated belly of numbers, scores and reviews.

The disparity between beer reviewing platforms is apparent when comparing single beers, such as Champion's Shower Beer. This Czech Pilsener scored a 94 (out of 100) in The Beer Connoissseur's expert review, indicating an "Excellent" beer. However, RateBeer's user reviews only provided a score of 45, Untappd users gave it a 69 and BeerAdvocate's user reviews offered an 86. That's a wild swing for a beer that was awarded a Great American Beer Fest Gold Medal in the Bohemian Pilsener category!

A major part of the beer review mania that we are currently experiencing stems from uneven style guidelines. Two major ones are the Beer Judge Certification Program and the Brewers Association, which are the most frequently used guidelines. Both have their merits, but because they are vastly different (link to too many styles article), they add to the confusion and oversaturation that is typical of the beer review landscape.

Perhaps the most prestigious beer reviewing platform are the industry competitions, of which there are many. The Great American Beer Festival, World Beer Cup, U.S. Open Beer Championships and the World Beer Awards are just a few examples. Oftentimes these competitions award medals and honors based on beer styles and other categories, but once again, there are drawbacks.

Because the competitions rely on style categories, many breweries shoehorn a beer into a certain category to qualify for that competition – even when a different style might suit it better. Also, breweries often brew one-off creations to compete in certain categories and win medals or accolades, but these beers might never actually see the light of day when it comes to production.

Reviews found on websites, app and blogs are almost certainly the most popular and most populous of the reviewing platforms. Sites and apps such as BeerAdvocate, RateBeer and Untappd provide users with an almost infinite number of beers to review from all over the world, from the crustiest macro crustaceans to the freshest, dankest and stickiest hop-filled IPAs from the smallest microbreweries.

These platforms serve as a stellar example of popular reviews, an important factor that holds a lot of weight. Much like Amazon or IMDB user reviews, these user reviews gauge the temperature of the public’s opinion for various beers. These popular reviews are also usually much more “of the moment.”

Because new craft beers are being released pretty much nonstop, these user-led platforms are far more dynamic and flexible than industry competitions and independent media reviews because of the simple fact that users provide the reviews themselves. This means if a user is planted at a brewery who is releasing a draft-only brewery exclusive, they can quickly and easily add that beer to the site’s database and post their review.

New Glarus Dancing Man Wheat is a standard bearer for creative American wheat beers. This does not mean it is immune to vastly different ratings across reviewing platforms. While BeerAdvocate and RateBeer's users provide near-lockstep scores of 96 and 99 respectively, Untappd users disagree, offering up a score of 75. This could be due to the Untappd user base skewing towards younger drinkers who prefer flashier, more eye-catching styles. The Beer Connoisseur review splits the difference, giving the beer an 85.

In this case, the user has “scooped” any outside media source and provided an exclusive review that might draw other people to that brewery to try the new beer. By that same token, however, there is a drawback to this method: There is nothing stopping people from reviewing innumerable beers that they’ve never actually had. There is no stumbling block in place for users to review 20 beers in a sitting while they are sipping a sugary soda – as opposed to an actual beer.

Independent media reviews seem to be an excellent middle ground between the rigidity of industry competitions and the widely unpoliced user review-led online platforms. Media outlets such as All About Beer Magazine, DRAFT Magazine, Craft Beer & Brewing Magazine and The Beer Connoisseur all have varying approaches to beer reviews. The Beer Connoisseur has a stable of Beer Judge Certification Program-certified judges of National rank or higher, who have years of experience imbibing and judging beers based on the BJCP’s style guidelines.

All About Beer, on the other hand, employs a cadre of craft beer luminaries such as Brewers Association founder Charlie Papazian and Brooklyn Brewery founder Garrett Oliver as judges for its beer reviews.

Both approaches have their merits and drawbacks, and neither one is vastly superior to the other. The Beer Connoisseur review aligns with the BJCP guidelines, which are quite rigid and not as varied as the Brewers Association styles. All About Beer’s reviewing method, while novel and interesting, does feature major craft beer figures reviewing the work of other craft beer makers, which can create an unintentional bias since some are, in fact, business competitors, and the reviewers know from personal experience how hard it is to create a craft beer.

Deep Ellum's Hop Seeker is a rotating Pale Ale that features a different bill of hops every time it's brewed. For the 2015 rendition, The Beer Connoisseur scored the beer an 80. BeerAdvocate's users upped the rating slightly to an 83, but Untappd and RateBeer users were nonplussed -- providing the beer with scores of 74 and 64, respectively. Even for a relatively unknown and unheralded beer, opinions and reviews oscillate wildly depending on the platform in which they were scored.

So what does it all mean? While all of these methods and platforms have their various boons and banes, there is no uniformity at all between any of these reviewing platforms. Sure, they often achieve the same results – just with different trappings – but they come to these conclusions via extraordinarily different ways.

In summation, should there be a uniform, international standard for reviewing beer?

Currently, one does not exist. If one did, would the reviews mean more because each craft beer went through the same rigorous process using the same criteria across all platforms?

While expert reviews (whether employing BJCP or BA guidelines) certainly convey the opinion of a highly-trained craft beer imbiber, popular reviews cover the opinions of the masses, and both of these methods are an important part of the modern craft beer industry.

In the end, having multiple review sources benefits the entire craft beer industry and its consumers because it allows for the dissemination of more information about beer to a larger audience. This increases demand and pushes brewers to create higher-quality beer and more diverse style selections. It’s important for readers to find review sources that they trust that can provide useful information in helping them make the best beer purchases.

What the craft beer world really needs is a platform similar to Rotten Tomatoes or Metacritic that compiles reviews from various different sources and posts them all together in an easy-to-read, well-designed page. That would be a site worth bookmarking.