Subjectiveness of Beer Judging

Mase

Well-Known Member
Trial Member
Joined
Mar 16, 2017
Messages
1,616
Reaction score
1,853
Points
113
In reading the "Commercial Calibration" section of the latest edition of Zymurgy (March/April 2018, page 77) I read significant differences between the 4 "Grand Masters of the BJCP". Specifically in their descriptions of "Aroma" and "Flavor" characterizations. Enough so, that I disappeared down a rabbit hole and put together a crude spreadsheet to see how far off from each other the Grand Masters were. Trying to keep myself from going too far down the rabbit hole, I only focused on the Aroma and Flavor descriptions of each of the judges and found that of the roughly 20 +/- unique characteristic descriptors identified by all the judges combined, they concurred on only about 25% of them. The remaining 75% were a unique characteristic's to the individual judges.

Again, these judges "...have attained the rank of Grand Master in the Beer Judge Certification Program", and they only concur on ~25% of all the characteristics they cumulatively identified.

My wife and I have discussed several times whether we want to submit any of our beers to a competition, be it a local event (County Fair), or a National competition and have (for now), maintained that we just aren't interested in doing so. Not for fear of not getting a medal or being negatively judged, rather it's a hobby that allows us to drink what we want, rather than focus on brewing to get a medal. Don't get me wrong, I applaud those who do like to have their beers judged as that's what a hobby is all about.... you can do what you want with your hobby.

Nevertheless, the results from my crude spreadsheet rabbit hole adventure makes me wonder how non "Grand Master" judges can evaluate a beer, when the Grand Masters concur only 25% of the time.
 
You and I should get together and share some homebrew. One thing is for sure: I’ll like mine and you’ll like yours! :D
 
In reading the "Commercial Calibration" section of the latest edition of Zymurgy (March/April 2018, page 77) I read significant differences between the 4 "Grand Masters of the BJCP". Specifically in their descriptions of "Aroma" and "Flavor" characterizations. Enough so, that I disappeared down a rabbit hole and put together a crude spreadsheet to see how far off from each other the Grand Masters were. Trying to keep myself from going too far down the rabbit hole, I only focused on the Aroma and Flavor descriptions of each of the judges and found that of the roughly 20 +/- unique characteristic descriptors identified by all the judges combined, they concurred on only about 25% of them. The remaining 75% were a unique characteristic's to the individual judges.

Again, these judges "...have attained the rank of Grand Master in the Beer Judge Certification Program", and they only concur on ~25% of all the characteristics they cumulatively identified.

My wife and I have discussed several times whether we want to submit any of our beers to a competition, be it a local event (County Fair), or a National competition and have (for now), maintained that we just aren't interested in doing so. Not for fear of not getting a medal or being negatively judged, rather it's a hobby that allows us to drink what we want, rather than focus on brewing to get a medal. Don't get me wrong, I applaud those who do like to have their beers judged as that's what a hobby is all about.... you can do what you want with your hobby.

Nevertheless, the results from my crude spreadsheet rabbit hole adventure makes me wonder how non "Grand Master" judges can evaluate a beer, when the Grand Masters concur only 25% of the time.
I've read of experiments where psychologists have taken cheap plonk, put it in a pricey bottle, served it to those guys who write the reviews that make or break wines in magazines like Wine Spectator and want to know what happened? The rating they gave the wine was proportional to the perceived cost of the bottle. Even more telling, they ranked it differently depending on which bottle it came from even though it was the exact same two-buck chuck. In a second experiment of its kind, they took white wine, with its own vocabulary of flavors, dyed it red, with a different vocabulary of flavors, gave it to the same set of "experts" and want to guess what happened? They began to evaluate it using the red wine lexicon. I've taken the same beer, dyed half of it brown, given it to my homebrew club for evaluation and seen them rate the brown beer differently than the yellow one.

I've judged and agree with what you say about the individual characteristics, I've had a beer take gold medals in two competitions then be scored a 24 in a third. Judging is always subjective, always. What you can believe, if you can get it, is the ordinal ranking in a flight or a competition. If your beer is first in the flight, it's very likely the best in the flight. I like that Zymurgy publishes those for the simple reason it demonstrates how everyone has different tastes. If I had the time, I'd repeat your analysis over time and see how much they agree using statistical methods but for now, I tend to agree. Judging is subjective where style is a factor. In a purely hedonistic competition, the best beer usually wins.
 
One of the major advantages to entering a competition with trained judges (probably not your county fair) is to get feedback on the beer's strength and flaws.

What I mean is good BJCP judges will be able to provide objective feedback on what could improve the beer. Maybe they will mention some astringency in the finish, and give you ideas for what could cause that. A good scoresheet is much more valuable than a friend tasting it and saying "This is great!" because they try very hard to give objective feedback.

The categories you mentioned ARE the most likely to be subjective while body, flavor, etc are not as likely to be as subjective.

As an example, a guy from another forum about 10 years ago was thrilled with his lagers. And he sent a couple to me to judge. I tried to be polite and nice, but the beer was just not very good. I simply said something to him like "I didn't score it out, but it had a very harsh finish and a bite to it". He said, "oh, yeah, my wife and I noticed that. But isn't the beer good?" Um, no. It would have probably scored a 25 at absolute best. I didn't want to spend 15 minutes judging it, since he wasn't open to any pointers anyway. It was most likely related to water chemistry- a too-high mash pH and a too high sparge pH that created that harshness.

I call this the "Ugly Baby Syndrome". We love our beer because we made it, and it's good! And I think that's fine. But it reminds me of those new parents who have UBS. They may have the world's ugliest baby, but it's theirs, and it's precious and beautiful. Brewers are like that too. With taprooms springing up all over, I've had plenty of bad beer in new breweries but it does sell a little and the brewers are proud of their wares.

If you're happy with your beer and don't want neutral feedback, I don't see any reason to enter a competition. That's what is really helpful, and it's not about winning or losing. Some brewers want to win, and are angry when their beers are judged as "good" and not "exceptional". But a person who wants honest feedback will benefit from a competition.
 
All valid points, person from that state up north.

And I indeed have ugly babies that I wouldn’t change for the world! No judges needed here.
 
They all look like Dwight Eisenhower, except for my grandchildren, who are too cute for words....
(My beers are all special, too!).

I've seen a lot of UBS in my time in homebrewing. My stock answer is, "It's interesting...." (while desperately searching for the dump bucket and something to get the taste out of my mouth) The issue of the variability between Grand Master beer judges is disconcerting but the feedback is honest, anonymous and sometimes hard to take. You may agree or disagree, your choice, some people are blind to or even like DMS (witness the popularity of Miller, which to me tastes like cabbage stew). Where I tend to have issues with judges, myself included, is with styles. I feel beers are styled incorrectly, that there's too much overlap and ambiguity and that there are far too many of them. I'll give an example: There's little difference between a Kentucky Common and a Dusseldorf Altbier, yet they're judged in different categories. The difference in outcome between a Kolsch and a Helles are minute, yet they're different categories. And all the IPAs.... But that's my opinion and I don't run the BJCP. And bias and subjectivity are problems in any judged competition.
 
I made a Best Bitter that scored 24 on my most recent entry and I can't honestly argue with it. I was rather underwhelmed myself but by god I bottled them so they're getting judged! I did a Saison in December that 3 judges all found unrelated and sometimes mutually exclusive flaws in but still did well enough. I'm mostly interested in seeing if I can win some swag, and if not see if there are any objective things I can improve upon.

My wife was actually worried about my emotional state when getting my beer judged, but I'm one of apparently the few that actually want to know if they made horse piss. I can't improve if i don't know I screwed up.

Edit: Was the ginger beer that scored 24, the Bitter scored 26. I actually take issue with the ginger beer because it's really damned tasty, but I don't have a clue what the actual style calls for so that may be the issue. Also there was no grain in it, just a shitload of ginger and other spices so it was probably cheating to enter it in a beer competition.
 
One of the major advantages to entering a competition with trained judges (probably not your county fair) is to get feedback on the beer's strength and flaws.

What I mean is good BJCP judges will be able to provide objective feedback on what could improve the beer. Maybe they will mention some astringency in the finish, and give you ideas for what could cause that. A good scoresheet is much more valuable than a friend tasting it and saying "This is great!" because they try very hard to give objective feedback.

The categories you mentioned ARE the most likely to be subjective while body, flavor, etc are not as likely to be as subjective.

As an example, a guy from another forum about 10 years ago was thrilled with his lagers. And he sent a couple to me to judge. I tried to be polite and nice, but the beer was just not very good. I simply said something to him like "I didn't score it out, but it had a very harsh finish and a bite to it". He said, "oh, yeah, my wife and I noticed that. But isn't the beer good?" Um, no. It would have probably scored a 25 at absolute best. I didn't want to spend 15 minutes judging it, since he wasn't open to any pointers anyway. It was most likely related to water chemistry- a too-high mash pH and a too high sparge pH that created that harshness.

I call this the "Ugly Baby Syndrome". We love our beer because we made it, and it's good! And I think that's fine. But it reminds me of those new parents who have UBS. They may have the world's ugliest baby, but it's theirs, and it's precious and beautiful. Brewers are like that too. With taprooms springing up all over, I've had plenty of bad beer in new breweries but it does sell a little and the brewers are proud of their wares.

If you're happy with your beer and don't want neutral feedback, I don't see any reason to enter a competition. That's what is really helpful, and it's not about winning or losing. Some brewers want to win, and are angry when their beers are judged as "good" and not "exceptional". But a person who wants honest feedback will benefit from a competition.

What's really odd to me is that the "Grand Master" judges are so far apart in their descriptions of Aroma's and Flavors, yet their scores were significantly closer to each other. So you may get a good (and arguably reliable) score, but I wouldn't think their descriptions would be helpful at all. So how could you take away anything on what to improve upon based on those two categories. In other words, If I get a score sheet back, regardless of the score, the descriptions are unreliable at best. Based on the article, I would be very hesitant to change any Aroma or Flavor additions as 4 Grand Masters can only agree 25% of the time (let alone a less than Grand Master reviewing your brew), and if you go and alter your recipe based on a judges opinion, you could take the beer in a completely different or wrong direction.

I might add that in the case of the "Grand Master" descriptors for the Ommegang's Abbey Ale Dubbel, save for one Judge, the added ingredients (from the brewery) of Coriander, Sweet Orange Peel, Star Anise, Licorice Root and Cumin were not even noted as an Aroma or Flavor in their descriptions. Sweet orange Peel was identified in the Aroma (not flavor) by only one of the 4 judges.

I'm not knocking what people do with their beers, rather, I would take the descriptions with regards to 64% of the overall score (or more based on "overall impression"), with a grain of salt...
 
Last edited:
No UBS here , if anything I'm pickier on my own brew than I am on others .
I was rather pleased to have judges call me over to a table to sample a delicious American amber ( mine ) to help sort out the score gap so helpfully pointed out the low levels of diacetyl and slight harsh bitterness that resulted in a lower score but a fair and honest one .

I quite enjoy stewarding at local comps , I get samples of the best and the worst without the palate fatigue and paperwork
 
my only comp ive entered i got one really good score from a beer and another score mid 20s but i valued the non biased feedback for sure;).
 
When I judge, I make an anonymous promise to every entrant: I will judge the beer objectively and provide honest, polite feedback to the entrant. I have my blind spots and biases just like anyone else but I will treat the entrant's beer as I want my own treated and provide the kind of feedback I would want if I'd brewed that beer. More than that I can't promise.
 
When I judge, I make an anonymous promise to every entrant: I will judge the beer objectively and provide honest, polite feedback to the entrant. I have my blind spots and biases just like anyone else but I will treat the entrant's beer as I want my own treated and provide the kind of feedback I would want if I'd brewed that beer. More than that I can't promise.
id be honored to have you judge my brews Nosey i know for sure as per you posts on this site they would be unbiased honest up front judgements.
 
One thing that just occurred to me: I wonder if those judges know what they're judging? When I judge, aside from the style the beer should be, I'm completely blind to the brewer, the ingredients (unless special ingredients are listed), I know nothing about what the beer "should be". If the Grand Masters know what they're judging, they have expectations and the expectations are very powerful influencers.
 
A stock answer of" its interesting" could also be" wow there is a lot going on in this beer."A good friend who is a retired commercial brewer used that one on a Belgian I shared with him. It took a while to finally understand exactly what he meant, "to much flavor malt, infected oxidized awful beer. But it was a subtle well meant correcting point in my early brewing years that was appreciated.
 
I've had some awful store bought beer, even micro brew so I wouldn't get to worked up on judges scores, it can happen to anyone, even the best
 
How do we expect judges to deal with a beer that's been entered in wrong category ?
The question came up recently , I'm not talking about beers that land in the overlap between some styles but for example judges getting a great stout that's entered as a Czech pils .
I'll leave off on neipas and where they fit into guidelines
 
I've had some awful store bought beer, even micro brew so I wouldn't get to worked up on judges scores, it can happen to anyone, even the best
I'm wary of buying imports for this reason , very hard to get many beers in my part of the world particularly those that don't travel well .
Poor Handling can ruin a beer and I've seen pallets of beer left piled outside on 40 + degree days by certain retailers
 
here’s another point, I had a strange one, I brewed a very good beer, perfect everything in the keg, pour, head, taste but bottled 5 gallons from the keg already carbed, perfect sanitation on the bottles then let those bottles set in the garage at about 40F for a week, when we drank the beer, the keg was fine but the bottles had bottle bombs, almost undrinkable from the foam so strange things can happen to the best of beers, especially while traveling, caps can leak, caps can taste funny, a bug can creep in from handling, you have a cold and breath wrong :(
 
How do we expect judges to deal with a beer that's been entered in wrong category ?
The question came up recently , I'm not talking about beers that land in the overlap between some styles but for example judges getting a great stout that's entered as a Czech pils .
I'll leave off on neipas and where they fit into guidelines
How we deal with it? We give them a crappy score then in the comments, tell them it's a good beer but not to style. I like the GABF judging over BJCP in that one respect: If it's a good beer, it gets a good score. Their criterion for goodness is will someone buy this beer.
 
I'm wary of buying imports for this reason , very hard to get many beers in my part of the world particularly those that don't travel well .
Poor Handling can ruin a beer and I've seen pallets of beer left piled outside on 40 + degree days by certain retailers
Yep, my 24 after receiving two gold medals for the same beer.... Same brew, even!
 

Back
Top