The Puck Stops Here
Next entry: A New Worst Player This Season
Previous entry: Montreal Canadiens Power Play
A Few Words On +/-
by PuckStopsHere on 11/22/08 at 10:57 AM ET
Comments (5)
There has been a series of posts on the blogosphere about the value of the +/- rating stat. Since I have a longstanding interest in sabermetrics and hockey I thought I would comment on it. The series of posts begin with David Staples of the Edmonton Journal writing on his Cult of Hockey blog. He wrote a post called Why Plus/Minus is a Rotten, Useless, Misleading and Irrelevant Stat for NHL Players. If we go beyond the hyperbole of the title, he makes the point that +/- ratings can be misleading because (like any other statistic) there is a context to the numbers which is not clear from one number alone.
As an example, he writes:
For instance, this year Marian Hossa of the powerful Red Wings is plus-8. But last season, playing with the hapless Thrashers most of the year, Hossa was minus-14.
Hossa is pretty much the same strong player this year as he was last year, but his plus/minus this year would make you think he’s utterly outstanding, while his plus/minus last year would make you think he was a mediocre bum.
The plus/minus stat tells us a lot about the strength of the team that a player is on, but very little about the strength of an individual player. It’s a team stat, not an individual stat, yet it’s used constantly by fans, reporters and commentators as an individual stat.
This is a bit of a strawman argument in that nobody with any hockey sense would ever argue that Marian Hossa was a mediocre bum last season. It is quite clear that the main difference in his +/- rating from one year to the next is the change in teams. That in no way makes +/- useless. It merely means that the context of these numbers is important.
Responding to that post, James Mirtle responds that +/- can be junked as a stat because there are better metrics to judge the value of individual players. He is correct that better metrics exist. He is correct to point to the work of Gabriel Desjardins at behind the net. Desjardins treats */- as a rate stat by using +/- per 60 minutes of 5 on 5 play. He then compares a given player’s plus minus rate when he is on the ice to that of his teammates in order to get a roughly team independent number. He then goes further to establish a context for those numbers by calculating the time averaged quality of teammates and quality of opposition. He cannot give one number to rank NHL players (which is quite likely impossible in a complex continuous sport - as opposed to one where plays are discrete events as in baseball) but he makes a significant step in that direction and better allows us to understand what is going on in the hockey games.
Mirtle argues that +/- being readily published and used in analysis can lead to some ridiculous events. For example he sees the 2006/07 Selke Trophy votes that Thomas Vanek of the Buffalo Sabres received as an example of this. Vanek had a +47 +/- rating and led the league. However, he did not play a defensive role on his team. Largely, Vanek was an offensive player who played on Buffalo’s second offensive line. Usually, Daniel Briere and Chris Drury were on the number one line that drew the opposition’s best checkers. Vanek played very well in his role playing against weaker opposition.
The problem is a fundamental misunderstanding of what +/- captures. +/- ratings are the difference between the goals scored and goals allowed when a player is on the ice (largely in even strength situations). Because they are the only of the well reported hockey stats (GP, G, A, Pts, PIMs) that has any attempt to capture defensive play. However, it does not directly capture defensive play at all. A top scorer on a good team who plays merely passable defence will usually get a good +/- rating. That said it should be possible to isolate players who play well defensively from looking at +/- ratings in the right context.
There is far too much information available in the behind the net +/- analysis to be published in a simple boxscore or in the hockey statistics usually listed in a newspaper. I think the best single number summary available might be +/- ratings. It conveys some information. It is clear that a player with the highest +/- on his team is doing something right and the player with the worst +/- on his team probably has some problems. With a little context, deeper conclusions can be drawn.
I believe that a further step to understand +/- would be in this proposal. I proposed that individual goals against and goals for averages could be calculated for individual players. They could be compared to time averaged goals for and goals against averages for their opponents to give an offensive and a defensive +/- rating that is independent of the quality of opposition. This would be a further step in the understanding of hockey sabermetrics. It is a big job to calculate. In my spare time I made some progress before mothballing the project as the new hockey season got underway. I hope to find the time to return to it someday.
The behind the net analysis can be criticized. Since players are compared to their teammates (their team’s performance when they are off the ice) this can lead to problems. If we assume that the best team in the NHL is made up of 20 interchangeable players who are exactly the same as one another, we can see this. All of these players are the best players in the NHL. Each of these players will have behind the net +/- rankings of zero. There would be no difference between their on and off ice +/- ratings (as the players when they are off the ice are exactly the same as when they are on the ice). Thus it would appear that some lesser player on a different team would have a higher rating.
The question is how often do these situations occur in the NHL today? In a parity filled league the approximation that each team is not different enough from the others to cause considerable problems would be made. That isn’t too poor an approximation in today’s NHL. Nevertheless, David Johnson of Hockey Analysis is a critic. He criticizes the behind the net method, his main example of an argument against it is wrong. He argues that Kris Draper is incorrectly seen as a poor player last season in this method. The problem is that Kris Draper did not contribute much to the Red Wings.
In Detroit, the top line with Pavel Datsyuk and Henrik Zetterberg was very good defensively and usually played against the top offensive unit on their opposing team and scored well in that situation. That is what made the Wings such a successful team. Kris Draper played on a more traditional checking unit and was almost never on the ice alongside Datsyuk or Zetterberg. Since the top offensive unit played against Datsyuk and Zetterberg, Draper played against lesser opposition. Despite that he rarely scored (he has 17 points last year). He was not able to keep the weaker opposition from scoring enough to be too valuable to his team. While there are cases where the quality of teammates influence the on/off ice +/- ratings, they are not as big as this case that David Johnson incorrectly cites.
On the Kukla’s Korner website, we see the successes of the behind the net system. On The Forecheck listed the top MVP candidates based on a system derived from the behind the net system. He lists Alexander Ovechkin as the current MVP this season. That is a bit of a surprise considering Ovechkin is currently 11th in the scoring race. I would pick his teammate Alexander Semin as the MVP at this point. This is largely because he has more points and thus has been more clearly involved in the goals that he has scored - as he was credited with a goal or an assist more often than Ovechkin. It is quite possible that the difference between them in their behind the net ranking is largely due to random fluctuations and nothing meaningful at this point. It is necessary to note that Washington is likely a team that would lead to its stars having high behind the next rankings. They have a lack of depth. As such, the stars will be better than their teammates by a larger margin than players on deeper teams such as San Jose or Detroit.
Forechecker goes on to list the worst players so far this season by his metric. This group contains a large number of checkers. Players such as Zdeno Chara, Jay Bouwmeester, John Madden, Sami Pahlsson and PJ Axelsson, who are consistently lined up against the best offenses in the NHL. These player’s jobs are to hold the scoring rate of these stars down and while doing that have not scored much themselves. It is the tough quality of opposition that keeps their ratings down. I think the truly worst players in the NHL are known to their teams and protected from tough assignments. Thus they would have a hard time getting ratings as poor as these players on The Forechecker list. I submit that Brett Lebda has probably been the worst player in the NHL at this point - though his lead is small.
Is +/- a misleading stat? It can be without the context. At the same time goals can be a misleading stat without the context. In 2005/06, Jonathan Cheechoo of the San Jose Sharks led the NHL in goals with 56. There is no reasonable way to argue he was the best player in the NHL that season. In fact, the MVP was won by teammate Joe Thornton who only scored 29 goals. Would anyone argue that this makes goals a useless, misleading stat? I hope not. Anyone with some hockey sense would realize that a main reason Cheechoo scored so many goals is from Thornton setting him up. Cheechoo’s league leading goal total is a testament to how well Thornton played.
+/- can be a very useful stat. It is a starting point to some successful sabermetric analysis of hockey. Likely, further developments in hockey sabermetrics will come from a +/- direction. It requires a context to make sense of the number. Unlike some of the other numbers we see (such as goals scored) that context is a little more complicated so we often have less of an intuitive feel for it, but that doesn’t make it a poor statistic. It is very valuable and I have enriched my understanding of hockey by studying it.
Filed in: | The Puck Stops Here | Permalink
Tags: Joe+Thornton, Jonathan+Cheechoo, Kris+Draper, Marian+Hossa, Thomas+Vanek,
Comments
Your problem is one of signal to noise. There are plusses and minuses that do not reflect a given player on the ice, but that does not preclude a meaningful signal existing under that noise. Because of its meaningful signal +/- has significant value. You want to throw out the baby with the bathwater.
I would argue Vanek being +47 was a meaningful signal. It didn’t show that he is a two-way player as your incorrectly suggest - it showed he was very dominant against the opposition he faced. The issue is the strength of that opposition. It wasn’t particularly strong. That is an individual conclusion - it shows that there is both individual and team information in +/- ratings.
Posted by PuckStopsHere on 11/22/08 at 05:16 PM ET
While there are cases where the quality of teammates influence the on/off ice +/- ratings, they are not as big as this case that David Johnson incorrectly cites.
Any team that does not change up lines a lot have the potential to have their on/off ice ratings biased in some way, and with teams where they have significant top end talent that always play together, like Detroit, the bias can be misleading. Why should Draper get penalized for playing with Dallas Drake and not playing with Zetterberg? Don’t get me wrong, the behind the net ratings are significantly better than straight +/-, but it still has flaws.
The forwards Draper played the most against were Boyes, Stempniak , Tkachuk and Kariya, fairly good offensvie forwards, and in approximately 33 even strength minutes against each he didn’t give up a goal. The next 2 forwards he played the most against were Andy MacDonald and Patrick Kane, and he held both of them to below their normal scoring rates (well below in the case of Kane). In fact, most players he played against he held their offensive production well below their normal offensive production. He was also a key component of one of the NHLs better PK units. Knowing this you have a real difficult argument convincing me he is one of the worst players in the NHL last year. He wasn’t very good offensively last year but he was very good defensively which is exactly what he was asked to do. My rating system gave him a 1.29 defensive rating which is pretty good.
On the Kukla’s Korner website, we see the successes of the behind the net system. On The Forecheck listed the top MVP candidates based on a system derived from the behind the net system. He lists Alexander Ovechkin as the current MVP this season.
Though I haven’t updated my list in a couple days, so do I. Look at overall contribution at http://stats.hockeyanalysis.com/200809/top20_ratings.php. The main reason Ovechkin comes up as more valuable is because he plays 23 minutes a game vs Semin’s 19.
I give Lebda an overal rating of 0.58 which is pretty bad. I don’t explicitly keep track of who is the worst but Lebda is by far the worst on the Wings. Taking a quick look around I have Jeff Tambellini, David Jones and Jamal Mayers at 0.55 so they may be contenders as well.
Posted by David Johnson on 11/23/08 at 10:29 AM ET
David.
You often speak of the rankings your system would give players, but you have never (to my knowledge) explained all the nuts and bolts of your system.
Can you please give me details to allow me to calculate using your system and criticize or praise its methodology?
Posted by PuckStopsHere on 11/23/08 at 11:55 AM ET
I may one day when I get time and to add a few refinements to the methodology. But my method, in my opinion, takes it to the next level. You start with straight +/- which we all agree is useless because bad players on good teams can have better +/- than good players on bad teams. Behind the Net tries to remove team effects from +/-. But, as I point out, not all players on a team play with the same quality of linemates so while it is better it isn’t perfect. What I do is take it to the next level and instead of using team stats to fix +/- I use individual stats. Instead of looking at the team a player plays on, I look at the players a player plays with.
So, in evaluating Kris Draper, I will look at every player than Kris Draper plays with as well as every player he plays against. When calculating defensive ratings I will look at goals against. So basically I will calculate the ratio between his teammages goals against per 20 minutes when he plays with Draper - when he is not playing with Draper and I will sum that up over all teammates weighted by ice time played together. I will then do something similar for all the players Draper plays against but use opponents goals for per 20 minutes instead of teammates goals against per 20 minutes.
So, if opponents generally score fewer goals when they are playing against Draper than when they are not and team mates generally give up fewer goals against them when they are playing with Draper than when they are not, the Draper will get a defensive score above 1.00.
This to me seems much more reasonable than just comparing him to his team statistics, particularly when a player may not play very much with several members of his team or may play against different quality opponents, as is often the case in teams that use set lines and like to line match.
I also attempt to factor in penalty kill stats to defensive ratings and power play stats to offensive ratings.
What I still need to do, which is probably the most difficult thing, is attempt to factor in goaltending. Right now players on teams with outstanding goaltenders, particularly those that primarily use one goalie (like Vancouver) will likely have a positive bias in their defensive ratings.
Posted by David Johnson on 11/23/08 at 01:10 PM ET
Add a Comment
Please limit embedded image or media size to 575 pixels wide.
Add your own avatar by joining Kukla's Korner, or logging in and uploading one in your member control panel.
Captchas bug you? Join KK or log in and you won't have to bother.
Most Recent Blog Posts
The Flip Side To Best Teams Pre-Lockout
Future Hall Of Famer Brendan Shanahan Retires
About The Puck Stops Here
The Puck Stops Here was founded during the 2004/05 lockout as a place to rant about hockey. The original site contains over 1000 posts, some of which were also published on FoxSports.com.
Who am I?
A diehard hockey fan.
Why am I blogging?
I want to.
Why are you reading it?
???
Email:
Donate to Kukla’s Korner
With respect, all the context in the world doesn’t make plus/minus a useful stat.
It will always be problematic for the simple reason that it awards a plus to every player on the ice when a goal is scored, even if that player played no part in the goal being scored. He might have just stepped on the ice. He might never have touched the puck. He might even have made a mistake, but some other fellow made a great play and the goal was scored.
This isn’t a minor problem with the stat. I estimate that about 40 per cent of the time a player is assigned a plus but has had no hand in the goal being scored. At the same time, about 40 per cent of the time, a player is assigned a minus and yet he’s made no mistake, or he’s had no impact on the play, or he’s actually played extremely well but some other mistake by another play led to the goal being scored.
This is my major issue with plus/minus, and it’s the main reason the stat has little use, whatever context you want to put in in.
Plain and simple, it’s a team stat, not an individual stat.
Some weight can be attached to the statement, “The Red Wings were +40 at even strength over the season.” We know, for sure, that the Red Wings were strong at even strength by this measure.
But when someone says, “Vanek is a strong two-way player because he was +40,” almost no weight should be attached to that statement. We have no idea how much Vanek had to do with the goals being scored (save for the ones he scored himself or got an assist), and we have no idea whatsoever what role he played in the goals scored against when he was on the ice.
It’s a complete mystery, which plus/minus obscures by giving him credit and blame in a mechanical and error-prone fashion.
Posted by David Staples from The Cult of Hockey on 11/22/08 at 05:08 PM ET