This was really, really good. You put into words a bunch of ideas that have been bouncing around in my head for a while now.
As a fellow non-expert, your breakdown of immediate, effective, and mobile coverage maps pretty neatly to my own framework for thinking about goaltending. I think goalie skill/talent can be thought of along at least four dimensions:
- Pre-shot positioning aka immediate coverage. Not just where the goalie is standing, but also arm/leg/stick positioning to most effectively cover the available net and potential deflections or redirections.
- Anticipatory positioning/positioning your body to move to cover what's left open by immediate cover aka effective coverage. How well is the goalie assessing the entire rink in front of him, the shooting danger from the puck carrier, but also the possibility of a pass and the danger from that. Do they position themselves well to cover secondary threats or do they start to cheat a bit, opening themselves up to more danger from the shooter?
- Puck tracking/reacting to how to puck moves appropriately aka mobile coverage. On a desperation save, how well does the goalie cover the area of the net the shooter is most likely to hit? On a clear shot, how well does the goalie position their pads to effectively control or redirect the rebound to a safe space?
- Reaction time. Self-explanatory, but I think slightly different from each of the positioning skills. Essentially, how quickly does the goalie move between the three and how quickly do they react when positioning is not enough to make the save.
With individual goalies being varying degrees of good at each. Obviously these aren't totally independent of each other, but I think breaking things down further increases the overlap between each dimension/skill. One skill I've left off, but that I think falls somewhere between all three types of positioning/coverage is baiting, where a goalie may appear to leave an option open to the shooter with intent of tricking them into taking that shot.
Another complicating factor is that for an individual goalie, these skills may vary by body part. Sometimes analysts talk about defensive schemes being more or less goalie-friendly, but I get the impression that these are generalities rather than defensive systems tailored to a specific goaltender's strengths and weaknesses.
And then the last complicating factor is goalie-coaching. Is the goalie being coached to play in a certain way that they have difficulty with? In theory you'd hope each goalie coach recognizes their player's strengths and weaknesses and tries to leverage the strengths to cover for the weaknesses, but I think there's no question that sometimes a goaltender is asked to play in a manner that's not intuitive to them.
I almost feel like to really evaluate goaltenders they need to be wearing like full motion capture suits with sensors at all their joints, or something like that. The next best option is probably some kind of computer vision technology that can translate multiple 2D images into a composite 3D image.
Anyway, great article! It was a fun and informative read and it sparked a lot of thoughts of my own.
I'm with Robbie here and echo his thoughts 100%. What an incredible effort on a topic that just doesn't get enough attention in my view. Not going to say I understand all of it, but immediate pure admiration for the level of effort here at bringing key insights into a tough issue to crack.
Let me offer two personal thoughts in the spirit of further discussion here. First, I've always viewed consistency of performance (I'll sometimes use the word "sustainability" as well) as indicative of a skill. On the other hand, a lack of consistency/sustainability is indicative of something that lies somewhere between pure luck and a skill. This matters in every profession - lawyers, doctors, writers, computer programmers, baristas etc. Repeatability of performance matters to something we think of as a skill.
I made this comment years ago regarding soccer goalies and their wild fluctuations in performance from year to year. That was not well received, but did not move me from my observation. Hockey goalies are similar for me. Are they skillful or somewhat lucky over the long haul? For me and other Avs fans, the question can best be restated as "was Darcy Kuemper good or lucky in 2021-2022?"
This issue ironically came back to me during the NHL Draft this year when ESPN was leaving for a commercial and the host said, "we'll be back and maybe a goalie will be drafted in the first round" - to which the voice of the analytics commentator, Meghan Chayka, in the background said "I hope not." I'm not familiar with the work of Chayka or Stathletes, but it came across as a not particularly favorable view of the skill/luck dichotomy for goalies from someone claiming to have an analytics perspective.
So, I need to be convinced at a foundational level that what goalies do rises to a real skill that can be measured and projected. That's not to say that goalies are not valuable or don't possess skillful elements in the pursuit of their performance that can matter and differentiate e.g. reflexes, positioning etc. I'm also not saying that at their higher more scrutinized level NHL goalies are not consistently better at their performance than say collegiate one or high school ones.
It's whether a valid differentiation over time exists within a level of goalies that I'm interested in understanding. Should a team pursue a supposed "great" goaltender or take a replacement level player and acquire other more impactful assets?
While some goalies can be arguably "better" than others over some period, the fluctuations in their output appear to make future projections - particularly near-term ones - very challenging. This is why it is often said that the Stanley Cup depends on having a "hot" goalie versus a highly ranked goalie during the regular season.
Finally, the economics here are perhaps a great tell. The Jeremy Swayman drama was likely not fueled by Boston being un-analytical or failing to recognize his value or his abilities or even a desire to sign him...but his relative economic value given the general variance in goaltender performance.
Swayman wanted a lot of money for a relatively thin track record of only 132 games. While I haven't examined it for some time, most goaltenders - like those in soccer - seem to not often be awarded big contracts, and some decisions to grant them are quickly regrettable. The market appears to be skittish on the position as well. In any event, the Bruins are now stuck with him - for better or worse - for 8 years at $8MM+ per.
Again, amazing and important work here that I very much hope to continue reading and enjoying. Thanks again for such a thought-provoking topic!
Man. You know too much about goaltending. Total respect, but some of the stuff you were saying was making my head spin. My sister used to be a goaltender, so I used to hear a lot about goaltending, but it's been years...
Forgive me, because I'm not a hockey numbers guy. I love to watch and play ice hockey, but looking at the numbers is something I've never done, so I may sound like a fool for saying this, but it seems like the cork in this bottle is any data on shot trajectory. Does such data exist on an every shot basis? If we can't access any data on shot trajectory, it would seem to me that such a statistic as you're proposing would be entirely impossible.
Baseball does this with their ball trajectory, so surely hockey can do it too, but as far as I know I don't think they actually do. Right?
If shot-by-shot trajectory data is available, I think such a stat as you would like is closer within reach than you would think. It'd likely start with some kind of xGoals based on where the shot is going, angle to the ground, angle to the net, etc., and from there one could ascertain how good goaltenders are at getting in the way of more and less dangerous puck trajectories. I'm not a model builder myself, but such a thing doesn't sound very difficult, based on my experiences with people who are.
You've got a good idea going here buddy. I like where you're coming from. It's just regretful that the data on shots doesn't seem like it exists yet to be able to support this line of thinking.
Yeah, the data doesn’t seem to be available for this kind of thing. I think I’ve heard teams might have it, but I assume it’s all internal.
I understand much of what I’m suggesting is impossible right now. But best case scenario is we get a huge increase in data and can put models like these into action. Worst case scenario is literally the status quo, but at least we can see the shortcomings in the currently available models.
Indeed, but I think we ought to spread this idea as far as our limited platforms can allow. How come baseball can track trajectories but hockey can't? That doesn't seem fair to us as fans. Surely they can make a hockey puck with some kind of tracking chip in it, or use the fancy cameras like they do for baseball. This is not impossible, I don't even think it'd be difficult. It's just not being done.
I can answer this! There is a chip in the puck now, but the margin of error for the location is still something like six inches, which is close enough in the general case, but not for something like evaluating goaltending where the margins of error for the goalie are typically smaller than that. I believe it's the same reason the NFL doesn't use the chip in their balls to determine if a goal line scrum results in a touchdown.
As for using cameras, baseball has the advantage of the pitcher and batter always being in the same place, while shots in hockey can come from anywhere. Plus, baseball has lots of space between the playing surface and the stands to set up cameras. I think a setup using high speed cameras should be possible in hockey, but it would be a lot more complicated that what MLB uses right now.
The other advantage of baseball is that no one is standing between the pitcher's mound and home plate, so there's always a clear view of the ball, compared to hockey where generating traffic in front of the net is often an intentional offensive tactic.
This was really, really good. You put into words a bunch of ideas that have been bouncing around in my head for a while now.
As a fellow non-expert, your breakdown of immediate, effective, and mobile coverage maps pretty neatly to my own framework for thinking about goaltending. I think goalie skill/talent can be thought of along at least four dimensions:
- Pre-shot positioning aka immediate coverage. Not just where the goalie is standing, but also arm/leg/stick positioning to most effectively cover the available net and potential deflections or redirections.
- Anticipatory positioning/positioning your body to move to cover what's left open by immediate cover aka effective coverage. How well is the goalie assessing the entire rink in front of him, the shooting danger from the puck carrier, but also the possibility of a pass and the danger from that. Do they position themselves well to cover secondary threats or do they start to cheat a bit, opening themselves up to more danger from the shooter?
- Puck tracking/reacting to how to puck moves appropriately aka mobile coverage. On a desperation save, how well does the goalie cover the area of the net the shooter is most likely to hit? On a clear shot, how well does the goalie position their pads to effectively control or redirect the rebound to a safe space?
- Reaction time. Self-explanatory, but I think slightly different from each of the positioning skills. Essentially, how quickly does the goalie move between the three and how quickly do they react when positioning is not enough to make the save.
With individual goalies being varying degrees of good at each. Obviously these aren't totally independent of each other, but I think breaking things down further increases the overlap between each dimension/skill. One skill I've left off, but that I think falls somewhere between all three types of positioning/coverage is baiting, where a goalie may appear to leave an option open to the shooter with intent of tricking them into taking that shot.
Another complicating factor is that for an individual goalie, these skills may vary by body part. Sometimes analysts talk about defensive schemes being more or less goalie-friendly, but I get the impression that these are generalities rather than defensive systems tailored to a specific goaltender's strengths and weaknesses.
And then the last complicating factor is goalie-coaching. Is the goalie being coached to play in a certain way that they have difficulty with? In theory you'd hope each goalie coach recognizes their player's strengths and weaknesses and tries to leverage the strengths to cover for the weaknesses, but I think there's no question that sometimes a goaltender is asked to play in a manner that's not intuitive to them.
I almost feel like to really evaluate goaltenders they need to be wearing like full motion capture suits with sensors at all their joints, or something like that. The next best option is probably some kind of computer vision technology that can translate multiple 2D images into a composite 3D image.
Anyway, great article! It was a fun and informative read and it sparked a lot of thoughts of my own.
I'm with Robbie here and echo his thoughts 100%. What an incredible effort on a topic that just doesn't get enough attention in my view. Not going to say I understand all of it, but immediate pure admiration for the level of effort here at bringing key insights into a tough issue to crack.
Let me offer two personal thoughts in the spirit of further discussion here. First, I've always viewed consistency of performance (I'll sometimes use the word "sustainability" as well) as indicative of a skill. On the other hand, a lack of consistency/sustainability is indicative of something that lies somewhere between pure luck and a skill. This matters in every profession - lawyers, doctors, writers, computer programmers, baristas etc. Repeatability of performance matters to something we think of as a skill.
I made this comment years ago regarding soccer goalies and their wild fluctuations in performance from year to year. That was not well received, but did not move me from my observation. Hockey goalies are similar for me. Are they skillful or somewhat lucky over the long haul? For me and other Avs fans, the question can best be restated as "was Darcy Kuemper good or lucky in 2021-2022?"
This issue ironically came back to me during the NHL Draft this year when ESPN was leaving for a commercial and the host said, "we'll be back and maybe a goalie will be drafted in the first round" - to which the voice of the analytics commentator, Meghan Chayka, in the background said "I hope not." I'm not familiar with the work of Chayka or Stathletes, but it came across as a not particularly favorable view of the skill/luck dichotomy for goalies from someone claiming to have an analytics perspective.
So, I need to be convinced at a foundational level that what goalies do rises to a real skill that can be measured and projected. That's not to say that goalies are not valuable or don't possess skillful elements in the pursuit of their performance that can matter and differentiate e.g. reflexes, positioning etc. I'm also not saying that at their higher more scrutinized level NHL goalies are not consistently better at their performance than say collegiate one or high school ones.
It's whether a valid differentiation over time exists within a level of goalies that I'm interested in understanding. Should a team pursue a supposed "great" goaltender or take a replacement level player and acquire other more impactful assets?
While some goalies can be arguably "better" than others over some period, the fluctuations in their output appear to make future projections - particularly near-term ones - very challenging. This is why it is often said that the Stanley Cup depends on having a "hot" goalie versus a highly ranked goalie during the regular season.
Finally, the economics here are perhaps a great tell. The Jeremy Swayman drama was likely not fueled by Boston being un-analytical or failing to recognize his value or his abilities or even a desire to sign him...but his relative economic value given the general variance in goaltender performance.
Swayman wanted a lot of money for a relatively thin track record of only 132 games. While I haven't examined it for some time, most goaltenders - like those in soccer - seem to not often be awarded big contracts, and some decisions to grant them are quickly regrettable. The market appears to be skittish on the position as well. In any event, the Bruins are now stuck with him - for better or worse - for 8 years at $8MM+ per.
Again, amazing and important work here that I very much hope to continue reading and enjoying. Thanks again for such a thought-provoking topic!
Man. You know too much about goaltending. Total respect, but some of the stuff you were saying was making my head spin. My sister used to be a goaltender, so I used to hear a lot about goaltending, but it's been years...
Forgive me, because I'm not a hockey numbers guy. I love to watch and play ice hockey, but looking at the numbers is something I've never done, so I may sound like a fool for saying this, but it seems like the cork in this bottle is any data on shot trajectory. Does such data exist on an every shot basis? If we can't access any data on shot trajectory, it would seem to me that such a statistic as you're proposing would be entirely impossible.
Baseball does this with their ball trajectory, so surely hockey can do it too, but as far as I know I don't think they actually do. Right?
If shot-by-shot trajectory data is available, I think such a stat as you would like is closer within reach than you would think. It'd likely start with some kind of xGoals based on where the shot is going, angle to the ground, angle to the net, etc., and from there one could ascertain how good goaltenders are at getting in the way of more and less dangerous puck trajectories. I'm not a model builder myself, but such a thing doesn't sound very difficult, based on my experiences with people who are.
You've got a good idea going here buddy. I like where you're coming from. It's just regretful that the data on shots doesn't seem like it exists yet to be able to support this line of thinking.
Yeah, the data doesn’t seem to be available for this kind of thing. I think I’ve heard teams might have it, but I assume it’s all internal.
I understand much of what I’m suggesting is impossible right now. But best case scenario is we get a huge increase in data and can put models like these into action. Worst case scenario is literally the status quo, but at least we can see the shortcomings in the currently available models.
Indeed, but I think we ought to spread this idea as far as our limited platforms can allow. How come baseball can track trajectories but hockey can't? That doesn't seem fair to us as fans. Surely they can make a hockey puck with some kind of tracking chip in it, or use the fancy cameras like they do for baseball. This is not impossible, I don't even think it'd be difficult. It's just not being done.
I can answer this! There is a chip in the puck now, but the margin of error for the location is still something like six inches, which is close enough in the general case, but not for something like evaluating goaltending where the margins of error for the goalie are typically smaller than that. I believe it's the same reason the NFL doesn't use the chip in their balls to determine if a goal line scrum results in a touchdown.
As for using cameras, baseball has the advantage of the pitcher and batter always being in the same place, while shots in hockey can come from anywhere. Plus, baseball has lots of space between the playing surface and the stands to set up cameras. I think a setup using high speed cameras should be possible in hockey, but it would be a lot more complicated that what MLB uses right now.
The other advantage of baseball is that no one is standing between the pitcher's mound and home plate, so there's always a clear view of the ball, compared to hockey where generating traffic in front of the net is often an intentional offensive tactic.