The Dallas Stars love fancy stats. The various articles written about the depth of their burning desire to consume and use any and every analytical device designed for hockey over the past week are full of interesting tidbits.
"There's amazing stuff in the blogosphere. We sit down all the time and analyze it. Lindy and I are on the plane all the time and looking at this stuff - we look at it and track it to see if there's something there. Like I said, we're all very competitive, and we are all looking for the edge. And whatever's gonna help us is great for the game."
Travis Yost first had the very quality interview with Jim Nill that showed how much material the Stars consistently devour. Michael Florek of the Dallas Morning News added more specific details about what exactly the Stars do. It didn't fully go into what the Stars do because there is no reason whatsoever for the Stars to fully tell anyone what they do. It's a very interesting piece though, and if you haven't read it you should.
The piece is a bit confusing though.
"It always came down to, ‘How reliable is the information?’ " Nill said. "That’s always the question mark."
Early on Nill is quoted questioning the reliability of the information, but we have to be careful that we know what he and his staff are questioning. The next passage sheds more light.
The numbers flag certain elements the Stars staff might miss. From there, the staff examines the aspects further and comes to a decision that could determine a player's future.
"If it's a number you get out of right field, I think it's great for analyzing that," Nill said. "Why such a difference? If the numbers are closer, now you've got more of a comfort level to say, ‘OK, yes. There's something here. We made this decision as a staff and the numbers back it up.' It's a lot easier to make a decision then."
The Stars are not questioning the validity or reliability of statistical analysis. In fact, for the most part, no team in the NHL is going to do that. Friend of the internet Hope_Smoke relays this note from Bob McKenzie:
McKenzie "The analytics in the NHL, team by team for the most part, is so far beyond Corsi that it's not even a talking point"— Hope_Smoke (@Hope_Smoke) July 23, 2014
The Stars are very much invested in fancy stats (it was blatantly obvious prior to these revelations), but they want to make sure that the data on which they are making decisions is of the highest quality currently attainable. What the Stars are doing is carefully comparing the "computer" data to their own to give them an extra degree of confidence when it comes time to make a decision that will impact the club.
There is this idea out there that you have to use either statistics or scouting with very little notion of how to use both in concert with each other. You can't coach someone by looking at their shot differentials alone, but you also aren't as effective as you can be if you fail to recognize identifiable problem areas that can be more easily identified through statistical analysis.
What successful organizations do though is marry statistics and scouting. The process described by the Stars is but one of many ways an organization could marry these evaluation methods, and it's actually quite simple.
By undertaking this game tracking adventure the Stars are starting with the research-supported belief that the metrics they are tracking carry weight. If you're skeptical you have to keep in mind the McKenzie note that teams aren't at this point. They're more focused on refining and evolving their analysis than questioning the validity of analysis as a valuable expenditure of time.
The Stars start with a game and they essentially track it twice. The coaches manually track the game and whatever program they use, which is most likely based off of NHL Play by Play reports, tracks it a second time. This is the fancy stats version of checks and balances. This step increases the Stars degree of confidence when the data they are tracking matches.
When it doesn't match the Stars have an easy fix. As an example lets use scoring chances. Say the program records the scoring chances as 14-8 during a game, but the Stars record them as 10-6. The difference is significant. The Stars first can easily go back and double check the chances the program recorded to see if they should have been recorded. They could easily decide to adjust their own records, or ignore some of the chances recorded by the program. The objective is synthesizing the information into something useful.
Once they have the data synthesized the coaching staff can do their job like they always have, but now they have hard evidence that can help them more clearly define practice and game objectives. There is a difference between telling someone how to do something and showing them why they need to. The hard evidence satisfies both scenarios.
For those of you who are more visual, here is a flow chart to make the process seem more tangible.
The question becomes "Does this matter?".
The answer is a resounding yes.
Another way to think of this system is like a standardized test. Standardized tests have any number of problems in education, but they work well here because we know the results actually measure what they are supposed to measure.
The data the team gets from this model is like what a teacher would receive from a standardized test. It tells the coach the results of the game in very specific areas they want to track. Those results are then related to a set of standards that the team holds themselves to. They don't necessarily need to be written down in a formal way, but teams know how they are expected to play and are reminded of it constantly. The system makes it very easy for a coach to see what standards a player isn't meeting because the statistics they tracked show it clearly.
For instance, this is a very generic representation of what I am talking about.
This is a pretty generic idea of what a defensive standard could look like if someone did want to write it down. The standards are on the left, and the results on the right are some of what you can expect to see from a team playing solid defense. If the results are poor the coaching staff can easily break down film to see why the problem is occurring.
And that is the key to all of this. The statistical analysis the Stars are connected with easily fits within an existing structure as an extra data source. It doesn't replace coaching, much like standardized tests don't replace teachers. A student very likely can't just study for the test without taking a class and pass much like a player very likely can't just be told "your Corsi sucks, fix it". The coaching staff is still responsible for providing the "why" and the strategic decisions necessary to correct the identified problem(s).
It isn't always going to be that smooth, mind you. Nill explained further in the Yost article:
The Stars don't have a perfect system down, and it would be shocking it anyone did. It will take time for teams to figure out exactly how they want to integrate statistical analysis into the way they do business, but the Stars under Nill are clearly pretty deep into the process and they believe in the fanciest of fancy stats.
God bless them.