Recency Bias, Sample Size, and the Pacers
USA Today Sports
Despite their win against the effortless and tanking Pistons last night, the Pacers are in a free-fall. The once Eastern Conference Finals rival-apparent to the Miami Heat have had a -1.9 efficiency differential since the All-Star Break. They were at one point the stingiest defense of the modern era, but have regressed back to normal top of the league levels. And the offense has fallen off a cliff. The stat has been well publicized, but is worth mentioning again: only the Sixers–essentially a D-League team–have had a worse offense post-All-Star Break.
I might write off the Pacers, if not for the “post-All-Star Break” part of that sentence. Remember, over the first 52 games of the season Indiana had a +8.6 efficiency differential, a solid mark for a real title contender. How do we weigh 24 games–less than half the first sample size–of bad basketball. Does the fact that the bad basketball happened closer to the playoffs actually matter? The Bulls are a contrasting example. They were in the negative for the first 52 games of the season, and have been a +4.9 per 100 possession team in the following 23 games. Are the Pacers a team that was unsustainably hot and are now regressing to the mean? Or did the first 54 games indicate their “true talent”, this smaller sample of games being a flukey cold stretch? Are the Bulls a team that played at the mean for the first two thirds of the season and are now unsustainably hot? Or are they finally finding their stride after a draining start?
The answer, of course, lies somewhere in the middle. The Pacers weren’t going to be the best defense ever, but Paul George isn’t just a 40% shooter either. D.J. Augustin isn’t this good, but Joakim Noah is usually better on defense than he was over the first few months of the season. Understanding the concept of “true talent” can be a tough task in the recent event based world of sports.
And that brings up an important question: In the NBA, do events that happen more recently carry more weight in determining a team’s true talent, even if they come in a smaller sample size? To test this, I dug into regular season splits data and teams’ performance in the playoffs*.
*It’s important to note that playoff performance isn’t the best indicator of true talent. But it’s the only out of regular season sample data and it is what really matters; at every turn one encounters the “will the Pacers figure it out in the playoffs” question.
The conclusion? Sample size trumps recency.
In fact, how teams perform after the All-Star Break adds almost no predictive ability (Kevin Draper noted this as a strong argument to shorten the season). Here’s it broken down month by month too:
February suffers because of a smaller sample size, but March is essentially equal in predictive ability to November and December. And what’s up with January? I’m still trying to figure out why, but net ratings in January have the lowest correlation to net ratings in any other month.
I tested out all sorts of dummy variables–a team’s best month being November, worst month being March etc. None proved significant.
I looked at it from another angle too. First, I calculated a season “trend” for each team. The closer to one, the more a team has improved as time has gone on and the closer to negative one, the more a team has gotten worse. I then calculated expected playoff win percentages based on a team’s full season efficiency differential. I then looked for a relationship between teams over and under performing in their expected win percentages in the playoffs and the direction to their regular season. Nothing.
Additionally there’s no trend looking at playoff over/under performance and the difference between post and pre All-Star Break efficiency differential.
So this is all good news for the Pacers. But historical data is limited, and they might be a huge outlier. Let’s compare the Pacers to a few other teams that have suffered huge fall-offs between season segments in the past few years.
This is anecdotal evidence, but it says the drop off doesn’t ring a death knell for the Pacers. Still, the Pacers’ fall is fairly unprecedented, and it’s tough to get a good projection on such an outlier. An especially concerning factor is that this slide is largely because of the widening of what was before the only chink in the Pacer’s armor–their offense.
The historical data says that sample size trumps everything, and that if the Pacers were a really good team for two thirds of the season, they’re probably a pretty good team. But again, this sort of slide for a playoff team is unprecedented and how Indiana fares in the playoffs might not fit the pattern of the past. Only two more weeks until these questions can begin to be answered.