Since I started doing draft analysis I have been interested in ways that I could supplement the data I had from the box scores and demographic data to help better predict NBA success. A good amount of this comes from scouts, including high school ranks, draft board rank, and specific trait ratings. That reflects my thoughts that analytic models should be complementary with scouting, not a substitute.
Last year a feature at 538 had an interview with Ed Weiland, a draft analyst with his own site, Hoop Analyst. Weiland developed a benchmark system in various statistical categories with minimum achievement levels based on the player's position. For example, a shooting guard prospect is expected to:
- Hit at least 50% of his two pointers
- Hit 30% of his three pointers
- Score 18 points per 40 minutes
- Get 1.3 steals per 40 minutes,
- Get a sum of 7 in rebounds, steals and blocks per 40
- Have an assist to turn over ratio of at least .8
The rest of the criteria are at the 538 interview.
The basic premise is that the more benchmarks a prospect misses, the more skeptical we should be about them performing in the NBA. The benchmarks are an interesting addition to look at prospects in addition to the models, I believe, as they look across multiple categories rather than adding them up as most draft models do. That lets us look at the the players versatility, or alternatively potential fatal flaws. In the past I have modeled on a few versatility measures, which had minor effects in the context of the model and in the end didn't adopt any in part because of positional biases.
There are a couple of downsides to the benchmark approach compared to modeling. It limits information to yes/no on making or not the benchmark (and doesn't credit elite performance over adequate) and by drops demographic information like age (though Weiland is clear he considers informally)
In any case, contrasting the benchmark performance with my model estimate seemed like a good way to explore the current prospects as well as the flaws and benefits of each method. To do this I measured the percentage of benchmarks a player exceeds for his position. For example, a 1.0 indicating the player exceeds all benchmarks, or a .2 indicating he exceeds only 1 of 5 benchmarks. To be clear, this my own adaptation in order to be able to compare the benchmarks to my model results.
Generally, it's the cases where my model disagrees with the benchmarks that I think are the most informative. So below I have a chart of the players that are highly rated by model but miss on a couple of benchmarks.
The chart here visualizes the prospects performance using a standardized measures of statistical performance centered around a zero indicating average performance compared to other prospects.
Eight of the nine players under performing on benchmarks are big men.That likely says something about the difference between the two methods by position. Possibly that benchmarks miss style issues with the modern big man, or that the model is overly enamored of scoring or rebounding specialists. Seven of nine are younger than the average prospect, which is to be expected given that age isn't explicit in the benchmarks.
But one other contrast that stands out is the treatment of specialists like Monte Morris with passing and distribution (the grey bar above), or Tony Bradly and Ivan Rabb with rebounds (the orange bar). The reliance on one skill should act as caution flags, I think for the model's result.
Next are the opposite cases, players not highly rated by the model, but hit most of their benchmarks.
In contrast all of these players are wings, mostly small forwards and all of them are older than average prospects.
As the stacked bar chart shows, here the opposite issue comes up for benchmarking, these prospects, for the most part, are competent across the board but don't excel in any skill. In order to be top out at more than a possible 10th man, a prospect probably needs to have at least one elite skill.
Lastly, there were three players that did well on both the benchmarks and in my model, but were not highly rated in the Draft Express top 100, Josh Hart. Mikal Bridges and Ethan Happ.
Happ is an interesting case, he puts up numbers across the board. But he lacks the size to play center in the NBA and lacks the shooting to play a modern power forward, much less a on the wing. That, of course, raises the question about whether these particular benchmarks are the best possible for today's game. Neither big man position has any criteria for shooting, arguably free throw percentage could be used as a proxy with a minimum that varies by position.
In any case, the exercise has convinced me that benchmarks are worth looking at more and a useful way to examine prospects, as well as my model results.
