Counting the Baskets

Re-Drafting 2012

Retrodicting is a funny word modelers use for testing models against results that have already happened, but were outside of the model training.  The results aren't new, but they're new to the model.

The two draft models I built, the P-AWS model and the % Player model, used the draft from 2002 to 2011 as the training data.  The independent variable was the max Alternative Win Score (AWS) in the drafted player's third and fourth years, meaning that the 2012  draft was outside of the model parameters.  So that draft class is available for out of sample testing, or retrodicting.  

In truth, during the training phase the models were run multiple times with various draft classes dropped in order test the robustness of the model specification and ward against over-fitting the data, ie coming up with coefficients that only fit the player's in the drafts tested.  The results from one class are hardly definitive, but looking at a brand new draft class is still a good trial for a model.

The first issue is that the 2012 draft class has only completed their second season after being drafted, one less than the minimum for the independent variable.  As a result, the biggest issue in the retrodiction is that age is a significant factor in the draft model so younger players maybe expected to perform systematically worse  than projected.  So, the retrodiction looked at both the Actual-AWS and one with one year more year of the aging curve tacked on to the player's 2014 AWS.

There are a couple of different ways to look at the retrodiction to judge the performance of the model.  One is did the model recognize that Draymond Green was a better prospect than Harrison Barnes?

Let's see, Harrison Barnes with a P-AWS of 3.47 ranked 31st in the class and 30% odds of being a quality starter in the % Player model 32nd in the class and Green with a 4.08 P-AWS ranked 20th and a 46%  %Player ranked 16th. So, yes, it did, and Green indeed has performed better in Actual AWS in 2014.  Model validated, right?

Another simple and intuitive, though not necessarily the best, way to look at the performance is a scatterplot and the out of sample coefficient of determination.  There both models do well, especially the P-AWS model performing about as well as in the training data.  The P-AWS model has .394 R² with the actual AWS and a .47 R² with the Aged 2014 AWS.

 The %Player model performs a bit worse by this measure with a .34 R², primarily because the scale of the logit model doesn't capture the separation between Anthony Davis, the scatter dot on the top right above, and the rest of the class.  It still performs better than the best fit line for the actual draft order, which was .2.  But, Anthony Davis is precisely why R² is not necessarily the best measure of model success given that predicting his outlier performance has an outsized effect on the measure.

Since a team can only take one player with each pick I also took a look how the models perform in terms of model rank compared to the player's actual performance rank against his class.  It allows for a more direct comparison with the league's draft order.  For instance, Terrence Jones was projected by both P-AWS and % Player models ranked Jones as the 11th best prospect in the draft, while he was taken with the 18th pick by the Houston Rockets.  Based on his stats this year Jones rated with a 7.68 AWs, 3rd in the draft class after Anthony Davis and Andre Drummond.  So the deviation for the models is 8 spots and for the actual draft it is 15.

For the class the average absolute deviation for each model is 9.6 and the median absolute deviation is 8 places.  For the actual draft the average absolute deviation was 12.5 and the median was 13 places.  

It is probably as informative and more interesting to look at how the models, drafters and actual stats compare on individual players.

 

Biggest hits for the P-AWS model over the drafters are Kyle O'Quin, who has been an efficient player in Orlando and was pegged as a late first rounder by the model, Jared Sullinger, who actually fell due  to his back issues,  Austin Rivers, who the model didn't like but went in the lottery and has not played well, and Terrence Jones mentioned earlier.  

Biggest misses are Miles Plumlee, who the model didn't like due to his age, and Terrence Ross, who is out performing his college stats.

Has Philly Already Won the Lottery?

Originally Posted on Hickory-High on April 17, 2014

The NBA played the final games of the regular season last night.  For fans of the worst teams their long regional nightmare is over with bright shiny ping pong balls as their solace for a season of futility and "quality losses" (Or a series of humiliating nationally televised losses if they are Lakers fans).

 Those ping pong balls will get cashed in when the NBA holds its draft lottery on May 20th, rewarding some team for their inefficacy and suffering, planned or not, with the number one pick.  

The number one pick, of course, affords the team the opportunity to try to identify and take the best talent available that year. Obviously the team with the first pick isn't always successful identifying the best talent and, even if they can the talent available every year varies fairly significantly, one year the best player available is Anthony Davis or LeBron James, the next it's Kwame Brown.

This year the consensus both at the time of the draft and since is that the Cleveland Cavaliers did not identify the best player available, selecting Anthony Bennett ahead of a number of more heralded prospects (most of whom have had more successful rookie seasons).  After compiling and testing a couple of Draft Models based on statistical production, that was not calibrated with the 2013 draft class, I certainly agree.

According to my model Nerlens Noel was the best player available in the 2013 draft got traded on draft night and is yet to play a minute in the NBA, and, if fully recovered from his knee injury, may be the best rookie in the NBA next year. Before his season ending knee injury Noel was putting up numbers that, given his age, projected him as not only the best player available in the 2013 draft, but would arguably put him ahead of the most famous names in the 2014 draft class. On draft night last year the New Orleans Pelicans traded the rights to Noel along with their 2014 pick for Jrue Holiday in an attempt to "Win Now" that has gone perfectly to date.

While there is no consensus top pick this year, generally a bad sign by the way, Joel Embiid is one of the possible number ones this year and as a Center is the easiest to compare to Noel.  The center that actually projects the best in my models is Clint Capela, a Swiss 19 year old playing pro ball in France. The models are based on statistical production using Alternate Win Score (AWS) in the player's third or fourth of their NBA career year as the independent variable.  I chose AWS primarily because it has been shown to be one the least context sensitive efficiency metrics and it is relatively straight forward to calculate.  You can read more gory details here or here if you're so inclined.

P-AWS and % Player

There are two relevant models, the Predicted AWS (P-AWS) model and the % Player model, they're closely related with the only difference being the regression format.  Essentially the difference is that the % Player model is an ordinal regression model based on the chances of the player being a starting caliber player by the end of their rookie deal, the difference is primarily that this model is less influenced by outlier stars and gives the results as an estimated probability. Below are their ratings in the models:

 

The knee injury isn't factored in the model and has to be considered as a caveat, though Noel has claimed the knee is fully recovered. Embiid season was also cut short by an injury, but it is reportedly not expected to limit him in his rookie year.

Based on the stats, via Draft Express Noel is rated as a likely star while Embiid is projected as a solid starter. Interestingly both were lowered in the model projections by playing a low number of minutes.   The overall efficiency ratings were fairly close with Noel's somewhat higher even though they affected the game in different ways. Embiid being better on the offensive end scoring more and more efficiently and assisting more. Noel on the the hand racked up more steals and blocks and committed fewer mistakes with many fewer turnovers and personal fouls (all numbers pace adjusted).

Even though Noel was a freshman the year before Embiid, they are almost exactly the same age, meaning Noels stats were compiled when he was a year younger.  Normally age is a pretty important factor based on studies I have done outside of these models and in calibrating these models.  Of course, famously, Embiid is newer to the game than most twenty year old first round draft picks, though there is not enough data to tell if that is a significant factor in development with only a few examples of big men coming late to the game getting drafted in the first round, one being the great Hakeem Olajuwon and the less great Feb Melo.

Beating the Basketball PhDs

My draft model is currently a work in progress, but, as of the second pass it has beaten the actual GM's doing the drafting in the training period (2002-2011 drafts).  It does so even using the best fit line for the draft order, a logarithmic curve, that gives the maximum value to early round selections. It is important also to say that the draft model is built purely on public information that was available at the time of the draft, namely how old the players were and how they performed at the game of basketball.

Granted, my model measures success as the player's performance as measured by Alternative Win Score (AWS), and it's safe to say none of the B-Ball Professor's running teams in the oughts were looking to maximize AWS,  as Baxter Holmes noted in a recent SportVu article only a few front offices at the time had anyone doing analytics.  But, Layne Vashro has built a model that beats the front offices at selecting talent based on Wins Shares and RAPM.  And Arturo Galleti has built a couple that beat them as measured by Wins Produced, and so on.  The only thing I haven't seen is one built off of Yay Points! that beats the traditional GM's.

In fact it is almost distressingly easy to beat the Basketball PhDs at picking draft picks.  But, as always, there are caveats.

The Rarity of Jeremy Lin

The one thing that retro models have as an advantage of knowing who was drafted, and out of necessity when we build our retrospective models they are based on the players that were selected.  The drafters, of course, did not know this and have to do a filtering process that starts with a great many more potential picks than 60.  

It may be that there are many gems the scouts and front offices pass on that analytics would have revealed, but the truth is that players who go undrafted and catch on with a team after Summer League or a brief stint over seas and have a major impact in the NBA are very rare.  That surprise was a big part of the whole Lin-snaity phenomena.  

For that reason I keep my model to the Top 100 prospects and am somewhat leery of any claims that some completely obscure player with great numbers will be a star player.  The scouts and executives are clearly adding value, even to models that claim to beat them.

This is also seen in that when I include draft order in my model the results on the training set improve, and the coefficients adjust somewhat, as Vashro explained in a great write up of his model a couple of weeks ago. In my current working model when draft position is included age is greatly reduced as a factor, though still significant, and Usage, a measure of how many possessions the player finishes, becomes more negative than it is in the model not including draft position.  That indicates, basically, that the PhDs may have been over valuing volume scoring, which not a surprise.

The proper way for a front office to evaluate players is to combine the analytic approach and the information from the scouts.   In fact, there is no inherent reason a good portion of the private scouting data can't be evaluated analytically.  As qualitative and categorical data is often analyzed in social sciences and business.

 Modeling Frank Kaminsky

That gets to Wisconsin Center Frank Kaminsky, who has sat a top many statistical analyses, but has yet to make a mock draft board or the Draft Express Top 100 that I use as a proxy filter for my analysis.  After his continued strong tournament and efficient 28 point Elite Eight game, that may be about to change.

Kaminsky has definitely put himself in the conversation with the season he's had. Barely played his first 2 years, now a legit NBA prospect.

— Jonathan Givony (@DraftExpress) March 30, 2014

 Jonathan Givony also brought up a separate and intersting point on Kaminsky that may explain part of the reluctance of Draft Boards to add him to their list.  It is very rare for a player in college to receive as few minutes early in their career to go on to be drafted.  But, while playing time is an indicator of ability to play it is not absolute.  Maybe Kaminsky was a later bloomer, or maybe Bo Ryan was a little slow to trust him on the floor.  In any case if the scouts think he's worthy of inclusion, my model will slot him in.