Counting the Baskets

Free Agent Model Check

Given that the free agency period is winding down I decided to check in on the performance of the free agency models I built. Using data from the market over the last three years I built two different models to predict the average annual valuation of the contracts for this year’s free agent crop. One is a regression model. The other is a Bayesian machine learning variation of a random forest model.

The primary factors in both models at predicting the AAV were Win Shares, Age, Usage, cap spike and playing time. For both models I used the Percent of the Salary Cap for the first contract year as the target variable. The benefit of the regression model is that it gives straight forward coefficients, while the ML model gives the “importance” of each variable.  However, the R stats package I worked with also provides a partial dependence graph that gives an idea of the shape and direction of each variable’s influence in the context of the model. (Context of model is important since the partial dependence is shaped by the other variables included in the model as well as the sample being modeled)

Below are a couple of the more interesting variables in the ML model.

 

The age variable, for example, shows age with little effect on the percent of cap on the player’s contract until he hits twenty-nine, and then it declines quickly.

 

 

 

 

 

 

 

 

 

 

 

 

 

Win Shares also shows a nonlinear pattern in the model, taking off at around two.

And while minutes played looks like it’s relatively linear, being a starter takes one jump right around 41 games started, then stays flat.

 

In addition to learning about the free agency market, part of my motivation was to expand my modeling skills and experiment with ML model. And, of course, I was interested to see which one would get better results out of sample. In terms of measuring overall success I used a simple mean absolute error (MAE), which is the average error regardless of the direction of the error. The ML model has so far slightly out performed the regression model, with a MAE of 3.4 million dollars for the regression and 3.3 for the ML. But, as it turns out the error of the models averaged together is slightly better than the either at 3.2.

In an overall perspective the error on simply guessing that every player gets the average contract is 7.7 million dollars per player, so the models net a decent improvement.

But it does look like there are some systematic errors between the model and this year’s market. To start, so far the model has overestimated the contracts of centers and underestimated the contracts of point guards on average. Below is the blend of the two models plotted by position.

Whether that is a part of the league’s continuing evolution, or a reflection on this year’s free agent group is tough to say.

I then looked at the residuals compared to out of sample individual statistics I found that Usage was still undervalued and age and blocks were overvalued. Though 40 year old Vince Carter’s one year $8 million deal seems to be more or less responsible for age affect.

Lastly, there are the individual outliers. In the cases where the model is much lower than the player’s contract it’s not clear if it’s a poor projection by the model, or an overpay by the team. Last year there were cases like Timofey Mozgov that proved to be a warning of an overpay. However, the models just give a rough baseline of where the market may fall. This year one of the biggest "overpays" via the model was Stephen Curry, who is not only part of the undervalued point guard class, but receiving a Super Max contract that did not exist in the training data. The other two "overpays" via the model are short term contracts that probably are a bit high on a per year basis, but that is purposely mitigated by attaching fewer contract years. JJ Redick and Paul Milsap, The last two could potentially be a bit more concerning for the signing team given the length of contract, Blake Griffin and Jrue Holiday. Both were projected to be about $9 million lower than their AAV, and were given five year deals to stay with teams that had little to no leverage.

The best value contracts (or where the model was the most over), were Luc Mbah Moute and Ersan Illyasova at around $7 million less than projected. The link to the list is attached here.

 

Summer League Rookie Updates

I couldn’t quite wait for Summer League to be over officially, run the numbers to see what, if anything, we can take from the rookies’ performance. But with the Lakers sitting virtually everyone of note for the Vegas Championship, I figured it’s over enough. (Note to the NBA: Vegas Summer League is too long, this is why teams start sitting their lottery talent).

In the link here, I have the full run down of per 40 one number performances via Kevin Ferrigan’s Daily RAPM Estimate (DRE) and Alt WIn Score (AWS), a metric I use quite often in my draft models with data from RealGM. But, to me, the real focus of Summer League has to be the rookies.

Some of that is informed by a Kevin Pelton ESPN Insider article, indicating that Summer League adds a bit of predictive power to rookie performance but not to 2nd year players. It’s also plain that we just have less information on rookies, so new info relatively more valuable as is seeing them in a new team setting.

In order to get a quick and estimate on evaluating the stats out of SL, and maybe providing a bit of perspective, I re-calculated my Rookie performance draft model, first by substituting the SL stats for the college or European league stats and then used the SL stats to just update my original Rookie Model.

First the Summer League Only (SLO) run. The SLO model seems to match the buzz coming out of Vegas very well, with Lonzo Ball on top followed by Dennis Smith Jr, and Jayson Tatum, at two and three. Laker fans might want to frame this table with three Lakers prospects in the top 10.

    

Of course, for reference I should add that the scale of the projections line up so that a 5 is roughly equal to average production, or a 0 in plus/minus terms. Indicating that the SLO model projects every rookie but Ball to be below average next year.

For Celtics fans SLO is more mixed, Tatum comes out the third best, Zizic performed respectably, and uh, Semi Ojeleye was also there.

It is hugely unfair to project Markelle Fultz based on what amounted to about 64 minutes of playing time. Though he was actually helped by the the regression I applied, since he had performed below average in his brief court time.

So then, how much to actually adjust our expectations, if at all? Going again based on the Pelton article, I ran the rookie model with the SL numbers added, weighted at a 25% of the pre-NBA numbers. This gives a much more realistic evaluation to Summer League importance.

For example, in the tables below ordered by the SL Updated Rookie Model (appropriately, perhaps- SLURM) Fultz is still appropriately in the top 2. And the most any player has been adjusted up is three tenths, and down is five tenths (again the model results are scaled so that it’s roughly the same as being projected to be .3 better in a plus/minus model).

And below is the 2nd half of the summer league rookies by SLURM:

Maybe, we can hold off on the Kuzma Rookie Of the Year ceremony and wait to bury Fultz or Zach Collins. At least until the second game of Preseason.

New Celtics' Draftee Stats Compared to Peers

One way to get a sense of player's strengths and weakness is to chart them against their peers. The visual can give a more intuitive feel than numbers by themselves and it's much easier to take in information on a bunch of different players at a time.

I created one for the draft that groups the prospects stats in categories that are mostly independent and represent different aspects of my draft model's evaluation. All the stats are standardized in a z-score, with average players scoring at 0 and the spread between best and worst scaled by the standard deviation. That way the differences in each category  are set to the same visual scale.

The stats are then put in a box plot using position as the grouping variables, so a player's assists, for example is compared to guys at the same position, instead of comparing a power forward's assists to a point guard.

Below is an image of the rebounding category, with Celtics picks Semi Ojeleye and Jayson Tatum's mark in the chart highlighted with commentary.

The data  in the link is interactive so you can change the filters are hover over the screen to see more information on a particular data mark.

Among the highlighted stats for Jayson Tatum, he was:

• A very good scorer, among the best from the wing positions
• An above average rebounder at the three, but not great.
• Distribution was his biggest weakness, one the worst wings in that rating.
• Was above average for his position in blocks and steals rating.
• He was one of the youngest players in the draft, and played a very competitive college schedule.

For Semi Ojeleye highlights were:

• Very good rebounder as a three, combo forward potential
• One of the best wing scorers in the draft
• Weakness in blocks and steals
• Slightly above average for a wing in distribution
• Ojeleye is one of the older prospects in the draft.

 

Retrodictions Are Hard: Especially About Three Point Shooting

It's been almost three years since I posted my models to predict the three point accuracy of players coming into the NBA, a post called "Predictions Are Hard: Especially About Three Point Shooting".  Along with presenting the coefficients for the models, I also gave the results for some of the first round selections from that summers 2014 draft class. (The results for some of this year's class are up at a Nylon Calculus post here).

The models were trained on the shooting percentage first four years a player is in the league, so we're not quite to the time frame to give an exact test, or, retrodiction of those projections. But, it's too close for me not to give a little look in.

To test the models I compared the mean average error (MAE) of  model projections against the MAE of the player's pre-NBA (either college or overseas) three point percentage and the MAE compared to assuming the player is league average. The two models I presented performed virtually identically, both performing better than compared to he player's three point percentage and better than the simple average shooter assumption. The models had a MAE of 3% and 3.1%, while the MAE for the player's pre-NBA three point percentage was 5.7% and compared to the NBA average three point percentage the MAE was 4.2%.

The results for individual players featured in the original post are shown below, with their Pre-Draft three point shooting, NBA performance and the two models projections, then the error for each measure.

Overall this class has not been a good shooting class, with a 32.9% average player shooting percentage. Though slightly higher, 33.4%, if weighted for attempts. The models were slightly biased high compared to the actual performance projecting 34.9%, the player's pre-draft three point percentage was 35.7% and league average since 2002 at around 35.5% coming into this year. The biggest miss was on Adreian Payne, who never developed the three point shooting he needed to establish his game. James Young was a similar miss, under performing his projections in shooting (along with many other areas of the game).

Draft Benchmarks Compared to My Draft Model

Since I started doing draft analysis I have been interested in ways that I could supplement the data I had from the box scores and demographic data to help better predict NBA success. A good amount of this comes from scouts, including high school ranks, draft board rank, and specific trait ratings. That reflects my thoughts that analytic models should be complementary with scouting, not a substitute. 

 

Last year a feature at 538 had an interview with Ed Weiland, a draft analyst with his own site, Hoop Analyst. Weiland developed a benchmark system in various statistical categories with minimum achievement levels based on the player's position. For example, a shooting guard prospect is expected to:

• Hit at least 50% of his two pointers
• Hit 30% of his three pointers
• Score 18 points per 40 minutes
• Get 1.3 steals per 40 minutes,
• Get a sum of 7 in rebounds, steals and blocks per 40
• Have an assist to turn over ratio of at least .8

The rest of the criteria are at the 538 interview. 

 

The basic premise is that the more benchmarks a prospect misses, the more skeptical we should be about them performing in the NBA. The benchmarks are an interesting addition to look at prospects in addition to the models, I believe, as they look across multiple categories rather than adding them up as most draft models do. That lets us look at the the players versatility, or alternatively potential fatal flaws. In the past I have modeled on a few versatility measures, which had minor effects in the context of the model and in the end didn't adopt any in part because of positional biases. 

 

There are a couple of downsides to the benchmark approach compared to modeling. It limits information to yes/no on making or not the benchmark (and doesn't credit elite performance over adequate) and by drops demographic information like age (though Weiland is clear he considers informally)

 

In any case, contrasting the benchmark performance with my model estimate seemed like a good way to explore the current prospects as well as the flaws and benefits of each method. To do this I measured the percentage of benchmarks a player exceeds for his position. For example, a 1.0 indicating the player exceeds all benchmarks, or a .2 indicating he exceeds only 1 of 5 benchmarks. To be clear, this my own adaptation in order to be able to compare the benchmarks to my model results.

 

Generally, it's the cases where my model disagrees with the benchmarks that I think are the most informative. So below I have a chart of the players that are highly rated by model but miss on a couple of benchmarks.

 

 

The chart here visualizes the prospects performance using a standardized measures of statistical performance centered around a zero indicating average performance compared to other prospects.

 

 

 

Eight of the nine players under performing on benchmarks are big men.That likely says something about the difference between the two methods by position. Possibly that benchmarks miss style issues with the modern big man, or that the model is overly enamored of scoring or rebounding specialists. Seven of nine are younger than the average prospect, which is to be expected given that age isn't explicit in the benchmarks. 

 

But one other contrast that stands out is the treatment of specialists like Monte Morris with passing and distribution (the grey bar above), or Tony Bradly and Ivan Rabb with rebounds (the orange bar). The reliance on one skill should act as caution flags, I think for the model's result.

 

Next are the opposite cases, players not highly rated by the model, but hit most of their benchmarks.

 

 

In contrast all of these players are wings, mostly small forwards and all of them are older than average prospects.

 

As the stacked bar chart shows, here the opposite issue comes up for benchmarking, these prospects, for the most part, are competent across the board but don't excel in any skill. In order to be top out at more than a possible 10th man, a prospect probably needs to have at least one elite skill.

 

 

Lastly, there were three players that did well on both the benchmarks and in my model, but were not highly rated in the Draft Express top 100, Josh Hart. Mikal Bridges and Ethan Happ.

 

 

Happ is an interesting case, he puts up numbers across the board. But he lacks the size to play center in the NBA and lacks the shooting to play a modern power forward, much less a on the wing. That, of course, raises the question about whether these particular benchmarks are the best possible for today's game. Neither big man position has any criteria for shooting, arguably free throw percentage could be used as a proxy with a minimum that varies by position. 

 

In any case, the exercise has convinced me that benchmarks are worth looking at more and a useful way to examine prospects, as well as my model results.

2014 Draft Out Of Sample

A couple of years ago I put out my first draft projection in advance of the 2014 NBA draft. Now a few years gone by and I can run a preliminary test for the draft model on that 2014 draft class. The basic idea of the test is to compare the player's actual performance as measured by my minute adjusted AWS efficiency metric in their third and fourth year (just using the third year for this test of 2014 draft) to the rank placed by both my model and the performance of the actual NBA draft order. 

A couple of years ago I tested my draft model for the 2012 draft class, which was out of sample then from my analysis, against the performance of the actual draft. In that case the model out performed the NBA drafters, at least as measured by the metric that I use for my draft modelling.

In terms of average error, without regard to direction, the model predicted slightly better out of sample how well players rank by actual performance this year than the their spot selected in the draft. The average error for the model was 6.9 spaces from their "actual rank," while the draft order error was 8.1 spots. The full table is at the bottom of the page.

I want concentrate first on both the biggest misses and biggest hits for the model compared to the place they were selected in the draft. I think those cases give some of the most interesting lessons on what the model tells us relative to the draft decision makers.

 

Below are the five worst estimates for the model compared to where they were drafted:

 

• Jordan Adams, -21 places worse. my initial model loved Adams, placing him at the top of the class. Unfortunately Adams suffered a major leg injury that ended his NBA career, placing him among the least productive in the draft class. Injuries are a risk of the game and the draft. However, it depends on what question you are trying to answer as to whether you should include player's with career ending injuries in the model. A model on the average expected value of a pick should include them, because that is part of the draft risk. Translating college stats to NBA performance is a dicier question, however. There was nothing in in Jay WIlliams playing stats that would have let you know he would end his career early due to a motor cycle accident. In that case, there is a significant missing variable that is outside the question the model is trying to answer. In any case, Adams' rookie year was middling for a rookie, so it's not clear what stage he would be in career right now had he been healthy.
• Jarnell Stokes, -17 places worse: Stokes is an interesting case. He was a box score monster in the NCAA. But his fit with the modern NBA is problematic as a big man that doesn't excel at protecting the rim or have shooting range. He is under contract with Denver, but doesn't play
• Damien Inglis -18: Inglis was one of the youngest players in the draft and had decent numbers against grown men in France, big factor in his rating. Inglis also suffered a fairly major injury his rookie year. On the other hand he's yet to show significant development since returning to basketball that would lead one to think he warranted a high draft selection.
• Zach LaVine -13 places worse error. LaVine is actually performing better, at least as measured by my metric, than either his draft slot or the model projection as the 7th best of his draft class. LaVine is scoring at a high rate of efficiency and torching from three, both at a higher rate than we saw at UCLA.
• Rodney Hood -12 worse rank error. Hood is also better than his draft spot or the model would have predicted.

On the other side of the ledger, here are the five best model estimates compared to where the were drafted.

 

• Nikola Jokic, +31 in rank error. Jokic is rated as a top player in AWS terms this year. And as a passing young big man was very highly rated by my model. Often player's that dominate box score numbers and show flashes of everything but lack overt athleticism just don't stick around in the NBA. However, an interesting indication from a different study I did looking at "star" players and "busts" with both scout rankings and on court performance was that performance was relatively more important for becoming a "star", while scouting ratings were relatively more important for avoiding becoming a "bust." So, guys that put up numbers but aren't as good on the eye test might be the real high variance plays.
• Clint Capela, +19 in rank error. Capela has turned into a good young big man and was rated as 4th in AWS among this class. It is arguable that Capela is a guy that is overrated by a box score based rating, but he is definitely better than the 22nd selection where he was picked.
• Kyle Anderson, +15 in rank error, Anderson, with the nickname Slow-Mo is kind of the ultimate eye test vs analytics model player. In this case his efficiency rating is between the two measures, but closer to the rating of the model, whether the Spurs development machine plays into it or not.
• Nik Stauskas +14 in rank error, Stauskas  has resurrected his career to extent in Philly. However, he is still not one of the more productive players in his class, ranking 24th via AWS.  In this case, we shouldn't have necessarily trusted the crowd source process.
• Jerami Grant, +11 in rank error, Grant was rated as a late first round talent, which is about what he's been.
• Honorable mention to Adreian Payne, +10, as an older player that didn't dominate in college and looked like a reach in the middle of the first round.

I should note that in this case we're testing the model against the target variable for the training of the model, which gives the model an advantage even out of sample. So I quickly looked at the model vs win shares available from Basketball Reference. In that case the gap was much closer with the average error for the model being 7.2 places and the draft being 7.4.

Below is the full table:

 

Tableau:

 

end

Four Factors Projections

Useful analytics at the start of the NBA season are sometimes hard to come up with. Essentially you have a choice of noting how unprecedented Anthony Davis's start of the season is or how noisy any particular stat is.

There isn't anything wrong with either of those approaches, but I tend to want to estimate how much of this noisy data or unprecedented performance can we take forward. So, I have been experimenting doing in season projections with based on the expected regression using a Bayesian updating formula that weights the amount of noise in the sample against the stability expected in the entire season. The gist of the formula is that the noisier the sample is the less weight we give the new data.

I decided to apply that method to Dean Oliver's four factors, effective field goal percentage, rebounds, turn overs and free trow rate, on both offense and defense for each team as of Tuesday morning via Basketball Reference. I used a team based prior based on the last season regressed toward average based on the season to season correlation.

(See below for the individual team factors and  the expected percent adjustment)

The adjusted expected four factors were then converted into net rating using a formula developed by Evan Zamir. In the table I have each teams current margin of victory, or net rating, and the expected rating after weighting with the prior.

The four factor method has the benefit of being relatively straight forward, but it leaves out  few noisy areas that can especially distort early season success, three point shooting by the offense and even more by their opponents, both of which get hidden in eFG%, as well as my personal favorite, free throw defense. But the results above at least have air of reasonableness, at least for a sample based on three games for most teams, well the Warriors aside.

Individual four factors as of Tuesday, November 1 and the percent adjustment expected by the end of the season according to the model.

PT-PM numbers for 2015-2016

I am belatedly getting around to releasing the final Player Tracking Plus Minus (PT-PM) numbers for 2015-2016. PT-PM is my version of a statistical plus minus model, that I have developed over the last few years.

I wanted to have a place to reference the numbers for future posts and put them out there for anyone interested. First a quick visualization, that pits the Defensive PT-PM against the Offensive PT-PM, with the color representing the overall score and the size of each player's marker determined by the number of minutes he played over the season. The visualization is filitered for players with at least 300 minutes. Plus I think it looks kinda cool.

 

Finally, here's the table with the O and D PT-PM numbers as well as the total score adjusted for playing time (lower minutes players are regressed more). Table can be filtered by team or minutes played.

 

 

Rookie Projections Visualization

This rookie projection visualization, was originally posted on Nylon Calculus. But the dimensions on the host site aren't quite optimized for Tableau visualizations, or at least not the dimensions I am using, so I am giving it a try here to see if it's a bit more readable:

 

 

The basic explanation is at the Nylon post, but in short I tried to highlight a couple of different inputs to the model for projecting end of the rookie contract production from last year's rookies, age, efficiency, time on the court and pre-NBA efficiency.

Stacking Up the Celtics Picks

I went through a brief exercise to annotate the stats for the players picked by the Boston Celtics in this years draft using the same basic stats visualization I debuted prior to the draft. The added annotations bring those prospects into focus, and makes it easier to follow them across the different categories.

The main upshot is that Jaylen Brown's numbers at Cal were bad. We that already, but the categories bring it home, most notably on the distribution measure. None of this means Brown is doomed as a prospect, certainly, and as a Celtic fan I will be rooting for him and not for my models to be correct. But it indicates that him finding stardom would buck the historical trends, and reasons for optimism have to come from some where other than his performance at Cal

Other prospects:

• Guershon Yabusele, overall my projections like him much more than the consensus did, ranking the selection as less of a reach than the common reaction. Yabusele was rated close to average in nearly every category compared to other power forward prospects but is given a boost due to age and a higher competition level he faced against grown men in France.
• Ante Zicic was picked near his consensus status before the draft and close to where my model rated him. Unlike Yabusele, Zicic has a good deal more variance in his ratings, coming up higher in scoring and rebounding and lower in distributing the basketball.
• Demetrius Jackson fell quite a bit from his pre-draft ranking and was rated fairly well by my model. Again Jackson isn't a standout in any one area but displayed a solid overall game.
• Benjamin Bentil's one standout skill was scoring, but he below average witch everything else.

The scores are all standardized to average being zero with each category weighted to reflect the values in the model.