Evaluation of Defensive Projections

Here’s a question I’m curious to answer: What method better projects team defense, stat projections or scouting reports?

One thing I need to answer the question is an objective criterion to measure. I don’t think there is anything out there except team defensive efficiency record. If a team has very good defensive players, they should make more outs than average, and post a better DER. DER is not a perfect measure though. The ballpark has an effect on this, as does the types of batted balls your pitcher allows.

Here’s my method: Start with the league average DER. Park adjust this. Then look at how many groundballs, flyballs, line drives, and popups the team allows. The formula I used, along with the batted ball data from the Hardball Times team page, is this: expected batting average allowed = GB*.237 + FB*.16+ LD*.718 + pop *.01. xDER = 1- XBA. Then xDER/league average is your batted ball adjustment factor.

Now, let’s look at the projected defensive stats of the players on your team. I’m using preseason projections, but prorated to the actual 2008 playing time. For the stat projections, mine were made available shortly after the 2007 season on my site here. For scouting reports, I’ve converted the results of Tango Tiger’s scouting report by the fans into run values.

I need to use custom weights of the report’s attributes for each position. For 1st basemen and outfielders I ignore the arm ratings. It’s not that they are unimportant, especially for outfielders, but they have nothing to do with how a fielder turns a batted ball into an out, which is what DER, and most defensive metrics measure. For 2B, 3B, and SS, the arm as well as range determine how efficient a fielder is at recording outs. The weights used are:

1B: Instincts 44%, 1st step 44%, Speed 12%
2B: Instincts 22%, 1st step 22%, Speed 11% Hands 22% release 11% strength 6% accuracy 6%
3B: Instincts 15%, 1st step 15%, Speed 10% Hands 15% release 15% strength 15% accuracy 15%
SS: Instincts 18%, 1st step 18%, Speed 12% Hands 12% release 18% strength 12% accuracy 12%
OF: Instincts 17%, 1st step 33%, Speed 33% Hands 17%

Next I find the average of each position – I just take the simple average of all players at their listed primary position. Then I find every player’s rating at each position, and convert to runs. For most positions, (rating – lg average)/2 gives you ratings that appear to be on the same scale as the stat-based projections. For 1B, where a player gets fewer chances, replace the 2 with a 3, and use 1.75 for shortstop.

Next, I take both sets of projections, find out how many plays saved each team projects to, and modify the expected DER accordingly. For this exercise, in the interests of time, I only looked at players who played at least 250 innings in 2008. If a player did not have a stat projection at the position played, I used zero for his projection. I did the same for the fan projections, except that if a player was evaluated, I had a projection at any position he might play in 2008, regardless of whether he ever played there before.

Here are the results:

Team statDER fanDER RealDER
ARI  0.686 0.688 0.687
ATL  0.685 0.688 0.695
BAL  0.706 0.707 0.691
BOS  0.695 0.694 0.700
CHA  0.694 0.696 0.688
CHN  0.701 0.706 0.706
CIN  0.687 0.688 0.674
CLE  0.695 0.692 0.686
COL  0.678 0.672 0.679
DET  0.706 0.701 0.686
FLA   0.689 0.691 0.694
HOU  0.694 0.692 0.699
KCA  0.694 0.691 0.691
LAA  0.696 0.699 0.693
LAN  0.692 0.697 0.693
MIL  0.694 0.696 0.700
MIN  0.690 0.690 0.690
NYA  0.694 0.697 0.684
NYN  0.690 0.691 0.699
OAK  0.703 0.698 0.701
PHI  0.691 0.688 0.696
PIT  0.679 0.680 0.676
SDN  0.698 0.695 0.697
SEA  0.704 0.706 0.683
SFN 0.688 0.685 0.686
STL  0.685 0.685 0.697
TBA  0.712 0.712 0.712
TEX  0.678 0.680 0.673
TOR  0.696 0.695 0.706
WAS 0.685 0.685 0.690

By RMSE (Root mean squared error) the results are:
Stats: .0084
Fans .0087

Going without defensive projections, using the ballpark and batted ball adjustment only (BB), we get an error of .0091

Using correlation, the results are similar:
Stats: .552
Fans: .534
BB: .450

The fan report would probably do better if I used multiyear data to create the projections. I’m not sure what the proper weights by year should be. Most likely, an even better projection could be created by some combination of stats and fan reports. I hope that the framework used here makes sense, and can provide an objective measure to evaluate defensive projections.

I should note that this can only be used to evaluate defensive projections, not defensive ratings. By comparing to DER, a stat like my own TotalZone would probably look better than a far more detailed stat, such as UZR or John Dewan’s plus/minus. That is because TotalZone, once adjusted for ballpark and hit type, is pretty much the same as DER. Since the more detailed stats capture more fielding ability, and better strip out the luck, they should better predict next year’s DER than a simpler stat.