How to Evaluate Catcher Defense
At least part of it. I remain agnostic as to how much impact a catcher's game calling has on pitchers. I use retrosheet, though you can make a reasonable approximation with data from Baseball-reference, Fangraphs, or Hardball Times.
Data needed: Innings caught, Stolen bases allowed, caught stealing, passed balls, wild pitches, errors, and pickoffs. I have totals of these data for each catcher by pitcher handedness, one total for lefties and one for righties.
1. Find the league averages per inning caught for each of the data elements above, by pitcher hand.
2. Compare each catcher's totals to the league average prorated to his innings. If a league allows .12 steals per inning with RHP, a catcher has 1000 innings and allows 80 steals with RHP, then he has allowed 40 steals fewer than average. Do this for all data elements.
3. Apply run value. For recent years I use:
SB -0.20 ....CS/Pick 0.47 ....WP/PB/Err -0.275
Sum up the run values for all events and both righties and lefties, and you've got catcher runs. Modify as you wish (Don't believe a catcher can have any impact on preventing wild pitches? Leave that out).