(WAR Talk) A major discrepancy in Fangraphs vs. Baseball Reference

[Angelsjunky]

I considered starting a general thread on advanced statistics, but decided to focus on a single issue and didn't want to scare old-timers like Lou away.

The impetus behind this was discovering a rather striking anomaly in the world of WAR. As those reading this likely know, there are three main varieties of WAR, the two most popular being the Fangraphs version (fWAR) and the Baseball Reference version (rWAR), with the Baseball Prospectus version (WARP) being less used. 

The fWAR and rWAR are mostly different among position players because of different ways of evaluating defense; the hitting and baserunning components are roughly similar, I believe. The difference in pitching is more extreme, with rWAR tending to have higher values. Fangraphs bases their's on "FIP" (Fielding Independent Pitching) which tries to ascertain earned runs without defense, that is on the pitcher's power alone, whereas BR bases their's more on actual runs allowed. To be honest, I have no clue how BP calculates WARP.

So here's the anomaly: Aaron Nola. First of all, here are his stats the last two years:

2017: 168 IP, 9.86 K/9, 2.63 K/9, 3.27 FIP, 3.54 ERA

2018: 199.1 IP, 9.48 K/9, 2.39 K/9, 2.97 FIP, 2.44 ERA

According to Fangraphs, his WAR is 4.2 and 5.4 respectively. Meaning, he's been somewhat better this year, both because he's pitched more and better overall performance. But it isn't as wide as his ERA would entail, based upon a narrower margin in FIP.

Now here's where things diverge. Baseball Reference has him at 4.3 for 2017 and 9.8 for 2018. 9.8!!!! In other words, their version of WAR has him at more than twice as valuable than he was in 2017. (And by the way, I'm not including hitting, which brings his WAR down to 9.3).

To me this is just wacky. Nola's a really good pitcher, but has he really been worth about 10 WAR this season? According to BR, the last 10 WAR pitching season was Zack Greinke in 2009 (10.4). Before that you've got Randy Johnson in 2002 and 2001 (10.7 and 10.1, respectively), and Pedro Martinez in 2000 (11.7). 

I'm having trouble wrapping my head around the idea that Aaron Nola's 2018 campaign is the fourth best pitching season of the 21st century, better than any Kershaw or Scherzer years. Let's compare again:

Fangraphs: 5.4, 129th best pitching season of 21st century (2001-18)

Baseball Reference: 9.8, 4th best pitching season of 21st century (2001-18)

The difference between 4th best and 129th best isn't a minor discrepancy. And, to be honest, it makes me not take BR's version of the stat as seriously, at least for pitchers.

Assuming Nola has a good last start or two and surpasses 10 rWAR, here are the 10 rWAR pitching seasons by decade:

2010s (1): Aaron Nola (2018)

2000s (4): Randy Johnson x2 (2001, 2002), Zack Greinke (2009), Pedro Martinez (2000)

1990s (2): Roger Clemens x2 (1990, 1997)

1980s (2): Dwight Gooden (1985), Steve Carlton (1980)

1970s (8):  Tom Seaver x2 (1971, 1973), Wilbur Wood (1971, 1972), Ferguson Jenkins (1971), Gaylord Perry (1972), Steve Carlton (1972), Phil Niekro (1978)

1960s (6): Bob Gibson x2 (1968, 1969), Sandy Koufax x2 (1963, 1966), Juan Marichal (1965), Dick Ellsworth (1963)

1950s: None

1940s (2): Bob Feller (1946), Hal Newhouser (1945)

1930s (3): Lefty Grove x3 (1930, 1931, 1936)

1920s (5): Dazzy Vance x2 (1924, 1928), Dolf Luque (1923), Red Faber (1921), Pete Alexander (1920)

1910s (13): Walter Johnson x7 (1910, 1912-15, 1918-19), Ed Walsh x2 (1910, 1912), Eddie Cicotte (1917), Pete Alexander (1915), Smokey Joe Wood (1912), Russ Ford (1910)

1900s (8): Cy Young x2 (1901-02), Christy Mathewson x2 (1903, 1908), Jack Chesbro (1904), Rube Waddell (1904),  Ed Walsh (1908), Jo McGinnity (1903)

The total numbers become less meaningful once you get to the 1970s when the four-man rotation--and 40ish starts a year--was the norm. But again, according to BR, Aaron Nola's 2018 season is one of the nine best seasons from the 1980s onward, and one of only 54 seasons from 1901 to the present, the span of the two leagues.

So again, according to rWAR, Nola's 2018 is better than any season by Kershaw, Scherzer, Sale, Halladay, Maddux, and many others. Isn't there something wrong with that picture?

[Docwaukee]

rWAR heavily weighs run scoring environments ie parks, opponents and defenses.  they try to establish what an avg pitcher would have done under similar conditions for that pitcher, and that specific number is calculated for each individual pitcher.  

(what's funny is that Nola didn't pitch yesterday yet they just updated his stats on bRef and his WAR went down. )  

Anyway, the bRef calc for run scoring environment specifically for Nola showing that an avg pitcher would give up almost 0.8 more runs per nine innings than the one deGrom pitched in.  

bRef uses defensive runs saved for the calc and this years philly defense is at -127.  Worst in baseball by a lot.  Actually, it's the worst since DRS has been kept track of (2003).  They used total zone before 2003.  

bRef actually says in their page describing their WAR calc that they are looking to give credit for what actually happened and lesser so as to what is predictable going forward.  

so it really is back to being about the difference in the way they calculate defense.  

[Angelsjunky]

But such a massive difference, both relative to his prior year and to other pitchers? Seems screwy.

[hangin n wangin]

Personally, I prefer fWAR.

For the most part, both types of WAR usually reflect each other, however, there are certain cases where they are far off.

[Inside Pitch]

9 minutes ago, hangin n wangin said:

Personally, I prefer fWAR.

For the most part, both types of WAR usually reflect each other, however, there are certain cases where they are far off.

The way they measure defense really really makes for some wild variances between the two systems.   They current systems may be the best we have ever seen but man, they are still kind of a shit-show at times..

[AngelsLakersFan]

This feels like a flaw in rWar.

fWar uses FIP to separate out the defense from the pitching. Yes sometimes guys will give up a lot more runs than their WAR says they should've, but this is then reflected in poor defensive ratings for the position players.

Here with rWar this feels like a good example of the double counting I have a feeling is going on. If you give a pitcher credit for preventing runs from scoring you shouldn't also be giving him credit for pitching in front of a bad defense. This might seem counter intuitive but if you have a defense that sucks overall its still possible for them to play at an average or even above average level in small sample sizes (30 games per year is a small sample size for defensive stats). 

So in this example here we have a guy who's ERA is half a run better than his FIP, which would imply that he's pitching in front of a decent defense, or that balls in play have been hit right at defenders (the latter being what rWar assumes Nola deserves the extra credit for). rWar is looking at ERA as the starting point, and looking at the overall defensive numbers (which are terrible) and then calculating that Nola is probably unlucky with his defense. 

tl;dr Nola's defense has been average to above average while he's been on the mound this year; despite sucking overall. rWar is crediting him for pitching in front of a terrible defense that has actually helped save him runs.

[T.G.]

[WeatherWonk]

Dick Ellsworth. Amazing!

[greginpsca]

On 9/20/2018 at 11:40 PM, Angelsjunky said:

I considered starting a general thread on advanced statistics, but decided to focus on a single issue and didn't want to scare old-timers like Lou away.

The impetus behind this was discovering a rather striking anomaly in the world of WAR. As those reading this likely know, there are three main varieties of WAR, the two most popular being the Fangraphs version (fWAR) and the Baseball Reference version (rWAR), with the Baseball Prospectus version (WARP) being less used. 

The fWAR and rWAR are mostly different among position players because of different ways of evaluating defense; the hitting and baserunning components are roughly similar, I believe. The difference in pitching is more extreme, with rWAR tending to have higher values. Fangraphs bases their's on "FIP" (Fielding Independent Pitching) which tries to ascertain earned runs without defense, that is on the pitcher's power alone, whereas BR bases their's more on actual runs allowed. To be honest, I have no clue how BP calculates WARP.

So here's the anomaly: Aaron Nola. First of all, here are his stats the last two years:

2017: 168 IP, 9.86 K/9, 2.63 K/9, 3.27 FIP, 3.54 ERA

2018: 199.1 IP, 9.48 K/9, 2.39 K/9, 2.97 FIP, 2.44 ERA

According to Fangraphs, his WAR is 4.2 and 5.4 respectively. Meaning, he's been somewhat better this year, both because he's pitched more and better overall performance. But it isn't as wide as his ERA would entail, based upon a narrower margin in FIP.

Now here's where things diverge. Baseball Reference has him at 4.3 for 2017 and 9.8 for 2018. 9.8!!!! In other words, their version of WAR has him at more than twice as valuable than he was in 2017. (And by the way, I'm not including hitting, which brings his WAR down to 9.3).

To me this is just wacky. Nola's a really good pitcher, but has he really been worth about 10 WAR this season? According to BR, the last 10 WAR pitching season was Zack Greinke in 2009 (10.4). Before that you've got Randy Johnson in 2002 and 2001 (10.7 and 10.1, respectively), and Pedro Martinez in 2000 (11.7). 

I'm having trouble wrapping my head around the idea that Aaron Nola's 2018 campaign is the fourth best pitching season of the 21st century, better than any Kershaw or Scherzer years. Let's compare again:

Fangraphs: 5.4, 129th best pitching season of 21st century (2001-18)

Baseball Reference: 9.8, 4th best pitching season of 21st century (2001-18)

The difference between 4th best and 129th best isn't a minor discrepancy. And, to be honest, it makes me not take BR's version of the stat as seriously, at least for pitchers.

Assuming Nola has a good last start or two and surpasses 10 rWAR, here are the 10 rWAR pitching seasons by decade:

2010s (1): Aaron Nola (2018)

2000s (4): Randy Johnson x2 (2001, 2002), Zack Greinke (2009), Pedro Martinez (2000)

1990s (2): Roger Clemens x2 (1990, 1997)

1980s (2): Dwight Gooden (1985), Steve Carlton (1980)

1970s (8):  Tom Seaver x2 (1971, 1973), Wilbur Wood (1971, 1972), Ferguson Jenkins (1971), Gaylord Perry (1972), Steve Carlton (1972), Phil Niekro (1978)

1960s (6): Bob Gibson x2 (1968, 1969), Sandy Koufax x2 (1963, 1966), Juan Marichal (1965), Dick Ellsworth (1963)

1950s: None

1940s (2): Bob Feller (1946), Hal Newhouser (1945)

1930s (3): Lefty Grove x3 (1930, 1931, 1936)

1920s (5): Dazzy Vance x2 (1924, 1928), Dolf Luque (1923), Red Faber (1921), Pete Alexander (1920)

1910s (13): Walter Johnson x7 (1910, 1912-15, 1918-19), Ed Walsh x2 (1910, 1912), Eddie Cicotte (1917), Pete Alexander (1915), Smokey Joe Wood (1912), Russ Ford (1910)

1900s (8): Cy Young x2 (1901-02), Christy Mathewson x2 (1903, 1908), Jack Chesbro (1904), Rube Waddell (1904),  Ed Walsh (1908), Jo McGinnity (1903)

The total numbers become less meaningful once you get to the 1970s when the four-man rotation--and 40ish starts a year--was the norm. But again, according to BR, Aaron Nola's 2018 season is one of the nine best seasons from the 1980s onward, and one of only 54 seasons from 1901 to the present, the span of the two leagues.

So again, according to rWAR, Nola's 2018 is better than any season by Kershaw, Scherzer, Sale, Halladay, Maddux, and many others. Isn't there something wrong with that picture?

Garbage in, garbage out.