Bullpen analysis since 2010

[Docwaukee]

You don't have to spend a lot of money to have a good bullpen.  But if you don't, you have to have some arms in your minor league system to fill those gaps or you aren't going to do well.  Successful pens are very much a function of the pitching depth within your farm system.  Particularly in the upper levels.  

In the past, your failed starters became your relieves.  But more and more we see guys being converted to relief earlier in their development.  Even still, it's kind of a mixed bag as to when a minor league player will become a stud reliever.  

It was a saber truth that spending on a bullpen whether cash or prospects, was unacceptable.  Metrics across the board gave little value to players who would pitch all of 50-60 innings per year.  Replacement value was stingier than that allocated to starters as it was predicated on the everyone moving up a level without any adjustments for leverage.  Yet, over the past few years, we have seen even the most saber savvy teams spending huge on bullpens with big dollars and bountiful prospect hauls.  I think this tells us that the use of proprietary metrics among various teams has determined reliever values to be much higher than previously thought.  

So where have the halos fit into all this madness?  

First of all, let's consider performance.  Using WAR to evaluate relievers was the primary reason why they were undervalued initially.  WPA takes leverage into account but it's not predictive (or at least that's what the consensus was early on).  I think teams are using some hybrid that allocates additional value based on leverage.  

Since 2010, the halos have accumulated 10.5 WAR from their bullpen.  Ranking them 29th.  Their WPA during that time is 5.63 which may not mean much to many, but it's essentially a point system allocated to their performance with leverage included.  They are ranked 20th.  Another way to look at this is the SD to MD or shutdowns vs. meltdowns.  Whether a relievers performance increased or decreased their teams win probability by a specific amount (+ or - 0.06).  We ranked 19th in SD and 19th in MD.  

So the pen has actually out performed their WAR value which makes sense in that they've been better in higher leverage than lower leverage (+6.09 ranking them 12th in terms of the difference).  

I only have easily accessible data for bullpen payroll from 2013.  During that time, we've had a WAR of 8.2 (25th), and a WPA of 4.83 (15th).  Again, showing that we have performed much much better in high leverage situations and not so good when neutralized for leverage (22nd).  Another interesting characteristic of our pens is that even though we have been league avg by era in high leverage, we have the worst xFIP in baseball over the last 7 years.  

I think this tells us a couple of important things.  

One, pen management has been pretty good.  I know that flies in the face of how many here feel in regards to Scioscia, but the bullpen has actually done decently well in high leverage and mostly failed in less important situations.  In order to understand how that works though, you have to know when situations are high leverage vs. when they aren't and that, frankly wasn't intuitive to me until I looked at the charts.   It was quite surprising.  My biggest annoyance with Scioscia's pen management was/is bringing in crappy relievers when we were tied or down by a run.    

As an example, let's say we're down by a run going into the 7th innings and MS brings in Salas.  The leverage when he tows the mound with no on and no out is 1.0.  Conversely, If we are up by a run and we bring in a new reliever with no on and no out, the leverage is 1.7.  If we are on the road, it's even more dramatic.  Down by a run?  0.8.  Up by a run? 1.9.  

So this tells us that while Scioscia likes his pen roles, the right guy is generally pitching in the right spots.  Could it be a bit better?  Maybe, but the end result means that the job was getting done more than expected relative to the talent value of the players.  

Two, our bullpen depth sucked.  Even though the high leverage guys did about average, the rest of the pen pretty much sucked creating multiple problem like never giving the team a chance to come from behind.  Like having to stick to a schedule based on scenario and giving for very little flexibility.  And perhaps most importantly, not playing to the strength of your managers decision making process.  

The trend from 2010 to 2016 is what I found to be interesting.  

Total pen WAR league wide was 73.6 in 14,235.1ip in 2010 vs. 102.6 in 15,893.2 ip for 2016.

2016 had the highest WAR and # of relief innings in the history of baseball.  

And here is the way teams spent on bullpens from 2013-2016 

2013 - 546 mil with an avg of 18.2. Total of 415 players.  Halos were 18th spending 16.7mil

2014 - 560 mil.  Avg of 18.7 mil.  Total of 390 players.  Halos - 21 mil

2015 - 530 mil. Avg of 17.7 mil. Total of 428 players.   Halos - 16.7 mil

2016 - 550 mil. Avg of 18.3 mil.  Total of 417 players.  Halos - 11.2 mil.

So while the trend is to use relieves for more and more innings, the outlay and # of players used has fluctuated much over the last four years.  That said, I have a feeling we are going to see a substantial jump in total outlay but my guess is that the number of players used might be pretty similar.  

We can also see that the halos have trended down in bullpen expenditures even though the rest of the league has stayed the same.  Although likely a consequence of payroll restrictions, pen arms didn't usually cost all that much on the open market.  Which is probably why we are now seeing a market adjustment this year.   

There is no doubt that the Angels bullpen has cost them wins over the last several years.  Not as many wins as I expected in the from those late inning high leverage situations but more so from those opportunities to keep games close ie holding deficits and ties.  While it was pretty easy to shore up a bullpen in the past, the secret is out and teams are going to be spending more and more money on it because they are relying on those relief innings.  

Eppler is the self proclaimed finder of minor league free agent talent that helps major league teams, but we have seen him take it to the next level this year.  He has acquired a slew of once highly though of arms via his clean peanut detector (patent pending) as opposed to tapping the free agent market.  I have to think that even if we had the financial resources to pay for pen help, this would still be the route he takes as there is no clear correlation between how much you spend on a bullpen and how well they perform.....yet.  But I think that's coming soon.  Which means that, as with any major league team assembly method, there will be market inefficiencies to tap.  

Regardless of whether the SP holds up, 2017 is very important in determining if one of Eppler's primary theories is correct.  It's a litmus as to his worthiness and I feel a very important one because if he and his team are truly talented in this regard, it gives our farm system that much more time to develop and means that he has found a cost effective way (at least for the time being) of shoring up a very important facet of the game.  Pitching depth.  While I don't expect him to pull a front line starter out of his hat via this method, it could obviate the necessity for that 220 inning guy who pitches to a sub 3 era.  There could be a trickle down effect where the entire staff benefits.  I'd certainly invest in whatever airline is between SLC and John Wayne.  

 

[Stradling]

What a great read, thank you Doc. 

[Inside Pitch]

This post is a HR.  

[Stradling]

I'm not horribly surprised that Sosh is decent at bullpen management.  There's nothing easier than making a judgement on a manager when a relief pitcher comes in and blows a game.  They simply don't have enough quality arms to use in tie games or down a run.  It's also the reason we had so much success when we had Scot Shields and the rest.  

[AngelsLakersFan]

I honestly thought that the general consensus among the members of this board with a clue was that Scioscia was pretty solid at bullpen management, to the extent that he has the arms to work with.

[Stradling]

5 minutes ago, AngelsLakersFan said:

I honestly thought that the general consensus among the members of this board with a clue was that Scioscia was pretty solid at bullpen management, to the extent that he has the arms to work with.

You must not frequent the game day threads.  But yes the smart normal people understand he's average to good at bullpen management. During the season however the vocal majority crucify him for his pen decisions. 

[Oz27]

1 hour ago, Stradling said:

You must not frequent the game day threads.  But yes the smart normal people understand he's average to good at bullpen management. During the season however the vocal majority crucify him for his pen decisions. 

That is a hell of an overgeneralization. He probably isn't terrible compared to his peers (guys like Matheny and Baker exist, after all) but there is no shortage of inexplicable decisions. For example, Jose Alvarez faced more RHH than LHH last year when a 30 second glance at his Baseball-Reference page is enough to tell you that was a terrible decision.

[Stradling]

19 minutes ago, Oz27 said:

That is a hell of an overgeneralization. He probably isn't terrible compared to his peers (guys like Matheny and Baker exist, after all) but there is no shortage of inexplicable decisions. For example, Jose Alvarez faced more RHH than LHH last year when a 30 second glance at his Baseball-Reference page is enough to tell you that was a terrible decision.

Yep he should have gone with his better options.  The ones he doesn't have.  

[Oz27]

8 minutes ago, Stradling said:

Yep he should have gone with his better options.  The ones he doesn't have.  

Silly response and you know it. Alvarez vs RHH is worse than nearly any RHP would be.

[Oz27]

31 minutes ago, hangin n wangin said:

Smart normal people understand he's average to good at bullpen management? That's interesting. So it's only the smart normal people that can understand?? 

So those that think he isn't very good at bullpen management are dumb retards? I think I'm getting it. Put me in the dumb retard category.

You're just not smart enough to understand it.

[Stradling]

Ok Oz you jack off to stats except now when it says he's pretty good at bullpen management? That's pretty convenient.  

The guy has had shit bullpens for quite awhile now.  

[Docwaukee]

48 minutes ago, hangin n wangin said:

Smart normal people understand he's average to good at bullpen management? That's interesting. So it's only the smart normal people that can understand?? 

So those that think he isn't very good at bullpen management are dumb retards? I think I'm getting it. Put me in the dumb retard category.

what are you basing your bullpen management evaluation on?  The few times that he made the wrong choice which stick out because of the catastrophic effect?  Or should it be made on the general body of work where statistics support his general decision making process?  

Are there WTF moments?  yes.  Like oz pointed out, having Alvarez face a rhed hitter seems crazy.  but when you look at the numbers, it's interesting.  His peripherals show reverse splits.  a 3.38 xFIP and 3.01 FIP vs. Rhers and a 4.52 xFIP and 3.22 FIP vs. LHers.  A 9.42 k/9 vs. Rhers.  6.59 vs. LHers.  A 51.1% GB rate vs. Rhers.  and 35.9% vs. Lhers.  

Also, Alvarez threw all of 4.1 innings in high leverage and 41.2 innings in low leverage.  

Overall, his babip vs. rhers was .398.  

But of course, that's just one year.  overall, his splits are pretty equal with him being slightly better vs. lefties.  

Anyway, throw out some scenarios where you disagree and we'll discuss.  

[Oz27]

14 minutes ago, Stradling said:

Ok Oz you jack off to stats except now when it says he's pretty good at bullpen management? That's pretty convenient.  

The guy has had shit bullpens for quite awhile now.  

No need to be so aggressive. There are a lot of ways to evaluate bullpen management and I'm not sure I or anyone can conclusively say which is one is right or wrong. I've seen studies say that he is good, not good and in the middle. Doc's methodology and analysis is good, as usual. But I've seen similar analysis with a different, but still valid, methodology, reach contrasting conclusions so I don't take any of it as gospel for the moment.

Also, all I did was point to one clear example of mismanagement to raise a contrary point to your "us smart people know this" nonsense. No need to be so damn aggressive about it.

[Troll Daddy]

I'm guessing Maddon and Francona had many questionable bullpen decisions. You could probably throw Bochy in the mix too. 

[Oz27]

1 hour ago, Stradling said:

Ok Oz you jack off to stats except now when it says he's pretty good at bullpen management? That's pretty convenient.  

The guy has had shit bullpens for quite awhile now.  

I should have mentioned before as well the problems with using leverage and WPA to measure bullpen success. Do runs count more in the eighth than they do in the sixth? Of course not. But according to those stats, they do. As Joe Sheehan, former Baseball Prospectus guru, wrote a few days ago:

"Leverage is an interesting descriptive stat, but as a value one it's basically a restatement of saves, in the same way that Win Probability Added is nerd talk for RBIs. Leverage takes into account game state, weighting later innings more highly than early ones, because runs in the eighth inning count more than runs in the third. No, wait. That's not right. Leaving that problem aside, though, my bigger one is that leverage ignores the opposing hitters"

Are there better ways to judge bullpen management? Definitely. We just haven't perfected them yet. But it's wrong to think the stats have conclusively said anything beyond doubt, no matter how many snide comments you want to put in your post. To be clear I don't mean that as a criticism of Doc at all, I'm just saying we haven't really come up with a flawless way to evaluate this or to justify making sweeping, hyperbolic statements such as "the smart people know this" or to dismiss a contrary position as "convenient".

[Stradling]

You're right Oz, I was as bad as the guys that bitch in every game day thread when I said smart people and normal people.  As far as the stats I was using hangin, I was using the post that Doc used as the op.  Now I am not sure how calling people smart and normal is any different than hangin referring to Scioscia's use of the bullpen as pathetic.  If a manager makes a decision that doesn't work, it isn't always the wrong decision, it just didn't work.  When you have lousy pitching options, then you are really going to struggle to always make the right calls.  Oz, when it comes to stats you ignore some of them when you don't want to acknowledge them, or at least are in this thread.  Doc brings up Alvarez's numbers this year, using things like xFIP (I don't really know what that is, but I would assume it is his performance taking luck or bad luck out of the equation) and using that stat, which I have seen you use to destroy Weaver, shows Alvarez as not nearly as bad as you make him sound against righties.  Now I guess it is possible that xFIP holds less credibility with relief pitchers, I have no idea.  So that should answer most of the questions you guys have asked me.  Oz, sorry I got so aggressive.  

[ettin]

2 hours ago, Oz27 said:

That is a hell of an overgeneralization. He probably isn't terrible compared to his peers (guys like Matheny and Baker exist, after all) but there is no shortage of inexplicable decisions. For example, Jose Alvarez faced more RHH than LHH last year when a 30 second glance at his Baseball-Reference page is enough to tell you that was a terrible decision.

Disagree with this Oz. Alvarez had a 15.6% K%-BB% vs. RHP in 2016, which is 1.5% above league average. He also suffered from a hugely inflated BABIP of .398 which is nearly 100 points higher than the League average of .299. The decision to have Alvarez face RHP was not bad it was a victim of bad luck and not leaving enough hitters left on base. I am not saying that Jose is the best choice the Angels have to face RHP but as a set up guy there is potential there. Sample sizes with relievers in any season should always be taken with a grain of salt. As I learned recently during a Twitter exchange with Mitchel Lichtman and Tangotiger our perception(s) of long standing baseball "standards" or long held beliefs can be incorrect when you look at the numbers supporting them. Actual results matter of course but they belay the underlying nature of proper sample size (of which his 15.6% might be an outlier) and, perhaps, misguided perceptions (confirmation bias).

[Oz27]

32 minutes ago, ettin said:

Disagree with this Oz. Alvarez had a 15.6% K%-BB% vs. RHP in 2016, which is 1.5% above league average. He also suffered from a hugely inflated BABIP of .398 which is nearly 100 points higher than the League average of .299. The decision to have Alvarez face RHP was not bad it was a victim of bad luck and not leaving enough hitters left on base. I am not saying that Jose is the best choice the Angels have to face RHP but as a set up guy there is potential there. Sample sizes with relievers in any season should always be taken with a grain of salt. As I learned recently during a Twitter exchange with Mitchel Lichtman and Tangotiger our perception(s) of long standing baseball "standards" or long held beliefs can be incorrect when you look at the numbers supporting them. Actual results matter of course but they belay the underlying nature of proper sample size (of which his 15.6% might be an outlier) and, perhaps, misguided perceptions (confirmation bias).

I certainly agree with the latter part of your post. 

I'll be honest, I hadn't dived in as deeply into Alvarez's splits as I should have and the high BABIP vs RHH isn't supported by his much higher ground ball rate (relative to his performance vs LHH) and it is significantly different to his 2015 result. So yeah, there is definitely some room for improvement there. I'm not sure that it's safe though to dismiss it entirely as him being a victim of bad luck though. I do want to delve into his batted ball results a bit more to see if there is a cause for it. There can be causes for high BABIP beyond bad luck. It is also worth pointing out his multi-year true average against RHP is .269, vs LHP it is .228.

On a side note, it hadn't hit me how bad Alvarez was on the road in 2016. His peripherals were hideous and his raw results were too. Maybe that was just random chance but, wow, those road results were bad.

[ettin]

18 hours ago, Dochalo said:

You don't have to spend a lot of money to have a good bullpen.  But if you don't, you have to have some arms in your minor league system to fill those gaps or you aren't going to do well.  Successful pens are very much a function of the pitching depth within your farm system.  Particularly in the upper levels.  

In the past, your failed starters became your relieves.  But more and more we see guys being converted to relief earlier in their development.  Even still, it's kind of a mixed bag as to when a minor league player will become a stud reliever.  

It was a saber truth that spending on a bullpen whether cash or prospects, was unacceptable.  Metrics across the board gave little value to players who would pitch all of 50-60 innings per year.  Replacement value was stingier than that allocated to starters as it was predicated on the everyone moving up a level without any adjustments for leverage.  Yet, over the past few years, we have seen even the most saber savvy teams spending huge on bullpens with big dollars and bountiful prospect hauls.  I think this tells us that the use of proprietary metrics among various teams has determined reliever values to be much higher than previously thought.  

So where have the halos fit into all this madness?  

First of all, let's consider performance.  Using WAR to evaluate relievers was the primary reason why they were undervalued initially.  WPA takes leverage into account but it's not predictive (or at least that's what the consensus was early on).  I think teams are using some hybrid that allocates additional value based on leverage.

First of all let me say great post!

In regards to the statement above I agree that might be part of it but I really feel like teams have learned the value of pitching that can get strikeouts. You said it yourself, and I agreed, in the other thread that you like guys who can miss bats. League strikeout rates have increased steadily and consistently over the last 10 years. For the first time it exceed 8.0 K/9 innings in 2016. In addition to that Mike Fast, before he left for Houston, wrote about how 1 mph velocity increase can lead to .2 to about .4 of run prevention. When you combine leverage, whiffs, and increased velocity it validates the rising acquisition prices in trade and free agent prices particularly for a guy like Chapman.

I have an interesting note regarding strikeouts and balls in play in an upcoming Primer article that I found interesting and perhaps you will too.

18 hours ago, Dochalo said:

Since 2010, the halos have accumulated 10.5 WAR from their bullpen.  Ranking them 29th.  Their WPA during that time is 5.63 which may not mean much to many, but it's essentially a point system allocated to their performance with leverage included.  They are ranked 20th.  Another way to look at this is the SD to MD or shutdowns vs. meltdowns.  Whether a relievers performance increased or decreased their teams win probability by a specific amount (+ or - 0.06).  We ranked 19th in SD and 19th in MD.  

So the pen has actually out performed their WAR value which makes sense in that they've been better in higher leverage than lower leverage (+6.09 ranking them 12th in terms of the difference).  

I only have easily accessible data for bullpen payroll from 2013.  During that time, we've had a WAR of 8.2 (25th), and a WPA of 4.83 (15th).  Again, showing that we have performed much much better in high leverage situations and not so good when neutralized for leverage (22nd).  Another interesting characteristic of our pens is that even though we have been league avg by era in high leverage, we have the worst xFIP in baseball over the last 7 years.  

I think this tells us a couple of important things.  

One, pen management has been pretty good.  I know that flies in the face of how many here feel in regards to Scioscia, but the bullpen has actually done decently well in high leverage and mostly failed in less important situations.  In order to understand how that works though, you have to know when situations are high leverage vs. when they aren't and that, frankly wasn't intuitive to me until I looked at the charts.   It was quite surprising.  My biggest annoyance with Scioscia's pen management was/is bringing in crappy relievers when we were tied or down by a run.    

As an example, let's say we're down by a run going into the 7th innings and MS brings in Salas.  The leverage when he tows the mound with no on and no out is 1.0.  Conversely, If we are up by a run and we bring in a new reliever with no on and no out, the leverage is 1.7.  If we are on the road, it's even more dramatic.  Down by a run?  0.8.  Up by a run? 1.9.  

So this tells us that while Scioscia likes his pen roles, the right guy is generally pitching in the right spots.  Could it be a bit better?  Maybe, but the end result means that the job was getting done more than expected relative to the talent value of the players.  

Two, our bullpen depth sucked.  Even though the high leverage guys did about average, the rest of the pen pretty much sucked creating multiple problem like never giving the team a chance to come from behind.  Like having to stick to a schedule based on scenario and giving for very little flexibility.  And perhaps most importantly, not playing to the strength of your managers decision making process.  

The trend from 2010 to 2016 is what I found to be interesting.  

Total pen WAR league wide was 73.6 in 14,235.1ip in 2010 vs. 102.6 in 15,893.2 ip for 2016.

2016 had the highest WAR and # of relief innings in the history of baseball.  

And here is the way teams spent on bullpens from 2013-2016 

2013 - 546 mil with an avg of 18.2. Total of 415 players.  Halos were 18th spending 16.7mil

2014 - 560 mil.  Avg of 18.7 mil.  Total of 390 players.  Halos - 21 mil

2015 - 530 mil. Avg of 17.7 mil. Total of 428 players.   Halos - 16.7 mil

2016 - 550 mil. Avg of 18.3 mil.  Total of 417 players.  Halos - 11.2 mil.

So while the trend is to use relieves for more and more innings, the outlay and # of players used has fluctuated much over the last four years.  That said, I have a feeling we are going to see a substantial jump in total outlay but my guess is that the number of players used might be pretty similar.  

We can also see that the halos have trended down in bullpen expenditures even though the rest of the league has stayed the same.  Although likely a consequence of payroll restrictions, pen arms didn't usually cost all that much on the open market.  Which is probably why we are now seeing a market adjustment this year.   

I recently read an article and for the life of me I can't find it (I just spent about 40 minutes Google searching it, it was recent in last 30 days or so) which indicated that there is an incorrect perception that teams spent an exorbitantly high amount for free agent relievers like Chapman, Jansen, and Melancon. The article indicated that the reality is that free agent reliever prices are actually in-line with previous contracts (Papelbon being the previous waterline) it is just the fact that there haven't been significant free agent reliever signings in the last couple of years (mostly it has been trades).

I guess what I'm saying is that although we are talking about this now among ourselves teams have been valuating like this for a while now (I'd say at least three years probably more). Your total expenditure numbers above show little variance which sort of supports that perception. You are correct though that the outlay will probably rise some.

18 hours ago, Dochalo said:

There is no doubt that the Angels bullpen has cost them wins over the last several years.  Not as many wins as I expected in the from those late inning high leverage situations but more so from those opportunities to keep games close ie holding deficits and ties.  While it was pretty easy to shore up a bullpen in the past, the secret is out and teams are going to be spending more and more money on it because they are relying on those relief innings.  

Eppler is the self proclaimed finder of minor league free agent talent that helps major league teams, but we have seen him take it to the next level this year.  He has acquired a slew of once highly though of arms via his clean peanut detector (patent pending) as opposed to tapping the free agent market.  I have to think that even if we had the financial resources to pay for pen help, this would still be the route he takes as there is no clear correlation between how much you spend on a bullpen and how well they perform.....yet.  But I think that's coming soon.  Which means that, as with any major league team assembly method, there will be market inefficiencies to tap.

Dipoto actually tried to do this but honestly I think Eppler has done a far superior job of it.

In regards to the conversation above I actually think I may have identified a market inefficiency that Eppler is tapping which I will discuss in said future Primer article.

18 hours ago, Dochalo said:

Regardless of whether the SP holds up, 2017 is very important in determining if one of Eppler's primary theories is correct.  It's a litmus as to his worthiness and I feel a very important one because if he and his team are truly talented in this regard, it gives our farm system that much more time to develop and means that he has found a cost effective way (at least for the time being) of shoring up a very important facet of the game.  Pitching depth.  While I don't expect him to pull a front line starter out of his hat via this method, it could obviate the necessity for that 220 inning guy who pitches to a sub 3 era.  There could be a trickle down effect where the entire staff benefits.  I'd certainly invest in whatever airline is between SLC and John Wayne.  

 

Here I really think Eppler's knowledge of Yankees baseball analytics has transferred over to the Angels and I think we already are a better team for it and will get even better prior to the start of the season.

Good job again Doc and thanks for the stock tip!

[ettin]

1 hour ago, Stradling said:

You're right Oz, I was as bad as the guys that bitch in every game day thread when I said smart people and normal people.  As far as the stats I was using hangin, I was using the post that Doc used as the op.  Now I am not sure how calling people smart and normal is any different than hangin referring to Scioscia's use of the bullpen as pathetic.  If a manager makes a decision that doesn't work, it isn't always the wrong decision, it just didn't work.  When you have lousy pitching options, then you are really going to struggle to always make the right calls.  Oz, when it comes to stats you ignore some of them when you don't want to acknowledge them, or at least are in this thread.  Doc brings up Alvarez's numbers this year, using things like xFIP (I don't really know what that is, but I would assume it is his performance taking luck or bad luck out of the equation) and using that stat, which I have seen you use to destroy Weaver, shows Alvarez as not nearly as bad as you make him sound against righties.  Now I guess it is possible that xFIP holds less credibility with relief pitchers, I have no idea.  So that should answer most of the questions you guys have asked me.  Oz, sorry I got so aggressive.  

I suspect this has to do with sample size Strad. If they had more innings pitched it would be more accurate and there is variance from pitcher to pitcher that xFIP doesn't quite capture.

[Docwaukee]

2 hours ago, Oz27 said:

I should have mentioned before as well the problems with using leverage and WPA to measure bullpen success. Do runs count more in the eighth than they do in the sixth? Of course not. But according to those stats, they do. As Joe Sheehan, former Baseball Prospectus guru, wrote a few days ago:

"Leverage is an interesting descriptive stat, but as a value one it's basically a restatement of saves, in the same way that Win Probability Added is nerd talk for RBIs. Leverage takes into account game state, weighting later innings more highly than early ones, because runs in the eighth inning count more than runs in the third. No, wait. That's not right. Leaving that problem aside, though, my bigger one is that leverage ignores the opposing hitters"

Are there better ways to judge bullpen management? Definitely. We just haven't perfected them yet. But it's wrong to think the stats have conclusively said anything beyond doubt, no matter how many snide comments you want to put in your post. To be clear I don't mean that as a criticism of Doc at all, I'm just saying we haven't really come up with a flawless way to evaluate this or to justify making sweeping, hyperbolic statements such as "the smart people know this" or to dismiss a contrary position as "convenient".

I disagree with Sheehan's first point and agree on the second.  

Leverage does not take into account the hitter, but he's essentially contradicting himself between the two point.  If run context doesn't matter, then situational variances shouldn't be accounted for at all in determining value.  

And that's where I disagree.  

Here's an interesting statement:

In order to determine the value of a player, context matters for some and not for others.  And here's why.  

The average leverage a player finds themselves in is important.  The average leverage for a qualified hitter in within 10% in either direction from 1.0.  That's why clutch doesn't matter for hitters value.  Because over the course of 600 plate appearances, it gonna avg itself out.

The quality of a batter that a reliever faces is likely to average itself out for the most part unless the trend of having a manager preferentially select for certain relievers to face higher quality batters continues.  

The point is, is you preferentially select for a player to perform is more critical situations then that player is worth more.  Those situations are worth more.  

In baseball, context matters.  It's why you bring in your best reliever to face the other team's best hitters in situations where there are increased odds of scoring.  

Certain relievers find themselves in those spots.  Leverage doesn't account for all of it, but it's better than not giving credit at all.  

So every run counts the same.  But if you are a player who is constantly put into a situation where preventing that run significantly increases your team's chance of winning whereas if you don't it significantly decreases the chances of winning, then your value should go up.  

 

[Oz27]

55 minutes ago, Dochalo said:

I disagree with Sheehan's first point and agree on the second.  

Leverage does not take into account the hitter, but he's essentially contradicting himself between the two point.  If run context doesn't matter, then situational variances shouldn't be accounted for at all in determining value.  

I don't agree with this bit. The game situation is out of the control of the pitcher, all he can do is get the batter out. I don't believe we should be rewarding somebody more for getting an out with his team up by a run later in the game than we would if they were down by a run earlier in the game. In my opinion, that doesn't impact the difficulty of his task. But the quality of the hitter does impact that. It's more difficult for a pitcher to get Mike Trout out than it is for him to retire Carlos Perez, right? If we're going to contextualize stats, then to me that is the sort of context we should be going for - not factors that were out of a pitcher's control (I know the quality of the batter is out of his control, but what happens when he's facing that batter is within his control).

55 minutes ago, Dochalo said:

In order to determine the value of a player, context matters for some and not for others.  And here's why.  

The average leverage a player finds themselves in is important.  The average leverage for a qualified hitter in within 10% in either direction from 1.0.  That's why clutch doesn't matter for hitters value.  Because over the course of 600 plate appearances, it gonna avg itself out.

The quality of a batter that a reliever faces is likely to average itself out for the most part unless the trend of having a manager preferentially select for certain relievers to face higher quality batters continues.  

The point is, is you preferentially select for a player to perform is more critical situations then that player is worth more.  Those situations are worth more.  

In baseball, context matters.  It's why you bring in your best reliever to face the other team's best hitters in situations where there are increased odds of scoring.  

Certain relievers find themselves in those spots.  Leverage doesn't account for all of it, but it's better than not giving credit at all.  

So every run counts the same.  But if you are a player who is constantly put into a situation where preventing that run significantly increases your team's chance of winning whereas if you don't it significantly decreases the chances of winning, then your value should go up.  

 

Isn't that again rewarding/punishing a player for something outside of their control, though? Theoretically good pitchers should find themselves in high leverage situations but that depends on managers making sound decisions. Cam Bedrosian was clearly our best reliever last year but his gmLI was ninth on the team, behind many relievers he was much better than (in order, Salas, Street, Bailey, Smith, Valdez, Ramirez, Morin and Alvarez). Judging by ability, Bedrosian should have been in the highest leverage situations but that wasn't how he was used. Should that impact how we evaluate him? In my opinion, no.

It's worth as well noting the wide gap between WPA and xFIP. If what we're trying to do is evaluate true ability, check out our 2016 reliever WPA leaderboard.

1. J.C. Ramirez, 0.79 WPA (seventh best xFIP out of 18)

2. Andrew Bailey, 0.69 WPA (10th in xFIP)

3. Greg Mahle, 0.25 WPA (13th in xFIP)

4. Cory Rasmus, 0.12 WPA (16th in xFIP)

5. Cam Bedrosian, 0.04 WPA (first in xFIP)

For our 2016, a chart measuring our reliever xFIP and WPA results would look like random dots on a page. Admittedly, the wider correlation isn't quite that low. The base correlation across MLB between xFIP and WPA is .495, so there is a relationship. But it really isn't that strong given we're measuring two things from the same sample. So if WPA isn't reflecting actual pitcher ability, which we best judge through xFIP or DRA, what is it actually showing us? I like using WPA, it is a fun tool to play around with. But I'm not convinced the trend towards using it to evaluate individual relievers or even groups of relievers makes a great deal of sense.

[Brent Maguire]

Lots of good stuff being thrown around in this thread. I'm ambivalent on Scioscia as a whole but I do believe his bullpen management has always been a strength of his. When given the right weapons, he's able to utilize them in a sufficient manner(see, 2014). He's always seemed to get a lot out of his relievers too, which Doc highlights in the original post, which can partially be credited to park factors as well.  

Anyways, I'm really curious to see how Scioscia manages the staff next year. There's a lot of room to try out of the box plans with all the potential starter/reliever hybrids on the 40 man roster. I don't expect Scioscia to go full SABR or anything but I'm hoping he uses the pen in a more advanced manner and gets the most out of what could be a weak overall unit. 

[Docwaukee]

24 minutes ago, Oz27 said:

I don't agree with this bit. The game situation is out of the control of the pitcher, all he can do is get the batter out. I don't believe we should be rewarding somebody more for getting an out with his team up by a run later in the game than we would if they were down by a run earlier in the game. In my opinion, that doesn't impact the difficulty of his task. But the quality of the hitter does impact that. It's more difficult for a pitcher to get Mike Trout out than it is for him to retire Carlos Perez, right? If we're going to contextualize stats, then to me that is the sort of context we should be going for - not factors that were out of a pitcher's control (I know the quality of the batter is out of his control, but what happens when he's facing that batter is within his control).

and then so is the situation at that point.  It's easier to get out carlos perez with the bases loaded than Mike Trout with no one on, but the probability of giving up a run is much greater in the Perez situation.  

You are more inclined to use the better pitcher when the extremes are greater though like if you are up by a run vs. down by 3.  If you don't think there is a difference, then why have you pushed for the flexibility of relievers to be available in higher leverage situations.  I know you have coupled that to the batter, but your opinion in the past was that both were important.  

The closer the game, the greater odds that one play, one strike or one swing will change the outcome.  Each at bat becomes more and more critical as the game progresses because there are fewer opportunities to change the outcome.  The reliever can't control it yet a position player doesn't determine what position they are going to play.  Yet we punish the position player for doing poorly.  Why? because it affects the outcome of the game.  

You know I am as stat inclined as anyone, but even with my limited experience playing there is a factorial difference ie an increased degree of difficulty in performing the same task when there is less time to compensate in the future or in environments where the odds of scoring runs are greater.  

You don't think that the game situation affects performance?  I am not saying that there is basis for clutch or the same player performing differently in more difficult situations than in less difficulty ones.  I am saying that some relievers in particular are put more exclusively in those more difficult situations and regardless of whether they have control you give them credit for it just like you give credit to Escobar for playing 3rd even though he's got no control over it.  I am not saying it has to be positive credit.  Just credit.  

There should be a low correlation between xFIP and WPA because one measure value independent of leverage and the other is linear weights of what happened.  Neither takes batter quality into account btw.  Both assume that it will average out over the course of the season yet we all know that's not true when it comes to relievers.     Another problem with comparing WPA to xFIP is that one is cumulative and the other is rate based.  

The Cam example is pretty tough to get on board with considering that his leverage was really low early in the year in that he was coming off a pretty bad season where it was tough to trust him.  Then it escalated until he got hurt.  

I guess I am also implying that I don't think xFIP is a great tool to measure reliever quality.  Being a fly ball pitcher without giving up hrs is a skill.  Weak contact is a skill.  

I don't think WAR or WPA tells the entire story for relievers.  I have to believe that major league teams have come up with a way to conglomerate WAR/FIP/xFIP with leverage and batter quality.  I think they have incorporated that into their team WAR calcs along with their proprietary defensive metrics and their makeup metrics.  

I think that the amount of money and/or farm talent being spent on the truly elite relievers tells us that value is being allocated to hitter faced quality and leverage.  Smarter people that us are making these calculations and I am sure that the R squared correlates are strong.  Stronger than anything we have access to.  

It's just like I believe that Dave Roberts pen management was him trying some shit out as a first time manager.  There are value assignments to everything based on whether that thing correlates to runs scored or prevented.  

A lot of what I am concluding in terms of reliever value is based on the trends I see from what I consider to be pretty savvy teams.  Honestly, I have no clue as to the specifics, but it's hard for me to believe that teams like the Astros would give up what they did for Giles or the Red Sox for Kimbrel if there wasn't situational value applied to those players.  

 

[Oz27]

2 minutes ago, Dochalo said:

and then so is the situation at that point.  It's easier to get out carlos perez with the bases loaded than Mike Trout with no one on, but the probability of giving up a run is much greater in the Perez situation.  

You are more inclined to use the better pitcher when the extremes are greater though like if you are up by a run vs. down by 3.  If you don't think there is a difference, then why have you pushed for the flexibility of relievers to be available in higher leverage situations.  I know you have coupled that to the batter, but your opinion in the past was that both were important.  

The closer the game, the greater odds that one play, one strike or one swing will change the outcome.  Each at bat becomes more and more critical as the game progresses because there are fewer opportunities to change the outcome.  The reliever can't control it yet a position player doesn't determine what position they are going to play.  Yet we punish the position player for doing poorly.  Why? because it affects the outcome of the game.  

You know I am as stat inclined as anyone, but even with my limited experience playing there is a factorial difference ie an increased degree of difficulty in performing the same task when there is less time to compensate in the future or in environments where the odds of scoring runs are greater.  

You don't think that the game situation affects performance?  I am not saying that there is basis for clutch or the same player performing differently in more difficult situations than in less difficulty ones.  I am saying that some relievers in particular are put more exclusively in those more difficult situations and regardless of whether they have control you give them credit for it just like you give credit to Escobar for playing 3rd even though he's got no control over it.  I am not saying it has to be positive credit.  Just credit.  

Yeah, I believe the best pitchers should be pitching in the highest leverage situations. But I don't think we should be giving them extra credit for doing that. Isn't that what the baseball community mistakenly did with wins or save totals for so long? I don't consider the pitcher's job to be any different whether it is the fourth, the seventh or the ninth. Whenever it is, the job is still to prevent runs from being scored right? I don't see that as contradictory to the belief that the best pitchers should be in the game for the highest leverage moments. You want them in there them because that's when the team can get best value from him. But it seems unfair to give that pitcher any reward or punishment for happening to be brought in for that point of the game. If two pitchers face three batters and produce identical results (let's say out, solo homer, out, out) but one does it in the fourth and another does it in the eighth, then the way that is judged by WPA is totally different. But how is that fair? They've done the same thing. They have both given up a run and that run counts the same on the scoreboard. While I get what you're saying with the Escobar argument, I view it differently and I suppose I've already partially outlined the reasons why. Plus, none of the mainstream offensive stats we use credit Escobar based on the point at which he was playing 3B or at the plate. That isn't different to crediting a pitcher based on innings pitched instead of when those innings were.

You point out WPA does the same thing for hitters and I'd make very similar arguments about the flaws with that. For hitters, that's why WPA is used as more of a tool to make fun graphs with instead of something that tells us anything meaningful about a player, is it not? It may not be a terrible way to judge someone, but I don't think it's a particularly good one.

4 minutes ago, Dochalo said:

There should be a low correlation between xFIP and WPA because one measure value independent of leverage and the other is linear weights of what happened.  Neither takes batter quality into account btw.  Both assume that it will average out over the course of the season yet we all know that's not true when it comes to relievers.     Another problem with comparing WPA to xFIP is that one is cumulative and the other is rate based.  

The Cam example is pretty tough to get on board with considering that his leverage was really low early in the year in that he was coming off a pretty bad season where it was tough to trust him.  Then it escalated until he got hurt.  

I guess I am also implying that I don't think xFIP is a great tool to measure reliever quality.  Being a fly ball pitcher without giving up hrs is a skill.  Weak contact is a skill.  

I don't think WAR or WPA tells the entire story for relievers.  I have to believe that major league teams have come up with a way to conglomerate WAR/FIP/xFIP with leverage and batter quality.  I think they have incorporated that into their team WAR calcs along with their proprietary defensive metrics and their makeup metrics.  

I understand they are measuring things in different ways, but shouldn't WPA correlate more strongly with our best pitching metrics if it were demonstrative of ability? Also, shouldn't the year-to-year WPA correlation be stronger if it were telling us something about a pitcher's ability instead of just, well, noise? For pitchers the year-to-year correlation is .266, which is lower than the year-to-year correlation for every somewhat mainstream pitching stat (the lowest is ERA at .311, while SBERA is up at .433).

Out of interest, why do you differentiate xFIP for relievers as opposed to starters? We're measuring the same thing, right? Pitching is the ability to prevent runs being scored and, DRA aside (which I would always use in preference), it is the best measure we have to reflect past performance and, to an extent, indicate future performance. I agree with you that being a fly ball pitcher who doesn't give up homers is a skill, we see guys who regularly under or over perform their FIPs and we need to be mindful of that when measuring pitchers using a FIP-based stat. But I still believe it's the best individual metric we have, besides DRA, to measure a pitcher's performance. I'm sure you're right about teams having come much closer to perfecting the right mix of all the factors we are talking about in whatever numbers they are using. DRA is the closest the public sphere has come to that.