Da Book

[BPS]

About The Book, which I guess is supposed to be the current state-of-the-art for sybermetric work:

I've only looked at the first half of Chapter 2 on hit/cold streaks, but this one initial statistical study should give enough information about the quality of the work that appears later in the book.

What I have observed: the authors present lots of numbers and occasionally throw out terms like "statistically significant." I guess that's good.

But this study is clearly the work of amateurs.

I'll only point out one major issue: key to their work is a measure of xOBA that we would "expect" a hitter to achieve if they are not hot or cold. They want to compare this "expected wOBA" with the "actual OBA" for players in the games immediately following a five-game "hot" or "cold" streak.

They argue that IF a five-game hot/cold streak is predictive THEN a hot hitter should have a higher xOBA in the games after the initial five games than we would expect. They find that a statistically significant difference exists between the actual xOBA and the expected xOBA. This means hotness/coldness is indeed predictive.

But they then go on to muddy the waters by saying that they don't think the size of the effect is large enough to qualify the statistically significant effect as "predictive." But, unfortunately, they don't attempt to quantify what size is necessary for it to be important: they just say that, IN THEIR OPINION, it is too small. But a statistical analysis can't depend on such subjective unsupported claims...but they continue on as if they have provided compelling proof the size is too small. (Although they haven't).

But I want to point out one of many major weaknesses of their work on hitting streaks, to wit: how they determine the key measure "expected xOBA."

They state that they used as expected wOBA the average xOBA for 3 years (the years before the year of the streak, the year of the streak, and the year after the streak). But they don't explain why they use this 3-year period nor do they indicate how their results might differ if they used, say, only a two-year period. That is, they don't provide any reason to find their use of their measure of wOBA is appropriate. (Not do they, as far as I could tell, say what they do in the case of a player who didn't play the year before and/or the year after.)

But their results about hotness/coldness only makes sense if they can provide a compelling reason to accept that the 3-year wOBA average they use is a good measure of expected wOBA. But they don't even try.

They seem to think that merely uttering the statement, "We will measure X by using Y" that they have provided a compelling reason to accept X as a measure of Y. But good statistical work requires that they consider in detail whether X is indeed a good measure of Y and what happens if you use X' (or Z) in place of X. That's just standard operating procedure.

So, despite all the numbers that appear and despite the frequent use of the term "statistically significant" in the text, I must conclude the authors of The Book are not ready for prime-time.

I have a feeling that the standards for sybermetric work is pretty low and, I guess, by these standards The Book is pretty good. But from other standards, say of someone who does number crunching for a living, the book falls far short of the mark. I think they might have potential as number cruchers, but they need to expand their horizons about statistical work by looking outside the sybermetric world if they want to find models for good statistical work.

[noel33]

Originally Posted by BPS

But they then go on to muddy the waters by saying that they don't think the size of the effect is large enough to qualify the statistically significant effect as "predictive." But, unfortunately, they don't attempt to quantify what size is necessary for it to be important: they just say that, IN THEIR OPINION, it is too small. But a statistical analysis can't depend on such subjective unsupported claims...but they continue on as if they have provided compelling proof the size is too small. (Although they haven't).

This isn't really what they say in The Book. What is said is that cold players have a wOBA of .006 points lower than their 3 year average and hot players have wOBA of about .004 points higher on the game after a 5-game streak. They view this as predictive, but it is a small enough number to make it not worth benching a cold Alex Rodriguez in favor of a hot Cody Ransom. If you want to know exactly how much that's worth, the 2 differences correlate to a difference of 2 (.004) or 3 (.006) runs per 550 plate appearances. Assuming the batter is batting leadoff (4.79 PA's a game on average) the team will score .0035 more runs on average if he is "hot" and .0052 runs less if he is "cold".

I'm not really sure what you want them to use for expected wOBA. The 3 year average is commonly used in order to get a best guess of a players talent, which we obviously have no true number for. The "compelling reason" is that this is their best guess for the talent of the player. It might be possible to come up with a better number, but the number used is more than good enough to prove that streaks don't have a supernatural effect because the numbers presented, should if nothing else, tell you that the 3 year wOBA is a better predictor of future success than how well the batter was hitting during the streak. That chapter of The Book not intended to prove that streaks have an effect of exactly .00378978 on wOBA, it's intended to prove that streaks have a low effect on wOBA, and should only be considered when making roster decisions after more important variables (such as a 3 year average of the batters wOBA) after taking into account.

[BPS]

Originally Posted by noel33

This isn't really what they say in The Book.

We read: "However, what little predictive value there may be is so small as to be relatively unimportant, in our opinion." But their muddying the waters about the meaning of "prediction" is only a lessor concern of mine.

Originally Posted by noel33

I'm not really sure what you want them to use for expected wOBA. The 3 year average is commonly used in order to get a best guess of a players talent, which we obviously have no true number for. The "compelling reason" is that this is their best guess for the talent of the player.

It shouldn't be the reader's job to come up with alternatives nor to get the data and run alternative studies.

It is the statisticians' job to make a good faith effort to consider the pros and cons of the proxies they use AND to present information to the reader about the consequences on their conclusions when different proxies, variables, assumptions, etc, are used.

Justifying the use of one particular proxy by saying it is "commonly used" (with no further consideration) is a mark of an amateur number cruncher.

For all we know, IF they used a 4-year average wOBA or only the current year wOBA they would have got very different results. For all we know if they used batting average as their determinant of hotness/coldness they might have come up with different results. For all we know if they had used only the top and bottom 1% of hot/cold batters rather than the 5% they used, they might have come up with different results.

But we don't know this because they only presented information about the narrow approach they used (and we don't know if they did others but the results were not to their liking and so they didn't report them). So we really haven't learned anything other than that one narrow incomplete study showed...

Now...the way the authors of The Book did their number crunching likely meets the standards for sybermetric work. But this does not mean they did good work as the standard of sybermetric work appears to be that of a hardworking, but amateur, statistician.

The above is just a tip of the iceberg as far as problems with The Book's analysis of hotness/coldness but provides strong evidence that the authors would be well-advised to become familiar with the standards for empirical work used by more professional statisticians.

[noel33]

In response to the first point it was their view that 2 or 3 runs a season was worth little predictive value, you're obviously free to disagree if you'd like.

Secondly using batting average would be stupid, the whole point of linear weights is that they accurately represent offensive production, batting average does not. The reason they use 3 year wOBA is to show that 3 Year wOBA has a much greater predictive value than the evident by looking at the expected wOBA versus the wOBA during the streak, which is pretty obviously true from looking at the numbers presented. If they had used 5 year wOBA as well and found that it predicted it numbers far different than 3 year wOBA all that proves is that the 3 year wOBA was a far better predictor of future success than 5 year wOBA. If 5 year wOBA had been a better predictor all it would have shown was that streaks have even less predictive value.


I suppose you could change the parameters to the top and bottom 1% and run a further study on "super streaky" players but the fact that they didn't doesn't invalidate the research done on players in the top and bottom 5%, and you're invariably gonna run into sample size problems the more you tighten the bounds.

[BPS]

Originally Posted by noel33

In response to the first point it was their view that 2 or 3 runs a season was worth little predictive value, you're obviously free to disagree if you'd like.

This presumes their estimate of the size of the benefit of hotness/coldness are correct. But this is a big presumption.

But EVEN IF their study of five-game hotness/coldness was compelling, they are presuming that their finding is valid for, say, a 7-game period of hotness/coldness. But they have no way of knowing that. That is, they go from one narrow study (one particular way of measuring the predictiveness of a 5-game period of hotness/coldness) to the general case of all cases of hotness/coldness.

It's not valid to do that.

Originally Posted by noel33

Secondly using batting average would be stupid, the whole point of linear weights is that they accurately represent offensive production, batting average does not.

It would not be stupid at all. The common definition of hotness/coldness is most refers to batting average and, so, if they want to discuss hotness/coldness it makes sense to see if their results are the same if they use the BA instead of wOBA. This is because a finding about wOBA does not give any necessary information about BA.

Originally Posted by noel33

If they had used 5 year wOBA as well and found that it predicted it numbers far different than 3 year wOBA all that proves is that the 3 year wOBA was a far better predictor of future success than 5 year wOBA.

No. It would have showed the results they reported in TB were not robust (even with a small change in their proxy) and, so, was not adequate for making pronouncements to the value of paying attention to hotness/coldness.

Originally Posted by noel33

I suppose you could change the parameters to the top and bottom 1% and run a further study on "super streaky" players but the fact that they didn't doesn't invalidate the research done on players in the top and bottom 5%, and you're invariably gonna run into sample size problems the more you tighten the bounds.

I don't see any problem here with sample size. The number of observations they would be using the 1%/1% cut-off for hotness/coldness would still have been, I believe, pretty darn huge.

My point is that they haven't presented any compelling evidence about hotness/coldness because they didn't make any effort to see how robust their findings were to even small changes in their study.

Not presenting information about robustness = not compelling.

[Flying Dutchmen]

Here are my thoughts as someone who would not be considered an "amateur number cruncher":

Originally Posted by BPS

It shouldn't be the reader's job to come up with alternatives nor to get the data and run alternative studies.

That's what science is - presenting findings and encouraging other interested individuals to change assumptions, measures, specifications and to see whether a given set of findings holds up. No one ever performs a complete analysis in the first attempt.

Originally Posted by BPS

Justifying the use of one particular proxy by saying it is "commonly used" (with no further consideration) is a mark of an amateur number cruncher.

So as someone in academia, every time an article in any scholarly journal uses a commonly used indicator (say measuring political affiliation on a 7-point range) we are amateur number crunchers? Interesting.

Originally Posted by BPS

For all we know, IF they used a 4-year average wOBA or only the current year wOBA they would have got very different results. For all we know if they used batting average as their determinant of hotness/coldness they might have come up with different results. For all we know if they had used only the top and bottom 1% of hot/cold batters rather than the 5% they used, they might have come up with different results.

But we don't know this because they only presented information about the narrow approach they used (and we don't know if they did others but the results were not to their liking and so they didn't report them). So we really haven't learned anything other than that one narrow incomplete study showed...

Sometimes something that simple is noteworthy. An initial study that shows what they find, even if doesn't meet your extensive robustness check standards, is still worthy of publication and thought-provoking.

Roubstness is typically a second-level analysis or one reserved for pieces that focus solely on one topic in the scholarly world in my experience. Only in cases of statistical simulation is it a cardinal sin to not consider robustness. If the result truly doesn't hold up, any other or lay person can run a similar study with their own measures and standards and present their results. Scholars can then sabermetrician debate the two findings and science is advanced.

Feel free, of course, to disagree.

[noel33]

Using BA is dumb because the book is essentially presented as a how-to for managers. The first real chapter illustrates what wOBA is and why it is better than BA. If you want a book that addresses nifty things fan wonder about I recommend Baseball Between the Numbers though I believe some of the information within there has been refuted.

Regarding the 5 game hot streak point you apparently didn't read the next page . They repeated the study with a 7 game hot streak and found roughly the same results (a .004 rise in wOBA for "hot hitters" and a .008 decrease for "cold" hitters).

As for the rest of your argument I'll just point to what The Flying Dutchmen said, and to note that right now our best theory regarding hot streaks and cold streaks is that they have a relatively small amount of predictive value. This is based on evidence both in The Book and elsewhere. The best evidence for the theory that hot streaks and cold streaks have a large amount of value that I have seen is that a certain person played baseball and "was in the zone" when they were getting a bunch of hits over consecutive games which really doesn't hold up.

[BPS]

Originally Posted by Flying Dutchmen

That's what science is - presenting findings and encouraging other interested individuals to change assumptions, measures, specifications and to see whether a given set of findings holds up. No one ever performs a complete analysis in the first attempt.

I'm not calling for a complete analysis of all possible alternative explanations.

But for a given model using particular variables/proxies: you'd agree, would you not, that any empirical study that uses a debatable proxy would be stronger if it considered how sensitive its results were to changes in the proxy? Indeed, is it not reasonable to call a study that did not consider obvious sensitivity issues uncompelling, as I have?

Originally Posted by Flying Dutchmen

So as someone in academia, every time an article in any scholarly journal uses a commonly used indicator (say measuring political affiliation on a 7-point range) we are amateur number crunchers? Interesting.

Are you implying that the analysis of hotness/coldness presented in The Book meets the standards of your field? If you say "yes" I guess this reflects well on sybermetrics. Or, perhaps, it reflects poorly on your field. I leave it to the reader to derive the appropriate proof as an exercise.

And, re the use of "commonly used indicator" without thought: arguments have been made that doing such a thing possibly introduces bias and/or invalidates tests of statistical significance. If you're fine with that, well....

Originally Posted by Flying Dutchmen

Sometimes something that simple is noteworthy. An initial study that shows what they find, even if doesn't meet your extensive robustness check standards, is still worthy of publication and thought-provoking.

I'm not calling for the withdrawal of the book by the publisher! I'm merely saying their results on hotness/coldness is not compelling.

Originally Posted by Flying Dutchmen

Roubstness is typically a second-level analysis or one reserved for pieces that focus solely on one topic in the scholarly world in my experience. Only in cases of statistical simulation is it a cardinal sin to not consider robustness. If the result truly doesn't hold up, any other or lay person can run a similar study with their own measures and standards and present their results. Scholars can then sabermetrician debate the two findings and science is advanced.

We might have different notions of what "science" involves (and what intellectual activities are, and are not, property labeled "science"), but that is not unexpected.

Yes...in some fields little concern is given to the robustness of the statistical results presented. Instead, folks in some fields are impressed with the use of data and of terms such as "statistical significance." They might be impressed ("we're doing science!"), but that does not make the statistical work good.

I'm not saying people shouldn't buy, read, and like The Book. I'm just saying that the statistical work presented (at least the part I've read) is not compelling. I don't know the background of the authors of TB, but I'm not sure they see themselves as fully professional statisticians. That's okay, they can always learn more and become better at it. I bet that's their goal after all.

But if someone in MLB accepts their conclusion about the unimportance of hotness/coldness in lineup decisions (based on the analysis in TB), such a person is making a mistake.

Perhaps the book gets better as it goes along. Someday I'll find out when I have the time.

[BPS]

Originally Posted by noel33

...right now our best theory regarding hot streaks and cold streaks is that they have a relatively small amount of predictive value.

If I recollect correctly, studies on hot/cold streaks that reject their existence typically conclude that the frequency of hot/cold streaks are what would be predicted if things were the outcome of a random process.

That is different, however, from saying that streaks are caused by random processes. If you gather 1,000 people you'd find their heights are distributed as if they were produced by a random process. But this does not mean that a given person's height is indeed the product of a random process (it is often systematically related to the heights of his/her parents).

But, equally important, let's say player A believes he's on a hot streak. But his manager says, "sorry, I'm sitting you down for the next week because your hotness is really a false impression given by a random process and I want to give another player (who is now hitting .125) a shot. You're not really responsible for your good performance recently. Them's the facts, son."

Would such a player be happy? Of course, the answer is YES!... Because people ARE machines their behavior is exactly the same every at bat and no psychological or physiological issues are involved. People are just mindless, thoughtless, emotionless machines. No ball players would ever be upset when sent to the bench after 10 games of hitting .400.

[noel33]

Originally Posted by BPS

If I recollect correctly, studies on hot/cold streaks that reject their existence typically conclude that the frequency of hot/cold streaks are what would be predicted if things were the outcome of a random process.

That is different, however, from saying that streaks are caused by random processes. If you gather 1,000 people you'd find their heights are distributed as if they were produced by a random process. But this does not mean that a given person's height is indeed the product of a random process (it is often systematically related to the heights of his/her parents).

But, equally important, let's say player A believes he's on a hot streak. But his manager says, "sorry, I'm sitting you down for the next week because your hotness is really a false impression given by a random process and I want to give another player (who is now hitting .125) a shot. You're not really responsible for your good performance recently. Them's the facts, son."

Would such a player be happy? Of course, the answer is YES!... Because people ARE machines their behavior is exactly the same every at bat and no psychological or physiological issues are involved. People are just mindless, thoughtless, emotionless machines. No ball players would ever be upset when sent to the bench after 10 games of hitting .400.

To the first point random chance causes the hot streaks. The "process" is the at bats given the fact that the batter has an expected wOBA of x we would expect a a streak distribution of y. Such distributions can easily be explained by random chance. We use the random chance idea because right now it is the best possible explanation for hot streaks. Maybe in the future it will be shown that tall players over perform after hot streaks and short players underperform causing the trend that falls in the middle. At that point we can change the theory, but until then we should use the theory best supported by the available facts.

To the second point I certainly agree to an extent, but we have to be careful not to take this too far. As a Rangers fan I witnessed the hullabaloo surrounding Michael Young this year and his move to 3rd base after winning a gold glove at shortstop. That was a good move that made sense while pissing off a player. The manager's job ideally is to balance short term gains (playing the better hitter in this game) versus long term gains (not pissing off say a young hitter who thinks he's found his groove). This is not a black and white issue.

[The Wolf]

Originally Posted by BPS

But this study is clearly the work of amateurs.

I can't wait until Tom Tango sees _this_ thread.

[Curtis]

Originally Posted by BPS

But, equally important, let's say player A believes he's on a hot streak. But his manager says, "sorry, I'm sitting you down for the next week because your hotness is really a false impression given by a random process and I want to give another player (who is now hitting .125) a shot. You're not really responsible for your good performance recently. Them's the facts, son."

Would such a player be happy? Of course, the answer is YES!... Because people ARE machines their behavior is exactly the same every at bat and no psychological or physiological issues are involved. People are just mindless, thoughtless, emotionless machines. No ball players would ever be upset when sent to the bench after 10 games of hitting .400.

Are you a Mets fan? I ask because the Mets did this twice last week. In their 8th game of the season they sat rightfielder Church, who was hitting around .480, in order to get Sheffield, who was at .250, some playing time. In game twelve they sat leftfielder Murphy, who was batting around .360, for the same Sheffield, who was by now down to .143. The same game they sat second baseman Castillo, who was hitting just about .400 (.500 over his most recent eight games) to get Tatis, who was .000 some playing time.

Were Murphy, Church and Castillo happy? Maybe not, but they were professional enough not to let it bother them. Obviously some real life managers feel that making sure their subs don't get rusty is more important than playing the hot hand.

[Raidergoo]

Originally Posted by The Wolf

I can't wait until Tom Tango sees _this_ thread.

He does seem to know how to spell the word sabermetrics, eh?

[The Wolf]

Originally Posted by Raidergoo

He does seem to know how to spell the word sabermetrics, eh?

Like I said, I can't wait.

But then I always love discussions about "sybermetrics."

[BPS]

Originally Posted by noel33

To the first point random chance causes the hot streaks.

Actually, there is no evidence about the causal relationship. At best, some have shown that, sometimes, the evidence is consistent with randomness. But it is also consistent with other explanations.

At issue is the fact that a theory that PREDICTS (or is consistent with) something is not necessarily a theory that properly identifies the actual causal mechanism. Yes, some streaks are likely the result of pure randomness. But other streaks are not.

Originally Posted by noel33

To the second point I certainly agree to an extent, but we have to be careful not to take this too far. As a Rangers fan I witnessed the hullabaloo surrounding Michael Young this year and his move to 3rd base after winning a gold glove at shortstop. That was a good move that made sense while pissing off a player. The manager's job ideally is to balance short term gains (playing the better hitter in this game) versus long term gains (not pissing off say a young hitter who thinks he's found his groove). This is not a black and white issue.

Once one accepts that human performance is not constant like a machine but varies with a wide variety of human factors (emotion, confidence, anger, tiredness, etc) then the door opens phenomena like streaks being (sometimes and/or partly) determined by non-random factors.

But I agree that it is NOT a black and white issue...although pro-randomness people seem to think their issue is pretty black and white (but it's not).

[BPS]

Originally Posted by Raidergoo

He does seem to know how to spell the word sabermetrics, eh?

I guess the title of the tread, "Da Book," didn't clue in everyone that I wasn't going to use standard spelling for everything. (I guess I was too subtile, subtyle, subtle) Oh, well.

[BPS]

Originally Posted by Curtis

Maybe not, but they were professional enough not to let it bother them. Obviously some real life managers feel that making sure their subs don't get rusty is more important than playing the hot hand.

And you know it didn't bother them because...?

People are necessarily subjected to emotional, psychological, physiological forces, along with lots of other things. When a batter believes he is on a hot or cold streak, this very possibly affects his psychological and physiological responses when he's at bat. When a pitcher faces a batter who is known to be on a streak (hot or cold) this very possibly affects how the pitcher pitches to the batter.

Yes, some folks are less subjected to these factors than others, but it is rare the person who is totally free of such things. Thank goodness.

[noel33]

Wait what other explanations are streaks consistent with that aren't bull**** after the fact proclamations, or vague declarations like confidence plays a large role?

[noel33]

double post see below

[noel33]

Look there are two theories

Theory #1
Streaks are caused by randomness.

Reasons Given:
The amount of streaks correlates with the amount of streaks we would expect if the game was played by robots. After 5 games of hot hitting the batter shows minimal effects from the streak. Same with after 5 games of cold hitting. Also the same with 7 games of hot hitting, and 7 games of cold hitting. just for kicks - Basketball players show no increased ability to hit a shot after making the previous shot.

Theory #2
Streaks are caused by human factors

Reasons Given:
People are not robots.


And you're complaining about the amount of evidence given in the 1st case? The first theory has a mountain of evidence behind it. The second has well, none. It's a lovely thing to think and its intuitive but whats intuitive isn't always right. Thus we should act as though the first theory is true until someone else comes up with a theory that explains streaks better.