Quote:
Originally Posted by injury log
Tango's data only covers MLB; there is absolutely no reason to think the model applies to minor leaguers- it would be ridiculous to analyse data on the top 1% of a population and hope the conclusions would apply to the other 99%.
|
I decided that I need to respond to this, just because it's an argument that comes up several times and is not correct. It is, in fact, quite normal to use very small sample sizes to adequately analyze entire populations. Small sample sizes are the entire foundation of proper polling ("good"
national polls are often made up on less than 1,000 people and are considered accurate with a 95% confidence factor if the sample has been properly designed). The challenge is to ensure your sample population is comprised of a group that is truly representative of the entire population. In this case, we're attempting to find a population of baseball players who are representative of all players who played baseball.
The Tango data has a few areas of concern here, which I'll touch on. But realize that Tango data is definitely "correct" in that it is real and is taken from a sample that consists of all qualifying baseball players who played from 1919-1999. This is a pretty large number of baseball players. If you are going to turn your back on the idea of using Tango as your backbone I figure, even with the sample issues below, you really ought to be replacing it with a system supported by research that is at least as good as Tango's.
Issues related to the Tango data:
Selective Sampling: Those players in the sample were considered good enough that a manager played them enough to qualify. There certainly exists a chance that people who make the majors develop differently than people who do not, but realize there also exists a chance that all baseball players do develop in the same basic fashion. Since we have no other data to fall back on, we get to use our human intuition on what we think is the right bet. Personally, if I were forced to bet my house on which was more correct, I would bet on the data we've got as being right because as you look at the breakout of skill acquisition and degradation, it just makes sense in the big picture. Again, though...that last is my opinion.
Regression: In order to remove an element of random chance, Tango regresses the raw data to mean. There are always arguments that suggest one level of regression is better than the next. This influences the shape of the curves a bit, and could be tweaked one way or the other a little in order to enhance gameplay without drawing too many complaints from statistical wonks.
Apples & Oranges: Pitcher data (especially) is based on slightly different parameters than pure OOTP rating schemes are, hence require a bit of leeway in their application. This is why I really have not pressed really hard to have the curves match perfectly, just that the shapes are proper. This is why OOTP users are moderately comfortable with hitters--the shapes match Tango pretty well. Someday maybe we'll get there with pitchers.
I hope no one would point to Tango and say "That's the bible." I know Tango himself would not. But it is the best real data we have. I have yet to see any argument to go a different direction that seems to be based on anything beyond vaporous thoughts of how things "ought to work."
The closest we get is the very fine work Garlon has done with historicals, which is designed at the high level around the thought of "force the stats that really happened in 1922 to happen in 1922." Personally, I think that's the wrong way to go unless your goal is to design an air-tight historical sim and nothing else. But though I wouldn't write the game around it (I prefer a game the is based on sound processes rather than constrained by forced outcomes) I respect the idea immensely and find his work to be impeccable.