Dynamic Player Scouting -- A Study of Player Assessment IRL and OOTP

[sebastian0622]

Tired of toggling the ratings scales trying to get something realistic: a system that allows you, as a GM, to see the basic ratings of your players (and draftees) as judged by your scouts but still is inexact enough to make stats as vital to player evaluation as they are in real life?

You can't do it, because how we look at players in OOTP is all wrong! IRL, the scouting process is a very difficult inexact science. Scouts gain a better idea on how good players are with time, and this needs to be seperate from player development. Players' ability to develop is one issue, scouts' ability to pinpoint a player's ability as his career advances is another issue, one that isn't modeled in OOTP! In OOTP, your scouting report on Albert Pujols has as much a range for error as the scouting report on a 17-year old high school kid you're thinking about drafting who missed half of his senior season due to injury. That's nonsense.

Today, I was trying to tweak the ratings scales to make the scouting game more difficult and realistic. I hid star ratings, put abilities and others on a 1-5 scale, and talent on a 1-10 scale. This, currently, is as realistic as I think we can get because of the following limitations:

While you could play with talent only on, and rely on stats, that is not practical. It defeats the purpose of having scouts if the GM--or player in this case--has to pour over seasons of numbers, splits, game logs (to see how a player performs in different parks, and to see what quality of pitching he has faced), and other data just to arrive at a general idea of the player's ability to hit doubles. That is the scouts job! So stats only isn't too realistic either...at least not without scouting reports about 20 times more detailed than currently exist in OOTP and infinitely varied...and real-life scouting reports are basically ratings anyway, so the ratings system lends itself just fine to realism. The idea is to get the right hybrid of ratings/stats, however.

IRL (and therefore how it should be in OOTP!), players get more accurately scouted the older they get, the more games they play, the more time they spend in the minors and majors. Therefore, in a perfect OOTP system, I would like to see a dynamic ratings system. Here is the basic outline (edit: with bumping this and re-reading it, I actually think I like a system like I talk about in post #3, below, better):

There would be the following scouting accuracy levels, basically. While not set in stone, here are some preliminary ideas for how it could work. This would be subject to adjustment, as level D may be best as ability 1-5, and E as ability 1-10. Read on.

lvl...talent...ability...other...stars?...synopsis
A...1-5.....1-5......1-5.......NO....HS players
B...2-8.....1-5......1-10.....NO....College, most A players
C...1-10....1-5.....1-10......YES....Most AA and newer AAA players
D...1-10....1-10....1-100....YES....AAA-first few yrs in majors
E...1-10....1-100...1-100...YES....most major leaguers, veteran AAA
F...1-100...1-100...1-100...YES....very veteran players

Rookie draft, high schoolers: Level A

Not only have you realistically recreated the difficulty and vagueness of scouting high school players, you have just solved the age old problem of "You just look for the most stars and draft a player, and there are barely ever diamonds in the rough!" With this method, the top high schoolers would still be obvious standouts, but after the first round or two things would get a lot more difficult (and realistic). Better have a good scout, because your late 2nd round pick may be a bust if your scout thinks he's a 3 talent contact hitter with 5 talent power but he really ends up being a 2 talent contact hitter, and his 5 power is really about an 81 on the 100 scale (which you won't find out until later in his career, of course...keep reading)

Rookie draf, college players: Level B

Quite a bit better scouted than high schoolers, but still quite vague. Realistically recreates the RL dichotomy of "I have a pretty good idea this college player is a safe bet, but I'm going to take a risk on this high schooler who I'm not sure on because he might have more potential" and vice versa.

Minors, Low

After players first full season or thereabouts in the minors, they should move to the next scouting accuracy level. This means most of your college players who are a season out of the draft will be on level C (and have star ratings), likely around the same time many move to AA and your top pick or two might move to AAA. Your high schoolers will be as well scouted (Level B) as if they had played 4 years in college, after spending a full year in A ball with your scouts watching them everyday. *Note* Mind that you still have the "top 100" feature to see where your top prospects stand among others (although I think that list should be randomized, as in a 3rd party scout--the newspaper writers--being used to evaluate and list the players).

Minors, High

This is where things the importance of this system comes into play. I believe only after a player plays AAA ball for a season or two should scouts have an accurate enough report on him for him to be level D, and they should maintain this level of accuracy through the next 2-3 years in either AAA ball or the majors before moving up to Level E, which is where most of your major leaguers with 3-10 years of experience will fall. Only 30-somethings who have been in the league long enough to consistently be scouted should reach Level F. Many players may never be scouted that accurately. As in real life. Give me 10 scouts and have them report on Billy Koch, and you will get 10 varying opinions about his abilities but most will be somewhat close. Koch, despite being a veteran, is still only scouted accurately to Level C or D. On the other hand, there is a pretty good consensus on how good Jim Thome and Gary Sheffield are, and they are Level F. The possibilities to recreate real-life open up!

Keep in mind that this in no way affects the way players develop or how good they are at a certain age, it merely affects how they are scouted by your (and any) organization. You may have a 21 year-old Dontrelle Willis who you move right up through the minors and put in the majors when he is only scouted to Level C. You aren't able to pinpoint his abilities, and it is a risk putting him in the majors, but based on his performance in the minors, you decide to do it! And what happens then? Uncertainty. Realism!

This would be a major change I am sure, and probably not a viable upgrade at this time, but I believe this needs to be the future of baseball simulation.

[randomlefty10]

I wholeheartedly agree. If there was one thing I'd ask them to work on this would be it. I also understand that would be a major change, but it seems doable since it's all build on strict levels and it's not very vague. I think this would also neccesitate more varied scouting reports, but they would actually become useful in the game. I for one don't really ever look at scouting reports.

As a suggestion, if writing a nuch of new scouting reports was time trouble, I'm sure there are plenty of people here would help pen some standard fill in the blank phrases. There's a lot I would do for free if it meant making the game deeper in depth.

[sebastian0622]

bump

The above sort of suggestion covers precision. As for accuracy with scouts turned on: scouts should be able to more accurately scout a MLB player than a SS in some winter league in Cuba. There should be little variation in ratings for scouts studying Albert Pujols, to use an example in the original post. There should be greater variation based on how short a player has been playing professional ball.

And actually, thinking about this whole thing, the accuracy deal may be the best way to go, scrapping the precision parameters in the original post. If you make HS players scoutable on whatever scale the player has set in the options, but make those ratings very volatile, I think you accomplish the same things: making HS players riskier than college players in the draft, for one, as college players would be a little more accurately scouted. So a HS player with an actual contact rating of 60 might show up on your draft screen with any number in a range of 20 (50-70), but a college player with actual contact of 60 might return any number in a range of 15 (52-67). A player with a year in pro ball might return any number in a range of 10 (55-65). Then each year in the minors might cut the range down by two; each year in the majors might cut it down by 4. You get the drift.

Then minor league stats play a major role, as you don't know whether your SS in AAA who has been in the system for two years and who your scouts say has a contact of 51 is really a 43 contact (not ready for the bigs) or a 59 contact (ready for the bigs). So, you can go off of stats or let him sit another year in the minors and let the scouts watch him play a little more, which would make his player evaluation more accurate (if he stayed at 51, you'd know he was between 45-57), and more importantly, would yield a larger sample size of stats by which to decide. I think that creates realism.

You can play around with the ranges, but you get the idea.

[Mike Donlin]

These are all great ideas, but it is not as though a certain rating automatically translates into a certain performance, even on a 1-100 scale, so there is *already* built into the game a certain imprecision that IMO produces pretty realistic results.

For example, I play with 1-100 scale for ratings and talent, no coaches and scouts, and even with Jayzone's Rating Translator, my players rarely perform exactly as their ratings say they should. They perform within a range, to be sure, but with enough variance for my taste, I guess.

I always viewed ratings as some kind of Pecota-like projections or MLEs--what a player, given his talent and past statistical performance, is likely to do this year. Likely but not guaranteed.

So I'm a little worried that these additions would complicate things, but perhaps that's good, since the thing it seeks to recreate is in fact pretty complicated. Still.

[sebastian0622]

Originally Posted by Mike Donlin

These are all great ideas, but it is not as though a certain rating automatically translates into a certain performance, even on a 1-100 scale, so there is *already* built into the game a certain imprecision that IMO produces pretty realistic results.

For example, I play with 1-100 scale for ratings and talent, no coaches and scouts, and even with Jayzone's Rating Translator, my players rarely perform exactly as their ratings say they should. They perform within a range, to be sure, but with enough variance for my taste, I guess.

I always viewed ratings as some kind of Pecota-like projections or MLEs--what a player, given his talent and past statistical performance, is likely to do this year. Likely but not guaranteed.

So I'm a little worried that these additions would complicate things, but perhaps that's good, since the thing it seeks to recreate is in fact pretty complicated. Still.

The difference is that the current variation is a function of randomness. The ratings tell you the player's ability as is. So if you have a player in AAA who outperformed his ratings, that doesn't project to the future, it just means he got lucky. So basically, the stats mean nothing in terms of projection when you have the ratings to go off of.

A more realistic scouting system would make both stats and ratings valuable for projection.

[lstephen666]

Hi Sebastian,

Your idea seems really interesting and I like it from the point of view of developing players. I was wondering how you saw it as players declined in later years.

I think there needs to be some "inexactness" in determining whether a player is in a slump, or whether it is a sign of a declining ability.

I guess the issue is that if Pujols was on a 1-100 scale then you would immediately know whether it was just a slump or whether he was beginning his decline through age.

[sebastian0622]

Originally Posted by lstephen666

Hi Sebastian,

Your idea seems really interesting and I like it from the point of view of developing players. I was wondering how you saw it as players declined in later years.

I think there needs to be some "inexactness" in determining whether a player is in a slump, or whether it is a sign of a declining ability.

I guess the issue is that if Pujols was on a 1-100 scale then you would immediately know whether it was just a slump or whether he was beginning his decline through age.

I guess I didn't think of that, but you're right--the decline phase is also a possible application of the same sort of thing. I figure if this was type of scouting system was implemented in any way, the range of possible ratings would be highlighted on the traditional ratings bars for quick reference, as well as have numbers somewhere. Anyway, you might know Pujols' ratings with a real slim margin for error in his late 20's, but as he hit the other side of 32 or 34, or as it varies by player, you could have the range go back up again in the offseason to reflect a certain degree of uncertainty of whether or not he's going to come back at the same level.

Hell, you could even eliminate the super-unrealistic talent jumps and declines by instead implementing the range, which would provide a lot of the unpredictability of individual player development and decline present IRL. So a prospect's range in a potential area might go up a little, but that doesn't necessarily mean the talent actually went up, it could just be a bit of a fluctuation in scouting/difference of opinion. The ranges would basically represent the differences of scouting opinion IRL, I figure.

I think the best application of the range for veteran/old players would be during the offseason. I think once a vet gets through spring training and starts playing games, there is a pretty good idea of whether he's at the old level or not. But, it certainly could be a factor in the offseason.

OTOH, it could be a factor for every player every year as they come into spring training. Instead of just having some players increase or decrease a certain number of points to pinpointed values in the offseason and after ST, spring training could actually be used realistically-to get players AB's and IP's to try to pinpoint their abilities. So, each player could come in with a fairly high range (higher for guys who've never been in the bigs or haven't played much in the minors, pretty narrow for guys in their peaks), and the more AB's and IP's, the lower the range of that player's abilities heading into opening day.

I guess this has a lot of possible applications even at the MLB level. Another one is after injuries. Is the player back 100% yet, or does he need to go on a rehab assignment? Give players coming back after a lot of time off wide ranges of error and it will encourage rehab assignments--to both get a sample size to see if he's ready for the bigs yet and to decrease the error range.

I rambled. But yeah, most certainly it could be used in the decline phase, although I think it would be best used during the offseason (I wonder if Barry Bonds is going to come back in 2006 like he played in 2004 and briefly in '05? is a real-life application; a GM in OOTP shouldn't know Bonds' exact ratings right now) as opposed to having wide ranges in the middle of July when a guy already has 250 AB's.

edit: Oh, and yeah, maybe implementing it for declining players in prolonged slumps or prolonged streaks would work too, showing that you're not really pinpointing the level of decline. So the decline might be occuring within the range, but you wouldn't be able to pinpoint it because of the hitting streak/slump.

[jerrycapo]

I think the biggest problem with the way the scouting is modeled is we have ratings info in real time, when IRL scouts have to assess players over time. When a young ss goes from 33 to 52 in contact, we know it as soon as the scout evaluates the player. Now IIRC each minor league team is going to also have a scout. If that scout is used to judge the talent of your minor league team then the scout should be able to judge that fairly quickly, because he would have seen, the player over several at bats.

This brings me to my point. I think the accuracy of the scouting reports should be based on how many at bats or innings pitched a scout sees a player perfrom. I think most scouts should be able to reasonably predict a players talent level after watching him over 100 to 200 at bats regardless of his age. With that being said I do not believe a scout should ever be able to accurately guess a players ratings. There should always be a margin of error. The better, more experienced scouts should then obviously have a smaller margin of error than someone who has 1 or 2 years of scouting. Obviously scouts should develop over time just as the players do.

The next problem I have is I think we are being too limited with the ability to have only six scouts per major league team. First, different scouts serve different purposes. There are talent scouts who evaluate players mostly at the high school and collegiate levels. There are advanced scouts who scout a team about two series ahead of the team. The reason they scout two series ahead is to scout the pitchers the team is likely to face. Then there are cross checkers, who when one scout thinks he has found a top prospect will go and look at the prospect and give a second opinion. In the game we are going to be given six positions to do all of that. What I would like to see in the game in the future are different levels of scouts and assignements.

I would like to be able to hire a scout and assign him to Puerto Rico, have another scout to work in the Dominican, another in Venezuala, another in Canada, and a scout who moves between Japan, Korea, and other asian countries. If I do not want to spend as much money scouting I can hire one scout to work all Latin American countries. By doing this though my scout would find fewer players and his analysis should suffer. I would also like to hire scouts to work different regions of the US, say a northeast scout, midatlantic, southeast, midwest, southwest, west etc.

Using Sebastian's great idea then the scout would have ranges of accuracy on players. Not based on age, but by number of at bats.

I will submit a post later with my ideas for how to scout player regression. There are some great ideas in this topic.

[RonCo]

Before we go designing this stuff for Markus, does anyone know how the game actually works in the area of scouts? Has anyone studied the accuracy of various scouts' reporting various classes of players right now?

[sebastian0622]

Originally Posted by RonCo

Before we go designing this stuff for Markus, does anyone know how the game actually works in the area of scouts? Has anyone studied the accuracy of various scouts' reporting various classes of players right now?

I doubt anyone has, since the scouting is so poorly implemented that almost everyone turns it off.

But that's now just one suggestion among several made in this thread.

I'd be very surprised if I learned that certain players were scouted more accurately than others in the current version, based on my experiences with the editor and whatnot. It's not uncommon to have your scouts be off 10+ points on a veteran player with 10 years in the bigs and 300 AB's that year.

But even if they did vary as they should, I think the range implementation is a good idea to represent different scouting opinions or just the range of error of scouting assessment.

[matey040949]

It looks like Markus read this, or was at least thinking along the same lines. The new scouting system is far more robust than previously and relies upon a lot of the concepts discussed in the opening post here (inexact natures of scouting, scouting over time, making stats actually useful in scouting etc). I'm looking forward to what happens to the system as it gets tweaked in the future.