An open letter to Markus regarding AI evaluations/decision making

[red95vette]

I agree with you guys that all these nuances are what REALLY come into play IRL and thus would be the ULTIMATE form of AI if the computer could model it. I've even argued a similar point in the past insisting that variability is important when programming a "GM model" so that you don't, for instance, have one human player playing against 29 identical "clone" GMs.

However, the basic foundation must be in place and solid before anything else can be considered. And it's "nice" (from a programming perspective) when that foundation is comprised entirely of mathematical analysis, because computers can be programmed to do that quite well and unfailingly. It all starts from there... and then if you're an ambitious programmer... you create the deviations by taking that base foundation and having different GMs "use" that information differently.

You guys are right in that there are TONS of nuances that can be put into the model to make the AI act "more human". But, although that is great and I would love to see it, I would also just be happy if some of the things I mentioned were addressed. No one can argue that behaviors like THIS are unbelievable, under any circumstance:
- An AI GM refusing $500K offered for $1 asked.
- AI GMs passing over a draft pick for 5,6,7 rounds and then trading their 1st, 2nd, 3rd pick for it immediately after the draft.

These are two isolated examples of things that are very well defined "issues" that undoubtedly would have to be fixed before any hope at all is placed on trying to model any further "higher level thinking". You might not think RvR and ROI are the "end all" goals for programming computer GM AI, but it should be evident that they are basic necessities for the computer to have and use as tools before any more complex behavior is modeled.

If the foundation is cracked, you fix it... you don't try to build a skyscraper on top of it as it is.

Also, one last point... it's a non-sequiter I believe to say well IRL GMs have all these intangible traits (I agree) and these traits would be super complex and unreliable to model in AI (I mostly agree) .. so what we have now is really as good as it gets and adding RvR or ROI would not really improve the competitiveness of the AI (I firmly disagree).

In any event, I must say I've enjoyed this discussion and am quite pleased at the constructive tone it's maintained throughout. Thanks for that guys.

[injury log]

And to respond to the original post, I think several of the replies illustrate well the complexity of designing a good AI. In fairness to the game, the AI has improved pretty dramatically over the last year or two. Still, I think almost everyone here agrees, it still has a ways to go.

Before the game gets into GM personalities (which would be very interesting), some more fundamental things need to be fixed. The AI could do much better with player valuation, understanding of risk and reward, and understanding of team context (positional strength, and position in standings). I still notice that pitching prospects sometimes are given negative value (an AI GM refuses a trade; I then ask him to add a decent low-level pitcher and he accepts) which doesn't make any sense. Mid-level prospects are way undervalued; I can request them in trades almost at will. This is on 'Hard' trading, 'Heavily Favor Prospects'.

I'd really like to see the game model the 'style' of trades seen in real life. I've never seen AI teams propose, or make with each other, a trade any more ambitious than a 2 for 2 deal, and those always contain a couple of bad prospects. If the game could make trades in the style of this year's Bedard, Santana, Haren or Tejada trades, you'd see more teams on proper rebuilding arcs, and longer running dynasties.

Lest this post get too scattershot, I'll stop soon. Just to say that the game is certainly intended for solo play (I've even read online players complain that the game seems designed for solo play, and that online aspects seem like afterthoughts), I expect the AI will again improve markedly in the 2008 version of the game, and I can still make solo play challenging with the right house rules (trade settings as above, all ratings off except 'Other' ratings, take over expansion team). No one wants house rules to be necessary, of course!

PS- RonCo, if you want to rest an SP, you could drop him in a later slot of the starting rotation, even, I think, the 6th slot. The game won't use a pitcher in the rotation as a reliever if it has other options.

[atlbrave1]

Originally Posted by Syd Thrift

- Some GMs ought to have wildly different criteria for judging talent than others. This happens to some degree right now, but I want one guy who looks at guys based entirely on their tools, dammit. Each GM ought to have an ability or two that he prizes above all else, as well as a couple attributes he really doesn't care about.

This is something I would really like to see happen in the future of the game. I don't think these attributes should necessarily be shown either. But I would love to be able to offer a package of players to one GM and be rebuffed, while the same package is loved by another GM. To use the Billy Beane analogy, I offer him 3 prospects who have good potential for average, good speed, average-below average power, but don't have good eye potential and are a year or two away from the majors, for a very solid major league player. He sees the deal as too risky, because eye potential and power are his top 2 attibutes when loooking at players. Now I take this same package to another GM, we'll call him Jim Bowden, and his top categories are potential contact and speed, and not only does he accept it, but I am also able to get a decent middle reliever in the package. I think one way this could be done is by weighting some the stats (both past and projected, if possible) w/in the GM's personality. I think the big concern is that you don't want this to be random (i.e. Intentional Walks and Caught stealing are given the most weight - although that might explain the Reds-Nats trad of Kearns-Lopez for Majewski-Bray a couple of years ago), but there are enough stats/ratings to be able to create some GM variety without making the AI GM's idiots.

[ctorg]

Of course the best way to build the ultimate AI would be to create one that could learn. But we're a very long way off from that, I guess.

[red95vette]

Originally Posted by ctorg

Of course the best way to build the ultimate AI would be to create one that could learn. But we're a very long way off from that, I guess.

Then we could just start calling it "I".

[johnnybourbon]

Originally Posted by atlbrave1

This is something I would really like to see happen in the future of the game. I don't think these attributes should necessarily be shown either. But I would love to be able to offer a package of players to one GM and be rebuffed, while the same package is loved by another GM. To use the Billy Beane analogy, I offer him 3 prospects who have good potential for average, good speed, average-below average power, but don't have good eye potential and are a year or two away from the majors, for a very solid major league player. He sees the deal as too risky, because eye potential and power are his top 2 attibutes when loooking at players. Now I take this same package to another GM, we'll call him Jim Bowden, and his top categories are potential contact and speed, and not only does he accept it, but I am also able to get a decent middle reliever in the package. I think one way this could be done is by weighting some the stats (both past and projected, if possible) w/in the GM's personality. I think the big concern is that you don't want this to be random (i.e. Intentional Walks and Caught stealing are given the most weight - although that might explain the Reds-Nats trad of Kearns-Lopez for Majewski-Bray a couple of years ago), but there are enough stats/ratings to be able to create some GM variety without making the AI GM's idiots.

I agree on this, and think this sort of thing can go well with a combination of the "make this work" and "shop player" features. One thing in OOTP5 (trade AI has come a LONG way since that version) that was great was that you'd get a response to what players that you put on the trade table with a list of players that the AI would offer. By updating this sort of thing with a, say "Make this Work/Shop Package" in response to players that you offer, you can get a good idea what your package of players is worth to each GM.

[Syd Thrift]

Originally Posted by injury log

Not to distract from the good points in the rest of your post, but about the Yanks, that's not true, at least not any more. They'd barely have a farm system now without paying well above slot for guys who've dropped in the draft, and for high-priced international signs. From the draft alone, NYY has paid way above-slot bonuses to:

2005: Austin Jackson, Alan Horne
2006: Dellin Betances, Mark Melancon, Dave Robertson
2007: Andrew Brackman, Brad Suttle, Carmen Angelini

I completely agree with what you say about how the Yanks are drafting now. They weren't doing that 10 years ago. Who did they develop between Jeter and one of those guys you mention? Does Mike Lowell count?

[injury log]

Originally Posted by Syd Thrift

I completely agree with what you say about how the Yanks are drafting now. They weren't doing that 10 years ago. Who did they develop between Jeter and one of those guys you mention? Does Mike Lowell count?

I don't intend to be argumentative, because we largely agree, but-

I think it's true that the Yankees draft strategy has changed over the last few years- it seems in response to the Red Sox, who have become much more aggressive in recruiting amateur talent- but the Yanks have a few success stories post-Jeter. They've been particularly successful in signing amateur (international) free agents: Chien-Ming Wang, Robinson Cano, Dioner Navarro, Melky Cabrera, Juan Rivera, Yhency Brazoban, and Wily Mo Pena (originally signed by the Mets) all from the late 1990s/early 2000s. I suppose you could count Alfonso Soriano as well, though he had an interesting career trajectory. In the draft they picked up Nick Johnson, Eric Milton, Marcus Thames, Zach Day, and Phil Hughes (2004). All are players signed or drafted by the Yanks pre-2005 and post-Jeter who have had big league careers (some abbreviated by injury), and there are many lesser lights I won't bother to mention.

It is true that the Yanks had an absolutely brutal series of drafts leading up to the Hughes draft in 2004. Man, those were bad. But they were still picking up a lot of good international signs, even before the high profile Jesus Montero, Kelvin Deleon and Jose Tabata deals of the last three years. So, while I agree that the Yankees seem to have a different philosophy now to the draft, they never ignored amateur talent, particularly internationally, and the fact that their prospects were used as trading chits doesn't negate the fact that they recruited these prospects in the first place.

[SittingDuck]

This is a great thread. Thanks to 'vette.

[thbroman]

Brilliant thread . . . simply wonderful.

Markus, if you have a chance at some point, it would be great to hear your reactions!

[zekester91]

I think that the bug fix that was anounced over at FOF will go a long way to fixing some of these issues. it is a great thread though.

[darkcloud4579]

Great post with lots of interesting insights. Nothing to add, but..it's really revealed some good discussion.

[Jose Cardenal's Afro]

Originally Posted by RonCo

This seems like an incredibly stupid thing for an AI to do. I mean, not even Dusty Baker would go that far.

There is an interesting idea. Who would run a team better...Dusty Baker or the OOTP AI?

BTW this is a good discussion. The AI does need some updating.

[Khrog]

Originally Posted by Syd Thrift

A secondary thing... one thing I'd like the AI to do is game the "Win Now/Rebuild" thing a lot, lot harder. It's more challenging to play in a league against a couple of powerhouses and a few patsies than it is to play against a bunch of teams that oscillate between pretty bad and pretty good. It's also more realistic to baseball; up until free agency, you regularly, decade after decade, had 2 or 3 teams up at the top of each league beating up on everybody else. This gets more and more pronounced the further back you go. Although I do see OOTP teams occasionally get into the doldrums and not climb out of the second division for, say, 30 years, I see this happen to one team in 16, not 6.
Also... while I get the point about AI... I don't think it's entirely realistic to have teams engage in wholly rational RvR reviews every time it makes a trade. I guess that underneath the hood this should (and probably, to some extent, is) be a way to evaluate potential trades/free agent signings/etc. but there are other bits to consider:

- Some GMs ought to have wildly different criteria for judging talent than others. This happens to some degree right now, but I want one guy who looks at guys based entirely on their tools, dammit. Each GM ought to have an ability or two that he prizes above all else, as well as a couple attributes he really doesn't care about. Billy Beane might rank Patience highly and rank all the "little skills" like Baserunning and Bunting low (and would probably be ineffective if placed in the dead ball era). George Steinbrenner, in the days when he was so intrusive he was basically the Yanks' GM, might rank Popularity very high but not care much at all for some of the character ratings. In the end, the guy who is highly evaluating skills that nobody else is looking at is probably the person who has an edge.

Now this is something that I believe would add the most to solo league play. GM personalities/whims/etc would add a ton to the game because for some teams they prefer to build using the heavy hitters and others prefer to go after pitching/fielding, etc. Besides adding tons of immersive GM profiles for your competition...hell in a large enough league this can be populated from "real" data gleaned from their time in subordinate leagues.

[JimboJones]

Great thread!

I'd like to offer another perspective if I may. Because this is a computer sim, every single thing that happens in the game is a result of some random roll of the dice. This random number, which is in turn fed into a formula, produces some result - a strkeout, homerun, error, etc. There is no such thing as a human having the advantage of being able to use his "intuition". There is simply no such thing when playing against the CPU. These are 0's and 1's were talking about, not living, breathing, feeling humans.

Because the programmer knows the formulas used to determine the results, he also has the ability to program the UI to know exactly how much a players ratings are worth. Theoretically, you can take a players ratings, feed them into a formula, and calculate exactly how many wins that player will likely contribute to a team. Then (since baseball is a business after all) the only thing left is to calculate how much money a win is worth, both present wins and future wins. Each player is in effect given a cash value, which makes evaluating trades and contract negitiations a matter of balancing the two sides. Not the most romantic way to think of things, but remember, its all just 0's and 1's.

Because these types of caluclations are impossibly complex for a human player, even if he had access to the code and knew the formulas, the computer will always be a better player over the long run. Not that it will always be right, due to random events, such as career ending injuries. But over the long term, the computer can and should have the advantage of being able to play the percentages better. A human will get lucky and say he won by having the advantage of human intuition, but its all just a falacy.

[Syd Thrift]

Well... theoretically if the AI was perfect, sure. In reality, the AI is far from perfect and lots of players know various ways to "game" it.

[ravinhood]

I like the OP's idea of some players being "untouchable" for trades no matter what (unless of course the player cheats and forces the trade in commisioner mode). Something that is missing in this game that I've seen in other games are "franchise players" those that would normally be in an untouchable mode and no matter what the AI wouldn't trade them off until their stats/attributes start to fail. The AI all too many times will give up quality even playing Hard difficulty trades if the player is patient enough to run through 100's of trade setups to get that player he wants. The AI should be programmed to be aware that when the human player wants to trade with him the human player wants more than he's going to offer most of the time. lol The human player is going to use every exploit hook crook and wheeled deal to get what he wants, but, if the AI is programmed against this then the human player will have to walk away with much less than he wanted to bargain for that particular player he wanted.

The AI program needs to put more value on what it has and what it needs and placing starting pitchers at the top and make it doubley hard for the human player to wrest away those good pitchers the AI has unless it's rebuilding and then it should demand high value prospects still.

I would value trades by the AI in the following order and it should be very hard to get the top few levels by the human players.

1. Starting Pitchers
2. RBI/HR hitters
3. Prospects
4. Closers
5. Defensive position players
6. Base Running/Leadoff Batters/Base Stealers
7. Draft Picks
8. Relief Pitchers
9. Utility players
10. Money

The AI should be more willing to make trades for the botton 6, but, very hard against trades for the top 4. Values for the top four should probably be doubled as their values are usually worth a lot more than a 1 to 1 ratio in most cases vs the human player. I go by the principle that a bird in the hand is worth two (prospects) in the bush. Thus it would be nice if the AI went that direction in trading as well.

Of course the values can adjust and change depending on the AI's level of competition, if rebuilding there's good reason to get rid of some old players that just won't be around or still be good when the team finally does meld, but, teams that are in win now personalities should be very hard to trade with unless they are getting the better deal, just makes sense to me that's the way I would be.

[Markus Heinsohn]

Oh well, AI... that is a complex topic.

All I can say in public is that the AI in OOTP does a pretty good job IMO, especially when compared to other baseball sims. No other game handles this many rule sets, so the AI in OOTP is much more complex.

The backbone of the AI is a good player evaluation algorythm, and we have improved it greatly in OOTP 9, so the results will be better in every area of the game.

That $1 for $2 trade thing was a small bug by the way