File size in online leagues

[Corsairs]

For those of you in online leagues, what's the typical file size of the master file sent out by the commissioner? I'm running a start-up league with OOTP 8 (using latest patch) and we've just hit our first regular season. When we started our files were in the 25 MB range; now they've ballooned up to 125 MB. This is causing problems for a couple of my owners who still use dial-up (poor fellows)!

In terms of settings, how much do box scores/game logs affect the league file size? Our league has 208 teams in 11 leagues (humans run 96 of those teams in 4 leagues). Several owners mentioned they wanted to see box scores for all the different leagues, so I currently have these settings enabled (note we run 8 day sims):

• Save Box Scores: All
• Save Game Logs: Human
• Keep News/Injury/Transaction: All
• Keep Career Stats/Splits: All
• Delete Players Never Reaching Majors: Checked

How much would changing these options likely cut off the file size? I'd hate to lose the box scores, but if it makes a significant difference I'd be willing to make the sacrifice.

Pictures are another issue. What I've been doing after each sim is running the "Update / Generate Player Pictures" function. For our last update that resulted in 385 player pictures being updated (7.2 MB worth), and those pictures were included in the file I sent out. Do those of you who commish online leagues typically update pictures and include them with each new file sent out?

On the actual file creation screen I have Team/League News and Player History checked for all 11 of our leagues. Box Scores / Game Logs / Messages from the last 8 days are included. Neither Names.dat nor Retired.dat is included, but new and changed images are.

Then there's the issue of my .dat files. The four largest files are World.dat (14 MB), Players.dat (19 MB), Scouting.day (28 MB ... we have scouting ON), and Teams.dat (42 MB).

Regarding Teams.dat, isn't that very high for a 208-team league? Markus recently looked at my .dat files for a separate issue and noticed there were nearly 4000 teams in my database. During our initial league setup I experimented with CubbyFan's set (which contains thousands of teams) for ideas on league settings. I deleted all those extra teams before starting my league, but apparently their still floating around in my teams.dat file. I'm pretty sure I already know the answer to his, but here's my big question: Is there any way to purge unused team entires from your .dat files?

[Tony M]

In answer to your question, though I think I probably answered it in regard to someone else (or it may have been you) I have tinkered with trying to delete teams that are invisible but there must be something somewhere I haven't decoded to make it work.

In terms of file size I did make a batch of suggestions about how things can be optimised, and I have ideas about how the game log/box scores can be databased and generated on the fly (and I think my structure would be smaller than the associated text files), and also that for online leagues an incremental approach could be adopted for files.

Take for example a team in teams.dat that has got 20 years worth of batting history, pitching history, financials, etc...

In every sim those figures will never change but they'll get included in any update sent to GMs. The export part of the game only really needs to export the bits of the file that pertain to the current year and then the receiving game knows that it only needs to stick them in to the current year.

In players.dat one of the biggest sections is the At Bat Batting Stats. Once a game in a season has been played these records are fixed (until deletion at the start of the next pre-season) so they don't need to be sent back and forth for any future sim that season.

[Corsairs]

That's right, I do believe you mentioned to me earlier that you had a system for seeing "invisible teams" but hadn't figured out how to remove them. It's a real shame that we seem to be stuck with them, as it may make the difference between being able to include things like box scores/pictures or not.

Alright, so given that there's no way to remove the 3700+ phantom teams in my .dat files, what kinds of settings and methods do you online commishes use to keep your file size manageable? For that matter, what size is considered "reasonable" for online leagues? I'm sure most people aren't sending out 125 MB every update. Do you include box scores for leagues other than your main one? Updated player pictures? I could definitely use some tips!

[Corsairs]

I was probably too wordy with my original post, so let me condense it down to one important question I'd love to get an answer to:

For those of you in online leagues, what is the typical size of the compressed league file sent out by the commissioner?

[Stu]

My league file is 25 MB right now.

22 teams with 3 minors each for a total of 88 teams. We have scouting enabled. We sim 7 days at a time and I only include box scores, news and gamelogs from the last 7 days.

File size seems to grow exponentially when you have more teams. You have more players, more scouting data, more stats, etc.

For pictures I don't include them in the league file. I create a seperate zip of images 2-3 times per season and upload it separately. Owners then d/l it and unzip into their images folder.

[Corsairs]

Good tip with the separate picture updates. I did that for our original file but figured I'd be okay with including the updated pictures after that. Clearly I was wrong; I'll start doing it the way you suggest.

We have 24 human teams with 3 minors (96 human-run teams in all) and I include boxes/logs/news from the last 8 days in my updates. The one difference is I'm also including boxes and news (no logs) for our AI-run teams. At the moment we have exactly 100 of those playing games (another 12 are idle). I had been hoping to keep the boxes for those games, but if I have to lose them to keep the file size reasonable I'll do so.

If I can trouble you for one more question, would you mind telling me the uncompressed sizes of your Players, Scouting, Teams and World dat files? That would be an enormous help in figuring out how "normal" my league setup is.

[Stu]

Originally Posted by Corsairs

If I can trouble you for one more question, would you mind telling me the uncompressed sizes of your Players, Scouting, Teams and World dat files? That would be an enormous help in figuring out how "normal" my league setup is.

Uncompressed sizes...

Players: 22.1 MB
Scouting: 33.3 MB
Teams: 1.4 MB
World: 8.23 MB

[Corsairs]

Wow, that really drives home how this "phantom teams" bug is just killing my file size. My Players and Scouting files are a bit smaller than yours, and my World is a bit bigger. But my Teams.dat file is 40 MB larger than yours!

I have got to see if there's a way to purge those extra entires from my teams.dat file. Knocking off ~35 MB from the league file size would be huge. It's likely the difference between being able to include box scores or not.

[Corsairs]

Alright, I've got some feedback on this after running tonight's sim. I took Stu's advice and unchecked the "Include new and changed images" box when creating the latest file. The result was my archived file size dropping from130 to 23 MB!

In absolute shock of this result, I examined the larger file that included the changed images. What I discovered is that every file in my /images directory is being included in the archive! Now I can't believe that's the way it's supposed to work. The checkbox reads "NEW and CHANGED images". Not only is it including every image in my league every time, it's actually putting non-picture files in there too. For instance, I had a .txt note to myself from a month ago that I see has been included in the latest archive.

Surely this can't be right. Are other league commissioners experiencing behavior like this? My league is going on holiday break until Jan. 7. I'd love to be able to resume play knowing how to include just new and updated pictures in the bi-weekly league files I send out.

[Stu]

Originally Posted by Corsairs

Alright, I've got some feedback on this after running tonight's sim. I took Stu's advice and unchecked the "Include new and changed images" box when creating the latest file. The result was my archived file size dropping from130 to 23 MB!

In absolute shock of this result, I examined the larger file that included the changed images. What I discovered is that every file in my /images directory is being included in the archive! Now I can't believe that's the way it's supposed to work. The checkbox reads "NEW and CHANGED images". Not only is it including every image in my league every time, it's actually putting non-picture files in there too. For instance, I had a .txt note to myself from a month ago that I see has been included in the latest archive.

Surely this can't be right. Are other league commissioners experiencing behavior like this? My league is going on holiday break until Jan. 7. I'd love to be able to resume play knowing how to include just new and updated pictures in the bi-weekly league files I send out.

There was a similar bug with game logs and box scores prior to patch #3. Instead of including the files from the last X days, it was including all of them so the file grew very large over the course of a season. I assume this is being caused by something similar.

[Corsairs]

Originally Posted by Stu

There was a similar bug with game logs and box scores prior to patch #3. Instead of including the files from the last X days, it was including all of them so the file grew very large over the course of a season. I assume this is being caused by something similar.

If that's what's happening here, that's a nasty bug indeed! So I've got to ask: is there any kind of fix for this?

What I'm doing at the moment is sending out updates with no images, then uploading a separate archive with the new images from the latest sim. That's a workable but not great solution. A few of my owners aren't extremely computer savvy; adding steps to the process like this makes things difficult for then.

Is this a case of an old bug being re-introduced in the latest version of OOTP?

[Solonor]

We have 24 teams with 3 minors each (96 teams). I send out 7 days with all boxes checked (box scores, game logs, news) for all levels plus images. Our league update zip is around 5 or 6 mb.

We don't have images for anything other than major league players, though...

[Corsairs]

Originally Posted by Solonor

We have 24 teams with 3 minors each (96 teams). I send out 7 days with all boxes checked (box scores, game logs, news) for all levels plus images. Our league update zip is around 5 or 6 mb.

We don't have images for anything other than major league players, though...

When you create your league file, does OOTP just include the new images from your last sim period or is everything in your /images directory getting added to the file?

[Cooleyvol]

Ever thought of zipping the league file up the way we USED to do it? We did it ourselves. We didn't let some piece of software do it for us.

[Solonor]

Originally Posted by Corsairs

When you create your league file, does OOTP just include the new images from your last sim period or is everything in your /images directory getting added to the file?

Looking at the zip I did this past Thursday, it only contains images that changed. There are 13 png files in the news/html/images folder in the zip.

[Corsairs]

Thanks Solonor. I guess that cinches it: I'm experiencing a bug of some sort. I'll make a post in the tech forum. At least with the helpful tips from this thread I've gotten our league file down to a reasonable level (even if there is an extra step involved). Now hopefully it can be determined why everything plus the kitchen sink is getting tossed into the file when I try to include just the new images!

[Solonor]

Do you do something like regenerate images every time? I think it's basing the decision to update on the date of the png file. Since I manually update photos, it's only changing those I touch.

[Corsairs]

I run "Update / Generate Player Pictures" for all leagues just before each new league file I send out. But what I'm finding is that every image is being included, not just the ones with newer file dates. For instance, updating player pictures during my last sim resulted in 107 new pictures (going by file dates here). But when I check the "Include new and changed images" box when creating the league file, all 7,162 files in my /images directory are put into the file - even those with file dates from October. It was even including a .txt file from November that I had placed in that directory!

When you say you "manually" update photos, what do you mean by that?

[Solonor]

Originally Posted by Corsairs

I run "Update / Generate Player Pictures" for all leagues just before each new league file I send out. But what I'm finding is that every image is being included, not just the ones with newer file dates. For instance, updating player pictures during my last sim resulted in 107 new pictures (going by file dates here). But when I check the "Include new and changed images" box when creating the league file, all 7,162 files in my /images directory are put into the file - even those with file dates from October. It was even including a .txt file from November that I had placed in that directory!

When you say you "manually" update photos, what do you mean by that?

I wonder if the "update" task is resetting a flag in the game to look at all files in the folder. Maybe it's got a date in one of the dat files that it looks at and then compares that to the file dates in the folder?

We're a little unique in that not all of our players are "human" so I don't want the game to generate player photos for everyone and overwrite our custom pics. So, I right-click on players that I want to have photos and generate them. Like I said, we only have pics for the major leagues, and I just go through and update the new humans that have been brought up or the ones that need uniforms changed based on the transactions report.