Code reuse between building game data file and save game files

[monkey0506]

I've been meaning to look into this more than I've actually had an opportunity to, but it seems that the engine is currently reusing some functions in writing the game data file and in writing save game files. As most of you probably know, I've been working on porting the process of writing the game data file to the editor's managed C# code in order to make it easier for multiple target platforms to be built from within the editor. This has led me to the following questions:

* What data do the game data file and save game files currently have in common?
* What data do each of these (game data file and save game files) actually use/require?
* Aside from code reuse, what benefits are there behind keeping a similar/shared format between these files?

To be more specific, I'm pretty sure that GUI data (size, location, graphics, controls, etc.) is written the same way when building the game as it is when saving the game. While I appreciate that this removes duplicated code, some of the members used at run-time have no meaning or bearing whatsoever at design-time or compile-time while creating a bloated data file. These run-time members are written statically into the game data file as null/empty/junk values that are never used by the engine, but exist in the data file only as a form of padding to maintain similar offsets.

My current fork is still in the debugging phase, and when building larger projects there are several inconsistencies in the output game files (as compared to the pure native C++ build process), so I'm curious whether it is worth debugging if large portions may simply be replaced with calls back into the native code to maintain the existing format. Personally I think it would be better to split the format of the game data files and save game files, which could potentially result in drastically smaller game sizes (especially for larger projects I've helped out with like Gemini Rue or The Cat Lady), but I'd like to hear what others have to say before I make any final decisions (in the mean time, I'll keep debugging my work-alike port).

[Crimson Wizard]

I've splitted data and savegame serialization in "develop" branch, when I introduced new save format, removing excess save data (I removed some excess data from game data too, because on some occasions it wrote values that could be changed only at runtime).

My plan was to gradually move the changes from "develop" to "develop-3.4.0", to be able to review them and keep only "good" changes, which I started to do with changes to GUI.

I have not change anything in saves yet, because I wanted to move "code style" and "refactoring" changes first, and thus test every step separately.

I do not think it is worth to keep game data serialization in native code forever. OTOH I do not have any opinion towards time priority of C# reimplementation. Also it won't hurt anyone if this code will remain in engine for some time in a WIP branch. When the C# code will fully work, the code that is not used for saved games could be removed from engine all at once.

[monkey0506]

When the C# code will fully work, the code that is not used for saved games could be removed from engine all at once.

That's mainly why I was asking. I didn't want to create a bunch of duplicated code if it wasn't useful, but the reason I decided to move it to C# in the first place was that 1) the code was a mess, and 2) it makes more sense in the main editor code base anyway instead of a separate assembly (given that as far as building the data file it is only run at compile-time). As I'm sure you're aware, there were a lot of hard-coded paths and methods that would be significantly more difficult to update across the editor and native assemblies which hopefully will now (once I'm finished, that is) be able to be updated and maintained in a single assembly (the editor).

Major changes don't have to happen right away, but I'm trying to keep my changes focused in the right direction, hence this discussion.

[Snarky]

2) it makes more sense in the main editor code base anyway instead of a separate assembly (given that as far as building the data file it is only run at compile-time).

Isn't it better to keep the compiler separate from the editor as much as possible (in fact, ideally make it a standalone executable), the way it is with most compilers/IDEs? That would ultimately allow you to do things like running builds on a separate build server, scripting builds (maybe for multiple platform support), etc.

[Calin Leafshade]

Ideally, the compiler itself would be a separate assembly, a library.

Then a compiler exe could simply reference this library and provide a terminal front end for it and the editor can reference it and provide an in-editor front end.

[monkey0506]

Well, ultimately, yes. But the compiler is nowhere near being a standalone assembly ATM, so this is a step toward that.

[Snarky]

Fair enough. Cool!

[Crimson Wizard]

My current fork is still in the debugging phase, and when building larger projects there are several inconsistencies in the output game files (as compared to the pure native C++ build process),

What bugs me is that you are putting struct padding manually to (de)serialization process. I've wrote an AlignedStream class which calculates that padding automatically and greatly simplified my work. Additionally, I can now call read/write functions with either AlignedStream or any other stream class, therefore support some future data format with no extra bytes.

I was thinking that maybe you could write game data without padding already; we would need to increase format version, and the engine will switch to not reading padding if the format is high enough.

[monkey0506]

It's true I am writing the padding manually, but my goal was to first make an exact work-alike to be sure all the data was in a consistent format to what the engine currently expects (which, I suspect will be maintained for backwards compatibility) and then the deserialization methods in the engine could be updated to not require those padding bits (if game data format is the new version of course). After all, you're the one who keeps saying to keep separate tasks as separate commits. Removing the padding bits now would also require rewriting the deserialization methods, which I haven't taken on yet.


Edit: Reading back over this, it seems like you might be assuming that I'm changing how the game data file is read by the engine. I just want to make it perfectly clear that, for now, I am not making changes to how the data file is read, just from where (code-wise and language-wise) it is written.