[SOLVED] Trouble following build flow -- when is EXE actually CREATED?

[monkey0506]

I'm having a bit of a problem tracking down why my code to build the game's data file from C# is failing to produce a valid EXE. I've combed over the native C++ implementation and compared it to my C# implementation dozens of times, and I'm not seeing where the problem lies. What I have been able to determine is that the native code is (apparently) writing data to the EXE file well ahead of my current implementation, which is getting things massively out of order, and causing some data to not be copied at all. This may be a bit of a code dump, but any help sorting this out is appreciated.

When creating the EXE, the Editor calls upon the native code to build the data file.

Editor/AGS.Editor/NativeProxy.cs:388-391

Code: csharp

        public void CreateGameEXE(string[] fileList, Game game, string baseFileName)
        {
            _native.CreateDataFile(fileList, game.Settings.SplitResources * 1000000, baseFileName, true);
        }

Editor/AGS.Native/ScriptCompiler.cpp:116-139

Code: cpp

		void NativeMethods::CreateDataFile(cli::array<String^> ^fileList, long splitSize, String ^baseFileName, bool isGameEXE)
		{
			char **fileNames = (char**)malloc(sizeof(char*) * fileList->Length);
			for (int i = 0; i < fileList->Length; i++)
			{
				fileNames[i] = (char*)malloc(fileList[i]->Length + 1);
				ConvertFileNameToCharArray(fileList[i], fileNames[i]);
			}
			char baseFileNameChars[MAX_PATH];
			ConvertFileNameToCharArray(baseFileName, baseFileNameChars);

			const char *errorMsg = make_data_file(fileList->Length, fileNames, splitSize, baseFileNameChars, isGameEXE);

			for (int i = 0; i < fileList->Length; i++)
			{
				free(fileNames[i]);
			}
			free(fileNames);

			if (errorMsg != NULL)
			{
				throw gcnew AGSEditorException(gcnew String(errorMsg));
			}
		}

In particular, I noticed that all this is doing is forwarding the parameters to the unmanaged method, make_data_file (rather long, over 100 lines).

Editor/AGS.Native/agsnative.cpp:2448-2619

Spoiler

Code: cpp

const char* make_data_file(int numFiles, char * const*fileNames, long splitSize, const char *baseFileName, bool makeFileNameAssumptionsForEXE)
{
  int a,b;
  Stream*wout;
  char tomake[MAX_PATH];
  ourlib.num_data_files = 0;
  ourlib.num_files = numFiles;
  Common::AssetManager::SetSearchPriority(Common::kAssetPriorityDir);

  int currentDataFile = 0;
  long sizeSoFar = 0;
  bool doSplitting = false;

  for (a = 0; a < numFiles; a++)
  {
	  if (splitSize > 0)
	  {
		  if (stricmp(fileNames[a], sprsetname) == 0) 
		  {
			  // the sprite file's appearance signifies it's time to start splitting
			  doSplitting = true;
			  currentDataFile++;
			  sizeSoFar = 0;
		  }
		  else if ((sizeSoFar > splitSize) && (doSplitting) && 
			  (currentDataFile < MAXMULTIFILES - 1))
		  {
			  currentDataFile++;
			  sizeSoFar = 0;
		  }
	  }
	  long thisFileSize = 0;
	  Stream *tf = Common::File::OpenFileRead(fileNames[a]);
	  thisFileSize = tf->GetLength();
	  delete tf;
	  
	  sizeSoFar += thisFileSize;

    const char *fileNameSrc = fileNames[a];

  	if (strrchr(fileNames[a], '\\') != NULL)
		  fileNameSrc = strrchr(fileNames[a], '\\') + 1;
	  else if (strrchr(fileNames[a], '/') != NULL)
		  fileNameSrc = strrchr(fileNames[a], '/') + 1;

    if (strlen(fileNameSrc) >= MAX_FILENAME_LENGTH)
    {
      char buffer[500];
      sprintf(buffer, "Filename too long: %s", fileNames[a]);
      ThrowManagedException(buffer);
    }
		strcpy(ourlib.filenames[a], fileNameSrc);

	  ourlib.file_datafile[a] = currentDataFile;
	  ourlib.length[a] = thisFileSize;
  }

  ourlib.num_data_files = currentDataFile + 1;

  long startOffset = 0;
  long mainHeaderOffset = 0;
  char outputFileName[MAX_PATH];
  char firstDataFileFullPath[MAX_PATH];

  if (makeFileNameAssumptionsForEXE)
  {
	  _mkdir("Compiled");
  }

  // First, set up the ourlib.data_filenames array with all the filenames
  // so that write_clib_header will write the correct amount of data
  for (a = 0; a < ourlib.num_data_files; a++) 
  {
	  if (makeFileNameAssumptionsForEXE) 
	  {
		  sprintf(ourlib.data_filenames[a], "%s.%03d", baseFileName, a);
		  if (a == 0)
		  {
			  strcpy(&ourlib.data_filenames[a][strlen(ourlib.data_filenames[a]) - 3], "exe");
		  }
	  }
	  else 
	  {
    	if (strrchr(baseFileName, '\\') != NULL)
		    strcpy(ourlib.data_filenames[a], strrchr(baseFileName, '\\') + 1);
	    else if (strrchr(baseFileName, '/') != NULL)
		    strcpy(ourlib.data_filenames[a], strrchr(baseFileName, '/') + 1);
	    else
		    strcpy(ourlib.data_filenames[a], baseFileName);
	  }
  }

  // adjust the file paths if necessary, so that write_clib_header will
  // write the correct amount of data
  for (b = 0; b < ourlib.num_files; b++) 
  {
	Stream *iii = find_file_in_path(tomake, ourlib.filenames[b]);
	if (iii != NULL)
	{
		delete iii;

		if (!makeFileNameAssumptionsForEXE)
		  strcpy(ourlib.filenames[b], tomake);
	}
  }

  // now, create the actual files
  for (a = 0; a < ourlib.num_data_files; a++) 
  {
	  if (makeFileNameAssumptionsForEXE) 
	  {
		  sprintf(outputFileName, "Compiled\\%s", ourlib.data_filenames[a]);
	  }
	  else 
	  {
		  strcpy(outputFileName, baseFileName);
      }
      if (a == 0) strcpy(firstDataFileFullPath, outputFileName);

	  wout = Common::File::OpenFile(outputFileName,
          (a == 0) ? Common::kFile_Create : Common::kFile_CreateAlways, Common::kFile_Write);
	  if (wout == NULL) 
	  {
		  return "ERROR: unable to open file for writing";
	  }

	  startOffset = wout->GetLength();
    wout->Write("CLIB\x1a",5);
    wout->WriteByte(21);  // version
    wout->WriteByte(a);   // file number

    if (a == 0) 
	{
      mainHeaderOffset = wout->GetPosition();
      write_clib_header(wout);
    }

    for (b=0;b<ourlib.num_files;b++) {
      if (ourlib.file_datafile[b] == a) {
        ourlib.offset[b] = wout->GetPosition() - startOffset;

		Stream *iii = find_file_in_path(NULL, ourlib.filenames[b]);
        if (iii == NULL) {
          delete wout;
          unlink(outputFileName);

		  char buffer[500];
		  sprintf(buffer, "Unable to find file '%s' for compilation. Do not remove files during the compilation process.", ourlib.filenames[b]);
		  ThrowManagedException(buffer);
        }

        if (copy_file_across(iii,wout,ourlib.length[b]) < 1) {
          delete iii;
          return "Error writing file: possibly disk full";
        }
        delete iii;
      }
    }
	if (startOffset > 0)
	{
		wout->WriteInt32(startOffset);
		wout->Write(clibendsig, 12);
	}
    delete wout;
  }

  wout = Common::File::OpenFile(firstDataFileFullPath, Common::kFile_Open, Common::kFile_ReadWrite);
  wout->Seek(Common::kSeekBegin, mainHeaderOffset);
  write_clib_header(wout);
  delete wout;
  return NULL;
}

[close]

As far as I can tell, this doesn't even open the EXE file for writing until it is near the end of the function.

Editor/AGS.Native/agsnative.cpp:2554-2583

Code: cpp

  // now, create the actual files
  for (a = 0; a < ourlib.num_data_files; a++) 
  {
	  if (makeFileNameAssumptionsForEXE) 
	  {
		  sprintf(outputFileName, "Compiled\\%s", ourlib.data_filenames[a]);
	  }
	  else 
	  {
		  strcpy(outputFileName, baseFileName);
      }
      if (a == 0) strcpy(firstDataFileFullPath, outputFileName);

	  wout = Common::File::OpenFile(outputFileName,
          (a == 0) ? Common::kFile_Create : Common::kFile_CreateAlways, Common::kFile_Write);
	  if (wout == NULL) 
	  {
		  return "ERROR: unable to open file for writing";
	  }

	  startOffset = wout->GetLength();
    wout->Write("CLIB\x1a",5);
    wout->WriteByte(21);  // version
    wout->WriteByte(a);   // file number

    if (a == 0) 
	{
      mainHeaderOffset = wout->GetPosition();
      write_clib_header(wout);
    }
    // ...

Here we see that as soon as the file is opened, it is recording the file's length (line 21 in the above snippet, line 2574 in the file), writing a bit of data (the next three lines), and then if it is the EXE, recording the file's current position. If the file were being newly created then this wouldn't make sense, but as seen on line 15 of the snippet (2568) the EXE is being opened if it already exists. From my tests, it seems that the EXE is being created elsewhere first and having some 2178048 bytes of data written into it (as per the Length and Position of the Stream) before make_data_file is being called.

However, if I replace the call in NativeProxy.cs with a call to my C# implementation of make_data_file, the EXE is not being created first.

My question, if anyone can be bothered to read through all of this, is at what point is the EXE being created and written to when using the native implementation that isn't being invoked with my C# implementation? I'm at a total loss as to where this file is being created (and even when the necessary code is being invoked).

My C# implementation is up in my personal fork, if anyone wants to look through it.

[Crimson Wizard]

Eurgh... did I mention this part is a mess?

What Editor does, is copying acwin.exe from its program folder to '<game project>/Compiled' folder:
AGS.Editor.cs, line 1306:

Code: csharp

        private void CreateCompiledFiles(string sourceEXE, CompileMessages errors, bool forceRebuild)
        {
            try
            {
		string newExeName = this.CompiledEXEFileName;
                File.Copy(sourceEXE, newExeName, true);

And only then it runs an actual game compilation in separate thread:
AGS.Editor.cs, line 1338:

Code: csharp

BusyDialog.Show("Please wait while your game is created...", new BusyDialog.ProcessingHandler(CreateCompiledFiles), forceRebuild);

Don't be confused by similar names, there are two "CreateCompiledFiles" methods, different by number of parameters.
The second one calls CreateGameEXE in the working thread, and you know the rest.
The 2178048 bytes is the size of "acwin.exe" copy. Editor just opens it and appends the game data.


UPD: In my strongest opinion these two actions should be done in the opposite order:
1. Create a raw game data file (perhaps in a temp folder).
2. Copy both data file & platform-related executable to target folder and merge them somehow.

Of course that would require to fix the "startOffset" written at the end of file (since there will be extra exe binary prepended), but that should not be a problem, because the ending chunk is always at the same place related to file's end.

UPD2: Actually it does create a separate raw data with the call to CompileGameToDTAFile.
What it needs, is a change in a workflow that would optionally leave game data unmerged with any exe.

[monkey0506]

Actually it does create a separate raw data with the call to CompileGameToDTAFile.
What it needs, is a change in a workflow that would optionally leave game data unmerged with any exe.

I kind of realized this in retrospect... I lost sight of what I was actually trying to accomplish I think (obviously, I'm trying to separate building the EXE from building the actual game data file). I realized what the real root of my problem was though. The engine file operations only have file modes for "Open", "Create", and "CreateAlways". If the file mode is "Create" then confusingly (IMO) it actually does an Append operation:

Common/util/filestream.cpp:190-200

Code: cpp

    else if (open_mode == kFile_Create)
    {
        if (work_mode == kFile_Write)
        {
            mode.AppendChar('a');
        }
        else if (work_mode == kFile_Read || work_mode == kFile_ReadWrite)
        {
            mode.Append("a+");
        }
    }

I was treating this as "OpenOrCreate" which opens the file at the beginning. And apparently I never did actually check the Length of the stream (in my C# implementation), just the Position...

Edit: With one more minor correction (writing startOffset as a 64-bit long instead of 32-bit int), I can now confirm that my C# conversion of "make_data_file" is fully functional. Now I'll start working on actually splitting the game data out from the EXE during the build process, and appending them back together afterwards.

[Crimson Wizard]

The engine file operations only have file modes for "Open", "Create", and "CreateAlways". If the file mode is "Create" then confusingly (IMO) it actually does an Append operation

I think I was using some library for the reference... We may renamed it if it is a problem.

Edit: With one more minor correction (writing startOffset as a 64-bit long instead of 32-bit int)

This means that the engine should be fixed too, and a package format version changed.
Sorry for nagging (yeah, uh, I always do that), but I'd suggest to make it a separate change, because changing formats is not a part of refactoring.

[monkey0506]

No, the engine code was already correctly writing it as a 32-bit int. The error was in my refactoring because BinaryWriter.Write(long) writes a 64-bit value.

[Crimson Wizard]

No, the engine code was already correctly writing it as a 32-bit int. The error was in my refactoring because BinaryWriter.Write(long) writes a 64-bit value.

Oh. Somehow I read that like that you decided to change it from 32-bit to 64-bit to allow really large files.

[monkey0506]

That's not a bad idea, but like you said, it's a totally different feature. Sorry about the confusion.