[TEST] Editor with Linux compilation (NEW! AGS 3.4.0+Linux) 2 Nov 2014

Started by monkey0506, Fri 02/05/2014 23:40:42

Previous topic - Next topic

monkey0506

This is something that's been needed for a while now, and though it took a fair amount of ground-work, the process for building for Linux is actually quite simple.

You can select the target platform(s) under the General Settings pane (currently only Windows and Linux, "DataFile" option cannot be turned off).

NOTE: If your project uses any plugins, the editor will search EDITOR/linux/lib32 and EDITOR/linux/lib64 for the appropriate shared library file. This always matches the pattern "lib" + plugin_name + ".so". The appropriate variant for "agsteam.dll" is "libagsteam.so". If a required plugin is not found you will be prompted whether you wish to continue with Linux compilation.

NOTE: Currently the Linux folder is not emptied by the editor, so your files may get out of date if you choose not to build for Linux.

Download
Source

Edit: 2 November 2014, I have now uploaded a full build of the AGS Editor (including latest commits from develop-3.4.0 branch, this is post-3.4.0.1 but pre-3.4.0.2) with the option of building for Linux. The editor now builds the game data files separately from the Windows EXE, so this is also the first version to not rely on publishing the Windows EXE for Linux versions. NOTE: The ags32 Linux engine currently included in this build is NOT up-to-date (I believe it is from an earlier 3.3.1 build). I will make sure that updated Linux engine files for both 32- and 64-bit are included in future versions. Also note that the Windows compiled files are now placed in Compiled/Windows.

Edit: (3 May 2014) I uploaded a new version which places the Linux files into a subdirectory of the editor directory. This also includes a bugfix -- the original version was producing Linux scripts with Windows-style line endings, which has been corrected.

Problem

Wow, that's great!

I can conform that it works for me. I'm using Kubuntu (14.04) - I started the AGS Editor using Wine, and successfully compiled a test game for linux. But I had to change some file permissions because the starter script, ags32 and ags64 were not flagged executable. After I had done that, the test game worked like charm. :)
If you want, I can run some more tests on different Linux distributions.

Crimson Wizard

Would not it be better if the build targets were configured for every project separately, instead of preferences? Thinking about future, if more build targets will be added later this may become cumbersome to always build all targets for every project user works on.
This, and another thing: I have certain doubts about "Build EXE" always building for all platforms. Consider game developer wants to build only Windows build while developing/testing game, and all builds when preparing distribution. Maybe it's worth adding more build commands to menu?
This needs to be thought through, considering more build targets and different situations, IMHO.

monkey0506

Quote from: Problem on Mon 05/05/2014 10:03:34I had to change some file permissions because the starter script, ags32 and ags64 were not flagged executable.

Hmm, good point. I'm not terribly familiar with how this correlates to Windows' security, or if it does at all, but I could try setting some permissions on those files directly from the editor.

Quote from: Crimson Wizard on Mon 05/05/2014 10:28:30Would not it be better if the build targets were configured for every project separately, instead of preferences? Thinking about future, if more build targets will be added later this may become cumbersome to always build all targets for every project user works on.

Maybe you're right about this. Is there a way that I could perhaps add to the General Settings pane a dropbox that would have checkboxes for each available target platform? There's also the matter that the Preferences page is becoming a bit too cumbersome IMO for a single dialog. Perhaps if we could add tabs inside the dialog, or just have it open as a normal tab instead...but that's an entirely different matter.

Quote from: Crimson Wizard on Mon 05/05/2014 10:28:30This, and another thing: I have certain doubts about "Build EXE" always building for all platforms. Consider game developer wants to build only Windows build while developing/testing game, and all builds when preparing distribution. Maybe it's worth adding more build commands to menu?
This needs to be thought through, considering more build targets and different situations, IMHO.

No, no, I agree. I did it this way for now because I know that both Linux and Android reuse the Windows executable. I do like the idea of having something in the General Settings (I'll see what I can find to make this work well) and then having a separate "Build All" button (probably add a tooltip that it's "build all platforms" and not "rebuild all files").

None of this is set in stone, that's why I (a) put it in a separate branch of my personal fork, (b) haven't yet submitted a pull request, (c) posted this as a testing build, and (d) held my breath for three days waiting for someone to post some feedback about it. :P

I also modified this to use hard links for the linux/data copy of the game instead of directly copying the game files. For larger games this makes the compilation much faster, though it does have some side effects. Thinking about it now I will actually modify it to do a full copy of acsetup.cfg, but the other game files should be fine as hard links (unless anyone has any strong objection to this).

Additionally, I realized that for some reason I hadn't included the Linux licenses folder. It may not be necessary to distribute that with your game (I'm not sure), but at the very least it should be included in the files distributed with the editor.

This gives me something to work on. :D

monkey0506

#4
I now have gotten things sorted out so that the target platforms are selected from the General Settings pane. For now, Windows must always be selected as a target. I removed the option from the Preferences page, and instead included a "Rebuild all platforms" option under the Build menu. Thinking about it, I may rename it to "Rebuild target platforms".

Still on my to-do list:

- Make sure file permissions are set for Linux executables
- Do a full copy of acsetup.cfg instead of hard linking it
- Add a "Build target platforms" option to accompany the "rebuild" option
- Rearrange directory structure:
|-- If building for Windows only, put all compiled files in "Compiled" folder directly
|-- If building for multiple platforms, put compiled files in "Compiled/[Platform]" directory

For the most part this should be relatively easy (the last item here is probably the most difficult). I fully suspect I should have a new build pushed up soon, so it should be out in plenty of time to include in the next 3.3.1 alpha/beta.




Edit: Setting the relative build paths is a bit trickier than I thought it would be. Specifically, there are some hard-coded strings in the native assembly that refer to the "Compiled" directory as being the root of the compilation process. I'll need to approach that with caution.

monkey0506

#5
I've updated the first post with links to the most current version. Moving forward this will only be designed around AGS 3.3.1.

Note that this still compiles the game to a separate "linux" folder. I have not yet had a chance to refactor the native code to by-pass the hard-coded references to "Compiled".

bgordebak

#6
This is great!

I tried 3.3.1, and it mostly works. But in the resulting script to run the game, there's a problem. The script is in Compiled/Linux/data folder, but it searches the executables in $SCRIPTPATH/data/ which results in looking into Compiled/Linux/data/data, and it can't find the executables.

It should be very easy to fix I guess.

Edit: Actually moving the script out of the data folder solves the problem.

monkey0506

Sorry about the script being in the wrong folder. As you discovered it needs to be moved up a level.

I've finally gotten around to mucking with the native code (mostly, just adding a parameter here and there for the compiled folder name), and I think I've just about got it working. I need to work out a couple of bugs, and then I should have a working version ready to pull into the 3.3.1 dev branch.

bgordebak

Thanks. Keep up the good work.

I think this may be the ultimate way to deploy games for Linux.

Crimson Wizard

#9
Regarding the downloadable build, I am getting this error when I start the editor:

Quote
An unexpected error occurred trying to start up the AGS Editor. Please consult the details below and post the error to the AGS Technical Forum.

System.IO.FileLoadException: Could not load file or assembly 'AGS.Native, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log or use the command-line sxstrace.exe tool for more detail. (Exception from HRESULT: 0x800736B1)

File name: 'AGS.Native, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' ---> System.Runtime.InteropServices.COMException (0x800736B1): The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log or use the command-line sxstrace.exe tool for more detail. (Exception from HRESULT: 0x800736B1)
   at AGS.Editor.NativeProxy..ctor()
   at AGS.Editor.NativeProxy.get_Instance()
   at AGS.Editor.ApplicationController..ctor()
   at AGS.Editor.Program.startupTimer_Tick(Object sender, EventArgs e)
   at System.Windows.Forms.Timer.OnTick(EventArgs e)
   at System.Windows.Forms.Timer.TimerNativeWindow.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

I tested this on one computer so far, seeing that it works for others makes me think there's something specific about it.
Building Editor from source works fine for me.
E: this may actually be an good ole "missing matching VC redist" problem http://answers.microsoft.com/en-us/windows/forum/windows_7-pictures/error-the-application-has-failed-to-start-because/df019c0d-746e-42d0-ad68-465e18e3f3ef


Regarding internals. This may be too much for the first version, but I want to suggest introducing "target platform" interface and implementations instead of enum. Maybe this is also a way towards "target platform" plugin interface.

What is more important, frankly I am very dissapointed to see that you don't compile a raw game data file to use for all platforms, but copying Windows exe instead... this is something I hoped won't be happening :(.
I think the proper way to do this is to:
- Make a raw game data compilation to the temporary file.
- Run all target builds which take this game data.
- Optionally, this also will let us implement a "Only game data" output target.

Sorry for not telling this earlier... :-[ I sort of forgot about testing this out.


EDIT: found another thing. You are using CreateHardLink from kernel32.dll. I think that conflicts with the plan to make Editor cross-platform.

monkey0506

I agree with what you're saying, this was just more of a starting off point. Bear in mind that this is the first non-Windows build option that the editor has offered, so a lot of this code was initially experimental.

As to creating hard links, .NET doesn't provide any framework methods, so it will have to be done by native assemblies. Mono's framework provides a link method for *nix systems. We can use the preprocessor to select the right method based on the loaded .NET runtime (MS or Mono).

Calin Leafshade

I assume the preprocessor runs before the compilation to the IL so I dont think you can use it like that since mono runs precompiled binaries.

monkey0506

#12
The compilation of this assembly (to IL) would be referenced against the precompiled Mono or MS binaries, but the preprocessor would be running in this assembly, not in the framework binaries.

http://stackoverflow.com/a/5159714/1136311

Edit: Reading a bit further, I do see that one of the goals of Mono is for assemblies compiled against either the MS or Mono binaries to be able to run seamlessly while using either, interchangeably. In this case perhaps it is just best to avoid hard links since there isn't a consistent way of creating them from C#, though the extra copying is rather unfortunate.

Crimson Wizard

Quote from: monkey_05_06 on Wed 20/08/2014 03:24:57In this case perhaps it is just best to avoid hard links since there isn't a consistent way of creating them from C#, though the extra copying is rather unfortunate.
Well, when the game developer will preparing installation packages he will need to copy game files anyway. While testing game he may need only one copy.

I was thinking that at the moment you have a concept of building game for various platforms. Yet, in fact, AGS builds only one game for everything, the rest is just combining proper files together.
Maybe we should distinct building the game and deploying it. In this case we could have only a) game files b) config file in the "compiled" directory. The engine can run game from any location, if the path is one of the command line arguments. Therefore you do not really need to assemble all files before actually getting a distribution packages ready.

monkey0506

#14
Well to me, ideally, you should just be able to zip up the build folder and ship it off, sans any additional copying. In order for that to work, the editor will basically have to copy the game data file (whether or not the Windows engine is attached) to every platform's distribution/deployment folder. That's why I thought that using hard links would be nice, because it would save copying potential gigabytes of data (namely, commercial games). There doesn't seem to be a nice, consistent, cross-platform way of creating a hard link from the editor though. The end result is the Compiled folder growing by a factor of one for each additional platform (Windows only = factor 1; Windows and Linux = factor 2; Windows, Linux, and Android = factor 3). Hence, the additional copying is unfortunate.

I've been reading further into it, and I think I've found a reasonable (and yes, cross-platform) solution. Instead of relying on the preprocessor for conditional compilation, it seems we can use Process.Start with run-time checks against the framework that is in-use:

Code: CSharp
public void CreateHardLink(string destFileName, string sourceFileName)
{
    string fileName = "mklink";
    string args = String.Format("/h {0} {1}", destFileName, sourceFileName);
    if (IsMonoRunning())
    {
        fileName = "/bin/ln";
        args = String.Format("{0} {1}", sourceFileName, destFileName);
    }
    Process.Start(fileName, args);
}


This should allow building the assembly without any compile-time dependency on a specific framework variant.

Also, you have rather convinced me that I should refactor my work before doing any commits to the main repo. This will delay any official builds targeting anything other than Windows, but you've made your point. In response to that, I am working to implement a uniform structure for the build target classes. The biggest portion of the work will come from refactoring the native build code (targeting only Windows) to instead build only the data file, and then handle the rest from the new classes.

Alberth

#15
My Linux has other ideas about 'ln':

Code: bash
$ which ln
/usr/bin/ln
Hardcoded paths to system binaries is usually a bad idea.

You probably should use the link system call (see link(2)) to link a file, but if you really want to start a process for it, just let the standard search path find it for you, or setup a configuration utility like ./configure, to set the path to the program at configure time.

Edit: Oh, I just found out I do have /bin/ln as well, and it points to /usr/bin/ln. Your code would not fail thus at my system.

monkey0506

#16
I understand it's probably not "best practice" under normal circumstances, but as far as creating a hard link from a framework-agnostic C# assembly, there's not much of a better way AFAICT. I referenced the "/bin/" directory because in the SO article that gave me the idea, someone was complaining that Process.Start wasn't finding ln, but someone else suggested their PATH may have been mucked up. I could leave it to just "ln" if that's a better solution.

Crimson Wizard

Just a small idea I had recently.
This has been a tradition to supply a No-MP3 version of the engine separately of main distribution; but I was thinking, what if we let user select which version of the engine they would like to use in project compilation?

monkey0506

Probably a good idea. Simple enough to implement once I get the rest of this sorted out. ;)

Gurok

We could almost do that automatically based on whether the project includes references to MP3 files.
[img]http://7d4iqnx.gif;rWRLUuw.gi

SMF spam blocked by CleanTalk