Topic: File access on Linux

[Snarky]

I've had a report from someone using the TotalLipSync module that while it works on Windows and Mac, it doesn't work on a Linux build of the game. I can only assume that this has to do with how the module accesses the file system.

The way the module works (code available here, see lines 267, 524 and e.g. 321 in TotalLipSync.asc), it tries to access lipsync data files for reading, by default in $INSTALLDIR$/sync, though it can be customized (AFAIK the game sticks with the default). It uses File.ReadRawLineBack() to read the data.

My first suspicion is that $INSTALLDIR$ doesn't work properly on Linux. Or maybe any file system access at all? My second theory is that it has something to do with the linebreak differences between platforms, which might mean that File.ReadRawLineBack() doesn't return the expected result and parsing fails. (But if this was the case I would expect it to fail on Mac as well.)

I don't have a Linux box, so I can't really test any of this myself.

[Crimson Wizard]

Is there an actual game that may be used in a test?

[VampireWombat]

I wonder how it would work using the Windows build via Wine. For whatever reason, most AGS games run better for me that way than the Linux builds.
And you shouldn't need a Linux box to test. You should be able to just use a live cd of Ubuntu or whichever flavor of Linux.

[Crimson Wizard]

And you shouldn't need a Linux box to test. You should be able to just use a live cd of Ubuntu or whichever flavor of Linux.

That is pretty easy to create virtual machine with Linux (using "VMware Player" for example), but of course you will have to learn how to use Linux in general of you don't know yet.

[eri0o]

@Snarky, when you say fails, you mean crashes with an exception, or does not work (but doesn't loudly fails)? If it's failing silently, then a game is needed to test and findout what is wrong. If not, a minimal game is still desirable, but can also be made. Either way, if there is an exception, it's better to have it.

[Snarky]

@Snarky, when you say fails, you mean crashes with an exception, or does not work (but doesn't loudly fails)?

I don't know, but I think it just doesn't work. The module is written to be fairly fault-tolerant.

But I just chatted with the game dev, and they figured it out: It's because filenames on Linux are case-sensitive, unlike in Windows. Because the module uses the character script-names to generate the filepath it looks for and doesn't change the case (so cRoger becomes $INSTALLDIR$/sync/Roge1.pam and so forth), while the filenames were saved all in lowercase, it didn't find them.

[Crimson Wizard]

But I just chatted with the game dev, and they figured it out: It's because filenames on Linux are case-sensitive, unlike in Windows. Because the module uses the character script-names to generate the filepath it looks for and doesn't change the case (so cRoger becomes $INSTALLDIR$/sync/Roge1.pam and so forth), while the filenames were saved all in lowercase, it didn't find them.

Ahh, this was the first thing that came to my mind, but I thought that's too simple...

E: I am wondering, what if we make AGS to be case-sensitive intentionally, then all versions of the game work consistently and such problems will be found earlier.

[eri0o]

Wait, Mac is case sensitive too, isn't? Also Android and iOS too, no?

[morganw]

On a Mac is isn't by default, and a lot of system requirements for programs say the filesystem must not be case-sensitive in order for everything to keep working.
Isn't the solution here just to use the correct filenames when reading or writing? If you don't assume that case doesn't matter, it will work everywhere.

[Crimson Wizard]

There are lots of games made before Linux port became a thing, or before it became usual thing to consider.
Something I've just realized, if $INSTALLDIR$ paths (or other paths) are case sensitive, that will break backwards compatibility, because idea of a port was to be able to run all existing games.
Actually, many paths in AGS engine are using special processing that checks both case sensitive and case insensitive filenames on Linux. Probably it is an oversight that it does not do that when opening files in script. This is why IMO the problem described here is actually a bug.

My strong belief is that it's important to make engine work consistently and set up a rule for being always case sensitive or case insensitive regardless of the system. A lot of users who make games with AGS do not know these peculiarities and may got surprised that their games stop working when ported to Linux. AGS may ease it for them by either enforcing case-sensitivity everywhere (in which case they will notice errors early), or case-insensitivity everywhere (although that may be confusing for Linux users).

[morganw]

Maybe this is a good example of splitting between AGS3 and AGS4, rather than trying to accommodate the old behaviour and also fix the issue. I can't really see the benefit in making everything case-insensitive, and this is related to the file system rather than a platform (ReFS supports case sensitive filenames, and that is for Windows). I would vote for just removing all the 'helper' parts (so the consistency is achieved by not doing anything) and letting the filesystem deal with it. It looks like the line handling assumes Windows line endings too, which is probably an issue long term.

[Crimson Wizard]

I would vote for just removing all the 'helper' parts (so the consistency is achieved by not doing anything) and letting the filesystem deal with it.

But if AGS will rely on filesystem to do everything, then a game scripted and tested on one computer may not work on another without rescripting / renaming files. Having all those differences will continiously get users into trouble (already does).
I am coming at this from the perspective that AGS is also a tool for users who are not tech specialists, and so my primary intent is to find a way to make things easier for them, by making sure no matter what they script will work (or not work) same way on all platforms. Guarantee that if they have "game files" that work on one system, then same files will load on other.

I admit that don't know if that will be easy to implement. Maybe not even biggest priority too.
Defaulting to case-sensitive file system might probably be a right thing to do, although I wonder if it's possible to force AGS to treat filenames as case sensitive on case-insensitive system. Probably I am speaking about virtual filesystem here...

It looks like the line handling assumes Windows line endings too, which is probably an issue long term.

Yes, this is also a problem. I think for game data files there again must be a fixed line ending (unix style?). For temp files like logs AGS could use system default linebreak maybe?

[morganw]

I am coming at this from the perspective that AGS is also a tool for users who are not tech specialists.

Potentially you could just emit a warning when a read or write was only successful because the filesystem was case-insensitive. If the whole things is abstracted away into a virtual filesystem, that is really a different concept. In this example the read write functions are operating on the filesystem, with assumptions about how the filesystem operates. I guess it would be less of an issue if there was a try/catch wrapper in the scripting; if I copied a game to a FAT16 volume I wouldn't expect the engine to adapt to that.

[Alberth]

Defaulting to case-sensitive file system might probably be a right thing to do, although I wonder if it's possible to force AGS to treat filenames as case sensitive on case-insensitive system. Probably I am speaking about virtual filesystem here...

The other direction is feasible too (being case-insensitive on a case-sensitive file system). Either move implies you need to read the actual file and directory names from the disk, and check them against what the user supplied. Forcing case-sensitive means the user must type it exactly right; allowing case-insensitive means accepting other names that differ in case only. The latter (being case-insensitive) may be more user-friendly from the perspective of a windows user I think.
It does have the problem that a case-sensitive file system could have more than one file that matches, eg "ABC" and "AbC"; I think it would be fair to warn about it in that case, as these never come from a case-insensitive file system.

The generic approach is quite involved, if you only fix or check paths just before opening files, you probably get a long way already.

Other problems could be the "\" vs "/", and the "X:" drive things, but that may have been covered already. More subtle, unix and windows have a different set of allowed characters in a filename, unix allows anything but / afaik, windows tends to get very upset  with characters like ":", there may be more such characters.
Note that I assume not passing paths through shell or any environment variable or so, as in that case you get quoting and escape mess.

Yes, this is also a problem. I think for game data files there again must be a fixed line ending (unix style?). For temp files like logs AGS could use system default linebreak maybe?

Another solution is to accept any combination or \r and \n as a line-ending; if you get the same character a second time, it's a second line-ending ie \n\n or \r\n\r\n. I don't know what Mac does nowadays, but if it dropped using purely \r (ie it uses \n or \r\n or \n\r), you could consider \n to be "the" newline, and \r as discardable whitespace.