Feature: Include custom files into game package

[Crimson Wizard]

There is a request by Radiant: http://www.adventuregamestudio.co.uk/forums/index.php?issue=556.0
What he is asking is to let game developers include certain files into compiled game package and read (but not write) them at runtime. As far as I can guess, this is to ensure they are always present and not e.g. occasionally deleted or modified by players.

From the engine's side this is a very simple task, because AGS already has a working method to read anything from a package which may be utilized.
This method works this way: engine can look for the file in the package, and if it is not found there, then it will look for stand-alone file into directory.

Script-wise this may be achieved introducing new function:

Code: ags

static File* File.OpenResource(String name, optional enum priority);

Where "priority" is either "Game file" or "Directory". This is precisely like internal resource manager works in AGS.


On the Editor's side, most easier would be to use same method as with Speech: introduce new folder, e.g. "Resources", and automatically pack all the folder contents into game.
Pro: this method is very simple and straightforward.
Con: this method requires game dev to work outside of the Editor.

Alternative solution would be to add a new node in the Project Tree and let users fill it with file names. The files should still be copied localy, I think, similar to how AudioCache works.

There was also a question, whether we may need to have a script objects for such resources to access them in script by script names, e.g.:

Code: ags

File* f = rDataFile.Open();
String rDataFile.Filename;
// etc

I was playing with this idea for some time, but did not make up my mind fully yet. For now it looks for me like not giving much benefit.


Does anyone have any thoughts about the issue?

[monkey0506]

I could see how it could be useful, so my question would be how difficult would it be to add this as a feature. As you said, the engine can already read files from the package or directory, so it seems it wouldn't be too much work from the engine side.

As for the API, perhaps a Resource class could be introduced with a static method for look-up, and a global indexer:

Code: ags

static Resource* Resource.GetResource(const string resourceName, [i]optional[/i] ResourceLocation=eResourceGameOrDirectory);
static int Resource.ResourceCount;
static Resource* Resources[];

This would avoid flooding the global namespace with more script o-names. The Resource* could then be used for:

Code: ags

File* Resource.Open();
readonly int ID;
readonly ResourceLocation Location;
readonly String Name;

Of course, with the asset manager the way it is it might make more sense (i.e., be simpler) to avoid having instances of resources altogether and just have a static Open method which returns null for non-existent resources. Just thought this type of API might give a better end-user representation of the resource files in the script (similar to how audio clips are represented).

Regarding what you said about populating the resources outside the editor, it would be relatively simple to add a Resources branch in the project tree, and a leaf could be added by selecting a file which would be cached in a similar fashion to AudioClips. The properties pane could be used to assign copy behavior (game or directory) and a name (defaulted to file name).

Overall it doesn't sound like a terrible addition to me. I was just thinking of a way the editor could allow the user to specify arbitrary files to copy to the Compiled folder (perhaps specifying platform too). This would definitely prove handy for that.

[selmiak]

Would it be possible to have the compiler calculate the MD5 or SHA-1 hash of the files added to the game, store these values and check for the file integrity on startup of the game?
So no user can replace a video not compiled into the exe with a different video and screw things up. The worst case would be someone distributing the game with changed additional files somewhere on the net, so this could be avoided.
Depending on the filetype missing or changed the game could either abort with an error message complaining about the missing file or, for example if the file is a video, give an error message but continue without playing the video at the desired place in the game.

[abstauber]

I think this would be really helpful. Also my project has a few external files which I could hide inside the resource file.
Would those files be read-only?

[Crimson Wizard]

Would those files be read-only?

Of course they will. That is the point.

[abstauber]

Did I win the dumb question of the day award? Well at least I've qualified...

Anyway: cool, I'd love to see this in 3.4.x

[Crimson Wizard]

As for the API, perhaps a Resource class could be introduced with a static method for look-up, and a global indexer:

Code: ags

static Resource* Resource.GetResource(const string resourceName, [i]optional[/i] ResourceLocation=eResourceGameOrDirectory);
static int Resource.ResourceCount;
static Resource* Resources[];

I still think we need a static function that will return File object. Opening resource would be most common operation, yet with this proposal it will require writing 2 lines and creating 2 managed objects where you could have 1 line and 1 managed object:

Code: ags

Resource* r = Resource.GetResource("name");
File *f = r.Open();

instead of simply

Code: ags

File *f = Resource.Open("name");

Also, I still do not see any benefit of having a Resource type, at all. Is there any scenario when it may come handy?
And is there really a need for numeric ID?

[Calin Leafshade]

I agree with CW. Really there is no difference between a file stored in the game package and a file on the file system. They are both just byte streams with arbitrary data.

[Radiant]

I'm glad to hear this is feasible.

The reason I requested this is for level files for (the sequel to) Indiana Rodent; these are basically binary files that can be read from disk. I can pull some shenanigans to import them into a room file as strings, but I'd prefer this method. Personally, I don't need a numeric ID.

[monkey0506]

I suppose I was thinking that resources might have some additional properties that might need to be handled by a Resource class, though in retrospect I suppose that's probably not the case (especially for arbitrary files).

In that respect I would say that the "priority" or "location" should not even be a parameter to the function at all, but instead just a new "special tag". We already have $SAVEGAMEDIR$ and $APPDATADIR$ which are compatible with File.Open, so I would suggest a new tag such as $RESOURCES$ which would indicate that the file is an embedded resource of the game package. The "default" behavior of using the game's installation directory would be preserved simply by not using this new tag. If a path is given containing this new tag with a FileMode other than eFileRead, a null pointer is returned.

This keeps changes to the engine at a minimum (no actual API changes) and is completely consistent with the existing function.

Code: ags

File *f = File.Open("$RESOURCES$/myFile.txt", eFileRead); // assume valid embedded resource "myFile.txt"
if (f != null)
{
  // do stuff!
  f.Close();
}
else if (File.Exists("$RESOURCES$/myFile.txt")) // should never happen
{
  // presumably, File.Exists should also be able to check for the existence of embedded resources
  AbortGame("Game files corrupted");
}
f = File.Open("$RESOURCES$/myFile.txt", eFileWrite); // assume the same for eFileAppend
if (f != null) // should never happen, cannot open resource in write/append mode
{
  Display("Opened resource in write mode!");
  f.Close();
}
f = File.Open("$RESOURCES$/madeUpResource.dat", eFileRead); // non-existent resource file
if (f != null) f.Close();
else Display("Made up resource doesn't exist!"); // should always happen

[Crimson Wizard]

One thing: you may have "$RESOURCES$" folder, this is a valid directory name.
Well, same for "$SAVEGAMEDIR$" and "$APPDATADIR$".

[Radiant]

Perhaps, rather than having a $RESOURCES$ virtual directory, there could be a function File.OpenResource("myfile.txt") which would read from the resource file, and automatically assume Read mode (since Write and Append aren't valid anyway).
(edit) That's basically what Crimson Wizard said in the top post; I would prefer this approach to File.Open("$RES/foo").

It could also assume priority="Game file" since if the priority was "directory" there would be no need for this feature request

[monkey0506]

Well, as I said, I proposed the tag for consistency's sake. If we're adding a function then File.OpenResource(const string) is fine. As noted, non-embedded resources may be opened normally. Though I would add that a File.ResourceExists(const string) should also be included.

[RickJ]

I really like this idea and can think of a number of things I would use it for. 

Has anyone given any thought as to how resource files will be populated?  It could be done manually or in some cases it may make sense for AGS to just write the data to the file.  In the latter case something similar to monkey's suggestion may make sense as follows:

Code: ags

// The $RESOURCES$ tag contains the path where embedded resource files are 
// located.  The normal AGS file operations can be used to read/write/append 
// files at this location as one would expect.

// Open file system "myFile.txt" for read
File *f = File.Open("$RESOURCES$/myFile.txt", eFileRead); 

// Open file system "myFile.txt" for write or append
File *f = File.Open("$RESOURCES$/myFile.txt", eFileWrtite); 


// When the game is compiled files contained in the $RESOURCES$ folder are
// embedded into the game file(s).  The embedded files can be opened for 
// read using eResourceRead instead of eFileRead. 

// Open embedded resource "myFile.txt" for read
// The $RESOURCES$ tag is ignored but allowing it's inclusion makes it easy to
// switch between the embedded file and the actual file by simply changing the mode.
File *f = File.Open("$RESOURCES$/myFile.txt", eResourceRead); 

// Being able to exclude $RESOURCES$ tag is convenient when access to the actual 
// file is not required
File *f = File.Open("myFile.txt", eResourceRead);

This would allow the resource files to be populated by the game itself during development.  A variable could be used to easily switch between the embedded file and the actual file during development.  Changes could be made to the game as it is running , something that has been discussed several times in the past.  One mode would be added to the File object and the compiler would need to be aware of the $RESOURCE$ tag and embedded any files found there.

[Crimson Wizard]

A variable could be used to easily switch between the embedded file and the actual file during development.

Well, that is how internal AGS resource manager works now, and what I actually suggest to expose to script API.

I have to say I do not really like to have a string tag for this. I'd prefer to use explicit type (Resource.Open) or separate function (File.OpenResource).

There is also one idea I was thinking over some time ago: a "repository" type object, which presents a concept of generic "file location", each having its own purpose.
For example, we could have:

Code: ags

UserData.OpenFile(); // open file in the private user location (savedgames, etc)
AppData.OpenFile(); // open file in shared app location (highscores, player created data, etc)
Resource.OpenFile(); // open file in the game data location (main game data and custom resources)

[monkey0506]

Regarding the "repositories" idea, maybe what could be done would be to simply do away with the special string tags altogether in favor of another enum which would specify file location, added as parameters for File.Open, File.Exists, and File.Delete, all defaulted to game directory. I don't see a reason or significant benefit behind adding this as a new script object type.

[Crimson Wizard]

Well... this feature was somehow left behind...

I was thinking about it recently (and eager to implement into 3.4 at last).
For particular reasons I am leaning to the final monkey_05_06 suggestion:

In that respect I would say that the "priority" or "location" should not even be a parameter to the function at all, but instead just a new "special tag". We already have $SAVEGAMEDIR$ and $APPDATADIR$ which are compatible with File.Open, so I would suggest a new tag such as $RESOURCES$ which would indicate that the file is an embedded resource of the game package. The "default" behavior of using the game's installation directory would be preserved simply by not using this new tag. If a path is given containing this new tag with a FileMode other than eFileRead, a null pointer is returned.

The main reason I chose this way is that I cherish the concept of "Repository", an abstract file location, each having its own read/write permissions, which actual implementation is hidden from script API, as I mentioned few posts above. But introducing new "Repository" type into script does not present any immediate benefit, and having it monkey's way would perfectly match the current "repository" selection of AGS ($SAVEGAMEDIR$ and $APPDATADIR$).
So it be.

I will be adding this to 3.4 very soon.

[Crimson Wizard]

Test build (copy over 3.4.0.5)

http://www.mediafire.com/download/bzggz7b3jbbe427/AGS-3.4.0.5--feat-user-data.zip

NOTE: I renamed $RESOURCES$ to $DATA$, because its shorter and I liked it more...

Features:
* When compiling the game, editor attaches all files from the "Data" subfolder in game project directory, including nested subfolders.

* In the script, following file-related functions now support "$DATA$" tags, which defines game resource:
File.Open()
File.Exists()
ListBox.FillDirList()

The name of the resource is always "$DATA$/[subdirs/]filename".

For example, in your game project create "Data" folder, then "Levels" folder inside. Put "test1.dat", "test2.dat" and "test3.dat" files in the "Data/Levels" (the contents does not matter).

In your game add a gui with list box and do this at some point in script:

Code: ags

ListBox1.FillDirList("$DATA$/Levels/*.dat");

This will list all "dat" files.

To open the file for reading do

Code: ags

File *f = File.Open("$DATA$/Levels/test1.dat", eFileRead);

NOTE: you cannot open DATA files for writing. Trying to do so will result in null pointer returning from File.Open.

[eri0o]

I occasionally look into old threads to see how things changed, perceptions and all...

Anyway just to mention that this was mapped to the issue here - I mean not really because this discussion is from before me, but I happen to have had the same idea/need

https://github.com/adventuregamestudio/ags/issues/1227

CW implemented the feature a few years ago

https://github.com/adventuregamestudio/ags/pull/1372

It then made into the 3.6.0 release when it was in its early alpha stages, in 3.6.0.7!

https://github.com/adventuregamestudio/ags/releases/tag/v.3.6.0.7

Anyway, I think this thread can be locked now

Hopefully if someone finds this thread through a search engine they can now know the feature exists.

[Crimson Wizard]

Anyway, I think this thread can be locked now

Right, I forgot, will lock now.