Adventure Game Studio | Forums

AGS Development => Editor Development => Topic started by: Crimson Wizard on Mon 17/08/2015 09:52:42

Title: Changing the way Editor references original files
Post by: Crimson Wizard on Mon 17/08/2015 09:52:42
There was a request made by Calin Leafshade: http://www.adventuregamestudio.co.uk/forums/index.php?issue=520.0
Basically it is about switching from using absolute paths to original sprite/audio files to relative ones.

While it is totally clear that this will benefit users who store original assets relative to the project folder (locally to project folder, or close to it), the question that bothers me is, would not that break a use of shared resources for someone?

Is this a common method for some of the AGS users to have a shared resources folder somewhere on their computer to take sprites/audio files from, regardless of where the game project itself is located?

I came to this question because I personally have a "library" of sprites for creating small test games, and this library is located in some place which is completely irrelevant to location of my AGS projects. Is this uncommon to how people work with AGS?
I am worried that changing to relative paths in all cases could break work for someone.
Title: Re: Changing the way Editor references original files
Post by: RickJ on Mon 17/08/2015 10:45:13
I can see a need for both. Could we not have a "path" setting containing multiple paths, both absolute and relative?   
Title: Re: Changing the way Editor references original files
Post by: ChamberOfFear on Mon 17/08/2015 10:48:50
That's a fairly good idea actually, I was worried there was no win-win situation in this. But if we add a new project specific property that lets the game developers choose if the path should be stored as relative or absolute we would meet the needs of both situations and not break previous projects.
Title: Re: Changing the way Editor references original files
Post by: Crimson Wizard on Mon 17/08/2015 10:51:22
Yes, I totally agree, this will be optimal, if its possible to implement. :)
Title: Re: Changing the way Editor references original files
Post by: ChamberOfFear on Mon 17/08/2015 10:53:38
I'll take that as we're in agreement then :) I'll take a look at it
Title: Re: Changing the way Editor references original files
Post by: cat on Mon 17/08/2015 10:59:58
Would it be possible to detect if the path is a sub-path of the main game folder and in this case keep it relative? In any other case it could be absolute.
Title: Re: Changing the way Editor references original files
Post by: ChamberOfFear on Mon 17/08/2015 11:08:48
Quote from: cat on Mon 17/08/2015 10:59:58
Would it be possible to detect if the path is a sub-path of the main game folder and in this case keep it relative? In any other case it could be absolute.

I believe so yes, to be more specific I imagine the end solution something like this.

If absolute paths is checked:



Files and folders outside the projectAbsolute paths
Files and folders inside the projectRelative paths

If relative paths is checked:



Files and folders outside the projectRelative paths
Files and folders inside the projectRelative paths

Files and folder located on another disk will always return an absolute path.
Title: Re: Changing the way Editor references original files
Post by: RickJ on Mon 17/08/2015 15:57:56
It should be a search list rather than be either or, as shown in the example below.

Code (ags) Select

SpritePath=PATH=C:\Ags\Standard\Sprites;.MyGameSprites;


[edit]
I may have mis-rtead your comments ChamberOfFear.
[/edit]
Title: Re: Changing the way Editor references original files
Post by: Crimson Wizard on Mon 17/08/2015 16:09:34
Quote from: RickJ on Mon 17/08/2015 15:57:56
It should be a search list rather than be either or, as shown in the example below.

Code (ags) Select

SpritePath=PATH=C:\Ags\Standard\Sprites;.MyGameSprites;


Quote from: ChamberOfFear on Mon 17/08/2015 10:48:50
But if we add a new project specific property that lets the game developers choose if the path should be stored as relative or absolute we would meet the needs of both situations and not break previous projects.

Sorry, ChamberOfFear, I seem to misread what you say. I was thinking you are going to do what RickJ suggested in his first post here.

I do not think it will be convenient to use project-specific switch, for two reasons:
1) it may be confusing to set this for every new project.
2) it seems like a valid scenario if the project uses assets from different locations, some of which may be relative and some absolute.


E: erm, or we all are confused? maybe we should restate what we have decided again?
Title: Re: Changing the way Editor references original files
Post by: ChamberOfFear on Mon 17/08/2015 16:32:11
Quote from: Crimson Wizard on Mon 17/08/2015 16:09:34
E: erm, or we all are confused? maybe we should restate what we have decided again?

I think I was the one that got confused. For safe measure, please restate how the implementation should be.
Title: Re: Changing the way Editor references original files
Post by: cat on Mon 17/08/2015 16:33:50
May I make a suggestion?

The user adds a new resource.
Scenario 1: It is stored in a subfolder of the game folder -> it is stored as .\sounds\bang.ogg
Scenario 2: It is stored somewhere else -> it is stored as C:\Users\cat\Pictures\logo.bmp

I wouldn't ask people to mess with define the path type or even define their own path settings.
Title: Re: Changing the way Editor references original files
Post by: Crimson Wizard on Mon 17/08/2015 17:05:21
Quote from: cat on Mon 17/08/2015 16:33:50
May I make a suggestion?

The user adds a new resource.
Scenario 1: It is stored in a subfolder of the game folder -> it is stored as .\sounds\bang.ogg
Scenario 2: It is stored somewhere else -> it is stored as C:\Users\cat\Pictures\logo.bmp

I wouldn't ask people to mess with define the path type or even define their own path settings.

Well, that was initial idea; ChamberOfFear suggested that user may want relative path even if the original files are not inside the project folder: for example if there is a shared resources folder, and several project folders positioned near.
Title: Re: Changing the way Editor references original files
Post by: ChamberOfFear on Mon 17/08/2015 18:24:05
I'm almost certain I'm the one who has been misunderstanding things, probably got a little too overzealous when I realized it was possible to implement a solution that was satisfactory for both trouble scenarios. So let me try this again.

Every asset (sprites, audio and font) has a "SourceFile"-property which stores the file location for the asset in question, currently it stores absolute paths, always. The proposed fix is to store both absolute and relative path in a single string inside the "SourceFile"-property.

Using this we can find an asset by reading the SourceFile-property as follows
Code (pseudo) Select

if relative_path == valid
  use relative path
else if absolute_path == valid
  use absolute path
else
  could not verify path


Building off RickJ's example the format of the string should be as follows
Code (text) Select

SpritePath=PATH=absolute_path;relative_path;
AudioPath=PATH=absolute_path;relative_path;
FontPath=PATH=absolute_path;relative_path;


Example (The referenced file is inside the game project folder):
Code (text) Select
SpritePath=PATH=C:\MyGame\Assets\Sprites\sprite001.png;Assets\Sprites\sprite001.png;

Example (The referenced file is outside the game project folder, but on the same disk):
Code (text) Select
SpritePath=PATH=C:\Assets\Sprites\sprite001.png;..\Assets\Sprites\sprite001.png;

Example (The referenced file is outside the game project folder, and on a different disk):
Code (text) Select
SpritePath=PATH=D:\Assets\Sprites\sprite001.png;D:\Assets\Sprites\sprite001.png;
Title: Re: Changing the way Editor references original files
Post by: Crimson Wizard on Mon 17/08/2015 19:36:59
Quote from: ChamberOfFear on Mon 17/08/2015 18:24:05
Every asset (sprites, audio and font) has a "SourceFile"-property which stores the file location for the asset in question, currently it stores absolute paths, always. The proposed fix is to store both absolute and relative path in a single string inside the "SourceFile"-property.

Oh...
Well, this is where our understanding differs.
What I thought RickJ is proposing is a kind of global PATH setting which directs to folders containing resources, where the sprites should be searched for. His last comment (http://www.adventuregamestudio.co.uk/forums/index.php?topic=52573.msg636519215#msg636519215) kinda suggests this.
But I would wait for him to clarify this himself.

And I must admit, that your variant is interesting too.


IDK, what we will choose, but I was thinking on how RickJ's solution (in my understanding) could be useful, and I came to following opinion.

What is the difference between absolute paths and relative paths from the end-user perspective?
Absolute paths are good for any occasional file. For example, you desperately needed a sprite, browsed your disk and imported one from some random location. Does relative path has any meaning in such case? I guess no, we need only absolute path.
Relative paths, on other hand, are for organized repository. It is when you know for sure that there is a directory structure that you need for your project(s).

So, what I may suggest, is to have a list of only relative paths (either global, or project-specific). If the file was taken from any of those directories (and their subdirectories), then the relative path is stored. Otherwise - absolute path is stored.
The default value for this PATH setting would be ".\", i.e. "project directory".
Title: Re: Changing the way Editor references original files
Post by: ChamberOfFear on Mon 17/08/2015 20:03:24
Quote from: Crimson Wizard on Mon 17/08/2015 19:36:59
What I thought RickJ is proposing is a kind of global PATH setting which directs to folders containing resources, where the sprites should be searched for. His last comment (http://www.adventuregamestudio.co.uk/forums/index.php?topic=52573.msg636519215#msg636519215) kinda suggests this.
But I would wait for him to clarify this himself.
In retrospect that makes a lot of sense :P

Quote from: Crimson Wizard on Mon 17/08/2015 19:36:59
So, what I may suggest, is to have a list of only relative paths (either global, or project-specific). If the file was taken from any of those directories (and their subdirectories), then the relative path is stored. Otherwise - absolute path is stored.
The default value for this PATH setting would be ".\", i.e. "project directory".
I think this is the best solution proposed so far however, it's simple and elegant, and has little or zero chance to break previous games. But a couple of details needs to be hammered out.
Title: Re: Changing the way Editor references original files
Post by: Crimson Wizard on Mon 17/08/2015 23:03:20
Quote from: ChamberOfFear on Mon 17/08/2015 20:03:24

  • Should the list be global, project-specific or both? To me it makes sense for it to be project-specific.
  • Where in the editor should the list be accessible for the user?
Well, if we relate to existing IDEs, these have a IDE-wide path settings, and project-wide. If it would be possible, we could have a setting in Editor Preferences that would serve default value for new projects; and a setting in project's General Settings. There does not seem to be a suitable group for this now. Maybe create a new one called Source Files, or something like that?
Title: Re: Changing the way Editor references original files
Post by: selmiak on Tue 18/08/2015 00:03:10
I like RickJ's choose your own destinypath system, set it in generalsettings somewhere and use your relative or absolute path for this project and set it different for another project. Everything fine, just define the default setting... (btw, I like relative paths)
I don't understand what ChamberOfFear's suggestion offers besides making it more complicated. I don't quite understand it already so this must be really complicated... :wink:
Title: Re: Changing the way Editor references original files
Post by: ChamberOfFear on Tue 18/08/2015 00:29:31
Quote from: selmiak on Tue 18/08/2015 00:03:10
I like RickJ's choose your own destinypath system, set it in generalsettings somewhere and use your relative or absolute path for this project and set it different for another project. Everything fine, just define the default setting... (btw, I like relative paths)
I don't understand what ChamberOfFear's suggestion offers besides making it more complicated. I don't quite understand it already so this must be really complicated... :wink:

Doesn't matter :P I've already discarded it.
Title: Re: Changing the way Editor references original files
Post by: sonneveld on Tue 18/08/2015 03:02:59
I think anything that stores absolute paths should be in a separate file such that is obvious that it only for that machine/user.  That way you can ensure you don't commit machine/user specific information into your repository.
Title: Re: Changing the way Editor references original files
Post by: Crimson Wizard on Tue 18/08/2015 09:35:21
Quote from: sonneveld on Tue 18/08/2015 03:02:59
I think anything that stores absolute paths should be in a separate file
AGS saves this information forever since it was first created.
Splitting the project file is a rather larger topic... this would require to rewrite serialization mechanism.
Title: Re: Changing the way Editor references original files
Post by: RickJ on Tue 18/08/2015 13:22:19
I think the confusion comes from looking at this from two different perspectives.  I wasn't thinking much about how sprite import currently works but I think ChamberOfFear was thinking in these terms.

If there is a Sprite Path property then one could specify a number of locations where sprites can be found which can be specified relative to the project folder or be an absolute specification. As selmiak says settle on a good set of defaults so that most people won't notice.

Sprites would be imported relative to the path which could include both absolute and relative specifications.  There are a couple of situations that must be handled properly though. 

1) Suppose the same file name is used in more than one folder.  The path property is essentially a search list, so to re-import the file one would search the folders specified by the path property until  a match is found.  Now if the original file was imported from a folder appearing latter in the list then the wrong file would be found.  This could be solved in a number of different ways.  For example, each folder in the path specification could be assigned a logical or symbolic name, and that name would be stored with the sprite. Alternatively one could ask the user to select from a myriad of options if such a situation occurs.

2. Navigation when the import dialog opens would need to be aware of the path specification and probably select the first folder in the list by default. The user would then be able to select sub-folders and eventually a file.  When the sprite is imported it is save with a path that is relative to selected path folder. No need to look everywhere for sprites when the path specification tells where they are.  However, there should be provision to import a sprite from an arbitrary location as well.

There are a couple of use scenarios where this would come in handy.

Multiple Games with Common Assets (RON)
In a situations such as "Reality on the Norm" the same game assets are used on multiple projects. So one could easily imagine a repository external to the game folder that contains these sprites.  Of course there are always game specific sprites such as room objecfts, inventory items, etc that are game specific.;  These sprites could be kept in a repository residing in the game folder where they are used. 

Multiple Artists
If a project had multiple artists creating sprites e then each one could have their own repository.  If re-import mechanism had an option to select folder from the path specification then it would be possible to re-import everything from this artist but not that other artist. There are of course other kinds of groupings possible.

Sprite Override
Suppose there is a template for a specific kind of GUI.  The Module author provides a default set of button, panel, and other sprites used in the GUI.  The default sprites could be kept in an absolute location (i.e. the AGS template repository or AGS install folder).  Anyone or all of these sprites could be replaced simply by making s copy and modifications in the project's sprite repository.

As is usually the case there are likely more detail to iron out but hopefully the above explains my thoughts more fully.

Title: Re: Changing the way Editor references original files
Post by: Crimson Wizard on Tue 18/08/2015 13:56:40
I can see that the simplistic fix, initially proposed by Calin, has grown to a serious design change in this thread. This is all terribly confusing. I guess its what happens when the design is not properly discussed before changing something.


Let me point out that the AGS never had a full-scaled "re-import from source" mechanism. It was not long ago when "Replace from source" command appeared for sprites in AGS 3.3.0. It was made for simple convenience, without having big changes in mind.
Similarly, in respect for audio files: the connection between source file and Audio cache was introduced only in AGS 3.2.

There are a lot of questions rising when you start to think of this thing in whole. For example: why is there this bind between source and cached versions? What is the purpose of this duplication? Currently it appears that the cache (at least sprite cache) is actually a main version, because re importing sprites could be nearly as complicated as importing sprites anew. So maybe it will be better to let Editor work with original source instead, and use cache only if source is not found?
What to do with tiled imports, or imports from clipboard? Should these be copied to some "source" folder too first?
Should modified source be copied to sprite cache automatically? because now it does not.

Take absolute paths, why anyone would need them in a collaborative project at all? Maybe introduce enviroment variables, like "PATH_TO_DEFAULT_SPRITES", and redirect sprite search there?

Etc, etc, etc.




This is something you need to have in mind: AGS simply does not have the complete feature of binding "source" material and project "cache". Therefore, it is not correct to propose modification to existing design, because there is practically none; if one wants this work differently and being organized, then a full design plan should be written from scratch, and discussed.
All this is beyond the scope of the initial idea of a fix.

If you are interested in discussing this, I have a suggestion to actually separate the fix issue and design issue, because the latter will take more time to flesh out and implement.
The fix we were originally speaking of is very much wanted: I was noticing this was mentioned several times on forums. The purpose of it is mainly to let people keep source material inside project folder and reference that with relative paths.
We may go with a more simple decision for current release (AGS 3.3.5), something like cat's proposal; and spend a proper amount of time designing the actual re-import mechanics for the next big release.


Maybe we should first agree on what we are discussing here, before continuing?
Title: Re: Changing the way Editor references original files
Post by: Snarky on Tue 18/08/2015 14:49:35
I think CW has it right. All this duplication is bad, and to really get a satisfactory solution a redesign from the ground up is needed.

In the long run, I think AGS should move to having the source files as part of the project, so they can be updated simply by updating the file (how to deal with animated gifs, tiled sprites, and transparency settings is a bit of an open question, as is the question of how resource indexing would work if a file goes missing), which would also improve source control support. AGS should also use the file system folder structure to organize sprites, scripts, audio files, etc., rather than forcing users to duplicate this organization themselves. All file references should be internal to the project, and relative to the root folder path, so that a project could be shared without breaking any references.

In the short run, I think adding another configuration setting so that everyone gets to have it however they like is not a good idea. One definition of design is to impose constraints on a problem, or in other words, to make decisions. But since Pumaman, CW and the other AGS developers have always been bombarded with different opinions and ideas, all too often the "decision" has been to have it both ways: to make it optional, configurable. In the long run, all these configuration options makes the system hard to learn and understand, difficult to support and debug (if someone has a problem, you first need to figure out exactly which of a million possible configurations they're using), and increasingly impossible to maintain.

So if we need a short-term solution before moving on to a (possible) future redesign, I would urge the devs to just consider the different arguments carefully, and then making a decision, not try to cover every individual preference as a separate setting. In this case, I think the simplest solution would be to store the relative path if the file is within the project folder (or a subfolder), and the absolute path otherwise. Users can address the other use-cases RickJ brings up by linking sub-folders of the project to a repository however they want; AGS shouldn't have to be involved in that. But that's just my opinion.
Title: Re: Changing the way Editor references original files
Post by: cat on Tue 18/08/2015 15:12:41
Quote from: Snarky on Tue 18/08/2015 14:49:35
In this case, I think the simplest solution would be to store the relative path if the file is within the project folder (or a subfolder), and the absolute path otherwise.
This.

Please don't make it overly complicated.
Title: Re: Changing the way Editor references original files
Post by: ChamberOfFear on Thu 20/08/2015 06:25:37
I agree that this has grown out of proportions, way beyond the scope of the original fix. I'd be happy to just apply what was originally going to be applied if we agree that the design-stuff will be re-visited later, more specifically I will be applying this:
Quote from: cat on Mon 17/08/2015 16:33:50
The user adds a new resource.
Scenario 1: It is stored in a subfolder of the game folder -> it is stored as .\sounds\bang.ogg
Scenario 2: It is stored somewhere else -> it is stored as C:\Users\cat\Pictures\logo.bmp

A final question to Crimson Wizard, on GitHub you requested this of me
Quote from: Crimson Wizard
Another thing, would it be possible to run an automatic fixup for all the existing references when project being opened and apply the relative path rule? This will help when importing existing projects with new Editor.
Should I still look into this? Or is it better to drop it for now?
Title: Re: Changing the way Editor references original files
Post by: Crimson Wizard on Thu 20/08/2015 09:33:20
Quote from: ChamberOfFear on Thu 20/08/2015 06:25:37
A final question to Crimson Wizard, on GitHub you requested this of me
Quote from: Crimson Wizard
Another thing, would it be possible to run an automatic fixup for all the existing references when project being opened and apply the relative path rule? This will help when importing existing projects with new Editor.
Should I still look into this? Or is it better to drop it for now?
I think this won't go out of scope, because it does not change the design itself, but enforces these changes you apply.
Yes, I would ask you try to implement this, because otherwise the game project may appear to have one half of paths following old rule, and another half following new rule.
Title: Re: Changing the way Editor references original files
Post by: Monsieur OUXX on Thu 20/08/2015 14:01:39
Regardless of the final implementation and decisions taken here, I think that if the "path" property was made editable in the editor, that would be really cool.


PS: what we do in our project is that every person working on the project can put the sprites wherever they want but then they have to mount a network drive so that the (absolute) sprite import path is always something like Y:\sprites\...
Title: Re: Changing the way Editor references original files
Post by: tzachs on Fri 21/08/2015 15:16:55
I support cat's proposal for the "short term" solution, I think it's simple and intuitive.
I also like Monsieur OUXX suggestion of making the path editable in the editor.

Now for the long term, regarding the cache/source folders duplication:
I think we need to separate this into two different duplications:

1. Duplication of organization: I agree with Snarky, this is completely unnecessary imo. I already have everything organized in my folders, the sprite window should either reflect that (i.e adding a folder in the sprites window will add a folder in windows explorer) or be scratched altogether by just having all the assets accessed from the tree.
The only disadvantage that I see for this is losing the ability to bind to absolute paths, as all assets would need to be inside the project (but this could be solved if it's really needed).

2. Duplication of content (the actual need for both using the source folders and the cache):
- The source folders are needed for development convenience (easily replacing assets with less work to bind them in the editor).
- The cache is needed for deployment- you want all assets embedded in the game file and not out in the open for the users to play with (or at least, we want the developer to have the option to embed the resources in the game file).
I don't see a need for the cache during actual development, though. So I think a possible solution would have the game to take the assets directly from the folders when developing, and adding a deploy button which will embed the resources (or have a checkbox of whether to embed the resources). We'll also be able to add more features for the deployment in the future (like an optional installer, choosing the platforms, etc).
Title: Re: Changing the way Editor references original files
Post by: Monsieur OUXX on Mon 31/08/2015 13:22:00
Quote from: tzachs on Fri 21/08/2015 15:16:55
1. Duplication of organization: I agree with Snarky, this is completely unnecessary imo.
Yeah, "virtual" folders are always a source of confusion in every IDE.


Quote from: tzachs on Fri 21/08/2015 15:16:55
I don't see a need for the cache during actual development, though.
Yeah but sometimes you want to share your game project with other developers, or simply back it up, without worrying that you forgot some of the other "external" folders where you keep your pictures, or without worrying that your backup is in a consistent state. There are countless examples of such features in other fields (for example Adobe software, that lets you zip in all the font files you use in your text editor project alongside it, not just strictly your project file). That's what the cache file does.

HOWEVER a possibility would be to simply tweak the cache file code, that (if I'm not mistaken) already has mechanisms to check if the file is outdated, to simply display a warning in the editor at compile time (in the Errors pane), that would tell you if the file is 1) missing or 2) outdated .