Use symbolic links to share modules between projects

[Monsieur OUXX]

Imagine you have this utility script that you're fond of, let's call it "StringUtility", that has a few generic functions that perform simple stuff on Strings. You use that module in every AGS project, because it's so simple yet so useful.

PROBLEM: you're developing several AGS projects at the same time, and from time to time you want to add a function to StringUtility. But if you modify that script in your first project, then you also have to change it in your other project. Quickly that can become a problem if you don't remember which version is not the most up-to-date.

If only there was a way to mirror changes to that script to every AGS project that uses it!
Well, there is.


=======

FOR WINDOWS VISTA/7/8
1. create that directory structure :

Code: ags

//Note: "<your_path>\Documents\AGS" can be any other folder you wish!
<your_path>\Documents\AGS\MyAGSProject1 //that folder contains your game, i.e. MyAGSProject1.agf
<your_path>\Documents\AGS\MyAGSProject2 //that folder contains your game, i.e. MyAGSProject2.agf
<your_path>\Documents\AGS\COMMON //that folder will contain all the modules you wish to share

2. Move the modules to the COMMON folder

For example, move "StringUtility.ash" and  "StringUtility.asc",
...from <your_path>\Documents\AGS\MyAGSProject1
... to <your_path>\Documents\AGS\COMMON

3. In the game folder, create a symbolic link to the COMMON folder

- Open a command prompt as Administrator (Start-->All programs-->Accessories-->Command prompt-->right-click-->run as administrator)
- Go to your game folder: cd <your_path>\Documents\AGS\MyAGSProject1
- type those two commands :
  mklink StringUtility1.1.asc ..\..\_COMMON\StringUtility.asc
  mklink StringUtility1.1.ash ..\..\_COMMON\StringUtility.ash
- now two files have appeared in your game folder! They look as shortcuts, but they are symbolic links. They act as real files.
- Repeat the same steps in your other game directory.

4. Test

- Open your first game in the AGS editor: <your_path>\Documents\AGS\MyAGSProject1\MyAGSProject1.agf
- Open the module's script files there. Modify them.
- Close AGS and now open your other game (MyAGSProject2.agf). Open the module files.
- They've changed here too! hurray!

5. Automate the process

- It's better if you can re-create those symbolic links in one click, just in case they get removed for some reason.
- To do so : create a .bat file in your game folder (e.g. <your_path>\Documents\AGS\MyAGSProject1\create_links.bat )
- Put all the commands you've typed in the command prompt into it.
- BUT precede them with cd %~dp0. That's because you'll need to run this batch file as Administrator, but unfortunately when you do so Windows makes the .bat file start in some random system folder. Using %~dp0 gives you the real folder where it got executed in the first place (i.e. : your game folder)
- I recommend you add a timeout at the end to be able to see if everything went fine. Note: "pause" won't work when running a bat file as admin. That's a safety measure from Windows, to forbid crazy batch files that don't require human interaction from getting blocked and preventing machines to be remotely shut down.

Code: ags

@echo off 
cls

REM the line below changes directory to the directory where the batch was run
pushd %~dp0
REM the line below makes sure we're not in C:\windows\system32 anymore
if "%CD:~0,2%"=="C:" goto wrongdrive

mklink StringUtility1.1.asc ..\..\_COMMON\StringUtility.asc
mklink StringUtility1.1.ash ..\..\_COMMON\StringUtility.ash

goto ok

:wrongdrive
	echo === Wrong drive! ===
        REM we use timeout instead of pause because pause doesn't work as Administrator
	timeout /t 20
	goto end

:ok
	timeout /t 20
	goto end

:end

6. If you want to do it like a Boss

I suggest you create two more .bat files :
- One that removes the symbolic links

Code: ags

@echo off
cls

pushd %~dp0
if "%CD:~0,2%"=="C:" goto wrongdrive

rem delete all symbolic links  (/AL) in this folder. /P is for confirming each file
del /AL /P *.*

goto ok

:wrongdrive
	echo === Wrong drive! ===
	timeout /t 20
	goto end

:ok
	timeout /t 20
	goto end

:end

- One that duplicates the whole game folder to some new output folder, and also copies over the real module files from "COMMON". This way, all you have to do is to zip up that output folder to distribute your full game code.
(note: this one doesn't need to be run as administrator)

Code: ags

@echo off
cls

REM delete Distrib folder and everything it contains
rd /s/q ..\Distrib
REM recreate it
mkdir ..\Distrib
REM copy contents (/s for subfolders)
xcopy *.* ..\Distrib /s

timeout /t 20

[tzachs]

Ooh, neat trick!

This can work nicely as a plugin:
Right clicking a script will give you the option to make it into a "common" script which will create the link to a predefined folder.
Right clicking a "common" script will give you the option to make into a normal script which will destroy the link (cause sometimes you want to break out of the mold and have some special hack in your script).
And also an option to "Import common script" which will scan the predefined folder and give you all the common scripts that are located within to choose from.
And finally (optional?) a deploy button that will copy all the used common scripts with the game files to the deployed folder.

[Monsieur OUXX]

Ooh, neat trick! This can work nicely as a plugin:(...)

I was intending on doing it as an Editor plugin but it turned out I'm too lazy for that

[Calin Leafshade]

I've been considering a NuGet like system for AGS for a while. I think it would be pretty simple to do. That would negate the need for symbolic links and such.

The main problem with that is the lack of forward declarations so the ordering of scripts is important.

[Monsieur OUXX]

a NuGet like system for AGS would negate the need for symbolic links and such.

I'm not an expert of NuGet. How would it replace symbolic links? The point here is not only to share existing modules, but also that if you modify a module in one of your AGS projects, then it gets updated in all the projects. Does NuGet enable that?

[Monsieur OUXX]

UPDATE FOR WINDOWS XP

    The tips I gave in the first post work for Windows 7. Of course, as soon as it started working, fate made my computer melt down and I had to revert to my "emergency" Windows XP computer. Windows XP does not fully support symbolic links but it's not too hard to fix.

1. How to delete symbolic links formerly created with Windows 7 in Windows XP

This is only if you're the same situation as me, after you've created links in Windows 7 and want to switch to Windows XP. Those links won't work.
The command remains the same. Officially, Windows XP does not know the "L" attribute, but it still works, and it does NOT delete regular files by mistake. So it's all good.

Code: ags

del /AL *.*

2. How to create Windows XP symbolic links

Code: ags

fsutil hardlink create <link> <real file>

e.g. : 
fsutil hardlink create module.ash "..\_COMMON\module.ash"

3. How to list existing Windows XP hard links

That's if you want to check in your project folders which files are really there and which files are actually symbolic links. KEep in mind that you don't really need that. Simply delete the symbolic links as explained before, and you'll have a clear vision of your folder.
There is no built-in function in Windows XP. You need to use a third-party tool : Hlscan.exe here : http://www.microsoft.com/en-us/download/confirmation.aspx?id=19815
i. Install it
ii. Run it :

Code: ags

REM This command lists all symbolic liks formerly created with 'fsutil hardlink create'
"C:\Program Files\Resource Kit\Hlscan.exe" /file *.*

[monkey0506]

This is kind of what I was doing with the Libraries editor plugin (except just keeping a central store of modules and copying them, not soft/hard linking them), but nobody really seemed interested so I just kind of gave up on it.

Also, just a note but you're using two different methods here for linking. On Windows 7 you're soft linking files (which sets the 'L' attribute) whereas on Windows XP you're hard linking files (which does not). The differences between them may have minor or insignificant side-effects for what you're doing here, but of particular note is the fact that hard links won't have the L attribute set (unless, perhaps, you've hard linked a soft link, which I'm not entirely sure is possible) so your method for deleting links won't always work and also if the original file is deleted then your soft links will be broken.

To sum it up, a soft symbolic link is like a file pointer, and the file only exists in one place; a hard symbolic link is an actual file, it can just "exist" in more than one place on the disk without consuming extra memory (e.g., appearing in separate directories).

I'd recommend switching to hard links on Windows 7 by passing the "/H" flag to the mklink utility. If you want to delete them, on Windows 7 and XP you can delete a hard link just as you would a normal file (because they are, in fact, normal files). If you want to break the link then you can copy the files (destination or source, they're the same), delete them (from the destination, not the source), and then move the copies back to the destination. This will create separate copies that will no longer modify the source files.

[Monsieur OUXX]

Thanks for the clarifications. I thought I was doing hard links all the time.

[monkey0506]

Yeah, there are some nuances with soft vs hard links and the mklink utility that probably aren't immediately apparent, but in general hard links are what you would want for this type of thing (and for that, the mklink tool needs the "/H" switch ). For what it's worth, you also brought to my attention the fact that Win XP doesn't have the mklink utility, so I put in a bugfix for the latest editor for that case as well.

And with the help of Reflector (the .NET disassembly tool), I've managed to recreate and update the source for the Libraries module plugin into a mostly functional state. I have a couple of things to do to get it working with the latest editor and then I'll gladly upload the source to GitHub. And if you'd like it would probably be pretty easy to tweak it to use hard links instead of copying the files (the way I'm doing it right now is a huge mess anyway, honestly...what was I thinking 4 years ago?).

[Monsieur OUXX]

I have a couple of things to do to get it working with the latest editor and then I'll gladly upload the source to GitHub. And if you'd like it would probably be pretty easy to tweak it to use hard links instead of copying the files

That would be cool. Not only for me but for everyone. For now, the solutions I've provided above work like a charm and are not risky for the end-user.

[monkey0506]

As an update on this, I completely abandoned the old source code. Everything I was doing was such a mess, it was just easier to start from scratch.

I did have one major face-palm moment though. As I was doing in the prior version of the plugin, I attempted to write some data to the LocalApplicationData folder (SYSTEM_DIR/Users/AppData/Local), except this time I was trying to create a hard link. The problem I ran into was one I was already well aware of but had simply failed to accommodate. Hard links cannot exist across physical disks. My AGS files are on a separate physical disk from my Windows installation, so I can't create hard links there.

I was able to add a dialog to select an alternate folder for caching (hard linking) the library scripts, and I write out some meta-data to the system folder instead (this is also necessary to keep track of the extra module info like author and description).

The new version is almost functional. So stay posted for that.

[Monsieur OUXX]

Come on. Monkey. Come to daddy.

[RickJ]

Very interesting. Way long time ago VAX/VMS had something called logical name assignments. A very useful feature that seems to be missing from mklink is the ability to assign one name to multiple targets.  VMS would search through the list until it found a target that exists. 

Anyway I found a little utility from nirsoft
http://www.nirsoft.net/utils/ntfs_links_view.html

He seems to have quite a few useful little things.  I had used his shell command a while back and had a good experience with it.
http://www.nirsoft.net/utils/nircmd.html

[monkey0506]

Sorry, I may have been overthinking certain things on keeping module data in-sync. I will try to take a look at this next week during spring break.

[Monsieur OUXX]

Yeyyy (for both of you)

[monkey0506]

I haven't looked at the code in a while and it was in an incomplete transitory state... so I'm not entirely confident as to the overall status of this. In that respect, I hereby introduce: the Libraries plugin v2 ALPHA 1 (with source).

This relies on hard-linking the script files and it's not feature-complete, but it should get the general idea across. Sorry I didn't get to devote the time to this that I meant to, but I wanted to actually release something.

[Monsieur OUXX]

I'll have a look at that asap :-) Thanks for the effort!