Creating hard links from the editor (to reduce copying on build)

[monkey0506]

This is an extension of a previous discussion brought up in my thread about building for Linux.

Through research as well as trial-and-error, I've attempted to come up with the most thorough, complete, and yes, even portable solution to create hard links from within the editor code.

Why do we need this? Particularly now that the build process is being split up to allow targeting multiple platforms from directly within the editor, the issue of copying files becomes particularly relevant. If your game's data files amount to several hundred megabytes or even reach into the gigabytes, then copying all of those data files into the appropriate build folders for Windows, Linux, etc. would become utterly obscene. Using hard links means that the actual data files can exist in exactly one place on your hard drive, but when you zip up your Windows or Linux folder, it grabs those files from wherever they're at instead of forcing you to have multiple copies of the files (think of it like a shortcut on your desktop, except you can also treat it like a normal file for zipping, etc.).

I wanted to post this here just to draw particular attention to it. Here's what I've come up with:

Code: csharp

        public static bool CreateHardLink(string destFileName, string sourceFileName, bool overwrite)
        {
            if (File.Exists(destFileName))
            {
                if (overwrite) File.Delete(destFileName);
                else return false;
            }
            ProcessStartInfo si = new ProcessStartInfo("cmd.exe");
            si.RedirectStandardInput = false;
            si.RedirectStandardOutput = false;
            si.UseShellExecute = false;
            si.Arguments = string.Format("/c mklink /h {0} {1}", destFileName, sourceFileName);
            si.CreateNoWindow = true;
            si.WindowStyle = ProcessWindowStyle.Hidden;
            if (IsMonoRunning())
            {
                si.FileName = "ln";
                si.Arguments = string.Format("{0} {1}", sourceFileName, destFileName);
            }
            bool result = (Process.Start(si) != null);
            if (result)
            {
                // by default the new hard link will be accessible to the current user only
                // instead, we'll change it to be accessible to the entire "Users" group
                FileSecurity fsec = File.GetAccessControl(destFileName);
                fsec.AddAccessRule
                (
                    new FileSystemAccessRule
                    (
                        new SecurityIdentifier(WellKnownSidType.BuiltinUsersSid, null),
                        FileSystemRights.Modify,
                        AccessControlType.Allow
                    )
                );
                File.SetAccessControl(destFileName, fsec);
            }
            return result;
        }

This supports relative or absolute paths, and should work with Mono (although I don't think there's a build of the editor that fully works with Mono to be able to test this on, it doesn't use any native code and specifies the appropriate linker file for Linux). It also attempts to make sure that appropriate security permissions are set on the hard link itself.

If I've overlooked anything else please let me know.

[Alberth]

You seem to build a single string of the arguments without doing shell escaping at all.

Wouldn't

Code: ags

destFileName = "bla q\"bla";
sourceFileName = "'stu\\uf";

fail horribly? At the very least you no longer have 2 filename arguments in text like

Code: ags

/c mklink /h bla q"bla 'stu\uf

(this has 3 space-separated arguments).

[monkey0506]

Yeah, and I was thinking about that as well. I've been having some other trouble with it (which I don't think is related to this function itself) with files not being copied in time for me to set the permissions, and so I have quoted the parameters now, but it would probably make sense to strip or disallow any double-quotes from those parameters as well.

[Alberth]

Unix shells have a long list of awkward characters. You may want to do proper escaping of them. (Not using a command-line for executing the operation would be the preferred solution, but I guess that's blocked by the belief that a language should be cross-platform portable?)

As for your other problem, "Process.Start" only seems to say things about starting the process. It does not say anything about finishing the operation before it returns from the call (http://msdn.microsoft.com/en-us/library/system.diagnostics.process.start%28v=vs.110%29.aspx at least). Maybe it forks the command as asynchronous operation to be executed in parallel?

[monkey0506]

Well even calling Process.WaitForExit isn't solving it, which is why I think that bit probably isn't related.

And I can agree that avoiding command-line processing would be ideal, but the .NET Framework doesn't have a method for creating hard links, and I still hold that a sub-optimal method is preferable over copying and duplicating huge amounts of data for every single platform. If you can devise a better method that would be framework-agnostic (runs on MS .NET and Mono), then I'm all ears.

[monkey0506]

I think I sorted out the particular file that was causing problems, but here's a modified version of the function. This seems to be working as expected, and should disallow any "bad" file names.

Code: csharp

        public static bool CreateHardLink(string destFileName, string sourceFileName, bool overwrite)
        {
            if (File.Exists(destFileName))
            {
                if (overwrite) File.Delete(destFileName);
                else return false;
            }
            List<char> invalidFileNameChars = new List<char>(Path.GetInvalidFileNameChars());
            if (Path.GetFileName(destFileName).IndexOfAny(invalidFileNameChars.ToArray()) != -1)
            {
                throw new ArgumentException("Cannot create hard link! Invalid destination file name.");
            }
            if (Path.GetFileName(sourceFileName).IndexOfAny(invalidFileNameChars.ToArray()) != -1)
            {
                throw new ArgumentException("Cannot create hard link! Invalid source file name.");
            }
            if (!File.Exists(sourceFileName))
            {
                throw new FileNotFoundException("Cannot create hard link! Source file does not exist.");
            }
            ProcessStartInfo si = new ProcessStartInfo("cmd.exe");
            si.RedirectStandardInput = false;
            si.RedirectStandardOutput = false;
            si.RedirectStandardError = false;
            si.UseShellExecute = false;
            si.Arguments = string.Format("/c mklink /h \"{0}\" \"{1}\"", destFileName, sourceFileName);
            si.CreateNoWindow = true;
            si.WindowStyle = ProcessWindowStyle.Hidden;
            if (IsMonoRunning())
            {
                si.FileName = "ln";
                si.Arguments = string.Format("\"{0}\" \"{1}\"", sourceFileName, destFileName);
            }
            Process process = Process.Start(si);
            bool result = (process != null);
            if (result)
            {
                process.EnableRaisingEvents = true;
                process.WaitForExit();
                if (process.ExitCode != 0) return false;
                process.Close();
                // by default the new hard link will be accessible to the current user only
                // instead, we'll change it to be accessible to the entire "Users" group
                FileSecurity fsec = File.GetAccessControl(destFileName);
                fsec.AddAccessRule
                (
                    new FileSystemAccessRule
                    (
                        new SecurityIdentifier(WellKnownSidType.BuiltinUsersSid, null),
                        FileSystemRights.Modify,
                        AccessControlType.Allow
                    )
                );
                File.SetAccessControl(destFileName, fsec);
            }
            return result;
        }