Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows' path limits can be exceeded #20053

Closed
jstedfast opened this issue Feb 6, 2024 · 10 comments · Fixed by #20298
Closed

Windows' path limits can be exceeded #20053

jstedfast opened this issue Feb 6, 2024 · 10 comments · Fixed by #20298
Labels
bug If an issue is a bug or a pull request a bug fix windows-only The issue only occur on Windows
Milestone

Comments

@jstedfast
Copy link
Member

jstedfast commented Feb 6, 2024

Background

Windows has a file path limit of 260 characters and a directory path limit of 248 characters even though, technically, Windows can handle much longer path lengths than that.

The work-around is to prefix paths that would normally exceed this limitation with a magical string (@"\\?\").

For example, the following code will throw an exception:

File.Exists (@"C:\Users\jestedfa\AppData\Local\Xamarin\iOS\Archives\2024-01-19\XamariniOSAppWithLongFilePaths 1-19-24 12.27 PM.xcarchive\Products\Applications\XamariniOSAppWithLongFilePaths.app\Frameworks\BlazerMaterialWeb.framework\wwwroot\_content\BlazerMaterialWeb.bundled\roboto\fonts\roboto-v30-cyrillic_cyrillic-ext_greek_greek-ext_latin_latin-ext_vietnamese-100.woff2");

However, if you prefix the path string with @"\\?\", things will work as expected.

e.g.:

File.Exists (@"\\?\C:\Users\jestedfa\AppData\Local\Xamarin\iOS\Archives\2024-01-19\XamariniOSAppWithLongFilePaths 1-19-24 12.27 PM.xcarchive\Products\Applications\XamariniOSAppWithLongFilePaths.app\Frameworks\BlazerMaterialWeb.framework\wwwroot\_content\BlazerMaterialWeb.bundled\roboto\fonts\roboto-v30-cyrillic_cyrillic-ext_greek_greek-ext_latin_latin-ext_vietnamese-100.woff2");

(or if you were using a string variable: path = @"\\?\" + path;)

The bug in the VIsual Studio-side of things can be found here: https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1935561

Steps to Reproduce

  1. Create a Xamarin.iOS project on Windows that will end up bundling resource(s) with really long relative paths. For example, you can use a library like BlazorMaterialWeb.Bundled that contains a file with a long name: wwwroot\_content\BlazorMaterialWeb.Bundled\roboto\fonts\roboto-v30-cyrillic_cyrillic-ext_greek_greek-ext_latin_latin-ext_vietnamese-100.woff2
  2. Build & Archive the project using Visual Studio 17.10 Preview1 (older versions will also fail, but due to this same bug in Visual Studio code)
  3. Attempt to publish the archive via Ad-Hoc publishing workflow in Visual Studio 17.10 Preview1 (older versions will also fail but due to bugs in Visual Studio).

Expected Behavior

No errors caused by file or directory paths exceeding the Windows MAX_PATH length.

Actual Behavior

Two example exceptions that can happen:

Xamarin.VisualStudio.Progress.ProgressReportService Information: 0 : Cannot create an IOS archive 'MauiApp6'. Process cannot be executed on XMA server.
The "Unzip" task failed unexpectedly.
System.IO.PathTooLongException: The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.Directory.InternalCreateDirectory(String fullPath, String path, Object dirSecurityObj, Boolean checkHost)
   at System.IO.Directory.InternalCreateDirectoryHelper(String path, Boolean checkHost)
   at Xamarin.MacDev.CompressionHelper.TryDecompressUsingSystemIOCompression(TaskLoggingHelper log, String zip, String resource, String decompressionDir) in /Users/builder/azdo/_work/1/s/xamarin-macios/msbuild/Xamarin.MacDev.Tasks/Decompress.cs:line 200
   at Xamarin.MacDev.CompressionHelper.TryDecompress(TaskLoggingHelper log, String zip, String resource, String decompressionDir, List`1 createdFiles, String& decompressedResource) in /Users/builder/azdo/_work/1/s/xamarin-macios/msbuild/Xamarin.MacDev.Tasks/Decompress.cs:line 104
   at Xamarin.MacDev.Tasks.Unzip.ExecuteLocally() in /Users/builder/azdo/_work/1/s/xamarin-macios/msbuild/Xamarin.MacDev.Tasks/Tasks/Unzip.cs:line 61
   at Xamarin.MacDev.Tasks.Unzip.Execute() in /Users/builder/azdo/_work/1/s/xamarin-macios/msbuild/Xamarin.MacDev.Tasks/Tasks/Unzip.cs:line 43
   at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
   at Microsoft.Build.BackEnd.TaskBuilder.<ExecuteInstantiatedTask>d__26.MoveNext()

and

The "Unzip" task failed unexpectedly.
System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\Users\admin\AppData\Local\Xamarin\iOS\Archives\wwwroot\_content\BlazorMaterialWeb.Bundled\roboto\fonts\roboto-v30-cyrillic_cyrillic-ext_greek_greek-ext_latin_latin-ext_vietnamese-100.woff2\2024-01-19\MauiApp21 1-19-24 4.53 PM.xcarchive\Products\Applications\'.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.Directory.InternalCreateDirectory(String fullPath, String path, Object dirSecurityObj, Boolean checkHost)
   at System.IO.Directory.InternalCreateDirectoryHelper(String path, Boolean checkHost)
   at Xamarin.MacDev.CompressionHelper.TryDecompressUsingSystemIOCompression(TaskLoggingHelper log, String zip, String resource, String decompressionDir) in /Users/builder/azdo/_work/1/s/xamarin-macios/msbuild/Xamarin.MacDev.Tasks/Decompress.cs:line 200
   at Xamarin.MacDev.CompressionHelper.TryDecompress(TaskLoggingHelper log, String zip, String resource, String decompressionDir, List`1 createdFiles, String& decompressedResource) in /Users/builder/azdo/_work/1/s/xamarin-macios/msbuild/Xamarin.MacDev.Tasks/Decompress.cs:line 104
   at Xamarin.MacDev.Tasks.Unzip.ExecuteLocally() in /Users/builder/azdo/_work/1/s/xamarin-macios/msbuild/Xamarin.MacDev.Tasks/Tasks/Unzip.cs:line 61
   at Xamarin.MacDev.Tasks.Unzip.Execute() in /Users/builder/azdo/_work/1/s/xamarin-macios/msbuild/Xamarin.MacDev.Tasks/Tasks/Unzip.cs:line 43
   at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
   at Microsoft.Build.BackEnd.TaskBuilder.<ExecuteInstantiatedTask>d__26.MoveNext()

Environment

Windows 11 Enterprise 22H2

Build Logs

Example Project (If Possible)

FileSystem.cs

I started working on a solution you guys could use based on work I did in Visual Studio, but it's started becoming more work than I had hoped and I wasn't sure (without spending more time than I'd like) to figure out which Tasks, at minimum, would need to be updated to use my FileSystem.cs file that I've attached to get you guys started on this.

I'm not sure if you'll want to just make all File/Directory calls go through my FileSystem layer or if you'll want to just do it for the code that can run on the Windows side.

using System;
using System.IO;

namespace Xamarin.Utils {
	static class FileSystem {
		static readonly char[] Win32DirectorySeparators = new char [] { '/', '\\' };
		const string Win32LongPathPrefix = @"\\?\";
		const int WIN32_MAX_DIR_PATH = 248;
		const int WIN32_MAX_PATH = 260;

		// Note: We can't use Path.GetDirectoryName() because it will throw PathTooLongException if the directory path is longer than 248 characters.
		static string GetWin32DirectoryName (string path)
		{
			int slash = path.LastIndexOfAny (Win32DirectorySeparators);

			return slash == -1 ? path : path.Substring (0, slash);
		}

		static string GetWin32LongFilePath (string path)
		{
			// By default, a file path is limited to MAX_PATH characters. To extend this limit to 32,767 wide characters, prepend "\\?\" to the path.
			if (path.StartsWith (Win32LongPathPrefix, StringComparison.Ordinal) || (path.Length < WIN32_MAX_PATH && GetWin32DirectoryName (path).Length < WIN32_MAX_DIR_PATH))
				return path;

			return Win32LongPathPrefix + path;
		}

		static string GetWin32LongDirectoryPath (string path)
		{
			// By default, a directory path is limited to MAX_DIR_PATH characters. To extend this limit to 32,767 wide characters, prepend "\\?\" to the path.
			if (path.Length < WIN32_MAX_DIR_PATH || path.StartsWith (Win32LongPathPrefix, StringComparison.Ordinal))
				return path;

			return Win32LongPathPrefix + path;
		}

		public static string GetFilePath (string path)
		{
			if (Environment.OSVersion.Platform == PlatformID.Win32NT)
				return GetWin32LongFilePath (path);

			return path;
		}

		public static string GetDirectoryPath (string path)
		{
			if (Environment.OSVersion.Platform == PlatformID.Win32NT)
				return GetWin32LongDirectoryPath (path);

			return path;
		}

		public static string GetDirectoryName (string path)
		{
			return Path.GetDirectoryName (GetDirectoryPath (path));
		}

		public static void CreateDirectory (string path)
		{
			Directory.CreateDirectory (GetDirectoryPath (path));
		}

		public static void DeleteDirectory (string path, bool recursive = false)
		{
			Directory.Delete (GetDirectoryPath (path), recursive);
		}

		public static bool DirectoryExists (string path)
		{
			return Directory.Exists (GetDirectoryPath (path));
		}

		public static string[] GetFiles (string path)
		{
			return Directory.GetFiles (GetDirectoryPath (path));
		}

		public static string[] GetFiles (string path, string searchPattern)
		{
			return Directory.GetFiles (GetDirectoryPath (path), searchPattern);
		}

		public static string[] GetFiles (string path, string searchPattern, SearchOption searchOption)
		{
			return Directory.GetFiles (GetDirectoryPath (path), searchPattern, searchOption);
		}

		public static bool FileExists (string path)
		{
			return File.Exists (GetFilePath (path));
		}

		public static void CopyFile (string sourceFileName, string destFileName, bool overwrite = false)
		{
			File.Copy (GetFilePath (sourceFileName), GetFilePath (destFileName), overwrite);
		}

		public static Stream CreateFile (string path)
		{
			return File.Create (GetFilePath (path));
		}

		public static void DeleteFile (string path)
		{
			File.Delete (GetFilePath (path));
		}

		public static DateTime GetLastWriteTimeUtc (string path)
		{
			return File.GetLastWriteTimeUtc (GetFilePath (path));
		}

		public static Stream OpenRead (string path)
		{
			return File.OpenRead (GetFilePath (path));
		}

		public static Stream OpenWrite (string path)
		{
			return File.OpenWrite (GetFilePath (path));
		}

		public static byte[] ReadAllBytes (string path)
		{
			return File.ReadAllBytes (GetFilePath (path));
		}

		public static string ReadAllText (string path)
		{
			return File.ReadAllText (GetFilePath (path));
		}

		public static void WriteAllBytes (string path, byte[] bytes)
		{
			File.WriteAllBytes (GetFilePath (path), bytes);
		}

		public static void WriteAllText (string path, string contents)
		{
			File.WriteAllText (GetFilePath (path), contents);
		}
	}
}
@jstedfast
Copy link
Member Author

BTW, don't forget to update code like PDictionary.FromFile(path) to do PDictionary.FromFile(FileSystem.GetFilePath(path))
and plist.Save(path) to do plist.Save(FileSystem.GetFilePath(path))

@rolfbjarne
Copy link
Member

If we could get a test project we can reproduce this with from the command line, we can easily create a unit test to verify that it actually works (and doesn't regress, which is even more important). Would that be something you have or could create? I can handle the actual code fixing afterwards.

@rolfbjarne rolfbjarne added this to the Future milestone Feb 7, 2024
@rolfbjarne rolfbjarne added bug If an issue is a bug or a pull request a bug fix windows-only The issue only occur on Windows need-info Waiting for more information before the bug can be investigated labels Feb 7, 2024
Copy link
Contributor

Hi @jstedfast. We have added the "need-info" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

@jstedfast
Copy link
Member Author

I have not had time to verify that the attached project reproduces the issue, but it's the project I had been using to repro the VS-side of things and it is the project I asked CSI to use when testing and they are ones hitting this issue (I could have sworn this worked once I made the VS-side fixes, but apparent not if CSI is using this test project).

XamariniOSAppWithLongFilePaths.zip

If this doesn't repro, then I don't have a good test case handy.

@jstedfast jstedfast removed the need-info Waiting for more information before the bug can be investigated label Feb 7, 2024
@jstedfast
Copy link
Member Author

Oh, actually, if the above project doesn't repro, you might be able to get it to repro by making the default Archive path something longer. I think CSI mentioned they had done that (as that was the original way that they had been reproing this kind of issue).

@rolfbjarne rolfbjarne added the need-attention An issue requires our attention/response label Feb 8, 2024
@rolfbjarne
Copy link
Member

@jstedfast do you have a binlog that shows the failure?

I'm having a hard time reproducing this, because my devbox is happy to create long paths even if long path support is disabled in the registry :/

@jstedfast
Copy link
Member Author

@rolfbjarne I'll ask for CSI to provide one. When I fixed this in VS, I'm pretty sure it Just Worked(tm) for me after that - so I was surprised when CSI came back to me with this issue.

@rolfbjarne rolfbjarne added need-info Waiting for more information before the bug can be investigated and removed need-attention An issue requires our attention/response labels Feb 15, 2024
Copy link
Contributor

Hi @jstedfast. We have added the "need-info" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

Copy link
Contributor

Hi @jstedfast. Due to inactivity, we will be closing this issue. Please feel free to re-open this issue if the issue persists. For enhanced visibility, if over 7 days have passed, please open a new issue and link this issue there. Thank you.

@rolfbjarne
Copy link
Member

Reopening, info was provided in the devops issue.

@rolfbjarne rolfbjarne reopened this Feb 23, 2024
@rolfbjarne rolfbjarne added need-attention An issue requires our attention/response and removed need-info Waiting for more information before the bug can be investigated labels Feb 23, 2024
@rolfbjarne rolfbjarne added this to the Future milestone Feb 26, 2024
@rolfbjarne rolfbjarne removed the need-attention An issue requires our attention/response label Feb 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug If an issue is a bug or a pull request a bug fix windows-only The issue only occur on Windows
Projects
None yet
2 participants