Improved StorageFile APIs #8
Comments
|
For applications that are already compatible with sandboxing (i.e. only read from %APPDATA% or %PROGRAMDATA%, and only access files explicitly chosen by a user) I don't think a continuous reminder is warranted. This could perhaps be controlled via some sandbox capabilities options that are chosen at development time (similar to the macOS model). |
|
A good test of this feature would be, I can create an SQLite database in the user's documents folder. I shouldn't need permission to do this if it's a new file. This is such a common scenario that the sandbox and the politics around it have never tried to solve for. An app should never need "broadFileAccess" but they should also make it so users can easily open documents and see their saved file right there. |
Why not create in AppData folder itself? A lot of users have complained apps creating random folders in their library folders like Documents, Music etc. introduces clutter to their workspaces and they have more trouble in finding their Work folders. Also, this kind of behavior from developers is what prompted MS to limit storage access in first place. |
|
It’s really great that this is being looked into, and that it’s the first issue here. UWP file performance and permissions is the thing I’ve most wanted to see an improvement in since 2015! I imagine that any permissions changes will be difficult to deliver via NuGet, so hopefully something can happen on this as soon as possible in order to ensure we don’t have to wait too long before the users have the necessary OS updates. Below I’ve put together a collection of scenarios related to file system performance and permissions that are problematic in UWP. These are things I have encountered while building my photo-viewing app. (Maybe it’s a bit selfish of me to concentrate on these problems, but hopefully they cover a wide range of use cases). If this post is too long, feel free to move it to a separate issue. Also, I am very happy to discuss any of these issues/ideas further. The scenarios themselves come first, followed by the problems with them in UWP and some possible solutions. I hope that this project will make time to think more creatively about how to deal with some of the file permissions issues. Some of my suggestions below are just initial ideas, so don’t judge them too harshly. It’s such an important subject that I think it’s worth spending a lot of time thinking about. The scenarios0. The user does not necessarily keep all their images in their Pictures libraryI put this in as point 0 as it influences all the others. A key principle of my app is to allow users to organise their photos how they like and not to force them into any patterns. Therefore the app cannot assume that all the user’s images are in their Pictures library, and indeed it’s designed for viewing all images, even for web designers etc who are probably not keeping images in there. Also, I’m going to assume that broadFileSystem access is not used, because that is perhaps too unrestricted and also the current user experience for it is poor. 1. User selects a folder containing a large number of files (can include sub-folders) to view in the appMy app has quite a simple concept of allowing users to select files they want it to display. They can do this by selecting a folder, optionally including subfolders, or selecting individual files etc. They can make multiple selections. It is not uncommon for a folder selected to contain 10,000+ image files. The app needs to maintain a list of the selected files (including the order they were selected in etc), therefore it needs to be able to list files in folders, and the quicker and less resource-intensive this is, the better. Obviously the app needs to be able to access the contents of these files alter on, but at this stage a path and file name is all that’s needed. 2. The app needs to be able to watch for files added/removed/modified in the folders that the user has selectedMy app has the option to auto-refresh the selected files to match the contents of the folders that the user selected to view. This can include modifications to files as well as files added/removed/renamed. 3. The app makes heavy use of StorageFile.GetBasicPropertiesAsync StorageItemContentProperties. RetrievePropertiesAsync to get properties such as DateTaken and OrientationThis should ideally be fast. Also, it expects to be able to read properties simultaneously. 4. The user can select files/folders via drag and drop, and expects to be able to edit these filesThis is self-explanatory. 5. The app offers a ‘Duplicate file’ functionThis is self-explanatory. There is a button in the app that allows the user to duplicate the file they are currently viewing in the app. (The file is one of those selected in scenario 1). This should place a copy of the file in the same directory as that file, with the name suffixed by ‘ Copy’ or similar. It should be a one-click operation as it’s a convenience function for the user (e.g. to make a copy to edit). 6. The app offers a button to load the other files in the same folder as the file that the user is currently viewingThe user may have selected an individual file to view in scenario 1. The app would like to show a button that would load the remaining files from the same folder in this case, ideally via a 1-click operation. 7. The app maintains a ‘quick access’ or ‘favourites’ list of files. Generally, clicking on a file in this list should also load the neighbors, as launching from File Explorer does.Users who make heavy use of this list may add hundreds or even thousands of files to it. 8. The app would like to offer a feature that allows users to save their session to a file, e.g. to create a slideshow that they can bring up straight awaySuch as slideshow may consist of thousands of images, which were originally selected in multiple different ways. The saved data file would store the file paths of these images, and when the data file is loaded it needs to be able to load the associated images. 9. The user can load files by double-clicking on a file in file explorer. This should load the neighboring files.The user needs to be able to do the same things with these files that they can do with files selected another way. An important feature of this is that the neighboring files are ordered in the same way as in File Explorer. The neighboring files list should be resilient to problems with the file system indexer. The challenges faced in a UWP appScenario 1The challenge here is simply that listing the StorageItems in a StorageFolder is incredibly slow and resource intensive compared to using the standard .Net/Win32 API to list the file paths. A quick test showed that in .Net it takes about 0.003ms per file, whereas with UWP StorageItems it takes about 2ms per file, which is around 700 times slower. Also, looking at the Runtime Broker, UWP uses about 70kB memory per file, which is a huge cost when loading 10,000+ files, whereas .Net uses around 0.1kB per file (that’s a very rough estimate). Scenario 2UWP has the StorageFileQueryResult.ContentsChanged method for watching for file system changes. However, it has absolutely no configuration options for what sort of changes to watch for, so it will fire on any change, even something like last access time I think, and include subfolders. (I agree that if you have no configuration options that firing on any change is the best option). It also does not tell you what changed. Combined with the speed issues in scenario 2, this makes for a doubly bad experience. If you look at the .Net docs of file system watching, they give an example of watching for changes in “C:\”, which made me laugh. In UWP, assuming that you managed to get permission for this folder, in order to determine what changed, you would first have to enumerate all the contents, which would literally take several days, then on a change you would have to enumerate all the contents again, taking several days, and in fact several weeks if you also want to check the date modified to detect file modifications, and look at the difference. Also, at one stage, I tried adding a separate watching for each of 600 sub-folders that had been added to the app (in order to avoid having to query all the files for changes) and basically it ground to a halt – it seems to create a new thread for each watcher which is not necessarily very efficient. Scenario 3I added this because I’m not aware of a way to do this easily in .Net without writing interop code. Also, for any alternative to StorageFile in UWP, it needs to be possible to get these properties ideally without the overhead of creating a StorageFile. Also, there are some issues with these in UWP. Firstly, if you call any of the Get..PropertiesAsync methods apart from RetrievePropertiesAsync simultaneously on the same StorageFile instance, they will fail, even though they are read-only as far as I can see so have no reason to. Secondly, fetching System.Size and System.DateModified using RetrievePropertiesAsync after the first time does not return an up-to-date result. Thirdly, there can be issues with scenario 9, which I’ll mention later. Finally, using SorageFile.GetBasicPropertiesAsync to get the file modified data is incredibly slow compared to .Net. in .Net it takes about 0.05ms per file. In UWP, it takes about 5ms per file, which is 100 times slower, and also adds about an extra 60kB per file (if you keep the StorageFile), compared to about 0.05kB per file in .Net. Scenario 4Currently in UWP if a user drags and drop a file/folder from File Explorer into the app, the file/folder is read-only, so can’t be edited, unless the app has access to the file path via some other means. Scenario 5This duplicate button is not possible in UWP if the user opened the file individually or by double-clicking in File Explorer, unless the file is in the Pictures library (assuming the Pictures library capability). Worse, the app cannot even open the save file dialog to the current folder, since there is no way to set the starting folder of the save file dialog. Scenario 6This is a similar problem to scenario 5. If the user opened the file individually this is impossible, unless the file is in the Pictures library. Scenario 7The only general way to save access to files is via the FutureAccessList. However, this is beset with problems. It has quite a small limit of 1000 files which is easily exceeded. The developer has to write code to handle the case when this happens, which is hampered by the fact that any of the APIs surround the FutureAccessList are quite slow, typically taking several milliseconds, and hence such a function has to be carefully debounced so as not to impact performance of the app. Furthermore, the app has to maintain a list of the tokens it is using and ensure this is in sync with the list, which is awkward. It would be easier if the list could be accessed via file path, with a use count. This could be stored in the metadata currently but for some reason in order to obtain the metadata for a token you have to enumerate the whole list. I find that whenever I add a feature to my app that requires this list, it takes me a couple of days to get the logic right, and even then it’s full of annoying compromises. Compare this with a .Net app where you can simply forget about the whole thing and spend zero days on it. Scenario 8Due to the fact that the slideshow list would be stored in an external file which could then be deleted without the knowledge of the app, and also because this could exceed 1000 files, the FutureAccessList cannot be used for this purpose, so it is impossible unless the files are in the Pictures library. Scenario 9There are currently various issues with the neighboring files query, which I believe is related to the way it uses the file system indexer. Firstly, it is quite common for the file system indexer to stop working properly. On my old PC this happened after every major Windows update. In this case, the neighboring files query would contain no files or be missing some files. I found disabling and re-enabling file content indexing fixed the problem. Also, if you use RetrievePropertiesAsync to retrieve properties for files loaded in this way, you won’t be able to access some properties. Also, some properties like title will be limited to a certain number of characters (e.g. around 250 for System.Title). Another problem is that the allowed file extensions does not always include all media files .e.g. for some reason .webm video files are not included in the neighboring files list (as of 1909). A few suggestions for solutionsScenarios 1-3 are performance-related, and in the case of scenario 2 the APIs are simply incomplete for UWP and need to be completed. The performance issues should be covered by the suggestions in your proposal. In my opinion in this case it is important that all APIs involving StorageFile/Folder have versions which accept the lightweight alternative. This should include an alternative for RetrievePropertiesAsync. Also, once you have permission to access a file for the current app session, you should be able to access it via any means, unlike currently where StorageFile.GetFileFromPathAsync(myAlreadySelectedStorageFile.Path) will fail. (Memory usage should be taken into account if it becomes a problem, but hopefully it shouldn’t). Scenario 4 is a simple matter of change the default behavior to read-write, and I think most people agree with this. Scenario 9 could be covert by enabling the app to indicate that it doesn’t want the neighboring files query to use the file system indexer, or something like that. It could also be solved by just allowing access to the folder, as long as it was possible to get the current sort order. I know this is an increase in access rights, but it would also solve some other problems which is why I mention it. The issue of some media files being omitted could be solved by Windows having a list of such allowed extensions that could be updated independently of a feature update so we do not have to wait a year for such a simple change (assuming the list was then updated to add files such as .webm). Scenarios 5-8 are more tricky. Basically all of these require more permissive file access or additional capabilities. I have sketched out a few ideas, some of which you might consider outrageous, but I think this is a very important area that needs exploring.
Finally, my thoughts on periodic warnings for users about apps using capabilities - personally I find this kind of notification annoying, so I would be against it. Better to make it easy for a user to investigate this, without giving them an intrusive notification. Maybe it could go so far as something like the using your location icon in the task bar, although even that is potentially annoying if there are too many. Thanks for your patience in reading that! |
Sorry @soumyamahunt, but this is a limitation. What if the user wants to copy the file to a USB drive or put it in their OneDrive storage? I'm forced to build a file manager UI so they can "export" the document, when the file system provides all of this functionality already. AppData is for caches and app metadata, but not user documents that they want to access, move around or do as they please with. AppData should be considered off limits to users mucking around with anyway. |
You don't have to build a file manager UI, user can go to the AppData folder of your application and copy it. If you are insistent in creating folder/file anywhere other than that you can ask user to create a folder with file picker and then add the folder/file in FutureAccessList. Automatically creating folder anywhere is huge no for me. |
|
These are the limitations I see in UWP file-access APIs besides the ones that are pointed out above 1. The
|
The problem is, AppData is hidden and I don't believe it's even easy to traverse because the final document would be behind your package folder that could be anything. If anything, it's there, but it's very unfriendly to users. That's why we're writing our software right? The scenario I see is this,
This is not about letting an app write junk all over the place, this is about a document based application creating a good user experience. |
|
Hi. I'm the developer of Files (UWP). Our primary pain points with the current experience of integrating the native Win32 file-access APIs into a UWP app were the things the Win32 File APIs lack (at least for UWP apps): Also, if we're planning on keeping the file/folder pickers, they should really have a modern UI to look familiar in our WinUI apps. |
|
@duke7553 - thank you!
Have you tried out StorageFolder.TryGetChangeTracker ?
Does FindFirstFileExFromApp help here? |
|
@jonwis My app, Files, previously included a purely WinRT file-access implementation. (Meaning, we used StorageFile and StorageFolder) It was highly optimized at that because we retrieved storage items in batches, and prefetched properties for partial storage items, yet users complained this approach was slower than they were familiar with. As of January, we now use the FindFirstFileExFromApp() to load the items themselves, but we do still rely on StorageFile/StorageFolder to return thumbnails and other properties for the items in a directory. Regarding StorageFolder.TryGetChangeTracker, my understanding of it was that it works only when the user is accessing libraries, so something like ReadDirectoryChangesW was needed here. I implemented it in Files (UWP) with the last update, v0.10.1 . |
|
Sure, we can access items with the hidden attribute this way, but they are invisible to all of the WinRT file operation methods (i.e. Cut, Copy, Delete) P.S. I will admit my case is kind of extreme since I built a file explorer. :) |
|
@duke7553 - There's a whole raft of Hey @smaillet-ms - does StorageFolder's change tracker work on any arbitrary storagefolder, or only on libraries? |
|
There's also some weird restrictions on existing capabilities, like access to the (Similar to @benstevens48 Scenario 8 above), I think there's a challenge in having any sort of project system at all (and loading these types of file sets from any other source). If a file references the absolute or relative path of another file (or files think '*' wildcard in a csproj); you can't open those references in a UWP today unless you ask the user to select a parent folder that has access to all references. It does seem like a hard challenge to solve though with figuring out an access pattern for detecting and allowing this without needing the full |
|
A
|
|
@jonwis |
From what I remember, it is supposed to work on any folder. However, my last attempt to implement it failed. The issue I faced was that it did not trigger the event right after the changes. I remember reading something along the lines of ...it will wait till something else (don't remember what exactly) before the event is triggered. |
Considering WIndows Store has a dedicated subsection titled "File Managers", I would argue, this is not an uncommon scenario. |
|
Does CreateFileFromApp and then handing over the handle to System.IO within the UWP app not work? |
|
@duke7553 yes, AFAIK this behaviour still persists because .NET uses |
|
Can someone edit the original proposal to include everything that's been discussed so far? Perhaps close this issue and open a new thread? |
|
Apologies if this isn't useful information or an appropriate place to share this, but I just discovered something unusual. I have an internal hard drive formatted to exFAT and learned my UWP app has the same access to anything on it that a Win32 app does. broadFileSystemAccess is not enabled, I haven't opened it with the picker or anything. EnumerateFiles and all the System.IO stuff works. I can even start an exe using Process.Start. |
|
@seanocali yes, exFAT has no security features and no separation between users or apps. Typically internal drives are not formatted as FAT (I believe Windows requires NTFS for system drives these days?). External drives are more often formatted as FAT (for interop with cameras, phones, other OSes, etc.) but would normally be covered by the removable storage capability. The process you start should still be running in your AppContainer, though - meaning it can't elevate your permissions. If you find that's not the case, please let me know. (There's nothing in UWP that stops you creating processes in your own container). |
I don't know how to test for that. I just set an exe on the exFAT drive to "run as an administrator". If I launch it from the desktop I get a UAC prompt first. But if I launch it from my UWP app using Process.Start then it starts up without any UAC prompt. |
|
What process do you launch? |
|
@ptorr-msft I sent you an email. |
|
@seanocali It should launch the executable on your exFAT partition with minimal privileges; your app can't create an elevated processes (hence no UAC dialog). |
andrewleader
changed the title
|
This comment from me may end up being out-of-scope for the discussion at hand, but I want to elaborate on an important scenario we had when interacting with certain OS WinRT APIs that assume all items are backed by a Essentially, the app needed to construct a DataPackage for an WinRT has a concept of streamed If this is off topic or invalid, please let me know. |
|
I just discovered another terrible feature of the File System API in UWP (AppContainer) apps. If you create a file using StorageFolder.CreateFileAsync in a folder selected through the folder picker, it is marked as blocked. More precisely, open the properties of the file in File Explorer and under security in the general tab it says 'this file came from another computer and might be blocked...'. This hasn't been a problem for me so far but some users don't like their files having this attribute. Is there any way to avoid this attribute being added? Is it really necessary for the system to add this attribute? |
|
I believe you can avoid this by using Win32 APIs such as |
|
Well this has been here for a long time with no definitave answer and I wonder if this is even the correct repository. .Net repository is cross platform, so StorageFile.* is our of scope Either way. WinRT based file access is superior to System.IO just becase of what it can do (like pre-populate from the file indexer) - however, as stated by many, the performance is so bad that it makes it impossible to use. If we look at a WinUI 3 desktop project, calling WinRT IO is 10 times slower than system.IO and can't be multi-threaded due to the broker. Either get rid of the broker and improve the interop performance, or bring the functionality to System.IO please. |
|
CsWinRT is just a projection of the API, issues with APIs themselves wouldn't go in their tracker as well. |
|
So you're saying that this is the correct repository - and no confirmed action since May 2020. |
|
There is no correct repository; the reäl correct place to send feedback about built-in Windows APIs is the Feedback Hub. |
|
I don't think the WinRT API will get fixed - the issues raised here have been known and reported for years. Take one example - I want to get the thumbnail of a photo. System.IO doesn't even have the option. Image controls/classes (any version) can't be used as they try to read the file (causing hydration on a placeholder files) - so WinRT is the only option - but it is incredibly slow, it also has bugs in it- oh, and there's a memory leak that's existed since 2012. So... even bugs that affect the built in Photos app have never been fixed, so I'm going to assume it never will be - that doesn't stop me using the API, I just know the customer experience is sub-standard and I have to work around the bugs/performance. |
|
To be fair, the UWP app model is dead, so at the very least, the winRT StorageFile class is also dead, as .net or C++ can be used in its place. I don't think there will be any bug fixes planned, as nobody will be using this API in a few year's time. This thread started out on the assumption that UWP app model will stay and people were asking for System.IO in UWP, but Microsoft gave up on UWP and introduced WinUI 3, which essentially has both System.IO and StorageFile. |
|
So sad. Have you seen how complex it is to get access to the thumbnail data in a placeholder file ! - those WinRT Storage.* API were brilliant for that, now we have to learn 30+ Win32 API's that are very complex, especially the search API. :( |
|
For thumbnails i created my own index, essentially. But this works in my case only i suppose. |
|
Yep, .Net provides no way to get this basic stuff :( |
|
I believe this issue #8, #219 together with #13/(https://techcommunity.microsoft.com/t5/msix-feedback/machine-wide-provisioning-install-for-all-users/idi-p/1781716) are the topmost "Three Most Wanted" features ever since WindowsAppSDK/Project Reunion was created. Since UWPs/ What we need are replacements of
(just like now we have |
and others
Proposal: Improved file-access APIs for UWP and AppContainer
Summary
Provide an updated version of
StorageFilethat fixes performance and usability issues in the currentWindows.Storage.Storage*set of types. Support apps that think in terms of file paths & handles instead of StorageFile objects.Rationale
The current
Windows.Storage.Storage*types are designed to give programmatic access to regular files, shell namespace items, and other Windows entities backed by files. A single abstraction over all those spaces is interesting but comes with high costs - namely performance and impedence-mismatch with existing application code. Project Reunion should clearly explain why StorageFile is important and - where possible - provide lower-level access to the user's files.BroadFileAccesscapability is "scary" to customers but required even for limited UWP scenarios....FromAppAPI variants, but higher-level code (JS, C#) interop for them is very limited.Scope
...FromAppplatform APIsIStream/ISequentialStreamAPIs for these filesW.S.Storage*and types in this functionalityOpen Questions
Should UWPs be given access to the
IShell*family of APIs?As these APIs are not truly universal on all editions of Windows, if an app were to program against them directly the app would not work on editions like HoloLens or Xbox.
Should Reunion warn the user about apps using this capability?
Some platforms periodically remind users that "App X is using feature Y, are you OK with that?" How should Reunion make the user aware that a "low rights" application is using capabilities?
The text was updated successfully, but these errors were encountered: