Avoid icon/mime clobbering by using <icon name=... />

[tresf]

Running the latest appimaged from downloads, I'm observing the following behavior on Ubuntu 14.04 LTS:

1. Placing <project>.AppImage in ~/Downloads successfully triggers the appimage_register_in_system.
2. During registration, appimaged copies <project>.xml to $HOME/.local/share/mime/packages and names it appimage_<hash>_<project>.xml.
3. During registration, appimaged copies <mime-icon>.svg to $HOME/.local/icons/hicolor/scalable/mimetypes and names it appimage_<hash>_<mime-icon>.svg.

What seems to be missing is, the xml still references the original .svg by it's original name, so the icon never shows up in Nautilus, regardless of how many times you restart nautilus or logout of the Desktop.

Here's an example xml. For demonstration purposes, I'll call this project "starbright".

Mime Package

• mime/packages/starbright.xml:

  <?xml version="1.0" encoding="UTF-8"?>
  <mime-info xmlns="http://www.freedesktop.org/standards/shared-mime-info">
    <mime-type type="application/x-starbright-file">
      <!---          ^-- EXPECTS icon to be named e.g. application-x-starbright-file.svg  -->
      <sub-class-of type="application/xml"/>
      <comment>Starbright File</comment>
      <glob pattern="*.starb"/>
    </mime-type>
  </mime-info>

Mime Icon

• icons/hicolor/scalable/mimetypes/application-x-starbright-file.svg

What happens is appimaged does the following:

1. Copies
   <AppImage>/share/mime/packages/starbright.xml -->
   
   $HOME/.local/share/mime/packages/appimaged_d41d8cd98f00b204e9800998ecf8427e_starbright.xml
   ✅ Good!
2. Copies
   <AppImage>/share/icons/hicolor/scalable/mimetypes/application-x-starbright-file.svg -->
   
   $HOME/.local/share/icons/hicolor/scalable/mimetypes/appimaged_d41d8cd98f00b204e9800998ecf8427e_application-x-starbright-file.svg
   ⛔️ This causes problem

Now the mime type value inside the xml file is mismatched from its respective icon. Is this a limitation of the mimetype support for this OS, or is there a gap in functionality with the way the icon is currently being installed?

<?xml version="1.0" encoding="UTF-8"?>
   <mime-info xmlns="http://www.freedesktop.org/standards/shared-mime-info">
-     <mime-type type="application/x-starbright-file">
+     <!-- Help! How to reference after rename? -->
+     <mime-type type="appimaged_d41d8cd98f00b204e9800998ecf8427e_application/x-starbright-file">
         <sub-class-of type="application/xml"/>
         <comment>Starbright File</comment>
         <glob pattern="*.starb"/>
      </mime-type>
   </mime-info>
</xml>

If the svg name is copied to hicolor/scalable/mimetypes/application-x-starbright-file.svg (without the appimaged_d41d8cd98f00b204e9800998ecf8427e_ prefix), the icon will show up.

[probonopd]

Thanks @tresf for your detailed analysis. We should definitely fix this issue, but at the same time we need a way to (easily!) remove the files a) copied to $HOME/.local/share that belong to any given AppImage, and b) that belong to any AppImage. Also, multiple instances/versions of the same application's AppImages should not overwrite each others' files. I think we are running into limitations of the XDG way of doing things here, compared to, say, the database-driven Apple LaunchServices.

Possibly we should discuss the "right way" to do this on the XDG mailing list?

[tresf]

We should definitely fix this issue, but at the same time we need a way to (easily!) remove the files a) copied to $HOME/.local/share that belong to any given AppImage, and b) that belong to any AppImage.

How about append to the end instead of beginning...

e.g. application-x-starbright-file-<appimaged-hash>.svg and application/x-starbright-file-<appimaged-hash> respectively?

1. SVG:

   - application-x-starbright-file.svg
   + application-x-starbright-file-appimaged-d41d8cd98f00b204e9800998ecf8427e.svg

2. Mime name:

   - application/x-starbright-file
   + application/x-starbright-file-appimaged-d41d8cd98f00b204e9800998ecf8427e

This would honor the unique, uninstall-able namespace without confusing the application class prefix.

It would still require the rewrite of the DOM inside mime/packages/starbright.xml, which I don't think the daemon is doing currently.

[probonopd]

Yes, that sounds like a good solution to me. Other opinions?

[tresf]

@probonopd would appimaged also have to rewrite the <project>.desktop file?

  Terminal=false
  Type=Application
- MimeType=application/x-starbright-file;
+ MimeType=application/x-starbright-file-appimaged-d41d8cd98f00b204e9800998ecf8427e;
  TryExec=/home/ubuntu/Downloads/starbright_x86_64.AppImage

[probonopd]

Yes

[probonopd]

PR welcome since I don't have the time to do this at the moment.

[tresf]

This would involve opening files and doing a string replacement on them. Is there a existing, preferred method for this that would be consistent with the project codebase?

[probonopd]

Yes, we do a similar thing with the desktop files around here: https://github.com/AppImage/AppImageKit/blob/appimagetool/master/shared.c#L390-L576

[tresf]

Initial version here: AppImage/AppImageKit@appimagetool/master...tresf:appimagetool/master

What works:

• SVG name is corrected to append hash instead of prepend hash
• Desktop file MimeType string is corrected to reflect SVG file name

What doesn't work:

• XML file still points to old SVG path

If anyone has code examples using glib to replace a DOM attribute that would be greatly appreciated.

[probonopd]

Do we really have to parse the XML or could we get away with just using a regex on the path string?

[probonopd]

Otherwise, maybe check https://github.com/ooxi/xml.c/blob/master/test/example.c

[tresf]

Do we really have to parse the XML or could we get away with just using a regex on the path string?

I see no evidence of wildcard support in the freedesktop.org specification.

I have basic parsing written in tresf/AppImageKit@1071398. Replacing and rewriting will be some more research.

[probonopd]

I don't mean to put the regex into the file, i just meant we could use a regex to rewrite the path in the xml file without really parsing the xml.

[tresf]

i just meant we could use a regex to rewrite the path in the xml file without really parsing the xml.

Perhaps. We don't know or care about the actual icon file name (mime spec is a bit loose about the slashes and dashes), so ingesting and appending to the node's attribute is probably the most reliable and straight forward approach.

[carlos22]

Any updates on this? Wanted to use mime integration but fails to work with AppImageLauncher...

[tresf]

Any updates on this? Wanted to use mime integration but fails to work with AppImageLauncher...

No, not from me. I'm not a C coder so this would take me days to iron out to use the XML parsing libraries properly. A simple regex would probably be quicker as @probonopd has indicated. The consensus from the project maintainers tends to be in this nature:

PR welcome since I don't have the time to do this at the moment.

If anyone is interested in picking this back up, the WIP code is here: AppImage/AppImageKit@appimagetool/master...tresf:appimagetool/master

[TheAssassin]

@carlos22, no, there's no updates to this unfortunately. If you check out our issue tracker, we have almost 200 open issues of which only 1/3 are feature proposals. This is not the only repository we maintain, and there's very limited time for AppImageKit at the moment even though I work on these things full time.

This seems easy to fix, I don't see how a regular expression would be necessary, it's only basic string replacement. Doesn't seem to complex. I will put it on this week's TODO list, but I can't promise I'll get to working.

[tresf]

I don't see how a regular expression would be necessary, it's only basic string replacement.

You'll still need to cycle over the DOM though. application/x-starbright-file can be coerced to application-x-starbright-file by replacing slashes with dashes, but not the other way around, right? ... even then you have to make some assumptions, such as type= always comes directly after <mime-type ... as an example... if someone copies a node from mime.types, they may paste in something like this (valid to the spec, but could break "basic" parsing).

<mime-type xmlns="http://www.freedesktop.org/standards/shared-mime-info" type="application/x-starbright-file">

As an aside, the freedesktop spec does allow the explicit use of a separate <icon=...> element but it doesn't seem to be used in any examples I can find.

This is why a DOM parser (or regex) was originally proposed. @TheAssassin thanks for putting this on the TODO.

[TheAssassin]

@tresf so it should be enough to use a minimal XML parser, open the copied XML file, search for elements of type mime-type, and append the md5 value to the value to it like you described? That seems a bit awkward. We'd define tons of redundant MIME types.

Do you really think it's a solution to append some sort of suffix to MIME types and define new ones? That's actually really annoying desktop integration wise, because then tools like xdg-mime query filetype <...> may return different results. The user's system would be left in an inconsistent state, with all tools making use of MIME information yielding unreliable information. MIME types shouldn't interfere with each other.

I think we should just copy MIME icons with their given filename, and if there's an existing file, leave it alone. The problem with that approach is that when there'd be two AppImages referencing that file, unintegrating would either have to leave the file there, remove it always, or we'd need some sort of reference counting to delete it only when the last referencing AppImage is removed. The latter won't work reliably either, though, so I'd just leave the icons on the system.

[tresf]

Do you really think it's a solution to append some sort of suffix to MIME types and define new ones? That's actually really annoying desktop integration wise, because then tools like xdg-mime query filetype <...> may return different results. The user's system would be left in an inconsistent state, with all tools making use of MIME information yielding unreliable information. MIME types shouldn't interfere with each other.

I was basing this on my interpretation of a few things that I saw appimaged doing in other areas...

1. I assumed the appimage was using the md5 intentionally as to never clobber another appimage
2. I assumed the mime database would be appended and equally cleaned up as part of the standard process (usually a logon/logoff, but can be triggered manually)

When I was originally speaking with @probonopd he made this statement...

[...] copied to $HOME/.local/share that belong to any given AppImage, and b) that belong to any AppImage. Also, multiple instances/versions of the same application's AppImages should not overwrite each others' files.

So to truly "belong" to an appimage, I thought this was the right path. Note, it's not uncommon for two applications to fight for a mime-type and the spec has some techniques for establishing "weight" on this so they can fight appropriately. This gets more into the libmagic side of things so it wouldn't handle two identical appimages with the same mime.

I can see how redundant the mime for a file downloaded to ~/Downloads as well as ~/Desktop would be but I see this same problem occuring for the launcher icons and I don't hate it. For example, if I download (hypothetical example) FreeCAD 1.0 and FreeCAD 2.0 it may have been on purpose; they may not have the same icons between versions.

Unique icons allow proper cleanup and that seems to have been the original intent. If it's not the intent, the design can certainly be simplified, but the clobbering/cleanup scenario will become much harder to manage I feel.

[TheAssassin]

It's of course better to make sure the AppImage's integrated files won't interfere with each other, but in case of MIME support, this simply doesn't work in my opinion. What I suggested is a trade-off between both sides, taking into account that the icons may be outdated. Unless the MIME types definitions allow defining some sort of special icon name/path, we can't resolve this issue, we can only try to work around the limitations.

Redundant MIME types are not at all an option, @probonopd must agree here, he's had some similar issues with the magic database recently, where the file command's output was inconsistent across several distributions, he can confirm that this is a situation you shouldn't intentionally create.

[tresf]

Unless the MIME types definitions allow defining some sort of special icon name/path, we can't resolve this issue, we can only try to work around the limitations.

The proposal does. What's wrong with it?

Redundant MIME types are not at all an option, @probonopd must agree here, he's had some similar issues with the magic database recently, where the file command's output was inconsistent across several distributions, he can confirm that this is a situation you shouldn't intentionally create.

I'm interested in understanding the problem here. You'll always be fighting the stock mimes, so without knowing the bug this creates, it just seems like a 2nd, 3rd and 4th instance of an identical problem the system would have naturally. e.g. FreeCAD from dnf mimes conflicting with FreeCAD from AppImage. I would like to know exactly what problems two FreeCAD mimes from AppImage create.

[TheAssassin]

Imagine someone uses the MIME type definition to check a file's MIME type. Now, on a system without redundant MIME type definitions, there will be exactly one correct output. However, when you have multiple definitions, there might be other MIME types returned. If I'd want to check whether the file was of type application/xml but I'd get application/xml_appimage-XXXXX, e.g., a string comparison would no longer work. That's the issue: you could accidentally break some other software on the system. That's something we must avoid at all cost.

[carlos22]

it just seems like a 2nd, 3rd and 4th instance of an identical problem the system would have naturally. e.g. FreeCAD from dnf mimes conflicting with FreeCAD from AppImage. I would like to know exactly what problems two FreeCAD mimes from AppImage create.

Well the MIME type is not the problem. The icon for the mime type is. Most of the time the linux apps do not ship their icon, the theme just provides them for all known types. So the complete xdg/freedesktop icon spec is designed around the idea of having themes which provide icons for everything. The "hicolor" theme is the fallback if in the current theme a icon is not found. Registering different mime types for the same "magic" (i.e. file suffix) is a problem. Having multiple apps for the same MIME type is not a problem.

[tresf]

Now, on a system without redundant MIME type definitions, there will be exactly one correct output. However, when you have multiple definitions, there might be other MIME types returned.

Understood, but this is a false assumption. There are multiple definitions, that's the point of the spec. Apps currently can and do fight over this.

If I'd want to check whether the file was of type application/xml but I'd get application/xml_appimage-XXXXX

This is a good point and I understand the concern however there is a parent type that's part of the spec, so it should still be possible to keep the application/xml

  <mime-type type="application/x-lmms-project">
+    <sub-class-of type="application/xml"/>

That's the issue: you could accidentally break some other software on the system

I'm still unsure what would be broken in this case. You're not deregistering the old handler, just registering a new handler. Once removed, the old handler can and will be placed back on priority. If you're going to take ownership of a file extension, that's exactly what's going to occur. I'm still confused as to the underlying problem.

I can certainly see the merit in removing the redundancy, I'm just not sure how appimages would manage this stuff. It seems like you'll be reinventing the purpose of the mime database but perhaps I'm not seeing the described issue clearly. 😟

[TheAssassin]

@tresf so to avoid issues, we'd need to add also "subclass of ", and we should be able to avoid duplicate definitions of the same MIME type (name), right?

[tresf]

@TheAssassin the subclass of should already be part of the mimes already. For example, if you make a .ini file that's supposed to open by default in notepadqq, you'd make it a subclass of text/plain although the spec rather obscurely hides this in an implicit rule so this should be handled automagically if the text/ prefix is used

freedesktop.org:
Some subclass rules are implicit:

All text/* types are subclasses of text/plain.

All streamable types (ie, everything except the inode/* types) are subclasses of application/octet-stream.

I understand the concern though... for the application/x-foos of the world it's going to be relatively sane whereas the very common, root mime-types, it can and will pollute the system's rather clean handling of basic types.

Worse, the subclass of is only a guideline. It's not always going to work and it's not always going to be required. For example, LMMS ships a compressed mmpz file format and it's being falsely advertised as xml when it's closer to a zlib compressed file so that's a bug with that line of the mime file.

Perhaps as a stop-gap, we focus on a sane prefix like application/x-?

Here's a snippet from Inkscape installer showing how consisten this is...

mimetypes.add_type('application/x-msi', 'msi')
mimetypes.add_type('application/x-7z-compressed', '7z')
mimetypes.add_type('application/x-gimp-pallette', '.gpl')

According to this article, x- is for non-standard mimetypes so it should be relatively safer to the system.

[TheAssassin]

So, just to clarify this, if we'd append the suffix, and would then use the common tools to check the MIME type of some file, it'd yield a list of all matching MIME types? In that case, there won't be the "collisions" I was afraid of.

However, I'm not sure whether that'd be good news icon wise. We couldn't make sure the desktop environment would show the installed icon if there's multiple choices for it anyway, can we?

[tresf]

We couldn't make sure the desktop environment would show the installed icon if there's multiple choices for it anyway, can we?

I don't think there's a guarantee, no. It's hard to interpret the standard, but it appears to do a first, shortest match.

https://standards.freedesktop.org/icon-theme-spec/icon-theme-spec-latest.html

To @carlos22's point, there are a lot of "assumed" icons that would make no sense to touch...

https://standards.freedesktop.org/icon-naming-spec/icon-naming-spec-latest.html

But for apps like Gimp, Inkscape, etc, I assume the AppImage would want to take priority. I haven't tested collisions enough to know how the system reacts but I think the statement is correct....

We couldn't make sure the desktop environment would show the installed icon if there's multiple choices for it anyway

Although this would only be for collisions. If the mime database does a LIFO, we'd be in much more predictable shape. I'm still not sure we have a choice though. Even in today's world, there's the natural fight between xpdf, kpdf, acrobat, etc where they may choose to register their own icons and compete for ranking. AppImage is just doing it on a macro-scale, right?

[TheAssassin]

My point is, we can fix this, but I don't think it'll solve the core issue which is the icon displaying. Even worse, by changing the MIME type name, we might even confuse the payload software. That's my biggest concern.

[TheAssassin]

Could/would/should won't get us a solution for this issue, though.

[probonopd]

What I am trying to say is that this is possibly a Desktop Linux Platform issue and as such needs to be discussed and solved e.g., in the XDG forums. AppImage is not the place to work around architectural issues of the Linux desktop; instead we should start a discussion to get this fixed for everyone.

[TheAssassin]

@probonopd in theory, the integration code is still broken, though, as the icon name's changed. We must decide how to handle MIME files in the future.

[probonopd]

I can see the day coming where we write our own desktop environment, or at least, file manager, together with a LaunchServices equivalent, everything modeled after macOS, throwing out of the door all of the XDG stuff... heck, had they supported .app bundles or AppDirs 15 years ago in Nautilus and Konqueror, AppImage would never have been started ;-) /rant mode off

[TheAssassin]

Please provide some constructive feedback regarding the issue. I still think we should just stop rewriting the icon filename.

[tresf]

everything modeled after macOS, throwing out of the door all of the XDG stuff

macOS handles this quite similarly actually. They too have a background service and they too have the possibility of two apps fighting for the same file extension.

The major difference here is that XDG has a user-space standard, but not a well ironed out portability standard. As an analogy... just like Mac wouldn't want Chrome blindly replacing the icons for HTML documents, it's completely reasonable for AppImageKit to not start overriding mime types that could lead to unpredictable desktop results. Being mindful of the system defaults is very good point and a very responsible catch by @TheAssassin.

That said... the mime standard allows x-, so that's a safe space, let's use it. We can be smart enough to assume we only want to re-use x- that are in the mime and then we'll only deploy x- that are in the mime. I don't see any reason not to support this if the namespace was created for this exact purpose.

There's always a chance an irresponsible ISV comes along and steals association for .txt or .xml or .odt but those problems are going to exist in package-proper just as well. I can't see a way (nor a reason) to protecting against that and removing the AppImage should and will put the system back to normal.

[TheAssassin]

@tresf my concerns also cover that the payload application might expect the MIME types to be installed so they can use them, but AppImage's desktop integration does change the MIME type name, resulting in unexpected behavior.

Although, thinking about it, a payload application shouldn't actually depend on such information, it should work without being integrated, too.

Okay, you convinced me, let's change the MIME type!

[tresf]

@tresf my concerns also cover that the payload application might expect the MIME types to be installed so they can use them, but AppImage's desktop integration does change the MIME type name, resulting in unexpected behavior.

Good point. There are a few select applications that ask on first run or install to make them the default handler of a particular file type. One example is IntelliJ for .java files, but I'm sure there are other examples that are more common so better runtime-awareness could be an issue created out of this.

For example, currently LMMS.AppImage must make certain assumptions at runtime to handle working paths so a better registry system (in the generic sense, not the Windows-sense) may be warranted but I'm not sure how apps do this today so this is something to research. As an example, LMMS may eventually want to use libmagic to associate LV2, VST, SF2, etc and that could potentially end up being a runtime decision.

Okay, you convinced me, let's change the MIME type!

🎉

[azubieta]

Please move to appimaged.

[probonopd]

This is an important aspect of AppImageCommunity/appimaged#30.

[TheAssassin]

Moved into libappimage, as the solution is to modify some behavior within there.

[probonopd]

Note that moving around tickets changes the # numbers. This might confuse people and hence should be kept to a minimum.

[TheAssassin]

GitHub sets up redirections.

[probonopd]

But not in people's brains ;-)

[TheAssassin]

People don't memorize GitHub issue IDs.

[shoogle]

Changing the MIME was the solution I used to get MuseScore's file icons to show without clobbering existing icon files for the same mimetype.

However, as pointed out above, this does have the drawback of potentially changing the MIME displayed in the File Properties. Furthermore, if multiple icons are installed for the same file extension then there is still going to be a competition happening somewhere to determine which icons actually get displayed in the File Manager. (You'd hope it would be the icons for whichever program appears first in the open-with list, i.e. the default program used when you double-click the file, but I haven't tested this.)

It did occur to me that another way to install the icons would be to create a new icon theme rather than placing them all in hicolor. For example:

• Instead of: $HOME/.local/icons/hicolor/scalable/mimetypes/mime_myapp_appimaged.svg
• Use this: $HOME/.local/icons/myapp_appimaged/scalable/mimetypes/mime.svg

That allows you to install all the icons without changing MIMEs or patching XML files, but there's no guarantee that the icons will actually get used unless the user switches theme.

[probonopd]

Icon themes also tend to require a restart of the desktop environment.

[shoogle]

Looks like there is a way to avoid icon/mime clobbering after all!

As @tresf pointed out, the problem is that the system expects icons to be named like the mimetype:

<?xml version="1.0" encoding="UTF-8"?>
<mime-info xmlns="http://www.freedesktop.org/standards/shared-mime-info">
  <mime-type type="application/x-starbright-file">
    <!---          ^-- EXPECTS icon to be named e.g. application-x-starbright-file.svg  -->
    <sub-class-of type="application/xml"/>
    <comment>Starbright File</comment>
    <glob pattern="*.starb"/>
  </mime-type>
</mime-info>

However, that is only the default name for the icon. The Shared MIME Info Specification allows another name to be specified using <icon name="...">. For example:

<?xml version="1.0" encoding="UTF-8"?>
<mime-info xmlns="http://www.freedesktop.org/standards/shared-mime-info">
  <mime-type type="application/x-starbright-file">
    <icon name="application-x-starbright-file-appimaged-d41d8cd98f00b204e9800998ecf8427e"/>
    <sub-class-of type="application/xml"/>
    <comment>Starbright File</comment>
    <glob pattern="*.starb"/>
  </mime-type>
</mime-info>

This avoids clobbering icon files and it avoid having to change the mime type for each app. Of course the system will only actually display one set of icons for the given mime type, but hopefully it is clever enough to pick the icons for the app that is set as the default "Open with" program for this mime.

Observations:

• The icon name mustn't contain a slash (/). Mapping / to - is recommended.
• The icon name mustn't have a file extension (.png / .svg) but obviously the actual icon files will.

[azubieta]

Working on it, if you have any additional comment or idea feel free to share it.

[shoogle]

Maybe print a big fat warning if the original manifest didn't specify an icon name (i.e. if it didn't have a <icon name="..."/> tag) and was instead relying on the default name based on the mimetype. This would help raise awareness of the importance of that tag.

The Shared MIME Info specification is far from helpful here. It just says:

icon elements specify the icon to be used for this particular mime-type, given by the name attribute. Generally the icon used for a mimetype is created based on the mime-type by mapping "/" characters to "-", but users can override this by using the icon element to customize the icon for a particular mimetype.

Words like "generally" and "customize" make it sound like using a non-default name is a purely aesthetic decision. I can't have been the only person to have read that and thought, well if there is a default name then why would I want to change it?

The only reason I could come up with is if you wanted to use the same icon for multiple mime types (e.g. an audio file icon for both WAV and MP3 files). It wasn't until I saw this thread that I realised it was absolutely essential to specify a non-default icon name in order to prevent collisions with icons from other programs.

Maybe the Shared MIME Info people should change their specification so that it STRONGLY RECOMMENDS setting a non-default icon name...