Use Java 16+ unix sockets #145
Comments
|
I already read about that feature in newer Java versions. As this library still continues to support older versions of Java, using the native implementation is not possible. Increasing the required Java version was already on my todo list. One thing I could think of is to change the unix socket parts to use service loader to switch the "unix socket backend". |
Ok that needs to be checked, of course.
That's what I assumed. But I still had a little hope that (judging from the low number of open issues), the current version is mature and doesn't require that much attention, so focus can shift to a new version. 😉
While certainly good software design, this might not be that much less effort, since it would require at least two, maybe three new libraries:
|
Low number of issues does not mean there are no bugs or that there is no room for improvement. As far as I can see in the description of JEP 380, it seems that it does not support all feature required by dbus-java: Non-Goals
One issue would be: dbus-java supports abstract unix sockets, which is not part of JEP 380 because windows does not support that. |
|
I have some interest in this. I am using the embedded bus on Windows, and so have to use the TCP transport on this OS. It is noticeably slower communication between the client and server when compared to running on OS X or Linux both using domain sockets. So frankly any alternative on Windows would be great. I had actually started on implementation on Windows that used name pipes, but while this whould be relatively easy with JNA (there is code already written), I couldn't find enough documentation on JNR to fit in with dbus-java and lost interest! I can also confirm that you don't have to use an abstract path on Linux, you can connect to the system or session dbus fine or create your own embedded bus using a real path. OS X doesn't support abstract paths either. |
|
When just using the library as "client", abstract sockets are not really needed. But when using the I'll have to dig into this when I have some spare time left. Also I still want to support older Java versions (especially 11) when 17 will be released next month. I few years ago when I started with dbus-java there were still frameworks for IOT forcing the developer to provide their plugins with Java 1.7 compatibility. That was really annoying, so my policy is to at least support the previous Java LTS version. |
|
OS X does indeed throw an exception if you try to use an abstract, but I am successfully using As another vote for having Java 16 domain socket support ... I have a research project scheduled to see if the application I use The last time I looked, neither JNA or JNR were fully supported. Graal SDK does have it's own FFI, but that would mean a rewrite of |
|
I just started to take a closer look to native unixsockets in Java 16. Especially the authentication stuff ( More changes were required due to the fact that the Java 16 unix sockets uses NIO while the current implementation uses the old socket API. My first tests using TestAll/TestLowLevel on an Ubuntu Linux machine worked. I'm not sure about including or not including jnr-unixsocket as dependency.
I uploaded the current development state to a new branch ( |
|
Thanks for making a start on this. Ive given it a try on Linux, Windows and Mac. The only real success so far I've had is connecting to the system bus on Linux. As the primary purpose of dbus-java, this works as well as it ever did. However, I can't seem to get There were a few tweaks I needed to get it to compile in my environment. It seemed I needed to add On Windows, I noticed the path is stil The main problem now seems to be starting the embedded daemon, then getting a connection to export some services on (in the same JVM). This results in two exceptions, one on the daemon thread and on the main thread where I am trying to connect to the daemon. Regarding your points as to how this could be plugged in, my preference would be for |
|
Thanks for the feedback. I have to admit I didn't look at modules-info (and I still don't really understand JPMS). EmbeddedDbusDaemon is broken due to my refactoring. I will fix it and implement the ServiceLoader stuff as well. The thing with the '/tmp' path is simply wrong, and is also wrong in upstream. Instead of using java.io.tmpdir, "/tmp" is used hardcoded. I will fix this as well and post an update here when you can pull the branch again. |
|
Alright, it has taken some more time than I excepted, but I just pushed my latest changes. The library has been splitted into multiple parts:
The system property stuff used before has been removed. So the big change is that you have to add at least on of the transport modules as well as dbus-java itself. Adding both, jnr and native unix socket will cause dbus-java to throw an exception because there can only be one protocol provider at the same time. In this case both provide support for 'unix' protocol, so you have to choose one of them (otherwise it would be more or less random which gets used during runtime). Combining jnr or native with tcp is still possible. Maybe I'll rename dbus-java to dbus-java-core to make clear that it does not work without any of the transport implementations. All parts now contain a module-info.java file and will also require at least Java 11 (Java 16+ is required when using native transport). During my refactoring I saw that there are not so many tests running with the TCP version. Most tests use unix socket or fail due to authentication issues. E.g. when running TestPeer2Peer using TCP on both ends ( All unit tests are working with native/jnr when using linux. No other OSes tested yet. Feedback welcome! |
|
Thanks, its looking a lot more polished already, but I am still seeing more or less the same thing as I saw before though, which sounds like the same case as TestPeer2Peer. I have an app that is in two parts. The service part exports a DBus interface on the system or session bus if it's there, but it tries to start an embedded bus if one doesn't exist (i.e. usually Mac and Windows). It then tries to export the service on this embedded bus. So it's all happening in the same JVM. FWIW, the same now seems to happen with both JNR and native domain sockets so hopefully it's just an effect of the refactoring rather than a limitation of native sockets. I'll post if I can find why. Edit: Found one thing. TestPeer2Peer can be made to work, if you change Edit 2: While that makes TestPeer2Peer complete, it doesn't completely help in my application. It gets further, but then hangs trying to obtain a connection to the embedded bus (again using TCP). |
|
Using SASL.AUTH_ANON is not a proper solution as the DBus specifications says that you have to use the SHA method when connecting to DBus using TCP. And because it is possible to run the original dbus daemon in TCP mode, dbus-java should be able to connect to it. I did some more investigation and it seems that SASL is somehow broken when using TCP. If you use the current Some of the other tests still fail or get stuck (e.g. TestTwoPart). Maybe I'll take a look at that later. |
Besides depending on dbus-java, kdewallet has only JUnit and SLF4J as further dependencies. So, on modularizing kdewallet, the dependencies for one part/module would be significantly reduced. |
|
I add this change to my to do list. |
No plans to release any sort of unfinished version (alpha, beta ,whatever you wanna call it). I'm still investigating the issues when using TCP transport. I don't think it is a show stopper, as the failing/blocking tests were never designed or ran with TCP as transport type. |
|
Alright, I just pushed my latest changes. During developing I found some more issues introduced by some recent PRs. I fixed them in branch and upstream as well. |
|
Thanks for the update. The At this stage, I can only guess I am doing something in the setup of the embedded daemon. Here is a fragment of my code that deals with this (worked with 3.x branch fine). boolean startedBus = false;
if(daemon == null) {
BusAddress listenBusAddress = new BusAddress(newAddress);
String listenAddress = newAddress;
if(!listenBusAddress.isListeningSocket()) {
listenAddress = newAddress + ",listen=true";
listenBusAddress = new BusAddress(listenAddress);
}
log.info(String.format("Starting embedded bus on address %s (auth types: %s)", listenBusAddress.getRawAddress(), toAuthTypesString(authTypes)));
daemon = new EmbeddedDBusDaemon(listenBusAddress);
daemon.startInBackground();
log.info(String.format("Started embedded bus on address %s", listenBusAddress.getRawAddress()));
startedBus = true;
}
/* Because we cannot guarantee that the daemon is truly started and listening
* for connections at this point (startInBackground() was used), try multiple
* times.
*/
log.info(String.format("Connecting to embedded DBus %s", busAddress.getRawAddress()));
for (int i = 0; i < 6; i++) {
try {
conn = DBusConnection.getConnection(busAddress.getRawAddress());
log.info(String.format("Connected to embedded DBus %s", busAddress.getRawAddress()));
break;
} catch (DBusException dbe) {
if (i > 4)
throw dbe;
try {
Thread.sleep(500);
} catch (InterruptedException e) {
throw new IOException("Interrupted. ", e);
}
}
}Have you got any more pointers on what this might be? I will try to get an isolated unit test to demonstrate, will post back here. On a testing side note. I tried running the test suite, but it fails from command line I can make the direct Eclipse tests work by separating the 3 dependencies into 3 Maven profiles and selecting the one I want to use. It should be possible to keep the current behaviour of running all tests for all providers via Maven, but make this selectable by turning off profiles. Ill make a PR when this dust has settled on version 4.0.0. |
|
Thanks for your feedback. I could reproduce your issue and fixed in the branch (please pull). |
|
That's got it! Everything is working exactly as it was with 3.x now. I will be trying this out on Windows and OS X tomorrow. If I get some time I might throw together a performance test. I also took your polling code from Am particularly looking forward to seeing domain sockets on Windows. I picked dbus-java for a cross platform app that needed an RPC mechanism because I don't particularly like RMI, and found other frameworks too heavyweight. Having used DBus on Linux for native apps, it seemed like a good idea. This change makes it an even more viable answer for people who might want cross platform RPC. DBus has it short comings of course, but for many purposes its great. Regarding Edit: @overheadhunter asked about builds. Anybody who might want to try this out is welcome to use my employers public Maven repository. It has builds of 4.0.0-SNAPSHOT that are up to date as of now. The repository is available long term, but I can't guarantee when updates might be uploaded. I tend to use dbus-java often so it should be fairly often. <repository>
<id>jdaptive-ext-snapshots</id>
<name>artifactory.javassh.com-snapshots</name>
<url>https://artifactory.jadaptive.com/ext-snapshots-local</url>
<snapshots/>
<releases>
<enabled>false</enabled>
</releases>
</repository> |
|
Have some feedback on OS X. It does all work, but there are is a change in behaviour that is causing me problems. My app is in two parts. The service runs as root. It starts It turns out previously, I was using This works ok on Linux. My deployed app uses the system bus and a policy file to open up the exported interface. In the absence of having policy files with the embedded daemon, and now no I noticed that the UID is retrieved using |
|
I updated I don't understand your issue with |
|
Fantastic, thanks. I'll give that a try now. I was indeed using And apologies, I missed that |
|
That all works good. Now Linux, OS X and Windows are all behaving in the same way as they were with the 3.x.x version, i.e. unix and linux can use domain sockets and windows can use TCP sockets. However, I seem to have hit a snag on Windows at the very last minute when using native sockets. My service successfully starts an Embedded Daemon, connects to that and exports a service on it. I can see a file in the file system. If I then try to connect a client to this, either as an administrator or normal user, the connections appears to just hang. It's like only a single connection is being accepted on the server end of the socket. Hopefully this isnt a limitation of either the JDK, or Windows domain socket support. |
|
Can you create a sample for reproducing this behavor? I created 2 classes ( My client side looks like this: try (DBusConnection conn = DBusConnection.getConnection(twopartAddress)) {
// repeat until JVM is killed
while (true) {
IExport remoteObject = conn.getRemoteObject(RunTwoPartDaemon.EXPORT_NAME, "/", IExport.class);
System.out.println("Remote side says: " + remoteObject.sayHello());
try {
Thread.sleep(500L);
} catch (InterruptedException ex) {
}
}
} catch (IOException | DBusException e) {
throw new RuntimeException("Could not connect to twopart daemon");
}I can connect multiple clients using native unixsocket without any blocking. All clients will print 'Remote side says: Hello'. I added my sample code to my sandbox repository, maybe you take a look at it. |
|
You are absolutely right, multiple connections do work. It turns out this problem only shows itself on the installed version of my app. Running the two parts as the same user from an IDE (Eclipse) using a domain socket is absolutely fine. I suspect it's something permissions related on the socket file. The component that embeds the dbus daemon runs on Windows as a service, and as the "Local System" account. This is not the same as "Administrator" apparently. The one thing that makes me doubt this is there are no exceptions, it just hangs. Will post back when I know some more. |
|
Confirmed as permissions. Adding this to the setup of the embedded daemon does the trick. /* WARN: Very loose permissions, anyone can access the bus */
if (busAddress.getBusType().equals("UNIX")) {
Path path = Paths.get(busAddress.getPath());
if(Util.isWindows()) {
AclFileAttributeView aclAttr = Files.getFileAttributeView(path, AclFileAttributeView.class);
UserPrincipalLookupService upls = path.getFileSystem().getUserPrincipalLookupService();
UserPrincipal user = upls.lookupPrincipalByName("Everyone") /* TODO Tighten this */;
AclEntry.Builder builder = AclEntry.newBuilder();
builder.setPermissions( EnumSet.of(AclEntryPermission.READ_DATA, AclEntryPermission.WRITE_DATA));
builder.setPrincipal(user);
builder.setType(AclEntryType.ALLOW);
aclAttr.setAcl(Collections.singletonList(builder.build()));
}
else {
Files.setPosixFilePermissions(path,
new LinkedHashSet<>(Arrays.asList(PosixFilePermission.values())));
}
} |
|
I thought I'd post the results of a very quick and dirty performance test too (on Windows). Using your and So that's more than 15 times quicker! I'm very happy with this, it's exactly the original reason I took an interest in this ticket. I'm sure sure the OP will be very happy with the cut down dependencies. Thanks for all your time on this change. dbus-java is better for it. Edit: Forgot to mention, I noticed a minor mistake in |
|
Thanks for your feedback. Sounds great that using unix socket will speed up everything on windows as well. I added some changes to allow setting the file owner, group and unix file permissions (obviously only on Unix/Linux systems) of the socket file created by btw. I also fixed the usage of the wrong args index in my sandbox sample, thanks for the hit ;) |
|
I've been using the dbus-java development version with java 16 unix sockets for some time now and haven't noticed any issues. |
|
No schedule yet. Maybe end of the month or early next year. |
|
Version 4.0.0 has been released and should be available in maven central soon. |
|
|
I learned via a downstream bug report, that this library transitively depends on a several JNR libs. Which is fine unless certain configuration problems arise. 😉
When looking at the dependency tree I noticed, that the only reason why dbus-java requires those is for accessing unix sockets. Now that we have native support for unix sockets in Java 16+, I'd ask whether it is possible to replace
jnr-unixsocketand get rid of a whole bunch of dependencies.I know, this would increases the required Java version significantly. But as the current version is mature and stable, I would propose to schedule this for a major release that could live in parallel to the current version. No changes in the public API, so users can switch between versions depending on their needs.
If you think this would be feasible from a maintainers perspective, I would offer our help to implement this.
The text was updated successfully, but these errors were encountered: