Add support for Dolphin/EnOcean PTM 215B

[pfink]

I have some battery-less wall switches based on Dolphin/EnOcean PTM 215B that I'd like to integrate (technical manual can be found here).

My setup:

• Raspberry Pi 3B
• openHAB 2.3
• Raspbian Stretch
• Bluez 5.43
• Generic BLE 4.0 USB adapter

Some information that I found out so far:

• Bluetooth version unclear for me (but maybe not relevant)? I guess BLE 4.0 or BLE 4.2.
• Currently, the wall switches don't get discovered at all. Neither via this binding nor via bluetoothctl (also not when trying to do radio-based commissioning according the docs, chapter 5.3). Other BLE devices are found without any issues.
• According to the docs, the wall switches send data telegrams on each press/release event on all three advertising channels. The data channels aren't used at all.
• Authentication should happen via AES-128 CCM (counter with CBC-MAC), the security key can be read manually via QR Code on the switch, via NFC or with radio-based/BLE commissioning. But radio-based/BLE commissioning is maybe disabled, depending on the manufacturer where the end product was bought. EnOcean's default is enabled. If disabled, it can be reenabled via NFC. Anyhow, for a start I'd prefer to manually read the security key via QR code and put it statically to a textual configuration, as it seems to be the simplest approach to start with.

@vkolotov: I'm a ESH/openHAB developer, but I'm quite new to the BLE protocol. If you could give me some hints and point me to the right direction, I may could come up with a PR.

[vkolotov]

Hi @pfink, thanks for that.

• BT version is not important.
• Since your switches are battery-less, this means that they only advertise data when you actuate them. So I'd suggest to try to continuously switch it on and off until you see in the OH inbox. Otherwise, you could try to add them manually. Let me know how it goes, if not working for you I'll try to explain how to do it manually.
• Re " advertising channels" that's good, this means that you don't have to be connected to them. This is expected, good design.
• Not sure why would you need any authentication though as you've mentioned that they advertise data without establishing any secure connection. Try what I suggested above. The binding supports some basic authentication methods, e.g. auth via pin code, but that should not be required in your instance.

I'm happy to help you, let's start with getting them into inbox first.

[pfink]

OMG... Before my first post, I troubleshooted two days to get the buttons discovered including that I tried to manually add them, everything unsuccessful. The solution was as dumb as easy: I did not press the button often enough :( As you suggested, I flooded openHAB now with very often button presses (> 10) and now they're discovered..... Thank you very much!

Via online channel and by putting online timeout to 1 sec, I'm now already able to detect whether a button was pressed at all. Next step would be to distinguish which button was pressed. My preferred solution would be to implement/offer a system.rawrocker channel.

Of course, I'm also open to simpler/easier solutions for a start :)

[vkolotov]

Good stuff! Very interesting to see battery-less devices to work with the binding :) What channels have been discovered? Can you post a screenshot?

[pfink]

Sure! This is a complete list:

• bluetooth:ble:xxxx:online
• bluetooth:ble:xxxx:connected
• bluetooth:ble:xxxx:authenticated
• bluetooth:ble:xxxx:connection-control
• bluetooth:ble:xxxx:connected-adapter
• bluetooth:ble:xxxx:rssi
• bluetooth:ble:xxxx:tx-power
• bluetooth:ble:xxxx:estimated-distance
• bluetooth:ble:xxxx:adapter
• bluetooth:ble:xxxx:location
• bluetooth:ble:xxxx:last-updated

Device Name is just the MAC address.

[vkolotov]

how about "unknown" channels?

That's how you do stuff with BT binding. The binding should discover some channels (BT characteristics) automatically. So that you can use them. I assume there will be several channels per each button. If it is like that, you only need to extend the binding with some XML files to get the channels to be recognised automatically, I assume they will be radioboxes/switches in OH.

See here: https://github.com/sputnikdev/eclipse-smarthome-bluetooth-binding/blob/master/gatt-extensions.md

[vkolotov]

Once you've changed "GATT parsing strategy" you will need to wait until the binding reloads and then refresh PaperUI page with your things to see new channels. IMPORTANT: they will appear only after the binding receives data, e.g. you will need to switch on/off your buttons to discover all unknown channels.

[vkolotov]

This would be an example of "unknown" characteristics (MiFlora device):

000071fe-0000-1000-8000-00805f9b34fb is an unknown characteristic.

[pfink]

I removed the Thing again, activated recognition of unknown channels as per your advice, and discovered + added it again. Where should I see these unknown channels? I don't find anything in PaperUI. I also checked /var/lib/openhab2/jsondb/org.eclipse.smarthome.core.thing.Thing.json, but there seems to be nothing more than I already posted.

[vkolotov]

Try to switch on / off more, continuously switch it for 30-60 secs let's say :) . Wait some time, reload browser tab with PaperUI to see new channels.

[vkolotov]

How many buttons does your switch have?

[vkolotov]

Right, I've skimmed the doc you have attached.

The way how your switch transmit data is via so called "manufacturer data". The binding supports it, a new channel (single) should be created for it. That channel contains the following (as per the doc):

Sequence Counter (4 byte)
The Sequence Counter is a continuously incrementing counter used for security
processing. It is initialized to 0 at the time of production and incremented for each
telegram (data telegram or commissioning telegram) sent.
 Switch Status (1 byte)
The Switch Status field reports the button action. The encoding of this field is described
in chapter 4.7.
 Optional Data (0 / 1 / 2 or 4 byte)
PTM 215B provides the option to transmit additional user-defined data within each
data telegram. This data can be used to identify user-specific properties.
The length of the Optional Data field is defined in the Configuration register as described
in chapter 6.7.7.
 Security Signature (4 byte)
The Security Signature is used to authenticate PTM 215B radio telegrams as described
in chapter 4.8

we are interested in Switch Status (1 byte).

[vkolotov]

I'll need to double check if the binding supports the "manufacturer data", should be.

[pfink]

I have rockers with 2 buttons and also those with 4 buttons. But I think there is the very same radio module behind them. Unfortunately, no additional channels get discovered. I guess this could be because the binding does not know the security key. I think the whole security and commissioning process described in the docs would make no sense if everyone can just publicly read the data ;-)

[pfink]

(see section 4.8 in the docs)

[vkolotov]

Right, I've checked it :) It is not implementing manufacturer data thingy, but it would be easy to add as this is very-very similar to an existing functionality (advertised service data).

Check this out:

eclipse-smarthome-bluetooth-binding/src/main/java/org/sputnikdev/esh/binding/bluetooth/handler/BluetoothDeviceHandler.java

Line 157 in 3bf8b0f

public void serviceDataChanged(Map<URL, byte[]> serviceData) {

That one implements "advertised service data" which is very similar to "manufacturer service data", in fact the later should be easier as there will be only 1 data point comparing to many for "service data".

Here is what needs to be implemented: https://github.com/sputnikdev/bluetooth-manager/blob/d239d59a5368d05d06f59dc62faeb3e355716f23/src/main/java/org/sputnikdev/bluetooth/manager/BluetoothSmartDeviceListener.java#L70

Let me know if it makes sense for you?

[vkolotov]

that argument manufacturerData will contain exactly this:

Sequence Counter (4 byte)
The Sequence Counter is a continuously incrementing counter used for security
processing. It is initialized to 0 at the time of production and incremented for each
telegram (data telegram or commissioning telegram) sent.
 Switch Status (1 byte)
The Switch Status field reports the button action. The encoding of this field is described
in chapter 4.7.
 Optional Data (0 / 1 / 2 or 4 byte)
PTM 215B provides the option to transmit additional user-defined data within each
data telegram. This data can be used to identify user-specific properties.
The length of the Optional Data field is defined in the Configuration register as described
in chapter 6.7.7.
 Security Signature (4 byte)
The Security Signature is used to authenticate PTM 215B radio telegrams as described
in chapter 4.8

[vkolotov]

Where key of that map would be 0x03DA, and the value is the rest of it:

[pfink]

I'll do a detailed review within the next days, thank you so far! In case you have more hints, don't hesitate to append them here :) The longer I read the more I recognize that I probably misunderstood the docs. Now I think that I don't need the security key to read the data, I just need it to verify that it comes from my device and not from an attacker that imitates my device (but for my use cases, I think it's not necessary to protect against such attack scenarios).

On the weekend, I'll setup a proper development environment and try to get it running.

[vkolotov]

Yep, the doc does not say that it is absolutely required to implement security checks. For basic usage, it should be more than enough. The telegram message contains signature part, that one can be used to validate if the message is sent by the original device by using keys that was exchanged initially. Not sure if we need it though :), however it is nice to have.

The receiver performs the same signature calculation based on sequence counter, source
address and the remaining telegram data of the received telegram using the security key it
received from PTM 215B during commissioning.
The receiver then compares the signature reported as part of the telegram with the signature
it has calculated. If these two signatures match then the following statements are
true:
 Sender (PTM 215B) and receiver use the same security key
 The message content (address, sequence counter, data) has not been modified

Effectively this ^^ is not required, the data should be unencrypted and available to read as far as I can see.

[pfink]

Yes, of course, I also see it definitely as a TODO for the binding. At the very latest when you use BLE devices to open your front door, you should have the posiibility to verify the origin of the telegrams ;-)

[vkolotov]

Have a look at the architecture design of the binding: https://github.com/sputnikdev/eclipse-smarthome-bluetooth-binding/blob/master/implementation-notes.md
This should give you a bit of overview.

[vkolotov]

Hey @pfink, I've added a basic support for advertised manufacturer data into the binding, have a look here:

eclipse-smarthome-bluetooth-binding/src/main/java/org/sputnikdev/esh/binding/bluetooth/handler/BluetoothDeviceHandler.java

Line 175 in 8bff492

public void manufacturerDataChanged(Map<Short, byte[]> manufacturerData) {

This creates binary channels like so:

0x59 here is an id of a manufacturer that device reported:

89	0x0059	Nordic Semiconductor ASA

Could you please grub this snapshot and try your switch with it?

https://oss.sonatype.org/content/repositories/snapshots/org/sputnikdev/org.eclipse.smarthome.binding.bluetooth/1.1.7-SNAPSHOT/org.eclipse.smarthome.binding.bluetooth-1.1.7-20181130.032312-3.jar

Obviously you will need to try to switch on/off multiple times like you did before in order to make your device to send some data so that the binding will pick it up.

[vkolotov]

Once you've discovered a new channel for your switch, we will need to think about how could we extend the binding so that it can parse binary telegram message into a set of switches.

Just so you know, at the moment the binding does not have ANY specific logic for any device, everything is made so that it is dynamically creates channels by using BT manager and GATT parser.

I'd like to continue this approach to make the binding as abstract as possible, however looking at the format of telegram message it seems to me it won't be possible as this format is not compatible with GATT specs due to that each field (switch state) depends also on the action type field. Looks like we need to design a mechanism for the binding that allows us to extend it. First thing that comes to my mind is to use osqi DI to load some custom handlers for unknown/custom characteristics / service data / manufacturer data. For example, once BT device handler comes across an unknown entity, it will try to find a handler in OSGi registry by using a unique ID (in our case if would be UUID 00000059-... ) and load it if found.

What do you think?

[pfink]

I've added a basic support for advertised manufacturer data into the binding

Thank you very much!

Could you please grub this snapshot and try your switch with it?

Sure! In this version, one additional String channel named 000003da-0000-0000-0000-000000000000 is discovered. Anyhow, this channel does never send/contain any data, disregarding how often I click the buttons.

Looks like we need to design a mechanism for the binding that allows us to extend it.

Absolutely!

For example, once BT device handler comes across an unknown entity, it will try to find a handler in OSGi registry by using a unique ID (in our case if would be UUID 00000059-... ) and load it if found.

I think it would be better not to let the main binding decide upon which criteria things are distributed to extensions. Probably it makes more sense that all extensions are subscribed to everything and decide theirselves if something that the BLE binding discovers is relevant for them. This provides more flexibility implementing an extension.

[pfink]

Another question: It seems that there is an issue with reliability. With zwave and HomeMatic, I'm used to at least 98% reliability. For the Dolphin/EnOcean PTM 215B, it's 85-95% when the switch is 2.5 meters away from the adapter. On 8 meters distance, it already drops to 75-85% measured on the real operation as a wall switch. Of course, this is extremely bad for a wall switch.

Under lab/debugging conditions, the reliability is much higher, I'd say > 95%. That's because often (but not always) the first click after the device was not active for a longer time does not work, while nearly all subsequent clicks are recognized correctly. I observed a very similar behavior when I use Flic Buttons together with my Flic Button Binding, so I wouldn't consider this issue related to a specific device. Also, it does not seem to be related to Bluez as Bluez is not used for the Flic buttons. Additionally, it does not seem to depend on the BT adapter, at least I don't notice any changes when switching BT adapters. Do you have any idea about this or any recommendations to increase reliability? I already tried to:

• Bluez: set controller mode to BLE: ControllerMode = le
• Bluez: set FastConnectable = true
• Binding: Increase Bluetooth devices update rate to 1s.

... but without any success / improvement in reliability, I believe.

[pfink]

Last but not least one more thing: With which bluez version would you recommend to setup my development environment? My Raspberry is working with 5.43. I'm tending to set it up based on Ubuntu Cosmic which provides the most recent version (5.50) which has significant changes compared to 5.43 (especially Mesh support).

[vkolotov]

Hi @pfink,

Good idea re giving the ability to choose if an extension can do something.

Re reliability, how do you calculate it if you don't even get the data right? :) That new channel, is it empty?

I think I know why you'd get a low reliability rate. Here is it why. I assume you are using a generic transport (not bluegiga transport)? Bluez framework is a terrific piece of junk, furthermore the generic transport talks to Bluez through TinyB library and DBus, this also adds a bit of complexity and less stability. Having said that, there is an issue with Bluez/Dbus/Tinyb (I have not figured out which component is contributing the most) when an object that is responsible for a device gets stale, it just does not report any failure just hangs like there was not any issue. So I have implemented a workaround for this case, it is very basic thing, the BM just checks how long ago was any communication from/to the object and if it exceeds a timeout, it initiates a reset for that object (releases it and gets a new "native" object). Can't remember the timeout, but it should be around 2 mins or so (maybe even less). That's why you probably loose the very first event from the device, because it tries to re-initialize a native object. This is to be confirmed though.

BTW, in general the blugiga transport is much more stable, however you will still get that "workaround" mechanism with it. I believe we need to come up with a setting that would control whether that workaround is on/off.

[vkolotov]

I'd recommend to switch to BLuegiga :) but I'd also recommend to stay with 4.43 as it proved to be the most stable.

[pfink]

Okay, ordered the BLED112 dongle now. It's always good for debugging to have the possibility to compare. Then I'll probably setup devenv based on 4.50 to have the newest and hottest shit which will probably not work, and after doing that I can use the BlueGiga for the serious stuff :D

[pfink]

reliability, how do you calculate it if you don't even get the data right? :)

As I said in the beginning, currently I'm using the online channel and an online timeout of 1s to just toggle the lights. With that setup, I have a "pilot" switch in my living room which I click from time to time. Then I capture if light got actually on or not.

That new channel, is it empty?

What do you mean with empty?

[vkolotov]

But I don't really understand why your workaround is necessary at all? Or what does it really work around? From my perspective, it's "by design" that BLE devices only send data if absolutely required.

Believe or not, but your case is an exception, normally BT devices are always online and advertise something. As I mentioned above, TinyB (I suspect) native objects gets stale, irrespectively of a type of the device. Hence there is a workaround.

[vkolotov]

Please grub this: https://oss.sonatype.org/content/repositories/snapshots/org/sputnikdev/org.eclipse.smarthome.binding.bluetooth/1.1.7-SNAPSHOT/org.eclipse.smarthome.binding.bluetooth-1.1.7-20181203.223729-4.jar

[vkolotov]

There is a project started by @xrucka to get rid of TinyB layer and talk to DBus directly: https://github.com/xrucka/bluetooth-manager-dbus.

I could not find enough time to test it, it was working for me mostly, but there were a couple of issues still... Would be great if we could ditch TinyB...

[pfink]

normally BT devices are always online and advertise

BT devices (without BLE) -> yes
BLE devices -> I'd say, sometimes, sometimes no

Hence there is a workaround.

But workaround for what? As I said, I think it's not a bug, it's feature! So what would happen if you just don't dispose things as long as they're added to openHAB? What would stop working then?

[vkolotov]

The bug is when you get an instance of a BT object from TinyB library, subscribe for some events, it works for some time (you get events) but then for some reason (not always though, it might work for days) it becomes stale and the object no longer notify you (even if the actual device is online). Hence the workaround to check when an "interaction" was last time and if it exceeded a timeout, destroy the object and acquire new one.

[pfink]

Ahh, okay. Now I understand :)

[vkolotov]

Let me know if the build above fixes the issue, I do not have time to test it now, I'm having a course now till the end of this week.

[pfink]

It doesn't fix the issue. It seems that it gets filled once, but then it's not updated anymore.

[xrucka]

@pfink @vkolotov I pretty much rewrote it from the scratch, however have not published it yet (there is still a lot of prooving-ground catch-all code involved), I'll push it to the main repo once I get home today.

[xrucka]

@pfink : if you want to try it:
xrucka/bluetooth-manager-dbus branch rewrite_all + xrucka/eclipse-smarthome-bluetooth-binding-dbus-transport rewrite_all

Should work with bluez-5.48+, emulating the battery service. It's not however updated against Vlad's most recent changes, so some things might not work out of the box.

[pfink]

Tried BlueGiga with the online channel meanwhile. First test results are worse than tinyB in terms of reliability, it seems like it suffers even more from the "first click not recognized" issue. Will gather more data on the weekend.

EDIT: This time, reducing Bluetooth devices update rate seems to have a quite significant effect. Will continue observing...

[vkolotov]

I'll try to fins some time this weekend to implement that setting which turns off the workaround for things.

[pfink]

@vkolotov: If you find no time, pls tell me, then I can implement it :) But I'd highly recommend you to set a license for this project before, otherwise you'll run into a legal mess that you can't get out without asking all contributors for permission...

[pfink]

FTR: I'm currently testing with BlueGiga and an own build where I completely deactivated the reset of stale devices. First results are quite good. Is there a reason why the "stale devices" workaround is enabled by default for the BlueGiga adapter as well? Doesn't it generally effect reliability negatively, e.g. if telegrams arrive exactly at the moment when the device is reset?

[vkolotov]

Hi @pfink, the reason is simple. Transport implementations are supposed to be as simple as possible, focusing on providing an abstract layer between BM and hardware, hence less threads/logics. I'd suggest to keep that logic there in BM for now and implement a setting for device governors so that we can switch it off (a setting per bt device in OH).

I'm hoping that when we Dbus transport is done and stable, we will get rid of that logic completely.

[vkolotov]

But I'd highly recommend you to set a license for this project before, otherwise you'll run into a legal mess that you can't get out without asking all contributors for permission...

what would be your recommendations? All bits and pieces are licensed with Apache v2.

[pfink]

what would be your recommendations?

As licensing is a quite complex topic, I carved out the answer to #74 :)

[pfink]

I'm hoping that when we Dbus transport is done and stable, we will get rid of that logic completely.

Okay. I'll start testing bluetooth-manager-dbus as soon as I completed the reliability test of my current setup. This will take probably take at least one more week, because I have to test under everday life conditions.

[vkolotov]

Great, thank you, let us know how it goes.

[pfink]

Reliability of BlueGiga with deactivated stale device removals is very good :) Out of 60 clicks in the last days, just a single one failed (with online channel).

@vkolotov: Anyhow, ManufacturerData still do not arrive at the item.
@xrucka: Tested with dbus, but the buttons do not get discovered. Log output:

2018-12-21 22:01:25.709 [INFO ] [.manager.transport.dbus.BluezFactory] - Found running bluetooth daemon, connecting & populating...
2018-12-21 22:01:30.709 [DEBUG] [.manager.transport.dbus.BluezFactory] - Discovered devices:
2018-12-21 22:01:30.709 [DEBUG] [.manager.transport.dbus.BluezFactory] - Discovered adapters:

(repeats every few seconds)

I tested it with Bluez 5.50.

Now I'll start to test tinyB with deactivated stale device removals as well.

[vkolotov]

Great news. Keeps up posed please.

Re manufacture data, I believe there is a bug somewhere. I might have some time next week to work on it, it should be something obvious...

[pfink]

Findings regarding manufacturer data so far:

• This code is not called at all
• This function is called when channel is created

[pfink]

More findings:

• With BlueGiga transport binding, the buttons get discovered as generic bluetooth devices, not as BLE devices. To get it to work, they have to be added manually with thing type ble
• Issue does not always occur. When doing it in the following order, it works fine:
  1. Remove Thing (if present)
  2. Restart openHAB
  3. Add Thing
  4. Press buttons to get the channel added
  5. Link item to a channel
• Anyhow, if this order is not respected, often it does not work. Issue can be safely reproduced by
  1. Add Thing
  2. Press buttons to get the channel added
  3. Restart openHAB
  4. Link item to a channel (can also be done before the restart, it does not matter)
• Technically, what happens is the following:
  • ServiceHandler only listens to changes if the channel is attached to the item
  • This is the method which activates listening to events. It uses the private property channelHandlers. But, e.g. after openHAB restart, this variable does not contain the ServiceHandler for the dynamic channels. It just contains the static channels which get initialized during instantiation. So, the method tries to attach the ServiceHandler, but it fails because it cannot find it.
  • This error does always occur if the channel is already present when it gets discovered first. This is because in other cases GattChannelHandler does the channel registration (but it skips it, when the channel is already present).

[pfink]

I did setup PR #75 now which fixes the issue. Anyhow, I'm not sure if this is a proper fix or just a workaround. @vkolotov: Would be cool if you could take a look on it :)

[pfink]

Good news: With the bugfix and some Binary parsing with Jython / JSR223, I'm now able to obtain which button was pressed / released 👍

[pfink]

FTR, Textual Configuration works like this:

Thing bluetooth:ble:E215000053B1 "My EnOcean PTM 215B Button" {
    Channels:
        Type characteristic--------readable-000003da-0000-0000-0000-000000000000-null: 3DA-3DA "Binary Button Data"
}

[pfink]

FTR, this is a Python / JSR223 script which parses the binary data and fills other items with human-readable String's that tell what actually happened:

from org.slf4j import LoggerFactory
import re

scriptExtension.importPreset("RuleSupport")
scriptExtension.importPreset("RuleSimple")

class MyRule(SimpleRule):
    def __init__(self, rawItemName, targetItemName):
        self.rawItemName = rawItemName
        self.targetItemName = targetItemName

        self.triggers = [
            TriggerBuilder.create()
                    .withId(rawItemName + "-changed")
                    .withTypeUID("core.ItemStateUpdateTrigger")
                    .withConfiguration(
                        Configuration({
                            "itemName": self.rawItemName
                        })).build()
        ]
        
    def execute(self, module, input):
        state = str(ir.getItem(self.rawItemName).getState())
        num_of_bits = 8
        switch_status_hex = state.replace("[", "").replace("]", "").split(", ")[4]
        switch_status_bin = bin(int(switch_status_hex, 16))[2:].zfill(num_of_bits)
        buttons = {
            3: "B1",
            4: "B0",
            5: "A1",
            6: "A0",
        }

        for button in buttons.keys():
            if switch_status_bin[button] == "1":
                button_activated = buttons[button]                

        operation = "PRESSED" if switch_status_bin[7] == "1" else "RELEASED"

        LoggerFactory.getLogger("de.pfink.PTM215BParser").debug("Button Operation: " + button_activated + " " + operation)
        events.postUpdate(self.targetItemName, button_activated + " " + operation)


items = ir.getItem("PTM215B_Button_Events").getAllMembers()

for item in items:    
    targetItemName = item.getName()
    rawItemName = targetItemName + "_Raw"
    automationManager.addRule(MyRule(rawItemName, targetItemName))

Sample Item Configuration:

Group PTM215B_Button_Events
String Button_Bedroom_Raw  "Button_Bedroom_Raw [%s]"    { channel="bluetooth:ble:E215000053D1:3DA-3DA" }
String Button_Bedroom     "Button_Bedroom [%s]"    (PTM215B_Button_Events)

This is how it works: For all items that are member of group PTM215B_Button_Events,it tries to find a corresponding item with the same name and the suffix _Raw. Then, a rule to that item is attached and listens to the incoming binary data. If the rule is trigerred, it parses the data and writes human-readable strings into the item which is member of PTM215B_Button_Events.