migrated persistence services from openhab1-addons

[kaikreuzer]

Depends on openhab/openhab-core#1387

Signed-off-by: Kai Kreuzer kai@openhab.org

[openhab-bot]

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/how-to-develop-a-new-persistence-addon/73105/7

[ssalonen]

PR for DynamoDB: kaikreuzer#4

[kaikreuzer]

@davidgraeff Question to you as the MQTT expert: The MQTT persistence is linked to the MQTT 1.x binding. Is there already any similar functionality in 2.x? Are people using it at all / is it still useful?
Will we need to adapt the persistence service to the new MQTT support?

[davidgraeff]

Is there already any similar functionality in 2.x?

No not yet.

Are people using it at all / is it still useful?

People are not using it currently. There was not a single complain with the big OH 2.4 MQTT introduction that the mqtt1 persistence broke for anyone.

Will we need to adapt the persistence service to the new MQTT support?

I'm not entirely sure what this persistence is supposed to do to be honest. A 5 lines rule can already push every item change to specific MQTT topics and I want to add OH<-->OH eventbus synchronisation via MQTT as well. I really don't see a reason for this extension?!

[kaikreuzer]

I agree that this isn't a very good match for a persistence service - so let's drop it here and we can still re-introduce it, if somebody comes up with a good use case for it.

[kaikreuzer]

@derekadams: Trying to migrate the Sitewhere persistence service, it deemed to me that this code is quite old and might not be working with the new Sitewhere 2.0 service anymore - the used MQTT communication rather seems to be replaced by a REST API, and the referenced tutorial at https://sitewhere.io/integration/openhab.html meanwhile results in a 404. Instead of putting much effort into it, I'll rather remove it here. If you believe that it is worthwhile to update it and keep it available for users, please let us know and help on getting it up-to-date again!

[kaikreuzer]

Putting this on hold for the moment, see openhab/openhab-core#680 (comment).

[derekadams]

@kaikreuzer We are planning to do a new persistence service for the SiteWhere 2.0 architecture since some of the interaction patterns have changed. Thanks for taking a look at the existing code. You're correct that we leverage the updated REST APIs to do much of the work now rather than sending the payloads via MQTT. We may support both options, but will know more once we dive back into the persistence service code. Thanks!

[kaikreuzer]

Thanks @derekadams, I'll let you know once this is merged (after the 2.5 release) and I'll remove the old Sitewhere service, so that we can add the new one then!

[JueBag]

I observed an issue concerning rrd4j ( filed here ).
.rrd files are created when performing a REST call although the items are not persisted.
Not sure if that is an issue or a feature.

[openhab-bot]

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/poll-which-oh1-x-addons-do-you-use/79121/1

[openhab-bot]

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/problem-with-new-addon-and-dependencies-packaging/85658/2

[kaikreuzer]

@ssalonen, @lewie and others: Now that those services are migrated and available in the 3.0-SNAPSHOT distro, we absolutely count on your to check whether your persistence services are functional - see my comment above (#5275 (comment)). Would be great, if you could go for this asap to make 3.0 a true replacement for 2.x!

[lewie]

@kaikreuzer,
very cool, an important step!

My detailed test scripts are written in Javascript and as far as I know there is currently no rule binding that can handle them for openHAB 3. Or is there? Now I tested the general installation and basic things like reading, writing and historical functions. The only thing is the CallType, which did not work, but it is outdated anyway?!

I have tested now, all JDBC Binding variants.

Tested persistence JDBC add-ons:

Title	Sign	Driver/Installation	read/write
DynamoDB	dynamodb	untested	untested
InfluxSB	influxdb	untested	untested
JDBC H2	jdbc-h2	ok	ok
JDBC HSQLDB	jdbc-hsqldb	ok	ok
JDBC MariaDB	jdbc-mariadb	ok	ok
JDBC MySQL	jdbc-mysql	ok	ok
JDBC PostgreSQL	jdbc-postgresql	ok	ok
JDBC SQLite	jdbc-sqlite	ok	ok
JPA	jpa	untested	untested
MapDB	mapdb	untested	untested
MongoDB	mongodb	untested	untested
MySQL	mysql	untested	untested

So, all tests I have done, turned out as expected, as in openHAB 2, in fact everything worked! 👍 💯 points!
Thank you very much, Kai for porting this!

One more remark. The second mysql binding (also JDBC) is redundant. At that time I built the JDBC binding for derby, h2, hsqldb, mariadb, mysql, postgres, sqlite, this was based on the original mysql binding. It was done just to reduce the redundancy of the same code for all JDBC based Drivers for databases. Even the database structure is fully compatible. Since the second obsolete mysql binding has no advantages, I suggest to finally let it die in openHAB 3 and recommend the JDBC-MySQL binding to users. What do you think about this saving of redundancy to maintain the code?

[kaikreuzer]

Great, thank you very much for testing, the result is much better than I had hoped!

written in Javascript and as far as I know there is currently no rule binding that can handle them for openHAB 3

Good question, I am actually not fully up to date on the status. But I am just working on removing the old rule engine and making the NGRE be installed by default and I think once we are there, it should not take long to have JS scripting back as well.

The only thing is the CallType, which did not work, but it is outdated anyway?!

It is correct that the CallType does not exist anymore. But the CallItem is still there, it now only uses the StringListType internally. So it would be important to test this item type as well (and adapt the code to handle the StringListType correctly, which I indeed haven't done).

What do you think about this saving of redundancy to maintain the code?

Yes, if the JDBC-MySQL service works as well as the other, I absolutely agree that we should remove the specific one! I'll create a PR immediately...

[kaikreuzer]

... done: #7306

[ssalonen]

It is correct that the CallType does not exist anymore. But the CallItem is still there, it now only uses the StringListType internally. So it would be important to test this item type as well (and adapt the code to handle the StringListType correctly, which I indeed haven't done).

The dynamodb integration tests failed with the CallItems due to this due to sorting differences, which is again due to changed serialization format (dynamodb persistence uses state.tostring). But this made me think..

Do we need to handle this in persistence bindings ? Otherwise old data is not readable.

Are there other breaking changes in State.toString with other item types?

[kaikreuzer]

Do we need to handle this in persistence bindings ? Otherwise old data is not readable.

Pfew, good question. I actually doubt that anybody ever wants to read old CallItem states back, so I would say, we can document this as a breaking change and be fine with that. It is really a very small edge case and imho not worth to carry migration code in the add-on with us.

Are there other breaking changes in State.toString with other item types?

No, not that I am aware of any.

[lewie]

@kaikreuzer,
I am really happy, CallType works with string as value without errors! The data is stored as chars in the databases anyway. 👍

Great, thank you very much for testing, the result is much better than I had hoped!

...not just you!

...it should not take long to have JS scripting back as well.

I can hardly wait!

Yes, if the JDBC-MySQL service works as well as the other, I absolutely agree that we should remove the specific one! I'll create a PR immediately...

Thanks, I think it's a good time for the deletion! And yes, there is also a point in the documentation about this: migration-from-mysql-to-jdbc-persistence-services

Fragmentarily, I have followed the discussions the last Month. But now first time I have dealt with the current state of openHAB 3, great work, compliments to you all!
I am extremely excited about the new parts like the speed of the surface, that is contemporary!

[ssalonen]

@kaikreuzer @lewie how to configure the persistence addon in 3.0? Finally got the IDE resolved and the demo app launched but I do not see any obvious way to configure the addon. There is no runtime/conf/services folder anymore.

Trying to test out the dynamodb with the full setup.

[kaikreuzer]

There is no runtime/conf/services folder anymore.

There has never been. the folder is conf/services and it is unchanged from openHAB 2. So the configuration is just done as before, no changes to that.

[digitaldan]

configuration is just done as before, no changes to that.

Trying use our new UI in a config file free use case made me realize we do not have the ability to configure persistence services through our API's, is this correct? Do you think this will be needed for 3.0 ?

[ssalonen]

I will try to place the file to old place and report back.

I think we used to have PaperUi UI for dynamodb persistence configuration but not anymore with 3.0 admin UI?

EDIT: I have trouble enabling persistence with openHAB3.0 :(

This is what I have configured for the demo app:

$ cat ~/src/openhab-master/git/openhab-distro/launch/app/runtime/conf/services/dynamodb.cfg                                                                               
service.pid="org.openhab.dynamodb"
accessKey=AKIAIOSFODNN7EXAMPLE
#secretKey=3+AAAAABBBbbbCCCCCCdddddd+7mnbIOLH
region=eu-west-1
tablePrefix=oh3-

I get no log entries at all (log4j configured with DEBUG root logger). I would expect error due to missing key secretKey

The shell does not seem to have the usual bundle:list command available so I'm not sure how to debug whether everything is picked up correctly...

Is there admin UI for configuring the persistence addon similar to PaperUI? DynamoDB should have the description for the UI (config.xml)

[ghys]

Is there admin UI for configuring the persistence addon similar to PaperUI?

What does it look like in Paper UI? I have never seen it.

[ssalonen]

@ghys
This is how it looks like:

[ghys]

So that's a service. It should then appear under "other services" in the main settings menu.

[ssalonen]

thanks @ghys. Unfortunately I do not see it there, only "Rule voice interpreter" and "Basic UI"

I have added the dynamodb persistence bundle in Run requirements of the demo app, nothing else.

As mentioned earlier, I'm not sure whether the bundle is loaded or not, since the shell does not seem to have the usual commands to diagnose running bundles...

[ghys]

I've done the same, installed the DynamoDB add-on, and the configurable service should appear in the call to /rest/services but I don't see it...

I'd be interesting to check if it's the case in 2.5.x so we can pinpoint what the actual issue is.

[ghys]

Gave it a try on the latest 2.5.5 snapshot, and the service does indeed appear:

So there's something wrong in 3.0 wrt to these services, when it's fixed it will appear in the UI.

[wborn]

The ESH-INF dirs should be renamed to OH-INF and they should be in src/main/resources:

$ find . | grep ESH-INF 
./bundles/org.openhab.persistence.influxdb/src/main/resources/ESH-INF
./bundles/org.openhab.persistence.influxdb/src/main/resources/ESH-INF/config
./bundles/org.openhab.persistence.influxdb/src/main/resources/ESH-INF/config/config.xml
./bundles/org.openhab.persistence.dynamodb/ESH-INF
./bundles/org.openhab.persistence.dynamodb/ESH-INF/config
./bundles/org.openhab.persistence.dynamodb/ESH-INF/config/config.xml

I'll test if that fixes it and create a PR for it.

[kaikreuzer]

Oops, seems if I had missed those when migrating, sorry...
Thanks for fixing it. @wborn!

[ssalonen]

Thank you all. Can confirm that UI & file works, success with persistence store and query.

Posted #7598 for some small improvements (mainly related to java8 -> java11)

[cweitkamp]

@kaikreuzer Can you briefly if there was a special reason for adding the JDBC version properties in this file? It created confusion in #9810 (comment).

Can we remove them? Or add them in the in the feature file of the bundle?

[kaikreuzer]

I don't recall, sorry. If it works without it, we can remove them.

[cweitkamp]

The code compiles successfully when I remove the properties from this file. I did not test installing the features.