Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LwM2mNodeSenMLDecoder for SenML-JSON with unit tests. #681

Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,327 @@
/*******************************************************************************
* All rights reserved. This program and the accompanying materials
* are made available under the terms of the Eclipse Public License v1.0
* and Eclipse Distribution License v1.0 which accompany this distribution.
*
* The Eclipse Public License is available at
* http://www.eclipse.org/legal/epl-v10.html
* and the Eclipse Distribution License is available at
* http://www.eclipse.org/org/documents/edl-v10.html.
*
* Contributors:
* Boya Zhang - initial API and implementation
*******************************************************************************/
package org.eclipse.leshan.core.node.codec.senml;

import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;
import java.util.Map.Entry;

import org.eclipse.leshan.core.model.LwM2mModel;
import org.eclipse.leshan.core.model.ResourceModel;
import org.eclipse.leshan.core.model.ResourceModel.Type;
import org.eclipse.leshan.core.node.LwM2mMultipleResource;
import org.eclipse.leshan.core.node.LwM2mNode;
import org.eclipse.leshan.core.node.LwM2mObject;
import org.eclipse.leshan.core.node.LwM2mObjectInstance;
import org.eclipse.leshan.core.node.LwM2mPath;
import org.eclipse.leshan.core.node.LwM2mResource;
import org.eclipse.leshan.core.node.LwM2mSingleResource;
import org.eclipse.leshan.core.node.codec.CodecException;
import org.eclipse.leshan.senml.SenMLJson;
import org.eclipse.leshan.senml.SenMLPack;
import org.eclipse.leshan.senml.SenMLRecord;
import org.eclipse.leshan.util.Base64;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class LwM2mNodeSenMLJsonDecoder {

private static final Logger LOG = LoggerFactory.getLogger(LwM2mNodeSenMLJsonDecoder.class);

public static <T extends LwM2mNode> T decode(byte[] content, LwM2mPath path, LwM2mModel model, Class<T> nodeClass)
throws CodecException {
String jsonStrValue = content != null ? new String(content) : "";
SenMLPack pack = SenMLJson.fromJsonSenML(jsonStrValue);
return parseSenMLPack(pack, path, model, nodeClass);
}

@SuppressWarnings("unchecked")
private static <T extends LwM2mNode> T parseSenMLPack(SenMLPack pack, LwM2mPath path, LwM2mModel model,
Class<T> nodeClass) throws CodecException {

LOG.trace("Parsing SenML JSON object for path {}: {}", path, pack);

// Extract baseName
LwM2mPath baseName = extractAndValidateBaseName(pack, path);
if (baseName == null)
baseName = path; // if no base name, use request path as base name
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

About "request path as basename, if basename is absent", I read quickly the specification §7.4.4 JSON, I didn't find anything about that.

It seems this part has changed with the new JSON format. I guess this is to respect more the SenML rfc. (If this is the case, I approve this change)

Anyway, I maybe missed it so if you find anything let me know.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read-Composite is introduced in LwM2M 1.1. Table: 7.4.4.-6 JSON payload in the server request to read manufacturer name, battery level and registration lifetime shows an example of Read-Composite request like below:

[{"n":"/3/0/0"},
{"n":"/3/0/9"},
{"n":"/1/0/1"}]

In response to the above Read-Composite request the client will return a JSON payload as below:
[{"n":"/3/0/0", "vs":"Open Mobile Alliance"},
{"n" :"/3/0/9", "v":95},
{"n":"/1/0/1", "v":86400}]

In this case, the base name is absent, because of resources belong to different object instances, and object instances belong to different object. Do you think SenML JSON/CBOR encoder/decoder shall consider Read-Composite/Write-Composite/Observe-Composite now? Will appreciate if you share you opinion.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect our current API is not really compatible with this.
So I would go to let this aside for now : one step at a time.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(when I say "let this aside", I mean support of Read-Composite/Write-Composite/Observe-Composite)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(when I say "let this aside", I mean support of Read-Composite/Write-Composite/Observe-Composite)

Understood! Is there any plan or on-going contributing activity related with Composite operation now?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no plan about that for now.
About on-going contribution, I just saw this #563 (comment)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After extract base name from record array, I am trying to update value of name field of each of record to full path. Therefore, rest of logic can ignore base name anymore. Do you have any comments on that?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure to understand. I think we should remove extractAndValidateBaseName(pack, path) because we could face cases where there are several basenames ?

look at this https://tools.ietf.org/html/rfc8428#section-5.1.6

   [
     {"bn":"2001:db8::2/","bt":1.320078429e+09,
      "n":"temperature","u":"Cel","v":25.2},
     {"n":"humidity","u":"%RH","v":30},
     {"bn":"2001:db8::1/","n":"temperature","u":"Cel","v":12.3},
     {"n":"humidity","u":"%RH","v":67}
   ]

if we want a more LWM2M one we could imagine something like this :
Reading object /2 (ACL) containing 2 instances

   [
     {"bn":"/2/0/","n":"0","v":3},
     {"n":"1","v":1},
     {"n":"3","v":123}
     {"bn":"/2/1/","n":"0","v":4},
     {"n":"1","v":1},
     {"n":"3","v":124}
   ]

(we could add unit tests like those)

So basename would be now handle in groupRecordsByInstanceId, maybe like this :

    private static Map<Integer, Collection<SenMLRecord>> groupRecordsByInstanceId(Collection<SenMLRecord> records)
            throws CodecException {
        Map<Integer, Collection<SenMLRecord>> result = new HashMap<>();

        String basename = "";
        for (SenMLRecord record : records) {
            // update basename if present
            if (record.getBaseName() != null)
                basename = record.getBaseName();

            // Build resource path
            String path = record.getName() == null ? basename : basename + record.getName();
            LwM2mPath nodePath = new LwM2mPath(path);

            // Validate path
            if (!nodePath.isResourceInstance() && !nodePath.isResource()) {
                throw new CodecException(
                        "Invalid path [%s] for resource, it should be a resource or a resource instance path",
                        nodePath);
            }

            // Get SenML records for this instance
            Collection<SenMLRecord> recordForInstance = result.get(nodePath.getObjectInstanceId());
            if (recordForInstance == null) {
                recordForInstance = new ArrayList<>();
                result.put(nodePath.getObjectInstanceId(), recordForInstance);
            }

            // Add it to the list
            recordForInstance.add(record);
        }

        return result;
    }


// Group JSON entry by instance
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a details, but several time in comments you are using JSON or JSON entry, I feel this would be more appropriate to use SenML or SenML record.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for that, I tried to replace or remove irrelevant comments, but still missed something.

Map<Integer, Collection<SenMLRecord>> recordsByInstanceId = groupRecordsByInstanceId(pack.getRecords(),
baseName);

// Create lwm2m node
LwM2mNode node = null;
if (nodeClass == LwM2mObject.class) {
Collection<LwM2mObjectInstance> instances = new ArrayList<>();
for (Entry<Integer, Collection<SenMLRecord>> entryByInstanceId : recordsByInstanceId.entrySet()) {
Map<Integer, LwM2mResource> resourcesMap = extractLwM2mResources(entryByInstanceId.getValue(), baseName,
model);

instances.add(new LwM2mObjectInstance(entryByInstanceId.getKey(), resourcesMap.values()));
}

node = new LwM2mObject(baseName.getObjectId(), instances);
} else if (nodeClass == LwM2mObjectInstance.class) {
// validate we have resources for only 1 instance
if (recordsByInstanceId.size() != 1)
throw new CodecException("One instance expected in the payload [path:%s]", path);

// Extract resources
Entry<Integer, Collection<SenMLRecord>> instanceEntry = recordsByInstanceId.entrySet().iterator().next();
Map<Integer, LwM2mResource> resourcesMap = extractLwM2mResources(instanceEntry.getValue(), baseName, model);

// Create instance
node = new LwM2mObjectInstance(instanceEntry.getKey(), resourcesMap.values());
} else if (nodeClass == LwM2mResource.class) {
// validate we have resources for only 1 instance
if (recordsByInstanceId.size() > 1)
throw new CodecException("Only one instance expected in the payload [path:%s]", path);

// Extract resources
Map<Integer, LwM2mResource> resourcesMap = extractLwM2mResources(
recordsByInstanceId.values().iterator().next(), baseName, model);

// validate there is only 1 resource
if (resourcesMap.size() != 1)
throw new CodecException("One resource should be present in the payload [path:%s]", path);

node = resourcesMap.values().iterator().next();
} else {
throw new IllegalArgumentException("invalid node class: " + nodeClass);
}

return (T) node;
}

/**
* Extract and validate base name from SenML pack
*
* @param pack
* @param requestPath
* @return
* @throws CodecException
*/
private static LwM2mPath extractAndValidateBaseName(SenMLPack pack, LwM2mPath requestPath) throws CodecException {
String baseName = null;
for (SenMLRecord record : pack.getRecords()) {
if (record.getBaseName() != null && !record.getBaseName().isEmpty()) {
baseName = record.getBaseName();
break;
}
}

// Check baseName is valid
if (baseName != null && !baseName.isEmpty()) {
LwM2mPath bnPath = new LwM2mPath(baseName);

// check returned base name path is under requested path
if (requestPath.getObjectId() != null && bnPath.getObjectId() != null) {
if (!bnPath.getObjectId().equals(requestPath.getObjectId())) {
throw new CodecException("Basename path [%s] does not match requested path [%s].", bnPath,
requestPath);
}
if (requestPath.getObjectInstanceId() != null && bnPath.getObjectInstanceId() != null) {
if (!bnPath.getObjectInstanceId().equals(requestPath.getObjectInstanceId())) {
throw new CodecException("Basename path [%s] does not match requested path [%s].", bnPath,
requestPath);
}
if (requestPath.getResourceId() != null && bnPath.getResourceId() != null) {
if (!bnPath.getResourceId().equals(requestPath.getResourceId())) {
throw new CodecException("Basename path [%s] does not match requested path [%s].", bnPath,
requestPath);
}
}
}
}
return bnPath;
}
return null;
}

/**
* Group all SenML record by instanceId
*
* @param records
* @param baseName
*
* @return a map (instanceId => collection of SenML Record)
*/
private static Map<Integer, Collection<SenMLRecord>> groupRecordsByInstanceId(Collection<SenMLRecord> records,
LwM2mPath baseName) throws CodecException {
Map<Integer, Collection<SenMLRecord>> result = new HashMap<>();

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following comment above, I feel that we don't need baseName parameter anymore.
And we should just use baseName from SenMLRecord.
Look at this part of the rfc to implement it correctly : https://tools.ietf.org/html/rfc8428#section-5.1.6

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above comments, I am trying to update value of name field of each of record to full path, then base name can be ignore, any comments for this change?

for (SenMLRecord record : records) {
// Build resource path
LwM2mPath nodePath = record.getName() == null ? baseName : baseName.append(record.getName());

// Validate path
if (!nodePath.isResourceInstance() && !nodePath.isResource()) {
throw new CodecException(
"Invalid path [%s] for resource, it should be a resource or a resource instance path",
nodePath);
}

// Get SenML records for this instance
Collection<SenMLRecord> recordForInstance = result.get(nodePath.getObjectInstanceId());
if (recordForInstance == null) {
recordForInstance = new ArrayList<>();
result.put(nodePath.getObjectInstanceId(), recordForInstance);
}

// Add it to the list
recordForInstance.add(record);
}

// Create an entry for an empty instance if possible
if (result.isEmpty() && baseName.getObjectInstanceId() != null) {
result.put(baseName.getObjectInstanceId(), new ArrayList<SenMLRecord>());
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there is no more baseName parameter, we will probably remove this part of the code.
And so we should handle the case where result if empy in calling method.


return result;
}

private static Map<Integer, LwM2mResource> extractLwM2mResources(Collection<SenMLRecord> records,
LwM2mPath baseName, LwM2mModel model) throws CodecException {
if (records == null)
return Collections.emptyMap();

// Extract LWM2M resources from JSON resource list
Map<Integer, LwM2mResource> lwM2mResourceMap = new HashMap<>();
Map<LwM2mPath, Map<Integer, SenMLRecord>> multiResourceMap = new HashMap<>();

for (SenMLRecord record : records) {
// Build resource path
LwM2mPath nodePath = record.getName() == null ? baseName : baseName.append(record.getName());

// handle LWM2M resources
if (nodePath.isResourceInstance()) {
// Multi-instance resource
// Store multi-instance resource values in a map
// we will deal with it later
LwM2mPath resourcePath = new LwM2mPath(nodePath.getObjectId(), nodePath.getObjectInstanceId(),
nodePath.getResourceId());
Map<Integer, SenMLRecord> multiResource = multiResourceMap.get(resourcePath);
if (multiResource == null) {
multiResource = new HashMap<>();
multiResourceMap.put(resourcePath, multiResource);
}
SenMLRecord previousResInstance = multiResource.put(nodePath.getResourceInstanceId(), record);
if (previousResInstance != null) {
throw new CodecException(
"2 RESOURCE_INSTANCE nodes (%s,%s) with the same identifier %d for path %s",
previousResInstance, record, nodePath.getResourceInstanceId(), nodePath);
}
} else if (nodePath.isResource()) {
// Single resource
Type expectedType = getResourceType(nodePath, model, record);
Object resourceValue = parseResourceValue(record.getResourceValue(), expectedType, nodePath);
LwM2mResource res = LwM2mSingleResource.newResource(nodePath.getResourceId(), resourceValue,
expectedType);
LwM2mResource previousRes = lwM2mResourceMap.put(nodePath.getResourceId(), res);
if (previousRes != null) {
throw new CodecException("2 RESOURCE nodes (%s,%s) with the same identifier %d for path %s",
previousRes, res, res.getId(), nodePath);
}
} else {
throw new CodecException(
"Invalid path [%s] for resource, it should be a resource or a resource instance path",
nodePath);
}
}

// Handle multiple resource instances.
for (Map.Entry<LwM2mPath, Map<Integer, SenMLRecord>> entry : multiResourceMap.entrySet()) {
LwM2mPath resourcePath = entry.getKey();
Map<Integer, SenMLRecord> entries = entry.getValue();

if (entries != null && !entries.isEmpty()) {
Type expectedType = getResourceType(resourcePath, model, entries.values().iterator().next());
Map<Integer, Object> values = new HashMap<>();
for (Entry<Integer, SenMLRecord> e : entries.entrySet()) {
Integer resourceInstanceId = e.getKey();
values.put(resourceInstanceId,
parseResourceValue(e.getValue().getResourceValue(), expectedType, resourcePath));
}
LwM2mResource resource = LwM2mMultipleResource.newResource(resourcePath.getResourceId(), values,
expectedType);
LwM2mResource previousRes = lwM2mResourceMap.put(resourcePath.getResourceId(), resource);
if (previousRes != null) {
throw new CodecException("2 RESOURCE nodes (%s,%s) with the same identifier %d for path %s",
previousRes, resource, resource.getId(), resourcePath);
}
}
}

// If we found nothing, we try to create an empty multi-instance resource
if (lwM2mResourceMap.isEmpty() && baseName.isResource()) {
ResourceModel resourceModel = model.getResourceModel(baseName.getObjectId(), baseName.getResourceId());
// We create it only if this respect the model
if (resourceModel == null || resourceModel.multiple) {
Type resourceType = getResourceType(baseName, model, null);
lwM2mResourceMap.put(baseName.getResourceId(), LwM2mMultipleResource
.newResource(baseName.getResourceId(), new HashMap<Integer, Object>(), resourceType));
}
}

return lwM2mResourceMap;
}

private static Object parseResourceValue(Object value, Type expectedType, LwM2mPath path) throws CodecException {
LOG.trace("Parse SenML JSON value for path {} and expected type {}: {}", path, expectedType, value);

try {
switch (expectedType) {
case INTEGER:
return ((Number) value).longValue();
case FLOAT:
return ((Number) value).doubleValue();
case BOOLEAN:
return value;
case TIME:
return new Date(((Number) value).longValue() * 1000L);
case OPAQUE:
return Base64.decodeBase64((String) value);
case STRING:
return value;
default:
throw new CodecException("Unsupported type %s for path %s", expectedType, path);
}
} catch (Exception e) {
throw new CodecException(e, "Invalid content [%s] for type %s for path %s", value, expectedType, path);
}
}

public static Type getResourceType(LwM2mPath rscPath, LwM2mModel model, SenMLRecord record) {
// Use model type in priority
ResourceModel rscDesc = model.getResourceModel(rscPath.getObjectId(), rscPath.getResourceId());
if (rscDesc != null && rscDesc.type != null)
return rscDesc.type;

// Then json type
if (record != null) {
Type type = record.getType();
if (type != null)
return type;
}

// Else use String as default
LOG.trace("unknown type for resource use string as default: {}", rscPath);
return Type.STRING;
}
}
Loading