Externalized SDK Helpers

[paulr34]

Problem:

SDK helpers are maintained within the ZAP source code, requiring SDK developers to write their code within the ZAP repository when they need a new ZAP API.

Initially, the SDK helpers were external. However, they were moved into the ZAP repository due to maintenance and design concerns:

1. To utilize ZAP helpers, the SDK needed access to the source code as it "required" files within the ZAP repository.
2. Any changes in ZAP, such as modifications to function names, involved corresponding adjustments throughout the entire SDK.

Design:

To prevent SDK developers from being constrained in their use of the ZAP API, a new design pattern has been devised to enable and empower them throughout their interaction with ZAP.

1. SDK developers have the flexibility to create their own externalized helpers and specify a relative path in gen-template.json for ZAP to consume. During binary creation, ZAP includes this file at runtime and can call it, passing in necessary data. This approach allows SDK developers to externalize a helper without requiring access to ZAP source code, eliminating the need for explicit "require/include/import" statements.
2. With the new design, ZAP can transmit data into external helpers, eliminating the necessity for SDK developers to include query files, etc. The data passed by ZAP would be an API object maintained within the ZAP repository. This object contains API definitions that abstract ZAP methods. Consequently, if an API developer needs to call a query and its name changes, the SDK developer remains unaffected as the API name within ZAP does not change. See snippet below.

In the provided code snippet, the SDK receives an API object containing "attributes" which invokes a query. In the event of a query name change, the ZAP developer updates the API, relieving the SDK developer from making any adjustments.

/**
 * Returns an array of attributes for a given cluster.
 * The cluster input is required to come from a script-api in this module.
 *
 * @param {*} context
 * @param {*} endpoint
 * @param {*} cluster
 */
function attributes(context, endpoint, cluster) {
  return queryEndpoint.selectEndpointClusterAttributes(
    context.db,
    cluster.clusterId,
    cluster.side,
    endpoint.endpointTypeRef
  )
}

[tecimovic]

Few comments...

• I would put the references to helper into gen-template.json, not the zcl.json.
• You are saying something about "including these files in snapshot" above. I don't understand that. THese files ARE on SDK and STAY on SDK. They are not part of binary distribution of zap. Binary zap loads them in at runtime, same way as running from source does.
• The "api" is being passed to these files. But we also need to define "entry point". Each of these files needs to have an entry point that will be a function or a symbol zap will access to retrieve information on helpers implemented inside this file.
• We need to have very detailed error messages, so any problems with the code inside these helpers will be clearly logged as such.
• The API, obviously, needs to be 100% backwards compatible at all times. But you can always add functions.

[paulr34]

• Changed to gen-templates.json
• Updated proposal to more accurately describe what happens
• Will add a new section about "entry point"
• agree
• agree

[tecimovic]

APIs we need to support:

• retrieve ALL available clusters/attributes/commands/events.
• retrieve SELECTED clusters/attributes/commands/events.
• retrieve endpoint configuration
• provide APIs to load in yaml/json files and access that data to act on.

[andy31415]

Do we have stability guarantees on API data returned?
Ideally we would enforce this via some unit tests:

• we want to clearly describe what is being fetched when asking for cluster/attribute/command/events
• want to ensure that selecting ALL gives us the same data as SELECTED (this seemed to vary last time I tried to update things for zap)

[paulr34]

@andy31415 my interpretation of ALL vs SELECTED is this

ALL means all of the Clusters/Attributes etc in the XML

SELECTED means all of the clusters/attributes etc in the .ZAP file

am I misunderstanding?

[bzbarsky-apple]

@paulr34 You are misunderstanding.

OK, you get "all" the clusters. What are you actually getting? It's not a list of cluster IDs; it's a list of JS objects that have some properties. What properties? Nothing documents that. So you have to do a bunch of reading ZAP code or guess-and-check to find out what things are available on those objects.

And the ALL vs SELECTED bits have different properties on them. And sometimes have the same information under different property names depending on whether you did ALL or SELECTED.

@andy31415's ask is that:

1. Which things you get is very clearly documented.
2. The same set of things/properties is available for ALL vs SELECTED (as much as possible)
3. If something/some-property is available for both ALL and SELECTED, it needs to be under the same name.

[paulr34]

Okay thank you for explaining @bzbarsky-apple

[s-mcclain]

@bzbarsky-apple Do you know if there are any plans for documenting these properties? Having just started going through trying to understand how this works, it's been a very cumbersome process for trying to create anything custom.