[Jira JQL] Add: support custom JIRA fields and enhance value mapping

[stefanseifert]

Enhancements to JQL Query Runner support:

• Do not skip fields starting with customfield_
• Convert list values to concatenated strings, and skip list items if they are dicts without specific mapping rules. This produces nicer outputs, and fixes errors with excel download when the result contained lists or lists with dicts in it.
• Add support for fieldMapping query property which allows to rename output columns or pick specific members from dict values
• the previous implementation contains some magic special mappings for key, name, watchCount. They are still applied if no field mapping is defined for a column for backwards compatibility.

example JQL:

{
    "fields": "summary,priority,customfield_10672,resolutiondate,fixVersions,watches,labels",
    "jql": "project = PVTC ORDER BY priority DESC, key ASC",
    "queryType": "select",
    "maxResults": 500,
    "fieldMapping": {
        "customfield_10672": {
            "name": "myCustomField"
        },
        "priority": {
            "member": "id"
        },
        "fixVersions": {
            "member": "name",
            "name": "myFixVersion"
        }
    }
}

the three field mappings define:

• rename field customfield_10672 to myCustomField in the result
• pick only a specific member 'id' from the priority field
• pick a specific member 'name' from the fixVersions field and rename it to 'myFixVersion'

[stefanseifert]

i've also provided a PR to update the documentation:
getredash/website#25

[arikfr]

Thank you for adding this! See my comments. If you have the time, it will be great to add some tests for the parsing logic.

Also, I would like to suggest an alternative syntax for fieldMapping:

"fieldMapping": {
   "customfield_10672": "myCustomField",
   "priority.id": "priority",
   "fixVersions.name": "myFixVersion"
}

The main change: instead of having an object to describe the mapping we use nested field name (priority.id) and parse it in the query runner. It makes the parser work a little bit harder, but makes it easier to write queries.

About custom fields - maybe we include only the ones that are in the mapping?

And last thing - 👍 major kudos on including an update to the documentation :-)

[arikfr]

Should be field_mapping (we use snake_case in the Python code).

[arikfr]

What if mapping['member'] not in listItem? Better to use listItem.get or check for this case and raise an explicit error.

[arikfr]

Oh, and I would love to understand how you use the JQL integration in Redash. We can do this in private if you prefer (arik at redash.io).

[stefanseifert]

thanks for the review! please note that this PR are the first lines of python i've ever written, so the code may be not very elegant or not conforming to standards.

i've updated the PR:

• apply snake_case naming
• adapted to the new query syntax as you proposed
• handled some cases where dicts members were accessed without checking of existence first
• added unit tests for field mapping parsing and issue parsing incl. the legacy special conversions

custom fields are only included when the are part of the "fields" string - or if not "fields" string is given at all. by the new mapping rules which never inject the dicts completely no "rubbish" should be included in the result, so in my opinion it's fine to apply no further filtering to custom fields. they do not harm, but users may get puzzled if they are missing.

we use JIRA internally a lot for all sorts of purposes, e.g. organizing projects or purchases. those issue types have custom fields with numbers e.g. costs and we want to use redash to visualize them.

[arikfr]

Thanks! Looking good. Appreciate your cooperation and the tests :)

[akakimeladze]

Hi guys. There's one issue that I stumbled upon for this particular feature. It doesn't allow me to reference a child element in JSON object if it is deeper within a child element itself. For example this custom field:

"customfield_12101": { "id": "15", "name": "Time to first response", "_links": { "self": "https://singular.atlassian.net/rest/servicedeskapi/request/58820/sla/15" }, "completedCycles": [ { "startTime": { "iso8601": "2017-08-31T05:02:18+0200", "jira": "2017-08-31T05:02:18.610+0200", "friendly": "Today 05:02", "epochMillis": 1504148538610 }, "stopTime": { "iso8601": "2017-08-31T09:00:29+0200", "jira": "2017-08-31T09:00:29.915+0200", "friendly": "Today 09:00", "epochMillis": 1504162829915 }, "breached": false, "goalDuration": { "millis": 3600000, "friendly": "1h" }, "elapsedTime": { "millis": 29915, "friendly": "0m" }, "remainingTime": { "millis": 3570085, "friendly": "59m" } } ] }

There is no way for me to get the element by writing:
"fieldMapping": { "fields.customfield_12101.completedCycles.0.breached": "time_to_response" }