v5.5.0 - Feedback wanted on executeQuery

[bigmontz]

Driver.executeQuery

In the version 5.5.0, we introduce a new experimental API to the driver. This is a simplified API for running queries against a database without needing to get into the detail of sessions and transactions.

The very basic usage example:

import neo4j from 'neo4j-driver'

const driver = neo4j.driver('neo4j://localhost', neo4j.auth.basic('neo4j', 'password'))

const { records, summary, keys } = await driver.executeQuery('RETURN 42 AS answer')

console.log('keys:', keys)
for (const record of records) {
  console.log('answer:', record.get('answer'))
}

console.log('result available after:', summary.resultAvailableAfter.toString())

This runs a the cypher query and return a EagerResult object. This object contains the list of records returned, the list of keys present in the records and the result summary. This return object shape is defined by the result transformer neo4j.resultTransfomers.eagerResultTransformer() which can be configured as following:

const { records, summary, keys } = await driver.executeQuery('RETURN 42 AS answer', {}, {
  // this should not change the default behaviour of the method
  resultTransformer: neo4j.resultTransformers.eagerResultTransformer() 
})

What is ResultTransformer

Result transformer is a async function which receives a Result and transform it to a desired output. In plain typescript, this type can be defined as:

type ResultTransformer<T> = (result: Result) => Promise<T>

For the driver user doesn't have to implement their own result transformer, the driver offers two implementations:

1. function neo4j.resultTransformers.eagerResultTransformer(): ResultTransformer<EagerResult> which provides the default result transformed used in driver.executeQuery.
2. function neo4j.resultTransformers.mappedResultTransformer(config: { map, collect }): ResultTransformer<T> which provides an result transformer with methods to map the records and collect the final result.

Using the mappedResultTransformer to improve the example

In the example bellow, we use an mappedResultTransformer to map our record to the final shape used in the for loop.

import neo4j from 'neo4j-driver'

const driver = neo4j.driver('neo4j://localhost', neo4j.auth.basic('neo4j', 'password'))

// This function can be reused in different `executeQuery` calls or direct in any result
const answerResultTransformer = neo4j.resultTransformers.mappedResultTransformer({
  map: (record) => record.get('answer').toString()
})

const { records: answers, summary, keys } = await driver.executeQuery('RETURN 42 AS answer', {}, {
  resultTransformer: answerResultTransformer
})

console.log('keys:', keys)
for (const answer of answers) {
  console.log('answer:', answer)
}

console.log('result available after:', summary.resultAvailableAfter.toString())

Although the code for printing the result was improved, the shape of the returning object is not quite what we need. We actually want a object with the answer, how long it takes for the answer be available and the keys, since we'd like to print the keys.

The collect can be used for defining a function which will receive the mapped records, the summary and the keys. See:

import neo4j from 'neo4j-driver'

const driver = neo4j.driver('neo4j://localhost', neo4j.auth.basic('neo4j', 'password'))

const answerResultTransformer = neo4j.resultTransformers.mappedResultTransformer({
  map: (record) => record.get('answer').toString(),
  collect: (answers, summary, keys) => {
    return { 
      answer: answers[0], 
      resultAvailableAfter: summary.resultAvailableAfter.toString(),
      keys
    }
  }
})

const { answer, resultAvailableAfter, keys } = await driver.executeQuery('RETURN 42 AS answer', {}, {
  resultTransformer: answerResultTransformer
})

console.log('keys:', keys)
console.log('answer:', answer)
console.log('result available after:', resultAvailableAfter)

NOTE: ResultTransformer can also be used in Session.run and Transaction.run.

Under the Hood

This API the common pattern used for run queries to neo4j using the driver. This very first example can be re-written as:

import neo4j from 'neo4j-driver'

const driver = neo4j.driver('neo4j://localhost', neo4j.auth.basic('neo4j', 'password'))

const session = driver.session()

try {
  const {records, summary, keys } = await session.executeWrite(async (tx) => {
    // the lines bellow are equivalent to 
    // return await neo4j.resultTransformers.eagerResultTransformer()(tx.run('RETURN 42 AS answer'))
    const result = tx.run('RETURN 42 AS answer')
    const { records, summary } = await result
    const keys = await result.keys()
    return { records, summary, keys }
  })

  console.log('keys:', keys)
  for (const record of records) {
    console.log('answer:', record.get('answer'))
  }
  console.log('result available after:', summary.resultAvailableAfter.toString())
} finally {
  // do not forget to close the session
  await session.close()
}

Configuration

For configuration see executeQuery api docs and query config api docs

Feedback wanted

This new API is currently marked as experimental.

We're definitely looking for feedback on this feature. Any thoughts you have are gratefully received. Specific questions we would like to ask:

1. Does this api fits well with your usage scenario?
2. Does the exposed result transformers helpful? Do you use for the mappedResultTransformer? Is it any improvement which can be done in its usage?
3. Do you think it will be easy migrate from executeQuery for using sessions and transactions (executeRead/executeWrite) in more complex use cases if you need to?

Let us know and we will correct course in the next releases!

[bigmontz]

The Driver.executeQuery has been released in the Neo4j Javascript Drivers 5.5.0.

[adam-cowley]

I can see a benefit to having many resultTransformer options for common use cases - eg get the first key on the first row MATCH (n) RETURN count(n) AS count.

    driver.executeQuery(cypher, params, {
        resultTransfomer: neo4j.resultTransformers.firstKey('count'),
    })

The first thing that strikes me is that the simplified API is actually more verbose and longer type than the current method. The implementation also adds the complexity of needing to know what a result transformer is, what the methods are and how to import it.

    // New - 260 chars, plus need to know what a result transformer is
    const res1 = driver.executeQuery(cypher, params, {
        resultTransformer: neo4j.resultTransformers.mappedResultTransformer({
            map: (record) => record.toObject(),
            collect: (records) => records,
        }),
    })
    console.log(res1)

    // Old - 208 Chars
    const session = driver.session()
    const res2 = await session.executeRead(
        tx => tx.run(cypher, params)
    )
    console.log(res2.records.map(record => record.toObject()))
    await session.close()

Personally, I would like to see plain functions that can be easily implemented as top-level options with type support.

    // 185 chars
    const res3 = driver.executeQuery(cypher, params, {
        recordTransformer: (record) => record.toObject(), // or mapRecord?
        resultTransformer: (records) => records
    })
    console.log(res3)
    // or recordTransformer -> map, resultTransformer -> collect for 179 chars

It may also help to have more understandable terms. For examplemapXxx rather than xxxTransformer

[bigmontz]

The two code snippets are not doing exact the same thing. When you use the mappedResultTransformer, you are mapping and filtering records while it is being streamed.

The equivalent code will be something like:

const session = driver.session()
const res2 = await session.executeRead(
    tx => new Promise((resolve, reject) => {
        const records = []
        tx.run(cypher, params).subscribe({
           onNext: record => records.push(record.toObject()),
           onCompleted: () => resolve(records),
           onError: reject
        })
    })
)
await session.close()

You are can also use the executeQuery without set any result transformer and use like:

const res1 = await driver.executeQuery(cypher, params)
console.log(res1.records.map(record => record.toObject()))

It might worth to change the ResultTransformer interface from type ResultTransformer<T> = (result: Result) => Promise<T> to

type ResultTransformer<T> = (result: Result) => Promise<T> | MappedResultTransfomerConfig<T>

In this scenario, you can transform record while stream like this:

const res1 = await driver.executeQuery(cypher, params, {
    resultTransformer: {
        map: (record) => record.toObject(),
        collect: (records) => records,
    },
})
console.log(res1)

// OR
const res2 = driver.executeQuery(cypher, params, {
    resultTransformer: {
        map: (record) => record.toObject()
    },
})
console.log(res2.records)

[adam-cowley]

The TypeScript implementation also seems a little problematic as the executeQuery method returns an EagerResult by default, which is one more thing I need to know about. If we keep with the current implementation, would prefer to see the record shape as a generic and have that passed to the EagerResult in return type of the executeQuery method.

executeQuery<T, ResultTransformerOutput = EagerResult<T>>(
  query: Query, 
  parameters?: any, 
  config?: QueryConfig<T>
): Promise<EagerResult<T>>;

This way the user doesn't need to include the extra type by default in order to type check

[bigmontz]

This is a good point, the shape of the record as the first param is quite helpful for handling the result.

[bigmontz]

Unfortunately, exposing the record shape through the executeQuery won't be possible without make the use of the mappedResultTransformer almost unusable and with a lot of types conflict since typescript doesn't support partial type inference.

Examples like bellow will simply doesn't work:

const personList = await driver.executeQuery<Person>("query", params, {
   resultTransformer: neo4j.resultTransformer.mappedResultTransformer({
     map: record => record.toObject(),
     collect: records => records
   })
})

For making it work, it will be need to do:

const personList = await driver.executeQuery<Person, Person[]>("query", params, {
   resultTransformer: neo4j.resultTransformer.mappedResultTransformer<Person, Person, Person[]>({
     map: record => record.toObject(),
     collect: records => records
   })
})

This is worse then what we have today, which is:

const personList = await driver.executeQuery("query", params, {
   resultTransformer: neo4j.resultTransformer.mappedResultTransformer({
     map: (record: Record<Person>) => record.toObject(),
     collect: records => records
   })
})

This is a kind of feature which will be really neat to have, however we lack of language support for doing it right.