Max long is rounded up

[librucha]

Hi.
I tried use jq for processing data from SOLR and found round problem for max long.

input.json

{
	"responseHeader": {
		"status": 0,
		"QTime": 0,
		"params": {
			"q": "_validTo:9223372036854775807",
			"rows": "2147483647",
			"wt": "json"
		}
	},
	"response": {
		"numFound": 147,
		"start": 0,
		"docs": [
			{
				"_id": "a6d40c35-8425-4c8a-8ff4-c8336ba6a70c",
				"_validFrom": 1487240174328,
				"_validTo": 9223372036854775807,
				"name": "COMPANY_NAME"
			},
			{
				"_id": "d247cec6-21a4-42b2-9255-75443af51d5d",
				"_validFrom": 1487240174020,
				"_validTo": 9223372036854775807,
				"name": "CUSTOMER_ID"
			}
		]
	}
}

Command:
jq '.response.docs' input.json

Result:

[
	{
		"_id": "a6d40c35-8425-4c8a-8ff4-c8336ba6a70c",
		"_validFrom": 1487240174328,
		"_validTo": 9223372036854776000,
		"name": "COMPANY_NAME"
	},
	{
		"_id": "d247cec6-21a4-42b2-9255-75443af51d5d",
		"_validFrom": 1487240174020,
		"_validTo": 9223372036854776000,
		"name": "CUSTOMER_ID"
	}
]

As you can see node _validTo is rounded up and overflow the max long number in Java.

[nicowilliams]

jq uses IEEE754 doubles for its internal number representation, which is why this happens.

[nicowilliams]

We may yet fix this. We'll see.

[librucha]

It was realy fast reaction. Thank you. 👍

[wtlangford]

It seems odd to me that it rounds this way, though. A quick test (I yanked jv_dtoa.h/jv_dtoa.c out, used the compile flags out of my Makefile, and called jv_strtod) implies that we should be getting 922337203685477580**8** if IEEE 754 was the only thing involved. Unless, of course, @librucha is running on a strange or interesting system?

Alternatively, the rounding could be a side effect of jv_dump.

And this turns out to be the case!
This test program (compiled with -ljq):

#include <stdlib.h>
#include <stdio.h>
#include <jv.h>

int main() {
	double thedbl = strtod("9223372036854775807", NULL);
	printf("printf:  %f\njv_dump: ", thedbl);
	jv val = jv_number(thedbl);
	jv_dump(val, 0);
	printf("\n");
	return 0;
}

Outputs:

printf:  9223372036854775808.000000
jv_dump: 9223372036854776000

Continuing down the call chain, it seems that it's jvp_dtoa_fmt() that does this. And that code is wholly unintelligible to me, having been made originally by David Gay (of Bell Labs fame)...

Why do we use this instead of something like sprintf?

[librucha]

I'm using Fedora 25

System:    Host: nb01.librucha Kernel: 4.9.10-200.fc25.x86_64 x86_64 (64 bit gcc: 6.3.1)
           Desktop: Cinnamon 3.2.8  dm: lightdm Distro: Fedora release 25 (Twenty Five)
CPU:       Quad core Intel Core i7-2670QM (-HT-MCP-) cache: 6144 KB
           flags: (lm nx sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx) bmips: 17561
           clock speeds: min/max: 800/3100 MHz 1: 1086 MHz 2: 925 MHz 3: 1018 MHz 4: 809 MHz 5: 912 MHz
           6: 958 MHz 7: 1056 MHz 8: 877 MHz

[nicowilliams]

@librucha: See #1327.

[bschlenk]

Are there any known jq ports to languages such as python that support arbitrary sized integers?

[nicowilliams]

@bschlenk: I seem to recall seeing some attempts to port to node/JS and such, but they're hard to find now. https://github.com/search?utf8=%E2%9C%93&q=jq+js+NOT+jquery&type=Repositories&ref=searchresults

There are bindings for Python, but I'm not aware of any ports to Python.

[nicowilliams]

I'm loathe to toss jv_dtoa.c or update it without a lot of input from @stedolan. Though I suppose we could try it and see if we can catalog some differences and decide whether it's OK.

I'm not sure that cases involving numbers that can't be represented exactly in IEEE754 doubles are too exciting, especially as to integers since we can and should add a 64-bit integer sub-type of the number type, which should take care of a large range of reports (though, obviously, anyone trying to deal with integers larger than 64-bit will still be unhappy; we can get at most a 65-bit integer range with low overhead, and we're not likely to have bignum support, though never say never, I guess).

[thapakazi]

waiting for changes to finalise on #1327
damn... I was unable to post tweets because of this 😠

thinking 🤔 of going with string version for now

[Alanscut]

hi @librucha
jq-master can handle the scenario you described normally. you can try it .

root@oss-001: echo '{"_validTo":9223372036854775807}' | jq .
{
  "_validTo": 9223372036854775807
}

[dmitrievav]

MacOS 10.15.7 | jq 1.6

echo '[{"id":1622073659203009936,"name":"output.log"}]' | jq -r .
[
  {
    "id": 1622073659203010000,
    "name": "output.log"
  }
]

[librucha]

On MacOS 11.4 and jq 1.6 is still wrong

echo '{"_validTo":9223372036854775807}' | jq .
{
  "_validTo": 9223372036854776000
}

[rmandvikar]

i'm hitting this too. any eta for the bug fix?

$ echo '{"long":1136492207025876992}' | jq .
{
  "long": 1136492207025877000
}

i'm using windows 10, jq 1.6 jq-win64.exe.

[pkoppstein]

The problem was addressed in a commit dated Oct 21, 2019, as mentioned in the jq FAQ:
https://github.com/stedolan/jq/wiki/FAQ#numbers. This commit has not yet made its way into an official "release" but Windows users can get a up-to-date .exe as per https://github.com/stedolan/jq/wiki/Installation#windows-using-appveyor

Please also note that the developers/maintainers of jq have historically regard this as an enhancement rather than a "bug fix; this probably explains why it has taken so long for the issue to be addressed.

[emanuele6]

jq 1.7 released with the fix. closing