Skip to content

Commit

Permalink
Fix floating-point value with leading dot(".001") faied to parse(USDA).
Browse files Browse the repository at this point in the history
Fix parsing quath, quatf and quatd typed Attribute(USDA)
Update fast_float
  • Loading branch information
syoyo committed Jul 30, 2023
1 parent 9061a87 commit f569c1b
Show file tree
Hide file tree
Showing 15 changed files with 1,783 additions and 1,020 deletions.
2 changes: 2 additions & 0 deletions src/ascii-parser-basetype.cc
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,7 @@ int parseInt(const std::string &s, int *out_result) {
}

nonstd::expected<float, std::string> ParseFloat(const std::string &s) {

// Parse with fast_float
float result;
auto ans = fast_float::from_chars(s.data(), s.data() + s.size(), result);
Expand All @@ -154,6 +155,7 @@ nonstd::expected<float, std::string> ParseFloat(const std::string &s) {
}

nonstd::expected<double, std::string> ParseDouble(const std::string &s) {

// Parse with fast_float
double result;
auto ans = fast_float::from_chars(s.data(), s.data() + s.size(), result);
Expand Down
26 changes: 23 additions & 3 deletions src/ascii-parser.cc
Original file line number Diff line number Diff line change
Expand Up @@ -2597,10 +2597,9 @@ bool AsciiParser::LexFloat(std::string *result) {
}
_curr_cursor.col++;

ss << sc;

// sign, '.' or [0-9]
if ((sc == '+') || (sc == '-')) {
ss << sc;
has_sign = true;

char c;
Expand All @@ -2621,9 +2620,14 @@ bool AsciiParser::LexFloat(std::string *result) {

} else if ((sc >= '0') && (sc <= '9')) {
// ok
ss << sc;
} else if (sc == '.') {
// ok
// ok but rescan again in 2.
leading_decimal_dots = true;
if (!Rewind(1)) {
return false;
}
_curr_cursor.col--;
} else {
PUSH_ERROR_AND_RETURN("Sign or `.` or 0-9 expected.");
}
Expand Down Expand Up @@ -2689,6 +2693,7 @@ bool AsciiParser::LexFloat(std::string *result) {
return true;
}


if (Eof()) {
(*result) = ss.str();
return true;
Expand Down Expand Up @@ -3852,6 +3857,21 @@ bool AsciiParser::ParsePrimProps(std::map<std::string, Property> *props, std::ve
&attr)) {
return false;
}
} else if (type_name == value::kQuath) {
if (!ParseBasicPrimAttr<value::quath>(array_qual, primattr_name,
&attr)) {
return false;
}
} else if (type_name == value::kQuatf) {
if (!ParseBasicPrimAttr<value::quatf>(array_qual, primattr_name,
&attr)) {
return false;
}
} else if (type_name == value::kQuatd) {
if (!ParseBasicPrimAttr<value::quatd>(array_qual, primattr_name,
&attr)) {
return false;
}
} else if (type_name == value::kPoint3f) {
if (!ParseBasicPrimAttr<value::point3f>(array_qual, primattr_name,
&attr)) {
Expand Down
23 changes: 23 additions & 0 deletions src/external/fast_float/LICENSE-BOOST
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
Boost Software License - Version 1.0 - August 17th, 2003

Permission is hereby granted, free of charge, to any person or organization
obtaining a copy of the software and accompanying documentation covered by
this license (the "Software") to use, reproduce, display, distribute,
execute, and transmit the Software, and to prepare derivative works of the
Software, and to permit third-parties to whom the Software is furnished to
do so, all subject to the following:

The copyright notices in the Software and this entire statement, including
the above license grant, this restriction and the following disclaimer,
must be included in all copies of the Software, in whole or in part, and
all derivative works of the Software, unless such copies or derivative
works are solely in the form of machine-executable object code generated by
a source language processor.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT
SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE
FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE,
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
DEALINGS IN THE SOFTWARE.
150 changes: 113 additions & 37 deletions src/external/fast_float/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@

## fast_float number parsing library: 4x faster than strtod
[![Fuzzing Status](https://oss-fuzz-build-logs.storage.googleapis.com/badges/fast_float.svg)](https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened&can=1&q=proj:fast_float)
[![VS17-CI](https://github.com/fastfloat/fast_float/actions/workflows/vs17-ci.yml/badge.svg)](https://github.com/fastfloat/fast_float/actions/workflows/vs17-ci.yml)
[![Ubuntu 22.04 CI (GCC 11)](https://github.com/fastfloat/fast_float/actions/workflows/ubuntu22.yml/badge.svg)](https://github.com/fastfloat/fast_float/actions/workflows/ubuntu22.yml)

The fast_float library provides fast header-only implementations for the C++ from_chars
functions for `float` and `double` types. These functions convert ASCII strings representing
Expand All @@ -21,8 +25,8 @@ struct from_chars_result {
```

It parses the character sequence [first,last) for a number. It parses floating-point numbers expecting
a locale-independent format equivalent to the C++17 from_chars function.
The resulting floating-point value is the closest floating-point values (using either float or double),
a locale-independent format equivalent to the C++17 from_chars function.
The resulting floating-point value is the closest floating-point values (using either float or double),
using the "round to even" convention for values that would otherwise fall right in-between two values.
That is, we provide exact parsing according to the IEEE standard.

Expand All @@ -40,7 +44,7 @@ Example:
``` C++
#include "fast_float/fast_float.h"
#include <iostream>

int main() {
const std::string input = "3.1416 xyz ";
double result;
Expand All @@ -53,39 +57,78 @@ int main() {


Like the C++17 standard, the `fast_float::from_chars` functions take an optional last argument of
the type `fast_float::chars_format`. It is a bitset value: we check whether
the type `fast_float::chars_format`. It is a bitset value: we check whether
`fmt & fast_float::chars_format::fixed` and `fmt & fast_float::chars_format::scientific` are set
to determine whether we allow the fixed point and scientific notation respectively.
The default is `fast_float::chars_format::general` which allows both `fixed` and `scientific`.

The library seeks to follow the C++17 (see [20.19.3](http://eel.is/c++draft/charconv.from.chars).(7.1)) specification.
The library seeks to follow the C++17 (see [20.19.3](http://eel.is/c++draft/charconv.from.chars).(7.1)) specification.
* The `from_chars` function does not skip leading white-space characters.
* [A leading `+` sign](https://en.cppreference.com/w/cpp/utility/from_chars) is forbidden.
* It is generally impossible to represent a decimal value exactly as binary floating-point number (`float` and `double` types). We seek the nearest value. We round to an even mantissa when we are in-between two binary floating-point numbers.
* It is generally impossible to represent a decimal value exactly as binary floating-point number (`float` and `double` types). We seek the nearest value. We round to an even mantissa when we are in-between two binary floating-point numbers.

Furthermore, we have the following restrictions:
* We only support `float` and `double` types at this time.
* We only support the decimal format: we do not support hexadecimal strings.
* For values that are either very large or very small (e.g., `1e9999`), we represent it using the infinity or negative infinity value.
* For values that are either very large or very small (e.g., `1e9999`), we represent it using the infinity or negative infinity value and the returned `ec` is set to `std::errc::result_out_of_range`.

We support Visual Studio, macOS, Linux, freeBSD. We support big and little endian. We support 32-bit and 64-bit systems.

We assume that the rounding mode is set to nearest (`std::fegetround() == FE_TONEAREST`).

## C++20: compile-time evaluation (constexpr)

In C++20, you may use `fast_float::from_chars` to parse strings
at compile-time, as in the following example:

```C++
// consteval forces compile-time evaluation of the function in C++20.
consteval double parse(std::string_view input) {
double result;
auto answer = fast_float::from_chars(input.data(), input.data()+input.size(), result);
if(answer.ec != std::errc()) { return -1.0; }
return result;
}

// This function should compile to a function which
// merely returns 3.1415.
constexpr double constexptest() {
return parse("3.1415 input");
}
```
## Non-ASCII Inputs
We also support UTF-16 and UTF-32 inputs, as well as ASCII/UTF-8, as in the following example:
``` C++
#include "fast_float/fast_float.h"
#include <iostream>
int main() {
const std::u16string input = u"3.1416 xyz ";
double result;
auto answer = fast_float::from_chars(input.data(), input.data()+input.size(), result);
if(answer.ec != std::errc()) { std::cerr << "parsing failure\n"; return EXIT_FAILURE; }
std::cout << "parsed the number " << result << std::endl;
return EXIT_SUCCESS;
}
```

## Using commas as decimal separator


The C++ standard stipulate that `from_chars` has to be locale-independent. In
particular, the decimal separator has to be the period (`.`). However,
some users still want to use the `fast_float` library with in a locale-dependent
particular, the decimal separator has to be the period (`.`). However,
some users still want to use the `fast_float` library with in a locale-dependent
manner. Using a separate function called `from_chars_advanced`, we allow the users
to pass a `parse_options` instance which contains a custom decimal separator (e.g.,
to pass a `parse_options` instance which contains a custom decimal separator (e.g.,
the comma). You may use it as follows.

```C++
#include "fast_float/fast_float.h"
#include <iostream>

int main() {
const std::string input = "3,1416 xyz ";
double result;
Expand All @@ -97,28 +140,61 @@ int main() {
}
```

You can parse delimited numbers:
```C++
const std::string input = "234532.3426362,7869234.9823,324562.645";
double result;
auto answer = fast_float::from_chars(input.data(), input.data()+input.size(), result);
if(answer.ec != std::errc()) {
// check error
}
// we have result == 234532.3426362.
if(answer.ptr[0] != ',') {
// unexpected delimiter
}
answer = fast_float::from_chars(answer.ptr + 1, input.data()+input.size(), result);
if(answer.ec != std::errc()) {
// check error
}
// we have result == 7869234.9823.
if(answer.ptr[0] != ',') {
// unexpected delimiter
}
answer = fast_float::from_chars(answer.ptr + 1, input.data()+input.size(), result);
if(answer.ec != std::errc()) {
// check error
}
// we have result == 324562.645.
```
## Reference
- Daniel Lemire, [Number Parsing at a Gigabyte per Second](https://arxiv.org/abs/2101.11408), Software: Pratice and Experience 51 (8), 2021.
## Relation With Other Work
## Other programming languages
The fast_float library is part of:
- [There is an R binding](https://github.com/eddelbuettel/rcppfastfloat) called `rcppfastfloat`.
- [There is a Rust port of the fast_float library](https://github.com/aldanor/fast-float-rust/) called `fast-float-rust`.
- [There is a Java port of the fast_float library](https://github.com/wrandelshofer/FastDoubleParser) called `FastDoubleParser`.
- [There is a C# port of the fast_float library](https://github.com/CarlVerret/csFastFloat) called `csFastFloat`.
- GCC (as of version 12): the `from_chars` function in GCC relies on fast_float.
- [WebKit](https://github.com/WebKit/WebKit), the engine behind Safari (Apple's web browser)
## Relation With Other Work
The fastfloat algorithm is part of the [LLVM standard libraries](https://github.com/llvm/llvm-project/commit/87c016078ad72c46505461e4ff8bfa04819fe7ba).
The fast_float library is part of GCC (as of version 12): the `from_chars` function in GCC relies on fast_float.
There is a [derived implementation part of AdaCore](https://github.com/AdaCore/VSS).
The fastfloat algorithm is part of the [LLVM standard libraries](https://github.com/llvm/llvm-project/commit/87c016078ad72c46505461e4ff8bfa04819fe7ba).
The fast_float library provides a performance similar to that of the [fast_double_parser](https://github.com/lemire/fast_double_parser) library but using an updated algorithm reworked from the ground up, and while offering an API more in line with the expectations of C++ programmers. The fast_double_parser library is part of the [Microsoft LightGBM machine-learning framework](https://github.com/microsoft/LightGBM).
There is a [derived implementation part of AdaCore](https://github.com/AdaCore/VSS).
## References
- Daniel Lemire, [Number Parsing at a Gigabyte per Second](https://arxiv.org/abs/2101.11408), Software: Practice and Experience 51 (8), 2021.
- Noble Mushtak, Daniel Lemire, [Fast Number Parsing Without Fallback](https://arxiv.org/abs/2212.06644), Software: Practice and Experience 53 (7), 2023.
## Other programming languages
- [There is an R binding](https://github.com/eddelbuettel/rcppfastfloat) called `rcppfastfloat`.
- [There is a Rust port of the fast_float library](https://github.com/aldanor/fast-float-rust/) called `fast-float-rust`.
- [There is a Java port of the fast_float library](https://github.com/wrandelshofer/FastDoubleParser) called `FastDoubleParser`. It used for important systems such as [Jackson](https://github.com/FasterXML/jackson-core).
- [There is a C# port of the fast_float library](https://github.com/CarlVerret/csFastFloat) called `csFastFloat`.
## Users
Expand All @@ -132,14 +208,14 @@ It can parse random floating-point numbers at a speed of 1 GB/s on some systems.
<img src="http://lemire.me/blog/wp-content/uploads/2020/11/fastfloat_speed.png" width="400">
```
$ ./build/benchmarks/benchmark
$ ./build/benchmarks/benchmark
# parsing random integers in the range [0,1)
volume = 2.09808 MB
netlib : 271.18 MB/s (+/- 1.2 %) 12.93 Mfloat/s
doubleconversion : 225.35 MB/s (+/- 1.2 %) 10.74 Mfloat/s
strtod : 190.94 MB/s (+/- 1.6 %) 9.10 Mfloat/s
abseil : 430.45 MB/s (+/- 2.2 %) 20.52 Mfloat/s
fastfloat : 1042.38 MB/s (+/- 9.9 %) 49.68 Mfloat/s
volume = 2.09808 MB
netlib : 271.18 MB/s (+/- 1.2 %) 12.93 Mfloat/s
doubleconversion : 225.35 MB/s (+/- 1.2 %) 10.74 Mfloat/s
strtod : 190.94 MB/s (+/- 1.6 %) 9.10 Mfloat/s
abseil : 430.45 MB/s (+/- 2.2 %) 20.52 Mfloat/s
fastfloat : 1042.38 MB/s (+/- 9.9 %) 49.68 Mfloat/s
```
See https://github.com/lemire/simple_fastfloat_benchmark for our benchmarking code.
Expand Down Expand Up @@ -180,36 +256,36 @@ You should change the `GIT_TAG` line so that you recover the version you wish to

## Using as single header

The script `script/amalgamate.py` may be used to generate a single header
The script `script/amalgamate.py` may be used to generate a single header
version of the library if so desired.
Just run the script from the root directory of this repository.
Just run the script from the root directory of this repository.
You can customize the license type and output file if desired as described in
the command line help.

You may directly download automatically generated single-header files:

https://github.com/fastfloat/fast_float/releases/download/v3.4.0/fast_float.h
https://github.com/fastfloat/fast_float/releases/download/v5.2.0/fast_float.h

## Credit

Though this work is inspired by many different people, this work benefited especially from exchanges with
Michael Eisel, who motivated the original research with his key insights, and with Nigel Tao who provided
Though this work is inspired by many different people, this work benefited especially from exchanges with
Michael Eisel, who motivated the original research with his key insights, and with Nigel Tao who provided
invaluable feedback. Rémy Oudompheng first implemented a fast path we use in the case of long digits.

The library includes code adapted from Google Wuffs (written by Nigel Tao) which was originally published
The library includes code adapted from Google Wuffs (written by Nigel Tao) which was originally published
under the Apache 2.0 license.

## License

<sup>
Licensed under either of <a href="LICENSE-APACHE">Apache License, Version
2.0</a> or <a href="LICENSE-MIT">MIT license</a> at your option.
2.0</a> or <a href="LICENSE-MIT">MIT license</a> or <a href="LICENSE-BOOST">BOOST license</a> .
</sup>

<br>

<sub>
Unless you explicitly state otherwise, any contribution intentionally submitted
for inclusion in this repository by you, as defined in the Apache-2.0 license,
shall be dual licensed as above, without any additional terms or conditions.
shall be triple licensed as above, without any additional terms or conditions.
</sub>
Loading

0 comments on commit f569c1b

Please sign in to comment.