-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding fail-fast feature to fast-serde #519
Adding fail-fast feature to fast-serde #519
Conversation
Codecov ReportAll modified lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #519 +/- ##
============================================
- Coverage 45.77% 45.77% -0.01%
+ Complexity 4439 4436 -3
============================================
Files 403 403
Lines 28070 28070
Branches 4622 4622
============================================
- Hits 12850 12848 -2
Misses 13664 13664
- Partials 1556 1558 +2 ☔ View full report in Codecov by Sentry. |
Hi @FelixGV @gaojieliu |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me, thanks for the change!
...vro-fastserde-tests-common/src/test/java/com/linkedin/avro/fastserde/FastSerdeCacheTest.java
Show resolved
Hide resolved
fastserde/avro-fastserde/src/main/java/com/linkedin/avro/fastserde/FastSerdeCache.java
Show resolved
Hide resolved
That's awesome! |
This PR aims to add "fail-fast" feature to the fastserde.
Normally when we have datum writer:
and it's used in a loop as:
the
datumWriter
under the hood uses standard avro writer until fast-serializer is generated on the background-thread.Currently if such generation fails due to any reason we have just one log entry and
datumWriter
still uses standard avro writer.We found this approach is sometimes too gentle because it hides an issue for a long time. It's easy to miss one WARN log-entry.
That's why we think "fail-fast" approach would be a nice feature. It can be turned on by setting one of system properties
avro.fast.serde.failfast
avro.fast.serde.failfast.supplier
By default it's disabled so that it's backward-compatible.
Commits:
Code cleanup
private Optional<String> compileClassPath;
changed toprivate String compileClassPath
;compileClassPath
extracted to helper method (will be re-used for the new system property)bugfixes
getFastGenericDeserializerAsync
no longer generated StackOverflow (shame on me!)AvroCompatibilityHelper
in fallback-serializers to fix an issue with missing c-tor in Avro 1.4.Generation errors logged on ERROR level.
[fastserde] Adding fail-fast feature.
Name "fail-fast" is a kind of industry standard however in this case it may cause confusion with "fast-serde" - we can use another name for this feature (fail-early / force-fastserde / etc.) or give it opposite meaning (e.g. ignore-generation-errors).