Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FluentD ingest #17

Merged
merged 15 commits into from
Apr 28, 2021
Merged

FluentD ingest #17

merged 15 commits into from
Apr 28, 2021

Conversation

sfc-gh-mpilman
Copy link
Owner

No description provided.

sfc-gh-ljoswiak
sfc-gh-ljoswiak requested changes Apr 27, 2021
View reviewed changes
Copy link
Collaborator

@sfc-gh-ljoswiak sfc-gh-ljoswiak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I understand the design here, but to recap:

  • There are different Ingestor types, ie FluentDIngestor or NoneIngestor, which expose a function to handle ingestion of some sample.
  • The ingestor implementation handles all sending of serialized samples.
  • There is a singleton class ProfilerConfigT which holds configuration set through the CLI, and also holds an instance of the ingestion class.
  • This configuration interface is also exposed through the special key space, where the user can change the ingestor by committing to the right key.
  • The profiler loop itself moves into the ProfilerImpl struct which now uses boost signals to change the frequency.

I left a few other comments for small changes.

fdbclient/ActorLineageProfiler.cpp Show resolved Hide resolved
fdbclient/SpecialKeySpace.actor.cpp Show resolved Hide resolved
fdbserver/fdbserver.actor.cpp Outdated Show resolved Hide resolved
okteto.yml Outdated Show resolved Hide resolved
.stignore Outdated Show resolved Hide resolved
fdbclient/ActorLineageProfiler.cpp Show resolved Hide resolved
fdbclient/ActorLineageProfiler.h Outdated Show resolved Hide resolved
fdbclient/FluentDSampleIngestor.cpp Outdated Show resolved Hide resolved
@sfc-gh-ljoswiak sfc-gh-ljoswiak merged commit 1c62a92 into features/actor-lineage Apr 28, 2021
sfc-gh-mpilman pushed a commit that referenced this pull request Nov 2, 2021
@sfc-gh-jslocum
Merge pull request #17 from sfc-gh-jslocum/bw_priorities
3eb49d3 
adding priorities to blob worker and fixing monitoring in blob manager
sfc-gh-mpilman pushed a commit that referenced this pull request Nov 29, 2022
@sfc-gh-anoyes @ammolitor
Fix an asan error
927cd18 
Previously we did not null out "attributes" in `Span& Span::operator=(Span&& o);`, but we destroyed the arena owning memory referenced by "attributes". Fix that by nulling out "attributes", and rewrite it in a way that's (hopefully) less error-prone.

ASAN diagnostic:
```
==24==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
=================================================================
==24==ERROR: AddressSanitizer: heap-use-after-free on address 0x60300033fa06 at pc 0x0000026eadcf bp 0x7ffca646fe50 sp 0x7ffca646f618
READ of size 9 at 0x60300033fa06 thread T0
    #0 0x26eadce in __asan_memmove /tmp/llvm-project/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:30:3
    #1 0xd43db76 in __copy<const unsigned char, unsigned char> /usr/local/bin/../include/c++/v1/__algorithm/copy.h:59:9
    #2 0xd43db76 in copy<const unsigned char *, unsigned char *> /usr/local/bin/../include/c++/v1/__algorithm/copy.h:72:13
    #3 0xd43db76 in write_bytes /mnt/ephemeral/anoyes/foundationdb/fdbrpc/include/fdbrpc/Msgpack.h:45:3
    #4 0xd43db76 in serialize_string(unsigned char const*, int, MsgpackBuffer&) /mnt/ephemeral/anoyes/foundationdb/fdbrpc/include/fdbrpc/Msgpack.h:109:6
    #5 0xd43d208 in void serialize_map<SmallVectorRef<KeyValueRef, 1> >(SmallVectorRef<KeyValueRef, 1> const&, MsgpackBuffer&) /mnt/ephemeral/anoyes/foundationdb/fdbrpc/include/fdbrpc/Msgpack.h:154:3
    #6 0xd42e2a2 in (anonymous namespace)::UDPTracer::serialize_span(Span const&, MsgpackBuffer&) /mnt/ephemeral/anoyes/foundationdb/fdbclient/Tracing.actor.cpp:157:3
    #7 0xd42c8f4 in (anonymous namespace)::FastUDPTracer::trace(Span const&) /mnt/ephemeral/anoyes/foundationdb/fdbclient/Tracing.actor.cpp:301:3
    #8 0xd41dfe2 in Span::operator=(Span&&) /mnt/ephemeral/anoyes/foundationdb/fdbclient/Tracing.actor.cpp:355:13
    #9 0xc3eb122 in (anonymous namespace)::ReadVersionBatcherActorState<(anonymous namespace)::ReadVersionBatcherActor>::a_body1loopBody1cont1(int) /mnt/ephemeral/anoyes/foundationdb/fdbclient/NativeAPI.actor.cpp:7180:9
    #10 0xc3e386d in a_body1loopBody1when2 /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbclient/NativeAPI.actor.g.cpp:32322:15
    #11 0xc3e386d in a_callback_fire /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbclient/NativeAPI.actor.g.cpp:32434:4
    #12 0xc3e386d in ActorCallback<(anonymous namespace)::ReadVersionBatcherActor, 1, Void>::fire(Void const&) /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/flow.h:1316:34
    #13 0x278af3f in void SAV<Void>::send<Void>(Void&&) /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/flow.h:655:23
    #14 0xdd2dc36 in send<Void> /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/flow.h:901:8
    #15 0xdd2dc36 in Sim2::execTask(Sim2::PromiseTask&) /mnt/ephemeral/anoyes/foundationdb/fdbrpc/sim2.actor.cpp:2319:15
    #16 0xdd2cf0e in Sim2::runLoop(Sim2*) /mnt/ephemeral/anoyes/foundationdb/fdbrpc/sim2.actor.cpp:1279:11
    #17 0x7ebdf2e in main /mnt/ephemeral/anoyes/foundationdb/fdbserver/fdbserver.actor.cpp:2276:17
    #18 0x7f62c7be6554 in __libc_start_main (/lib64/libc.so.6+0x22554)
    #19 0x266f028 in _start (/mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/bin/fdbserver+0x266f028)

0x60300033fa06 is located 6 bytes inside of 32-byte region [0x60300033fa00,0x60300033fa20)
freed by thread T0 here:
    #0 0x26eb352 in free /tmp/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:111:3
    #1 0xdd90118 in ArenaBlock::destroy() /mnt/ephemeral/anoyes/foundationdb/flow/Arena.cpp:466:6
    #2 0xdd8f0b0 in delref /mnt/ephemeral/anoyes/foundationdb/flow/Arena.cpp:173:3
    #3 0xdd8f0b0 in delref<ArenaBlock> /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/FastRef.h:95:7
    #4 0xdd8f0b0 in operator= /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/FastRef.h:147:5
    #5 0xdd8f0b0 in Arena::operator=(Arena&&) /mnt/ephemeral/anoyes/foundationdb/flow/Arena.cpp:119:36
    #6 0xd41dfed in Span::operator=(Span&&) /mnt/ephemeral/anoyes/foundationdb/fdbclient/Tracing.actor.cpp:357:8
    #7 0xc3eb122 in (anonymous namespace)::ReadVersionBatcherActorState<(anonymous namespace)::ReadVersionBatcherActor>::a_body1loopBody1cont1(int) /mnt/ephemeral/anoyes/foundationdb/fdbclient/NativeAPI.actor.cpp:7180:9
    #8 0xc3e386d in a_body1loopBody1when2 /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbclient/NativeAPI.actor.g.cpp:32322:15
    #9 0xc3e386d in a_callback_fire /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbclient/NativeAPI.actor.g.cpp:32434:4
    #10 0xc3e386d in ActorCallback<(anonymous namespace)::ReadVersionBatcherActor, 1, Void>::fire(Void const&) /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/flow.h:1316:34
    #11 0x278af3f in void SAV<Void>::send<Void>(Void&&) /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/flow.h:655:23
    #12 0xdd2dc36 in send<Void> /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/flow.h:901:8
    #13 0xdd2dc36 in Sim2::execTask(Sim2::PromiseTask&) /mnt/ephemeral/anoyes/foundationdb/fdbrpc/sim2.actor.cpp:2319:15
    #14 0xdd2cf0e in Sim2::runLoop(Sim2*) /mnt/ephemeral/anoyes/foundationdb/fdbrpc/sim2.actor.cpp:1279:11
    #15 0x7ebdf2e in main /mnt/ephemeral/anoyes/foundationdb/fdbserver/fdbserver.actor.cpp:2276:17
    #16 0x7f62c7be6554 in __libc_start_main (/lib64/libc.so.6+0x22554)

previously allocated by thread T0 here:
    #0 0x26ebeb2 in aligned_alloc /tmp/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:176:3
    #1 0xdd8e5e0 in ArenaBlock::create(int, Reference<ArenaBlock>&) /mnt/ephemeral/anoyes/foundationdb/flow/Arena.cpp:339:21
    #2 0xdd91139 in ArenaBlock::allocate(Reference<ArenaBlock>&, int) /mnt/ephemeral/anoyes/foundationdb/flow/Arena.cpp:322:7
    #3 0x2bbf49b in operator new[] /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/Arena.h:206:9
    #4 0x2bbf49b in StringRef /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/Arena.h:446:54
    #5 0x2bbf49b in Span::Span(SpanContext const&, Location const&, SpanContext const&, std::initializer_list<SpanContext> const&) /mnt/ephemeral/anoyes/foundationdb/fdbclient/include/fdbclient/Tracing.h:141:33
    #6 0x2bbef25 in Span::Span(Location const&, SpanContext const&, std::initializer_list<SpanContext> const&) /mnt/ephemeral/anoyes/foundationdb/fdbclient/include/fdbclient/Tracing.h:148:6
    #7 0xc1bd941 in Span /mnt/ephemeral/anoyes/foundationdb/fdbclient/include/fdbclient/Tracing.h:160:44
    #8 0xc1bd941 in ReadVersionBatcherActorState /mnt/ephemeral/anoyes/foundationdb/fdbclient/NativeAPI.actor.cpp:7125:6
    #9 0xc1bd941 in ReadVersionBatcherActor /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbclient/NativeAPI.actor.g.cpp:32621:6
    #10 0xc1bd941 in readVersionBatcher(DatabaseContext* const&, FutureStream<DatabaseContext::VersionRequest> const&, TransactionPriority const&, unsigned int const&) /mnt/ephemeral/anoyes/foundationdb/fdbclient/NativeAPI.actor.cpp:7098:26
    #11 0xc1b316a in Transaction::getReadVersion(unsigned int) /mnt/ephemeral/anoyes/foundationdb/fdbclient/NativeAPI.actor.cpp:7366:8
    #12 0xc19055f in getReadVersion /mnt/ephemeral/anoyes/foundationdb/fdbclient/include/fdbclient/NativeAPI.actor.h:315:44
    #13 0xc19055f in Transaction::get(Standalone<StringRef> const&, Snapshot) /mnt/ephemeral/anoyes/foundationdb/fdbclient/NativeAPI.actor.cpp:5314:13
    #14 0x5eca2f1 in a_body1 /mnt/ephemeral/anoyes/foundationdb/fdbserver/MoveKeys.actor.cpp:190:55
    #15 0x5eca2f1 in ReadMoveKeysLockActor /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbserver/MoveKeys.actor.g.cpp:1001:9
    #16 0x5eca2f1 in readMoveKeysLock(Transaction* const&, MoveKeysLock* const&) /mnt/ephemeral/anoyes/foundationdb/fdbserver/MoveKeys.actor.cpp:188:26
    #17 0x5ee590d in a_body1loopBody1 /mnt/ephemeral/anoyes/foundationdb/fdbserver/MoveKeys.actor.cpp:228:39
    #18 0x5ee590d in (anonymous namespace)::TakeMoveKeysLockActorState<(anonymous namespace)::TakeMoveKeysLockActor>::a_body1loopHead1(int) /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbserver/MoveKeys.actor.g.cpp:1421:49
    #19 0x5ecb801 in a_body1 /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbserver/MoveKeys.actor.g.cpp:1400:16
    #20 0x5ecb801 in TakeMoveKeysLockActor /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbserver/MoveKeys.actor.g.cpp:1807:9
    #21 0x5ecb801 in takeMoveKeysLock(Database const&, UID const&) /mnt/ephemeral/anoyes/foundationdb/fdbserver/MoveKeys.actor.cpp:216:34
    #22 0x542a421 in DDTxnProcessor::takeMoveKeysLock(UID const&) const /mnt/ephemeral/anoyes/foundationdb/fdbserver/DDTxnProcessor.actor.cpp:620:9
    #23 0x562aa75 in takeMoveKeysLock /mnt/ephemeral/anoyes/foundationdb/fdbserver/DataDistribution.actor.cpp:330:69
    #24 0x562aa75 in DataDistributor::InitActorState<DataDistributor::InitActor>::a_body1loopBody1(int) /mnt/ephemeral/anoyes/foundationdb/fdbserver/DataDistribution.actor.cpp:370:44
    #25 0x562a0ba in a_body1loopHead1 /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbserver/DataDistribution.actor.g.cpp:1116:49
    #26 0x562a0ba in DataDistributor::InitActorState<DataDistributor::InitActor>::a_body1(int) /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbserver/DataDistribution.actor.g.cpp:1083:16
    #27 0x55088c0 in InitActor /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbserver/DataDistribution.actor.g.cpp:1715:9
    #28 0x55088c0 in init /mnt/ephemeral/anoyes/foundationdb/fdbserver/DataDistribution.actor.cpp:367:26
    #29 0x55088c0 in a_body1loopBody1 /mnt/ephemeral/anoyes/foundationdb/fdbserver/DataDistribution.actor.cpp:611:39
    #30 0x55088c0 in (anonymous namespace)::DataDistributionActorState<(anonymous namespace)::DataDistributionActor>::a_body1loopHead1(int) /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbserver/DataDistribution.actor.g.cpp:2772:49
    #31 0x54e9cf1 in a_body1 /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbserver/DataDistribution.actor.g.cpp:2751:16
    #32 0x54e9cf1 in DataDistributionActor /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbserver/DataDistribution.actor.g.cpp:3976:9
    #33 0x54e9cf1 in dataDistribution(Reference<DataDistributor> const&, PromiseStream<GetMetricsListRequest> const&) /mnt/ephemeral/anoyes/foundationdb/fdbserver/DataDistribution.actor.cpp:586:26
    #34 0x54f8758 in a_body1 /mnt/ephemeral/anoyes/foundationdb/fdbserver/DataDistribution.actor.cpp:1533:39
    #35 0x54f8758 in DataDistributorActor /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbserver/DataDistribution.actor.g.cpp:11473:9
    #36 0x54f8758 in dataDistributor(DataDistributorInterface const&, Reference<AsyncVar<ServerDBInfo> const> const&) /mnt/ephemeral/anoyes/foundationdb/fdbserver/DataDistribution.actor.cpp:1517:26
    #37 0x88f3f88 in (anonymous namespace)::WorkerServerActorState<(anonymous namespace)::WorkerServerActor>::a_body1cont10loopBody1when6(InitializeDataDistributorRequest&&, int) /mnt/ephemeral/anoyes/foundationdb/fdbserver/worker.actor.cpp:2219:42
    #38 0x8871c15 in a_callback_fire /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbserver/worker.actor.g.cpp:13169:4
    #39 0x8871c15 in ActorSingleCallback<(anonymous namespace)::WorkerServerActor, 6, InitializeDataDistributorRequest>::fire(InitializeDataDistributorRequest&&) /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/flow.h:1338:34
    #40 0x33e13c0 in send<InitializeDataDistributorRequest> /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/flow.h:1004:29
    #41 0x33e13c0 in NetNotifiedQueue<InitializeDataDistributorRequest, false>::receive(ArenaObjectReader&) /mnt/ephemeral/anoyes/foundationdb/fdbrpc/include/fdbrpc/fdbrpc.h:702:10
    #42 0xd91b6c6 in (anonymous namespace)::DeliverActorState<(anonymous namespace)::DeliverActor>::a_body1cont1(int) /mnt/ephemeral/anoyes/foundationdb/fdbrpc/FlowTransport.actor.cpp:1042:15
    #43 0xd91a37d in a_body1cont2 /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbrpc/FlowTransport.actor.g.cpp:4333:15
    #44 0xd91a37d in a_body1when1 /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbrpc/FlowTransport.actor.g.cpp:4345:15
    #45 0xd91a37d in a_callback_fire /mnt/ephemeral/anoyes/build/foundationdb.linux.clang.asan.x86_64/fdbrpc/FlowTransport.actor.g.cpp:4366:4
    #46 0xd91a37d in ActorCallback<(anonymous namespace)::DeliverActor, 0, Void>::fire(Void const&) /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/flow.h:1316:34
    #47 0x278af3f in void SAV<Void>::send<Void>(Void&&) /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/flow.h:655:23
    #48 0xdd2dc36 in send<Void> /mnt/ephemeral/anoyes/foundationdb/flow/include/flow/flow.h:901:8
    #49 0xdd2dc36 in Sim2::execTask(Sim2::PromiseTask&) /mnt/ephemeral/anoyes/foundationdb/fdbrpc/sim2.actor.cpp:2319:15
    #50 0xdd2cf0e in Sim2::runLoop(Sim2*) /mnt/ephemeral/anoyes/foundationdb/fdbrpc/sim2.actor.cpp:1279:11
    #51 0x7ebdf2e in main /mnt/ephemeral/anoyes/foundationdb/fdbserver/fdbserver.actor.cpp:2276:17
    #52 0x7f62c7be6554 in __libc_start_main (/lib64/libc.so.6+0x22554)

SUMMARY: AddressSanitizer: heap-use-after-free /tmp/llvm-project/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:30:3 in __asan_memmove
Shadow bytes around the buggy address:
  0x0c068005fef0: fa fa fd fd fd fd fa fa fd fd fd fd fa fa fd fd
  0x0c068005ff00: fd fd fa fa fd fd fd fd fa fa fd fd fd fa fa fa
  0x0c068005ff10: fd fd fd fd fa fa fd fd fd fd fa fa fd fd fd fd
  0x0c068005ff20: fa fa fd fd fd fa fa fa fd fd fd fd fa fa fd fd
  0x0c068005ff30: fd fd fa fa fd fd fd fd fa fa fd fd fd fa fa fa
=>0x0c068005ff40:[fd]fd fd fd fa fa fd fd fd fd fa fa fd fd fd fd
  0x0c068005ff50: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068005ff60: fd fa fa fa fd fd fd fa fa fa fd fd fd fd fa fa
  0x0c068005ff70: fd fd fd fd fa fa fd fd fd fa fa fa 00 00 00 00
  0x0c068005ff80: fa fa fd fd fd fd fa fa 00 00 00 00 fa fa fd fd
  0x0c068005ff90: fd fd fa fa 00 00 00 00 fa fa 00 00 00 00 fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sfc-gh-ljoswiak sfc-gh-ljoswiak sfc-gh-ljoswiak approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@sfc-gh-mpilman @sfc-gh-ljoswiak