Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exception pause debug in lockstep scheduler in OSX (Visual Studio Code) #12661

Open
pedrogasg opened this issue Aug 8, 2019 · 4 comments
Open

Exception pause debug in lockstep scheduler in OSX (Visual Studio Code) #12661

pedrogasg opened this issue Aug 8, 2019 · 4 comments
Assignees
@julianoes
Labels
bug Sim: SITL software in the loop simulation stale

Comments

@pedrogasg
Copy link

Describe the bug
Running the simulator in debug SITL, the launch start but pauses several times with 3 different errors time_wait, result and `timeout.
The simulation is still running but you need to press continue(F5) until the process runs without pausing.
This doesn't happen in Ubuntu Bionic (18.04)

To Reproduce
Steps to reproduce the behavior:

  1. Install the toolchain for OSX (depending your libraries you may need to install gazebo 9)
  2. Install Visual Studio Code
  3. Clone Firmware
  4. Open Firmware folder with Visual Studio Code
  5. Install recommended extensions for Visual Studio Code
  6. Launch Debug SITL (gazebo iris)
  7. The program pause in pthread_cond_wait

Expected behavior
The program shouldn't pause in the wait of a lockstep.

Log Files and Screenshots
Debug console log :

Loaded '/usr/lib/system/libsystem_m.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_malloc.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_network.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_networkextension.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_notify.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_sandbox.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_secinit.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_kernel.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_platform.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_pthread.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_symptoms.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_trace.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libunwind.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libxpc.dylib'. Symbols loaded.
Loaded '/usr/lib/closure/libclosured.dylib'. Symbols loaded.
Loaded '/usr/lib/libobjc.A.dylib'. Symbols loaded.
Loaded '/usr/lib/libc++abi.dylib'. Symbols loaded.
Loaded '/Users/dev/projects/Firmware/build/px4_sitl_default/bin/px4'. Symbols loaded.
=thread-selected,id="2"
Execute debugger commands using "-exec <command>", for example "-exec info registers" will list registers in use (when GDB is the debugger)
The program '/Users/dev/projects/Firmware/build/px4_sitl_default/bin/px4' has exited with code 0 (0x00000000).

backtrace

  thread #1: tid = 0x691a2, 0x00007fff53b5924a libsystem_kernel.dylib`__wait4_nocancel + 10, name = 'px4', queue = 'com.apple.main-thread'
    frame #0: 0x00007fff53b5924a libsystem_kernel.dylib`__wait4_nocancel + 10
    frame #1: 0x00007fff53ad39de libsystem_c.dylib`system + 452
    frame #2: 0x00000001000036c6 px4`main [inlined] run_startup_script(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) + 1490 at main.cpp:520
    frame #3: 0x00000001000030f4 px4`main(argc=0, argv=0x00007ffeefbff8d9) + 5988 at main.cpp:276
    frame #4: 0x00007fff53a08015 libdyld.dylib`start + 1
    frame #5: 0x00007fff53a08015 libdyld.dylib`start + 1
  thread #3: tid = 0x691cf, 0x00007fff53b5806e libsystem_kernel.dylib`__close_nocancel + 10
    frame #0: 0x00007fff53b5806e libsystem_kernel.dylib`__close_nocancel + 10
    frame #1: 0x00007fff53a930e9 libsystem_c.dylib`fclose + 92
    frame #2: 0x0000000100188a3f px4`px4_daemon::Server::_server_main(this=<unavailable>) + 719 at server.cpp:215
    frame #3: 0x0000000100188766 px4`px4_daemon::Server::_server_main_trampoline(self=<unavailable>) + 6 at server.cpp:120
    frame #4: 0x00007fff53d20661 libsystem_pthread.dylib`_pthread_body + 340
    frame #5: 0x00007fff53d2050d libsystem_pthread.dylib`_pthread_start + 377
    frame #6: 0x00007fff53d1fbf9 libsystem_pthread.dylib`thread_start + 13
  thread #4: tid = 0x691d0, 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff53d21589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame #2: 0x0000000100189949 px4`LockstepScheduler::cond_timedwait(this=0x00000001005001f0, cond=0x00000001007002d0, lock=0x0000000100700290, time_us=10000000) + 185 at lockstep_scheduler.cpp:88
    frame #3: 0x000000010012774b px4`DriverFramework::HRTWorkQueue::process(this=<unavailable>) + 283 at DriverFramework.cpp:486
    frame #4: 0x000000010012761f px4`DriverFramework::HRTWorkQueue::process_trampoline(arg=<unavailable>) + 415 at DriverFramework.cpp:361
    frame #5: 0x00007fff53d20661 libsystem_pthread.dylib`_pthread_body + 340
    frame #6: 0x00007fff53d2050d libsystem_pthread.dylib`_pthread_start + 377
    frame #7: 0x00007fff53d1fbf9 libsystem_pthread.dylib`thread_start + 13
  thread #5: tid = 0x691d1, 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'hpwork'
    frame #0: 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff53d21589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame #2: 0x0000000100189949 px4`LockstepScheduler::cond_timedwait(this=0x00000001005001f0, cond=0x000070000635ce38, lock=0x000070000635ce68, time_us=962000) + 185 at lockstep_scheduler.cpp:88
    frame #3: 0x0000000100189abf px4`LockstepScheduler::usleep_until(this=<unavailable>, time_us=<unavailable>) + 191 at lockstep_scheduler.cpp:124
    frame #4: 0x000000010012d41a px4`work_hpthread(argc=<unavailable>, argv=<unavailable>) + 26 at work_thread.c:270
    frame #5: 0x000000010012a952 px4`entry_adapter(ptr=0x0000000100700000) + 66 at px4_posix_tasks.cpp:105
    frame #6: 0x00007fff53d20661 libsystem_pthread.dylib`_pthread_body + 340
    frame #7: 0x00007fff53d2050d libsystem_pthread.dylib`_pthread_start + 377
    frame #8: 0x00007fff53d1fbf9 libsystem_pthread.dylib`thread_start + 13
  thread #6: tid = 0x691d2, 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'lpwork'
    frame #0: 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff53d21589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame #2: 0x0000000100189949 px4`LockstepScheduler::cond_timedwait(this=0x00000001005001f0, cond=0x00007000063dfe38, lock=0x00007000063dfe68, time_us=962000) + 185 at lockstep_scheduler.cpp:88
    frame #3: 0x0000000100189abf px4`LockstepScheduler::usleep_until(this=<unavailable>, time_us=<unavailable>) + 191 at lockstep_scheduler.cpp:124
    frame #4: 0x000000010012d43d px4`work_lpthread(argc=<unavailable>, argv=<unavailable>) + 29 at work_thread.c:296
    frame #5: 0x000000010012a952 px4`entry_adapter(ptr=0x0000000100700030) + 66 at px4_posix_tasks.cpp:105
    frame #6: 0x00007fff53d20661 libsystem_pthread.dylib`_pthread_body + 340
    frame #7: 0x00007fff53d2050d libsystem_pthread.dylib`_pthread_start + 377
    frame #8: 0x00007fff53d1fbf9 libsystem_pthread.dylib`thread_start + 13
  * frame #0: 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff53d21589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame #2: 0x0000000100189949 px4`LockstepScheduler::cond_timedwait(this=0x00000001005001f0, cond=0x0000700006462e08, lock=0x0000700006462e38, time_us=1884000) + 185 at lockstep_scheduler.cpp:88
    frame #3: 0x0000000100189abf px4`LockstepScheduler::usleep_until(this=<unavailable>, time_us=<unavailable>) + 191 at lockstep_scheduler.cpp:124
    frame #4: 0x000000010012d01f px4`work_hrtthread [inlined] hrt_work_process + 20 at hrt_thread.c:222
    frame #5: 0x000000010012d00b px4`work_hrtthread(argc=<unavailable>, argv=<unavailable>) + 27 at hrt_thread.c:266
    frame #6: 0x000000010012a952 px4`entry_adapter(ptr=0x0000000100700370) + 66 at px4_posix_tasks.cpp:105
    frame #7: 0x00007fff53d20661 libsystem_pthread.dylib`_pthread_body + 340
    frame #8: 0x00007fff53d2050d libsystem_pthread.dylib`_pthread_start + 377
    frame #9: 0x00007fff53d1fbf9 libsystem_pthread.dylib`thread_start + 13
  thread #7: tid = 0x691d4, 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'wq:manager'
    frame #0: 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff53d21589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame #2: 0x000000010012b491 px4`::px4_sem_wait(s=0x0000000100600208) + 65 at px4_sem.cpp:89
    frame #3: 0x0000000100186010 px4`px4::WorkQueueManagerRun() [inlined] BlockingQueue<px4::wq_config_t const*, 1ul>::pop(this=0x0000000100600190) + 9 at BlockingQueue.hpp:69
    frame #4: 0x0000000100186007 px4`px4::WorkQueueManagerRun() + 295 at WorkQueueManager.cpp:181
    frame #5: 0x000000010012a952 px4`entry_adapter(ptr=0x00000001007004a0) + 66 at px4_posix_tasks.cpp:105
    frame #6: 0x00007fff53d20661 libsystem_pthread.dylib`_pthread_body + 340
    frame #7: 0x00007fff53d2050d libsystem_pthread.dylib`_pthread_start + 377
    frame #8: 0x00007fff53d1fbf9 libsystem_pthread.dylib`thread_start + 13
  thread #8: tid = 0x691f7, 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'dataman'
    frame #0: 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff53d21589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame #2: 0x000000010012b491 px4`::px4_sem_wait(s=0x000000010020a210) + 65 at px4_sem.cpp:89
    frame #3: 0x000000010003ff7a px4`task_main(argc=<unavailable>, argv=<unavailable>) + 650 at dataman.cpp:1368
    frame #4: 0x000000010012a952 px4`entry_adapter(ptr=0x00000001020002b0) + 66 at px4_posix_tasks.cpp:105
    frame #5: 0x00007fff53d20661 libsystem_pthread.dylib`_pthread_body + 340
    frame #6: 0x00007fff53d2050d libsystem_pthread.dylib`_pthread_start + 377
    frame #7: 0x00007fff53d1fbf9 libsystem_pthread.dylib`thread_start + 13
  thread #9: tid = 0x691fe, 0x00007fff53b5a0a2 libsystem_kernel.dylib`poll + 10, name = 'sim_rcv'
    frame #0: 0x00007fff53b5a0a2 libsystem_kernel.dylib`poll + 10
    frame #1: 0x00000001000d8fe2 px4`Simulator::poll_for_MAVLink_messages(this=<unavailable>) + 1186 at simulator_mavlink.cpp:775
    frame #2: 0x00000001000d6319 px4`Simulator::start(argc=4, argv=0x0000000100402c10) + 169 at simulator.cpp:114
    frame #3: 0x000000010012a952 px4`entry_adapter(ptr=0x0000000100402bf0) + 66 at px4_posix_tasks.cpp:105
    frame #4: 0x00007fff53d20661 libsystem_pthread.dylib`_pthread_body + 340
    frame #5: 0x00007fff53d2050d libsystem_pthread.dylib`_pthread_start + 377
    frame #6: 0x00007fff53d1fbf9 libsystem_pthread.dylib`thread_start + 13
  thread #10: tid = 0x691ff, 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'sim_send'
    frame #0: 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff53d21589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame #2: 0x0000000100189949 px4`LockstepScheduler::cond_timedwait(this=0x00000001005001f0, cond=0x00007000066f1d08, lock=0x00007000066f1cc8, time_us=1000000) + 185 at lockstep_scheduler.cpp:88
    frame #3: 0x000000010012b588 px4`::px4_sem_timedwait(s=0x00007000066f1cc8, abstime=0x00007000066f1c90) + 136 at px4_sem.cpp:137
    frame #4: 0x0000000100184cd8 px4`::px4_poll(fds=<unavailable>, nfds=<unavailable>, timeout=100) + 600 at cdev_platform.cpp:384
    frame #5: 0x00000001000d8962 px4`Simulator::send(this=0x0000000102800600) + 274 at simulator_mavlink.cpp:602
    frame #6: 0x00000001000d8850 px4`Simulator::sending_trampoline((null)=<unavailable>) + 16 at simulator_mavlink.cpp:578
    frame #7: 0x00007fff53d20661 libsystem_pthread.dylib`_pthread_body + 340
    frame #8: 0x00007fff53d2050d libsystem_pthread.dylib`_pthread_start + 377
    frame #9: 0x00007fff53d1fbf9 libsystem_pthread.dylib`thread_start + 13
  thread #11: tid = 0x6920d, 0x00007fff53b58d8a libsystem_kernel.dylib`__semwait_signal + 10
    frame #0: 0x00007fff53b58d8a libsystem_kernel.dylib`__semwait_signal + 10
    frame #1: 0x00007fff53ad3724 libsystem_c.dylib`nanosleep + 199
    frame #2: 0x00007fff53ad3618 libsystem_c.dylib`usleep + 53
    frame #3: 0x00000001001899ea px4`LockstepScheduler::TimedWait::~TimedWait() [inlined] LockstepScheduler::TimedWait::~TimedWait(this=0x00000001007008b0) + 38 at lockstep_scheduler.h:43
    frame #4: 0x00000001001899c4 px4`LockstepScheduler::TimedWait::~TimedWait(this=0x00000001007008b0) + 4 at lockstep_scheduler.h:23
    frame #5: 0x00007fff53a0ad59 libdyld.dylib`tlv_finalize + 49
    frame #6: 0x00007fff53d22163 libsystem_pthread.dylib`_pthread_tsd_cleanup + 463
    frame #7: 0x00007fff53d21ee9 libsystem_pthread.dylib`_pthread_exit + 79
    frame #8: 0x00007fff53d2066c libsystem_pthread.dylib`_pthread_body + 351
    frame #9: 0x00007fff53d2050d libsystem_pthread.dylib`_pthread_start + 377
    frame #10: 0x00007fff53d1fbf9 libsystem_pthread.dylib`thread_start + 13
  thread #12: tid = 0x6920e, 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'wq:hp_default'
    frame #0: 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff53d21589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame #2: 0x000000010012b491 px4`::px4_sem_wait(s=0x0000700006774e50) + 65 at px4_sem.cpp:89
    frame #3: 0x0000000100185b78 px4`px4::WorkQueue::Run(this=0x0000700006774dc0) + 56 at WorkQueue.cpp:108
    frame #4: 0x000000010018638c px4`px4::WorkQueueRunner(context=<unavailable>) + 92 at WorkQueueManager.cpp:166
    frame #5: 0x00007fff53d20661 libsystem_pthread.dylib`_pthread_body + 340
    frame #6: 0x00007fff53d2050d libsystem_pthread.dylib`_pthread_start + 377
    frame #7: 0x00007fff53d1fbf9 libsystem_pthread.dylib`thread_start + 13

Capture d’écran 2019-08-08 à 14 42 39

Capture d’écran 2019-08-08 à 14 44 59

Drone (please complete the following information):

  • Simulation

Additional context
Tested in different versions of OSX
@julianoes julianoes added bug Sim: SITL software in the loop simulation labels Aug 8, 2019
@julianoes julianoes self-assigned this Aug 8, 2019
@weekly-digest weekly-digest bot mentioned this issue Aug 11, 2019
Closed
@julianoes
Copy link
Contributor

I tried to reproduce this with make px4_sitl gazebo___lldb and could not see it. I'll have to set-up vscode on macOS and try it as well.

@julianoes
Copy link
Contributor

@pedrogasg have you found out more about this? I can reproduce an exception pause on macOS now but it doesn't seem to be the same backtrace from what I can tell.

Also see: https://discuss.px4.io/t/vscode-lldb-debugging-paused-on-exception/13512/7?u=julianoes

@pedrogasg
Copy link
Author

Hello, I actually start to working in a linux machine after this, my last try was to set the miDebuggerPath to lldb-mi, but I have some issue so I give up for the moment, if some is working to solve this maybe can follow this track...

@stale
Copy link

stale bot commented Feb 3, 2020

This issue has been automatically marked as stale because it has not had recent activity. Thank you for your contributions.

@stale stale bot added the stale label Feb 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@julianoes julianoes

Labels
bug Sim: SITL software in the loop simulation stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pedrogasg @julianoes