You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am attempting to develop and application which utilizes the 'register_recv_callback' API to setup a callback to receive data from multiple UDP sockets. This application is receiving data from 100-150 multicast sockets and processing the zero-copy packets within a callback function that was registered using 'register_recv_callback'. The application is receiving approximately 980,000 packets/sec from the multicast sockets and can run for hours at a time.
In the my callback function if I simply drop the packet (return VMA_PACKET_DROP) the application will work and never crash. When I modify the callback to return VMA_PACKET_HOLD and process/free the packet from another thread I see infrequent almost random exceptions and/or segmentation faults.
During long term testing using OFED 4.0-1.0.1.0 (VMA 8.2.8 included) I've seen the following errors:
VMA 8.2.8 - C++ Exception:
03-21-2017 23:47:51.558597 INFO: Pkts = 976557.208093, Drops=0, PDiffs=0 VMA INFO: bpool[0x806330]:568:buffersPanic() no circle was found in buffer_pool
VMA ERROR: 0 /lib64/libvma.so.8(_ZN11buffer_pool12buffersPanicEv+0x77) [0x7f4c7fc3d1c7]
VMA ERROR: 1 /lib64/libvma.so.8(_ZN11buffer_pool11put_buffersEP10vma_list_tI14mem_buf_desc_tXadL_ZNS1_18buffer_node_offsetEvEEEm+0x14e) [0x7f4c7fc3d51e]
VMA ERROR: 2 /lib64/libvma.so.8(_ZN11buffer_pool23put_buffers_thread_safeEP10vma_list_tI14mem_buf_desc_tXadL_ZNS1_18buffer_node_offsetEvEEEm+0x2f) [0x7f4c7fc3d55f]
VMA ERROR: 3 /lib64/libvma.so.8(_ZN6cq_mgr20return_extra_buffersEv+0x45) [0x7f4c7fc3f7e5]
VMA ERROR: 4 /lib64/libvma.so.8(_ZN6cq_mgr20reclaim_recv_buffersEP10vma_list_tI14mem_buf_desc_tXadL_ZNS1_18buffer_node_offsetEvEEE+0x78) [0x7f4c7fc41088]
VMA ERROR: 5 /lib64/libvma.so.8(_ZN11ring_simple20reclaim_recv_buffersEP10vma_list_tI14mem_buf_desc_tXadL_ZNS1_18buffer_node_offsetEvEEE+0x3f) [0x7f4c7fc5aaff]
VMA ERROR: 6 /lib64/libvma.so.8(_ZN12sockinfo_udp12reuse_bufferEP14mem_buf_desc_t+0x11f) [0x7f4c7fcaa32f]
VMA ERROR: 7 /lib64/libvma.so.8(_ZN12sockinfo_udp12free_packetsEP12vma_packet_tm+0x6a) [0x7f4c7fca944a]
VMA ERROR: 8 ./tx2test5() [0x405316]
VMA ERROR: 9 /lib64/libpthread.so.0(+0x7dc5) [0x7f4c7f9d2dc5]
VMA ERROR: 10 /lib64/libc.so.6(clone+0x6d) [0x7f4c7e8021cd]
VMA PANIC: bpool[0x806330]:581:buffersPanic() m_n_buffers(200001) > m_n_buffers_created(200000)
terminate called without an active exception
VMA 8.2.8 - Segmentation Fault 1:
03-22-2017 16:30:36.548321 INFO: Pkts = 976554.977319, Drops=0, PDiffs=0 VMA ERROR: vma_list_t.push_back() - buff is already a member in a list.
VMA WARNING: vma_list_t.erase() got NULL object - ignoring.
Segmentation fault
Noticing there had been changes made to many of the files since the VMA 8.2.8 release I downloaded and compiled VMA 8.2.10 and re-tested with updated VMA libraries. The likelyhood of occurance seems to be lower but I can seemingly still receive segementation faults:
I am attempting to develop and application which utilizes the 'register_recv_callback' API to setup a callback to receive data from multiple UDP sockets. This application is receiving data from 100-150 multicast sockets and processing the zero-copy packets within a callback function that was registered using 'register_recv_callback'. The application is receiving approximately 980,000 packets/sec from the multicast sockets and can run for hours at a time.
In the my callback function if I simply drop the packet (return VMA_PACKET_DROP) the application will work and never crash. When I modify the callback to return VMA_PACKET_HOLD and process/free the packet from another thread I see infrequent almost random exceptions and/or segmentation faults.
During long term testing using OFED 4.0-1.0.1.0 (VMA 8.2.8 included) I've seen the following errors:
VMA 8.2.8 - C++ Exception:
03-21-2017 23:47:51.558597 INFO: Pkts = 976557.208093, Drops=0, PDiffs=0
VMA INFO: bpool[0x806330]:568:buffersPanic() no circle was found in buffer_pool
VMA ERROR: 0 /lib64/libvma.so.8(_ZN11buffer_pool12buffersPanicEv+0x77) [0x7f4c7fc3d1c7]
VMA ERROR: 1 /lib64/libvma.so.8(_ZN11buffer_pool11put_buffersEP10vma_list_tI14mem_buf_desc_tXadL_ZNS1_18buffer_node_offsetEvEEEm+0x14e) [0x7f4c7fc3d51e]
VMA ERROR: 2 /lib64/libvma.so.8(_ZN11buffer_pool23put_buffers_thread_safeEP10vma_list_tI14mem_buf_desc_tXadL_ZNS1_18buffer_node_offsetEvEEEm+0x2f) [0x7f4c7fc3d55f]
VMA ERROR: 3 /lib64/libvma.so.8(_ZN6cq_mgr20return_extra_buffersEv+0x45) [0x7f4c7fc3f7e5]
VMA ERROR: 4 /lib64/libvma.so.8(_ZN6cq_mgr20reclaim_recv_buffersEP10vma_list_tI14mem_buf_desc_tXadL_ZNS1_18buffer_node_offsetEvEEE+0x78) [0x7f4c7fc41088]
VMA ERROR: 5 /lib64/libvma.so.8(_ZN11ring_simple20reclaim_recv_buffersEP10vma_list_tI14mem_buf_desc_tXadL_ZNS1_18buffer_node_offsetEvEEE+0x3f) [0x7f4c7fc5aaff]
VMA ERROR: 6 /lib64/libvma.so.8(_ZN12sockinfo_udp12reuse_bufferEP14mem_buf_desc_t+0x11f) [0x7f4c7fcaa32f]
VMA ERROR: 7 /lib64/libvma.so.8(_ZN12sockinfo_udp12free_packetsEP12vma_packet_tm+0x6a) [0x7f4c7fca944a]
VMA ERROR: 8 ./tx2test5() [0x405316]
VMA ERROR: 9 /lib64/libpthread.so.0(+0x7dc5) [0x7f4c7f9d2dc5]
VMA ERROR: 10 /lib64/libc.so.6(clone+0x6d) [0x7f4c7e8021cd]
VMA PANIC: bpool[0x806330]:581:buffersPanic() m_n_buffers(200001) > m_n_buffers_created(200000)
terminate called without an active exception
VMA 8.2.8 - Segmentation Fault 1:
03-22-2017 16:30:36.548321 INFO: Pkts = 976554.977319, Drops=0, PDiffs=0
VMA ERROR: vma_list_t.push_back() - buff is already a member in a list.
VMA WARNING: vma_list_t.erase() got NULL object - ignoring.
Segmentation fault
VMA 8.2.8 - Segmentation Fault 2:
03-22-2017 20:58:18.255651 INFO: Pkts = 976546.831317, Over=0, Drops=0, PDiffs=0
VMA WARNING: vma_list_t.erase() got NULL object - ignoring.
Segmentation fault
Noticing there had been changes made to many of the files since the VMA 8.2.8 release I downloaded and compiled VMA 8.2.10 and re-tested with updated VMA libraries. The likelyhood of occurance seems to be lower but I can seemingly still receive segementation faults:
VMA 8.2.10 - Segmentation Fault:
03-23-2017 20:44:54.231603 INFO: Pkts = 976554.117604, Over=0, Drops=0, PDiffs=0
03-23-2017 20:45:54.231729 INFO: Pkts = 976544.334772, Over=0, Drops=0, PDiffs=0
VMA ERROR: vma_list_t.push_back() - buff is already a member in a list.
VMA WARNING: vma_list_t.erase() got NULL object - ignoring.
VMA WARNING: vma_list_t.erase() got NULL object - ignoring.
< rows deleted >
VMA WARNING: vma_list_t.erase() got NULL object - ignoring.
VMA WARNING: vma_list_t.erase() got NULL object - ignoring.
Segmentation fault
The text was updated successfully, but these errors were encountered: