Example for using Zero Copy with Galactic

[k-gupta]

We are trying to find an example on how to use zero copy with Fast RTPS. Is there a simple example somewhere that we can refer? We tried to use Galactic to do it but saw this error.

Currently used middleware can't loan messages. Local allocator will be used.

[fujitatomoya]

@k-gupta thanks for the issue. you could take a look at #579, especially this comment

[k-gupta]

Hey @fujitatomoya Thanks for looking into this. I had tried the suggestion from the issue you mentioned but I am seeing an error when I run it. I am just running the demo node - talker_loaned_message. Is there something I am doing wrong? I am still pretty new to using ROS so I might be using the profiles incorrectly.

This is the error

terminate called after throwing an instance of 'eprosima::fastcdr::exception::NotEnoughMemoryException'
  what():  Not enough memory in the buffer stream

The command I ran, this is running on Galactic. Also I am using the docker container not sure if that matters - ros/galactic-ros-core

FASTRTPS_DEFAULT_PROFILES_FILE=/ros/profile.xml     RMW_FASTRTPS_USE_QOS_FROM_XML=1     RMW_IMPLEMENTATION=rmw_fastrtps_cpp ros2 run demo_nodes_cpp talker_loaned_message

My profile looks like this

<?xml version="1.0" encoding="UTF-8"?>
<dds xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles">
    <profiles>
        <data_writer profile_name="/chatter_pod">
            <qos>
                <data_sharing>
                    <kind>AUTOMATIC</kind>
                </data_sharing>
            </qos>
        </data_writer>

        <data_reader profile_name="/chatter_pod">
            <qos>
                <data_sharing>
                    <kind>AUTOMATIC</kind>
                </data_sharing>
            </qos>
        </data_reader>
    </profiles>
</dds>

[EduPonz]

Hi @k-gupta

The error you are seeing is to be expected the way you are configuring things, let me explain. By default, Fast DDS sets the history memory policy to PREALLOCATED, which means that it allocates memory for a certain amount of samples of a fixed size. Since ROS 2 types tend to have unbounded sequences in them, rmw_fastrtps overrides this set with PREALLOCATED_WITH_REALLOC, which allows Fast DDS to reallocate samples if the sequences go out of the preallocated bounds.

However, as explained here, when setting the environment variable RMW_FASTRTPS_USE_QOS_FROM_XML=1, rmw_fastrtps does not override the history memory policy, so if a sequence goes is larger than the preallocated memory, then it cannot be de-serialized, since the setting prevents Fast DDS to reallocate the buffer. As suggested in the README.md, you can set the history memory policy in the XML as well (see this).

[fujitatomoya]

@k-gupta

probably you can try the following xml setting. (this works with rolling if the data is PoD.)

<?xml version="1.0" encoding="UTF-8" ?>
<profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles">

  <!-- Default publisher profile -->
  <data_writer profile_name="default publisher profile" is_default_profile="true">
    <qos>
      <publishMode>
        <kind>SYNCHRONOUS</kind>
      </publishMode>
      <data_sharing>
        <kind>AUTOMATIC</kind>
      </data_sharing>
    </qos>
    <historyMemoryPolicy>DYNAMIC</historyMemoryPolicy>
  </data_writer>

  <data_reader profile_name="default subscription profile" is_default_profile="true">
    <qos>
      <data_sharing>
        <kind>AUTOMATIC</kind>
      </data_sharing>
    </qos>
    <historyMemoryPolicy>DYNAMIC</historyMemoryPolicy>
  </data_reader>
</profiles>

then it cannot be de-serialized, since the setting prevents Fast DDS to reallocate the buffer.

nitpick, this means serialization since the problem is observed on publisher LoanedMessage (stacktrace includes eprosima::fastcdr::Cdr::serialize(char const*)).

gdb stacktrace

#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=139760881099968) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=139760881099968) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=139760881099968, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007f1c9ddf0476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007f1c9ddd67f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007f1c9e09abfe in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f1c9e0a628c in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#7  0x00007f1c9e0a62f7 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007f1c9e0a6558 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
#9  0x00007f1c9cfb4cea in eprosima::fastcdr::Cdr::serialize(int) [clone .cold] ()
   from /root/ros2_ws/colcon_ws/install/fastcdr/lib/libfastcdr.so.1
#10 0x00007f1c9cfb743a in eprosima::fastcdr::Cdr::serialize(char const*) ()
   from /root/ros2_ws/colcon_ws/install/fastcdr/lib/libfastcdr.so.1
#11 0x00007f1c97f3f816 in eprosima::fastcdr::Cdr::serialize(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
   from /root/ros2_ws/colcon_ws/install/rmw_dds_common/lib/librmw_dds_common__rosidl_typesupport_fastrtps_cpp.so
#12 0x00007f1c97f3f767 in eprosima::fastcdr::Cdr::operator<<(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
   from /root/ros2_ws/colcon_ws/install/rmw_dds_common/lib/librmw_dds_common__rosidl_typesupport_fastrtps_cpp.so
#13 0x00007f1c97f3ef17 in rmw_dds_common::msg::typesupport_fastrtps_cpp::cdr_serialize(rmw_dds_common::msg::NodeEntitiesInfo_<std::allocator<void> > const&, eprosima::fastcdr::Cdr&) ()
   from /root/ros2_ws/colcon_ws/install/rmw_dds_common/lib/librmw_dds_common__rosidl_typesupport_fastrtps_cpp.so
#14 0x00007f1c97f40beb in rmw_dds_common::msg::typesupport_fastrtps_cpp::cdr_serialize(rmw_dds_common::msg::ParticipantEntitiesInfo_<std::allocator<void> > const&, eprosima::fastcdr::Cdr&) ()
   from /root/ros2_ws/colcon_ws/install/rmw_dds_common/lib/librmw_dds_common__rosidl_typesupport_fastrtps_cpp.so
#15 0x00007f1c97f40f38 in rmw_dds_common::msg::typesupport_fastrtps_cpp::_ParticipantEntitiesInfo__cdr_serialize(void const*, eprosima::fastcdr::Cdr&) ()
   from /root/ros2_ws/colcon_ws/install/rmw_dds_common/lib/librmw_dds_common__rosidl_typesupport_fastrtps_cpp.so
#16 0x00007f1c9da7d822 in rmw_fastrtps_cpp::TypeSupport::serializeROSmessage(void const*, eprosima::fastcdr::Cdr&, void const*) const () from /root/ros2_ws/colcon_ws/install/rmw_fastrtps_cpp/lib/librmw_fastrtps_cpp.so
#17 0x00007f1c9d9bd555 in rmw_fastrtps_shared_cpp::TypeSupport::serialize(void*, eprosima::fastrtps::rtps::SerializedPayload_t*) () from /root/ros2_ws/colcon_ws/install/rmw_fastrtps_shared_cpp/lib/librmw_fastrtps_shared_cpp.so
#18 0x00007f1c9d335477 in eprosima::fastdds::dds::DataWriterImpl::perform_create_new_change(eprosima::fastrtps::rtps::ChangeKind_t, void*, eprosima::fastrtps::rtps::WriteParams&, eprosima::fastrtps::rtps::InstanceHandle_t const&) ()
   from /root/ros2_ws/colcon_ws/install/fastrtps/lib/libfastrtps.so.2.6
#19 0x00007f1c9d335ee3 in eprosima::fastdds::dds::DataWriterImpl::create_new_change_with_params(eprosima::fastrtps::rtps::ChangeKind_t, void*, eprosima::fastrtps::rtps::WriteParams&) ()
   from /root/ros2_ws/colcon_ws/install/fastrtps/lib/libfastrtps.so.2.6
#20 0x00007f1c9d335fc0 in eprosima::fastdds::dds::DataWriterImpl::create_new_change(eprosima::fastrtps::rtps::ChangeKind_t, void*) () from /root/ros2_ws/colcon_ws/install/fastrtps/lib/libfastrtps.so.2.6
#21 0x00007f1c9d335ff9 in eprosima::fastdds::dds::DataWriterImpl::write(void*) ()
   from /root/ros2_ws/colcon_ws/install/fastrtps/lib/libfastrtps.so.2.6
#22 0x00007f1c9d9968c9 in rmw_fastrtps_shared_cpp::__rmw_publish(char const*, rmw_publisher_s const*, void const*, rmw_publisher_allocation_s*) () from /root/ros2_ws/colcon_ws/install/rmw_fastrtps_shared_cpp/lib/librmw_fastrtps_shared_cpp.so
#23 0x00007f1c9d98f15f in rmw_fastrtps_shared_cpp::__rmw_create_node(rmw_context_s*, char const*, char const*, char const*) () from /root/ros2_ws/colcon_ws/install/rmw_fastrtps_shared_cpp/lib/librmw_fastrtps_shared_cpp.so
#24 0x00007f1c9da61e1c in rmw_create_node ()
   from /root/ros2_ws/colcon_ws/install/rmw_fastrtps_cpp/lib/librmw_fastrtps_cpp.so
--Type <RET> for more, q to quit, c to continue without paging--
#25 0x00007f1c9dd7b0e6 in rcl_node_init () from /root/ros2_ws/colcon_ws/install/rcl/lib/librcl.so
#26 0x00007f1c9e437adb in rclcpp::node_interfaces::NodeBase::NodeBase(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<rclcpp::Context>, rcl_node_options_s const&, bool, bool) ()
   from /root/ros2_ws/colcon_ws/install/rclcpp/lib/librclcpp.so
#27 0x00007f1c9e431f75 in rclcpp::Node::Node(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rclcpp::NodeOptions const&) () from /root/ros2_ws/colcon_ws/install/rclcpp/lib/librclcpp.so
#28 0x00007f1c9e4339fc in rclcpp::Node::Node(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rclcpp::NodeOptions const&) () from /root/ros2_ws/colcon_ws/install/rclcpp/lib/librclcpp.so
#29 0x00007f1c9c1c37e3 in demo_nodes_cpp::LoanedMessageTalker::LoanedMessageTalker(rclcpp::NodeOptions const&) ()
   from /root/ros2_ws/colcon_ws/install/demo_nodes_cpp/lib/libtopics_library.so
#30 0x00007ffe34191890 in ?? ()
#31 0x0000558500303e00 in ?? ()
#32 0x00007f1c9c1c444c in rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::LoanedMessageTalker>::create_node_instance(rclcpp::NodeOptions const&) () from /root/ros2_ws/colcon_ws/install/demo_nodes_cpp/lib/libtopics_library.so
#33 0x00005584ffaef215 in main ()

[ZhenshengLee]

@k-gupta

probably you can try the following xml setting. (this works with rolling if the data is PoD.)

@fujitatomoya thanks for your hints.

this configuration with ASYNCHRONOUS publisherMode also enables the datasharing

I am testing this in https://github.com/ZhenshengLee/performance_test
you can see the test results in https://zhenshenglee.github.io/ros2_jetson_benchmarks/benchmark_results/galactic/pc_ros2_galactic_v5.13.0-41-generic_amd64/rmw_compare/

[k-gupta]

Thanks a lot for the info. We were able to fix this by setting the historyMemoryPolicy to PREALLOCATED_WITH_REALLOC. But we will also try to the setting you gave.

[fujitatomoya]

@k-gupta thanks. please close the issue once everything is confirmed and resolved.

[fujitatomoya]

Since Galactic is already E.O.L https://docs.ros.org/en/rolling/Releases.html, i will go ahead to close this issue.