[Issue]: can't hipMemPoolExportPointer signal memory?

[Epliz]

Problem Description

Hi,
I have allocated 8 bytes of signal memory with hipExtMallocWithFlags and am trying to share it with another process (as it seems like the stream wait value APIs would be perfect for cheaper sync between processes than events). But hipMemPoolExportPointer returns an error.

Is there any allocation size at which it would work?

Would it be possible for you to help me to get it working or if there is a limitation at the moment, to make it work?

Best regards,
Epliz

Operating System

Ubuntu 24.04

CPU

Intel xeon XE9680

GPU

MI300x x8

ROCm Version

ROCm 6.2.2

ROCm Component

No response

Steps to Reproduce

No response

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

No response

[darren-amd]

Hi @Epliz,

From my understanding, it seems that you are trying to use hipExtMallocWithFlags with hipMemPoolExportPointer, however, this would result in an error, as when you call hipExtMallocWithFlags, this would not be allocated in a memory pool and therefore trying to export a pointer to the data with hipExtMallocWithFlags would error out. Please see the documentation for memory pools here: Memory Pool Documentation. Specifcally, please see the function hipMallocFromPoolAsync which you can use to allocate memory in a memory pool. You can find an example of exporting and importing pointers from a memory pool here: Memory Pool Example.

Please give that a read and let me know if you have any questions, thanks!

[Epliz]

Hi @darren-amd ,

Thank you for your answer!
I looked into what you pointed at but:

1. I don't see how to pass the signal memory flags anywhere when creating a pool, or when doing an allocation from it
2. the memory return by hipExtMallocWithFlags, when allocating signal memory, seems to be CPU memory (by looking at the memory type attribute obtained using hipPointerGetAttributes). I don't think mem pools support CPU memory at all?

It would be great to support both points:

1. supporting CPU memory pools
2. adding the ability to pass flags when creating a memory pool so that it can be a pool for signal memory

[darren-amd]

Hi @Epliz,

Good questions, firstly, I don't see a particular way to pass in a signal flag to memory allocated in a pool. For the second question, hipExtMallocWithFlags should allocate on an accelerator and should be in GPU memory rather than CPU.

However, I believe hipIpcOpenMemHandle (found here) would be more appropriate for your use case. It will allow you to open an interprocess memory handle exported from another process and returns a device pointer usable in the local process, allowing you to share pointers between different processes. Please give that a try and let me know, as it should support signal memory.

[Epliz]

Hi @darren-amd ,
Could you please re-ooen te ticket? I still plan on trying what you suggested when I have time (probably during the end of year break).
I have some doubts that hipIpcOpenMemHandle will work as the allocated memory for signals seems to be in CPU mem.
Sorry for taking time to reply.

[darren-amd]

Hi @Epliz,

Of course, let me know if you have any questions!

[Epliz]

Hi @darren-amd ,

Here is a small program I tried:

#include <iostream>

#include <hip/hip_runtime.h>
#include <hip/hip_fp16.h>

int main(int argc, char** argv) {

  int device_count;

  if (hipGetDeviceCount(&device_count) != hipSuccess) {
    std::cout<<"Failed to get the numnber of GPUs"<<std::endl;
    return -1;
  }

  std::cout<<"Devices "<<device_count<<std::endl;

  hipError_t error;

  int supported;
  if (hipDeviceGetAttribute(&supported, hipDeviceAttributeCanUseStreamWaitValue, 0) != hipSuccess) {
    std::cout<<"Error getting the the property"<<std::endl;
    return -2;
  }

  std::cout<<"wait stream value: "<<supported<<std::endl;

  int* signal = nullptr;
  // apparently needs to allocate 8 bytes
  if ((error = hipExtMallocWithFlags((void**) &signal, 8, hipMallocSignalMemory)) != hipSuccess) {
    std::cout<<"Failed allocating signal memory"<<std::endl;
    return -3;
  }
  std::cout<<"signal ptr: "<<signal<<std::endl;

  hipPointerAttribute_t attributes;
  if (hipPointerGetAttributes (&attributes, signal) != hipSuccess) {
    std::cout<<"Failed getting attributes"<<std::endl;
    return -4;
  }

  std::cout<<"Memory type "<<attributes.type<<std::endl;

  hipIpcMemHandle_t mem_handle;
  if (hipIpcGetMemHandle (&mem_handle, signal) != hipSuccess) {
    std::cout<<"Failed getting mem handle"<<std::endl;
    return -5;
  }

  std::cout<<"got mem handle"<<std::endl;

  return 0;
}

Let me know if you see anything wrong, but for me, it fails when getting the memory handle:

Devices 2
wait stream value: 1
signal ptr: 0x755512bffd08
Memory type 1
Failed getting mem handle

And as I said before, the memory type seems to be CPU.

Best,
Epliz