vp9e: do not allocate internal resources via external allocator

[dvrogozh]

Please, switch vp9 encoder to use internal mediasdk core and avoid any external allocations for internal resources.

[xhaihao]

@dvrogozh @onabiull @DmitryMalishev is D3DDDIFMT_INTELENCODE_MBSEGMENTMAP (MFX_FOURCC_VP9_SEGMAP) an internal resource ? According to

MediaSDK/_studio/shared/src/libmfx_core_vaapi.cpp

Line 528 in f290d47

if (m_bSetExtFrameAlloc && request->Info.FourCC != MFX_FOURCC_P8)

, MediaSDK still tries to use the external frame allocator for D3DDDIFMT_INTELENCODE_MBSEGMENTMAP on linux

[DmitryMalishev]

Hi @xhaihao,
MFX_FOURCC_VP9_SEGMAP surface has MFX_MEMTYPE_INTERNAL_FRAME flag when inited:

MediaSDK/_studio/mfx_lib/encode_hw/vp9/src/mfx_vp9_encode_hw.cpp

Line 226 in b1c624e

request.Type = MFX_MEMTYPE_D3D_INT;

MediaSDK/_studio/mfx_lib/encode_hw/vp9/include/mfx_vp9_encode_hw_utils.h

Line 75 in b1c624e

MFX_MEMTYPE_D3D_INT = MFX_MEMTYPE_FROM_ENCODE | MFX_MEMTYPE_DXVA2_DECODER_TARGET | MFX_MEMTYPE_INTERNAL_FRAME,

It means that internal frame allocator should be used for this surface allocation:

MediaSDK/_studio/shared/src/libmfx_core_vaapi.cpp

Lines 560 to 563 in b1c624e

	if (m_bSetExtFrameAlloc && request->Info.FourCC != MFX_FOURCC_P8)
	{
	// Default allocator should be used if D3D manager is not set and internal D3D buffers are required
	if (!m_Display && request->Type & MFX_MEMTYPE_INTERNAL_FRAME)

Do you see in debug that it is not the case? Please let us know if so.

@eshiryae, could you please track this request if more help required?

[xhaihao]

@DmitryMalishev @eshiryae No, it is not the case because m_Display is not NULL.

Maybe MFX_FOURCC_VP9_SEGMAP should go into the else block too.

MediaSDK/_studio/shared/src/libmfx_core_vaapi.cpp

Lines 612 to 623 in b1c624e

	else
	{
	// Default Allocator is used for internal memory allocation and all coded buffers allocation
	if (request->Type & MFX_MEMTYPE_EXTERNAL_FRAME)
	return MFX_ERR_MEMORY_ALLOC;
	m_bUseExtAllocForHWFrames = false;
	sts = DefaultAllocFrames(request, response);
	MFX_CHECK_STS(sts);
	return TraceFrames(request, response, sts);
	}

[DmitryMalishev]

@xhaihao, thank you for the clarification.
Before solving it out we(@eshiryae) would like to ask you a couple of things:
[1] What is the impacted usage scenario? Do you have workaround?
[2] Is it urgent change request?
[3] Is it possible to make it work by returning an UNSUPPORTED status from the external allocator on this request to make it use the internal allocator then?

MediaSDK/_studio/shared/src/libmfx_core_vaapi.cpp

Lines 572 to 577 in b1c624e

	sts = (*m_FrameAllocator.frameAllocator.Alloc)(m_FrameAllocator.frameAllocator.pthis, &temp_request, response);
	// if external allocator cannot allocate d3d frames - use default memory allocator
	if (MFX_ERR_UNSUPPORTED == sts || MFX_ERR_MEMORY_ALLOC == sts)
	{
	// Default Allocator is used for internal memory allocation only

[xhaihao]

@DmitryMalishev we are enabling vp9 encode in both ffmpeg/qsv and gst-msdk which have an external allocator. We may workaround it by returning an UNSUPPORTED status however we have to provide the workaround in both ffmpeg/qsv and gst-msdk, another thing is we failed to find MFX_FOURCC_VP9_SEGMAP in mfx header files, so we have to define it in ffmpeg/qsv and gst-msdk. Hence I think it would be better to fix it in MediaSDK.

[eshiryae]

Hi @xhaihao, thank you for the details, we understand the problem and will work to solve it.

[eshiryae]

Hi @xhaihao, it is expected behavior for hw library and in this case you should return an unsupported error(MFX_ERR_UNSUPPORTED) for segmentation map allocation try. All resources which unsupported by external allocator are allocated by internal allocator.
According MFX_FOURCC_VP9_SEGMAP, you can define it as it's done here:

MediaSDK/_studio/mfx_lib/encode_hw/vp9/include/mfx_vp9_encode_hw_vaapi.h

Line 34 in fcc3426

MFX_FOURCC_VP9_SEGMAP = MFX_MAKEFOURCC('V','P','8','S'),