Unfortunately, AMAP_3D_TARGET appears to be useless. Internally, surface allocation comes up with the following capabilities for the surface: DDSCAPS_VIDEOMEMORY | DDSCAPS_LOCALVIDMEM | DDSCAPS_OFFSCREENPLAIN | DDSCAPS_3DDEVICE. Sadly, DirectDraw responds (might respond?) with E_NOTIMPL.
While one can advance one step further by removing DDSCAPS_VIDEOMEMORY | DDSCAPS_LOCALVIDMEM, another problem is hit that you are no longer able to create a surface with is both FourCC enabled and in the same time has 3D rendering capabilities. Remove your YUV four character code (such as NV12, YV12, YUY2 you would normally have as codec’s output) or fail miserably with DDERR_INVALIDPIXELFORMAT.
Back to the original problem, the AMAP_3D_TARGET flag failing to work makes it impossible to allocate 3D-enabled DirectDraw surface with DirectShow Video Mixing Renderer. While it might sound as deprecated technology, it is yet a supported Windows SDK component, the most efficient video rendering component, the least hardware sensitive video rendering component, and – the most immediately important – the only way an SDK I am working with can deliver its video overlay.
So, there is no easy way to request a 3D enabled surface through a custom allocator-presenter, with a hook in the IVMRSurfaceAllocator::AllocateSurface updating the allocation flags. It does not make much sense to replace the whole allocator-presenter either: while an earlier DirectShow/Windows SDK provides a decent base for custom allocator-presenter, it is going to eventually hit the same problems mentioned above. And you cannot blit from 3D surface into a plain offscreen surface either without getting E_FAIL.
What appears to still be possible is allocating an addition 3D enabled surface, such as within IVMRImagePresenter::StartPresenting and using it as a replacement surface within IVMRImagePresenter::PresentImage. Having a custom allocator-presenter receive video frame, you can blit the picture into 3D-enabled surface, do your thing and pass the updated VMRPRESENTATIONINFO to default allocator-presenter so that it presents the additional update surface, not the original one.
