[PATCH v3 0/7] MR61: winegstreamer: Dynamic transform stream format change support.

[Zebediah Figura]

On 5/24/22 01:07, Rémi Bernon (@rbernon) wrote:
> 
>>>      To
>>>      get zero-copy you'll have to add even more entry points, to provide
>>>      the output buffer beforehand, and release it after the processing.
>>
>> I'm confused about this, because this doesn't match my (somewhat dim)
>> awareness of what would be necessary for zero-copy. So maybe it would
>> help to hash that out now completely.
>>
>> My understanding is:
>>
>> * in some cases we can and should transform data in-place (perhaps
>> because the PE side expects it, not that I'm aware of any such cases).
>> In that case we should feed a writable GstBuffer into the pipeline, and
>> if the output buffer was not the same as the input buffer, blit the
>> latter into the former.
>>
>> * if we cannot transform data in-place (either because GStreamer doesn't
>> support it or because it'd confuse the PE side, not that I'm aware of
>> any such cases there either), we should try to let GStreamer elements
>> fill buffers allocated by the PE side. In order to achieve this we
>> should set up a GstBufferPool ahead of time containing a fixed number of
>> buffers. (I think we can guarantee we won't run out?) If the output
>> buffer comes from this pool, we can retrieve the corresponding input
>> buffer (the original intent of struct wg_sample) and return that pointer
>> to user space. If not, we should call gst_buffer_pool_acquire_buffer()
>> ourselves and blit the output buffer into it.
>>
>> In both cases, though, it doesn't make sense to force a specific output
>> buffer unless it's the same as the input buffer, which means there's no
>> need for an extra API. Is there some extra consideration I'm missing here?
>>
>> The other consequence that falls out from this is that the act of
>> pushing data will inherently cause data to be written; i.e. there is no
>> concept of "peek" quite like pipes. On the other hand, we can still
>> "peek" at the last output buffer (and, more importantly, its associated
>> metadata) without actually dequeuing it (and hence without changing the
>> wg_transform state at all), which is kind of what I was trying to
>> propose in the first place. It's not obvious to me why this is
>> problematic; am I missing something here as well?
> 
> There's no such thing as transforming in-place. Input buffers (as in
> provided by the app to ProcessInput) and output buffers (as in provided
> by the app to ProcessOutput) will never be the same. They just cannot
> in general simply because of a difference of size, but also because it's
> not how it's supposed to work: input buffers and kept referenced until
> they have been fully processed, and until one or more output buffers
> have been filled from it.

I'm looking at MFT_INPUT_STREAM_PROCESSES_IN_PLACE. I haven't checked 
whether anything interesting exposes it, though; maybe nothing does?

(I don't think any of the GStreamer elements we care about are capable 
of in-place transform either, but theoretically some of them could be.)

> 
> What zero-copy means however, is avoiding any unnecessary copy when
> passing data back and forth GStreamer and unix-side in general.
> 
> The only way this can be done, for ProcessOutput, like I described
> previously already, is by:
> 
> 1) providing the application provided output buffer to GStreamer
>     *before* processing any input, so that any future buffer allocation
>     can use it to write the decoded data.
> 
> 2) process the input buffers that were queued, checking for a format
>     change event, or whether the output buffer has been used.
> 
> 3) blit the output data to the output buffer if it wasn't picked up.
> 
> 4) remove the output buffer from any GStreamer buffer thay may still be
>     referencing it (This last step also possibly but hopefully doesn't
>     involve copying the data back from the output buffer to GStreamer
>     memory if the data isn't supposed to be discarded, this can happen
>     in some case, at least when format change event is returned so we
>     do not lose the data but not only).
> 

Okay, the thing I was missing was that in the case of Media Foundation 
transforms (or at least some of them), we can't actually choose the 
sample buffer, but rather it gets provided to us by the application on a 
per-sample basis. Same for DMOs I guess. (On the other hand, DirectShow 
can pool, and so can all three parser frontends. And Media Foundation 
transforms are capable of pooling, although neither of the ones we 
currently implement actually do.)

This is ugly to deal with. I guess you can provide a specific buffer by 
(ab)using GstBufferPool's acquire_buffer() method to block until 
ProcessOutput() is called and we have a buffer; I don't know if you had 
a better plan than that.

On the other hand, it's not clear to me how we're going to deal with 
MF_E_TRANSFORM_STREAM_CHANGE or MFT_OUTPUT_STATUS_SAMPLE_READY while 
maintaining zero-copy. What is your plan for those?