Add user-lockable frame memory to theora
SUMMARY
Currently it is not generally possible to display ('play') theora content without copying decoded frames at least twice - once from libtheora memory to user memory, and once from user memory to display memory.
This proposal suggests the addition of two API calls that add the concept of a "shared" buffer, allocated and freed by the user but lockable by libtheora. With such a buffer, one of these copies can be eliminated.
THE PROBLEM
libtheora provides decoded frames in memory allocated and owned by libtheora (through the theora_yuv_out function call). libtheora does not guarantee that this memory is valid following subsequent theora_packet_in calls. Hence, the player application must either copy the decoded frame directly to the screen, or copy it to a buffer for later display.
However, it is undesirable to copy the frame directly to the screen, as this will lead either to unpredictable display times or an inefficient use of available time (the player must either display the frame as soon as it is ready or wait until the display time of the frame arrives). Hence, players must in general incur the cost of copying the frame to a temporary buffer owned by the player application.
THREE PROPOSED SOLUTIONS
(1) One solution would be to add a single API call to libtheora:
theora_unlock_yuv_buffer(theora *, buffer *)
libtheora would then avoid freeing frames that have not been explicitly unlocked by the user. This approach has disadvantages:
- the behaviour of the library changes (legacy applications will stop working due to memory leaks)
- an additional function call per frame is required for normal operation of the library
(2) These disadvantages can be mitigated by requiring the user to lock buffers first:
theora_lock_yuv_buffer(theora_state *, yuv_buffer *)
theora_unlock_yuv_buffer(theora_state *, yuv_buffer *)
libtheora would exhibit the same behaviour in the absence of these functions being called, but would not free frames that have been locked by the user until subsequently unlocked.
(3) Another approach is to allow user-allocation of buffers:
theora_user_yuv_out(theora_state *, yuv_buffer *)
int theora_user_buffer_is_freeable(theora_state *, yuv_buffer *)
In this case, the user can provide user-allocated memory into which theora decodes the frame, and theora can indicate that a user-allocated frame should not be freed (for example, if the frame is a reference frame). Normal operation of the library is unchanged.
This approach has the additional advantage that the user can tailor memory use to the application and environment - restricted-memory embedded systems can avoid malloc and free, while directX-based systems can provide memory already allocated as a directX texture.
RECOMMENDATION
Given that this problem exists, and is fixable with minimal API modification and no perceived change for existing applications, I would suggest that we consider fixing it. I currently favour approach 3, as it is more flexible, however this approach also means more work for the application designer that wishes to use the new feature.