Package limn.video

Class VideoFrame

java.lang.Object
limn.video.VideoFrame
public final class VideoFrame extends Object
One decoded picture, borrowed from the producer that published it. A frame is a lease, not a value: after release() the same instance is refilled with a different picture, so nothing here may be cached, stored in a collection, or compared for equality: hold it, use it, release it.

Deliberately mutable and deliberately reused. At 60 pictures a second, allocating planes, buffers or wrapper objects per picture is a garbage collection pause every few seconds; publishing a frame writes primitives into an object that already exists and allocates nothing. For the same reason a frame is never an immutable bitmap: a renderer that caches an uploaded texture by object identity would show whichever picture happened to be in the slot first, for the life of the stream.

Not AutoCloseable: closing is expected to be idempotent, whereas releasing a frame twice returns one slot to the pool twice and must be loud. See release().

A picture has two possible shapes and kind() says which. Either it is planar samples a consumer can read, or it is an opaque handle to memory a device owns and a consumer cannot address at all, which is what a hardware decoder hands back. The two are not interchangeable and neither is a special case of the other: plane(int) fails on a handle rather than returning something plausible, handle() fails on planar samples, and toPlanar() is how a consumer that cannot use a handle asks for one to be read back into memory it can.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static interface 
    VideoFrame.Downloader
    How a producer reads one of its own device pictures back into memory a consumer can address, when toPlanar() asks it to.
    static enum 
    VideoFrame.Kind
    Which of the two shapes a picture has: samples in memory, or a handle to a device allocation.
    static interface 
    VideoFrame.Recycler
    How a producer gets a released frame's memory back.
    static final class 
    VideoFrame.Writer
    A producer's handle on one pooled frame: the only thing that can point a frame at memory or publish it.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final long
    PTS_UNKNOWN
    What ptsMicros() reports when the producer does not know the presentation time.

  • Method Summary

    Modifier and Type
    Method
    Description
    VideoColor
    color()
     
    PixelFormat
    format()
     
    long
    generation()
    A counter that changes on every publication and every release of this slot, odd exactly while a consumer holds the frame.
    long
    handle()
    The device allocation this picture lives in, as the opaque integer kind() names.
    int
    height()
     
    VideoFrame.Kind
    kind()
     
    ByteBuffer
    plane(int plane)
    The bytes of one plane, read-only, positioned at 0 with the limit at the plane's extent.
    long
    ptsMicros()
    Presentation time in microseconds measured from the start of the stream, not from the epoch, and not from when decoding began.
    void
    release()
    Hands this frame's buffers back to the producer that published it.
    int
    slot()
     
    int
    stride(int plane)
    Distance in bytes between the starts of two consecutive rows of plane, not samples and not pixels.
    void
    toPlanar()
    Makes this picture's samples readable, reading a device allocation back into memory if that is what it is.
    String
    toString()
    Never throws, released or not: VideoFrame[slot=2 1920x1080 NV12 IO_SURFACE pts=…]
    int
    width()
     

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

  • Field Details

    • PTS_UNKNOWN

      public static final long PTS_UNKNOWN
      What ptsMicros() reports when the producer does not know the presentation time.
      See Also:

  • Method Details

    • width

      public int width()
      Returns:
      visible picture width in pixels; still readable after release()

    • height

      public int height()
      Returns:
      visible picture height in pixels; still readable after release()

    • format

      public PixelFormat format()
      Returns:
      the plane layout of this picture, never null; still readable after release()

    • color

      public VideoColor color()
      Returns:
      how this picture's samples are to be interpreted, never null; readable after release

    • stride

      public int stride(int plane)
      Distance in bytes between the starts of two consecutive rows of plane, not samples and not pixels. At least the plane's byte width and often more, and an arbitrary byte count rather than a multiple of four, so a consumer that assumes four-byte row alignment skews every frame whose width is not a multiple of four.

      Still readable after release().

      Throws:
      IndexOutOfBoundsException - if plane is not a plane of format()

    • plane

      public ByteBuffer plane(int plane)
      The bytes of one plane, read-only, positioned at 0 with the limit at the plane's extent. Row r begins at r * stride(plane) and carries the plane's byte width of picture; the remainder of each row is padding holding anything at all.

      The memory belongs to the producer. It is valid until release() and is then refilled with another picture; copy or upload anything that must outlive the frame. The same buffer instance is returned for the life of the slot, so its position and limit are shared state; they are reset on every publication, and a consumer that reads relatively should duplicate the buffer first rather than leave it consumed for the next one.

      Direct or heap-backed, whichever the producer bound. Either way the view returned is read-only, so a heap-backed plane is not array-accessible: hasArray() is false and array() throws. Read samples with get(index), and test isDirect() before assuming there is an address to hand to a device.

      A handle-backed picture has no planes and this throws for one. It does not return an empty buffer, a stale one, or the planes of whatever the slot held last: a VideoFrame.Kind.PLANAR picture is the only one whose samples are in memory a consumer may read, and a consumer that meets any other kind either binds the handle() or asks toPlanar() to read it back. Answering plausibly here would put a wrong picture on the screen instead of an exception on the one line that can explain it.

      Throws:
      IllegalStateException - if the frame has been released, or was never published
      UnsupportedOperationException - if kind() is not VideoFrame.Kind.PLANAR
      IndexOutOfBoundsException - if plane is not a plane of format()

    • kind

      public VideoFrame.Kind kind()
      Returns:
      whether this picture is samples in memory or a handle to a device allocation; still readable after release(), and changed by toPlanar()

    • handle

      public long handle()
      The device allocation this picture lives in, as the opaque integer kind() names.

      What it is worth to a consumer depends entirely on the kind, and a consumer that does not recognise the kind must refuse rather than guess: the same long is a reference-counted CoreVideo surface on one platform and a file descriptor on another.

      It is borrowed for exactly as long as the frame is. The producer hands the allocation back to its own pool on release() and refills it, so a device object derived from this (a texture bound onto it, say) stops meaning this picture at that moment, and any device work that reads it must have completed first. Nothing here can enforce that: it is the consumer's, and it is the hardest lifetime in the subsystem.

      Returns:
      the handle, never 0 while the frame is held
      Throws:
      IllegalStateException - if the frame has been released, or was never published
      UnsupportedOperationException - if kind() is VideoFrame.Kind.PLANAR

    • toPlanar

      public void toPlanar()
      Makes this picture's samples readable, reading a device allocation back into memory if that is what it is. A VideoFrame.Kind.PLANAR picture is already readable and this does nothing.

      This is the download that every consumer without a device needs, and it is not free: it moves the whole picture out of the decoder's memory (3.1 MB at 1080p 4:2:0, 12.4 MB at 4K) across whatever bus separates them, per picture. A consumer that can bind handle() must do that instead; this exists so that the ones that cannot (a software converter, a writer, a test) are not simply broken by a decoder they did not choose.

      Afterwards kind() is VideoFrame.Kind.PLANAR for the rest of this lease and handle() throws. Releasing the frame is unchanged and still required exactly once.

      Whichever thread holds the frame, and not two at once: this re-points the frame's planes, which is otherwise something only a producer may do to a frame nobody is holding.

      Throws:
      IllegalStateException - if the frame has been released, or was never published
      UnsupportedOperationException - if the producer offers no download for this kind

    • ptsMicros

      public long ptsMicros()
      Presentation time in microseconds measured from the start of the stream, not from the epoch, and not from when decoding began. Non-decreasing across the frames one source delivers, because any reordering happens inside the producer; consecutive equal values are legal, so it is not strictly increasing.

      Microseconds because a 90 kHz container tick is 11.11 of them, finer than the container's own clock, while a long still spans far more than any stream's length. Milliseconds would quantize a 60-per-second frame interval into visible judder.

      Returns:
      the presentation time, or PTS_UNKNOWN when the source has no timing at all

    • slot

      public int slot()
      Returns:
      this frame's index in its producer's pool, stable for the life of the pool: the integer that identifies the memory to the side that owns it, for diagnostics and for a producer's own bookkeeping

    • generation

      public long generation()
      A counter that changes on every publication and every release of this slot, odd exactly while a consumer holds the frame. A holder that keeps a frame across several of its own frames reads this at delivery and compares before using it: an unchanged value means the picture is still the one it was handed. This is the only way to detect a stale reference, because a recycled frame is otherwise indistinguishable from a live one.
      Returns:
      the publication counter, never decreasing

    • release

      public void release()
      Hands this frame's buffers back to the producer that published it. The producer owns the memory, always; a consumer borrows a frame between delivery and this call and owns nothing. Exactly one release per delivered frame.

      A pool has a fixed, small number of slots. A frame that is never released removes one permanently, and the producer stalls as soon as the remaining slots are all in flight. A video that plays for a second and then freezes with no error and no exception is this bug, and it is why this call has no convenient alternative.

      Afterwards the frame is dead to the caller: drop the reference. plane(int) throws until the producer refills the slot, and once it does, this same instance carries a different picture. The descriptive accessors keep answering, so a released frame can still be logged.

      Any thread: a frame may be handed between threads and released on a different one from the one it was delivered to. Two threads racing to release the same frame cannot both succeed.

      Throws:
      IllegalStateException - if the frame is not currently held. A slot returned twice is handed to two producers at once, and the picture corruption that follows appears nowhere near this call.

    • toString

      public String toString()
      Never throws, released or not: VideoFrame[slot=2 1920x1080 NV12 IO_SURFACE pts=…]
      Overrides:
      toString in class Object