Package limn.video.ffmpeg

Class FfmpegMedia

java.lang.Object
limn.video.ffmpeg.FfmpegMedia
All Implemented Interfaces:
AutoCloseable
public final class FfmpegMedia extends Object implements AutoCloseable
An open container, and the tracks inside it: one picture stream, and whichever of its soundtracks is playing.

This type exists because nothing in limn.video models a container. VideoDecoder.openStream returns video and only video, which is right for a decoder facade and leaves a real MP4's soundtrack with nowhere to arrive. Rather than widen an SPI that five landed phases depend on, the module that demultiplexes publishes the pairing itself, in terms of the two types the toolkit already defines: 


 FfmpegMedia media = FfmpegMedia.open(Path.of("clip.mp4"));
 MediaPlayer player = new MediaPlayer(media.video());
 if (media.hasAudio()) {
     player.setAudio(media.audio(), PlayOptions.DEFAULTS.withBus(AudioBus.MUSIC));
 }
 player.start();
 // ... and when the application is finished:
 player.close();     // joins the decode thread, so nothing is reading any more
 media.close();      // and only then

Nothing above this module needs a line changed to use it: video() is a VideoStreamSource and audio() is an AudioStreamSource, and a player takes exactly those two.

Who closes what

This object owns the container, and closing it is what frees the decoder. Closing the video track does the same thing, because a caller that reached the track through Videos.open never sees this object and closing what it was given has to work.

Several soundtracks

A container may hold more than one audio track; a film with two languages is the ordinary case. audioTracks() lists every one of them and audio(int) opens the one asked for; audio() keeps meaning the container's default, which is what a caller that asks for nothing has always been given.

One is open at a time, and taking a second one ends the first. The source audio(int) returns is transferred, exactly as audio()'s is, so asking for another track while the engine is still streaming the last one is the case that has to be right, and what it does is end the old track rather than invalidate it: its readFrames reports 0, the engine treats that as the track finishing, and the engine closes it the way it closes every source it was given. Nothing is left for the caller to remember, and a close arriving from the superseded track does not disturb the new one.

Closing the audio track does not close the container. That is not an inconsistency, it is the only thing that can be right: handing a track to the audio engine transfers it, and the engine closes it on every path, so if that closed the container, a soundtrack ending would pull the decoder out from under the pictures. What closing the audio track does instead is tell the demultiplexer that nobody is reading that track, after which its packets are discarded as they are met rather than queued.

A call that arrives after the container is closed is answered, not punished. Reading video reports the end, reading audio reports zero frames (which is what the end of a track means to the audio engine) and releasing a picture does nothing. That is deliberate: the soundtrack may outlive the player, so the engine's streaming thread can genuinely still be inside a read when an application closes the container, and the alternative to answering it is a use-after-free in a thread the application does not know exists.

Subtitles

subtitleTracks() lists them and selectSubtitles(int) chooses one; subtitles() is where the cues come out, asked for by position. Nothing is selected when a container opens, whatever the file marks as its default: whether a viewer wants subtitles is not a fact about the file, and an unselected track's packets are freed as they are met, so a release carrying a dozen languages costs nothing for the eleven nobody reads.

What crosses is text and an interval, and drawing it is the application's. Nothing here rasterises a cue into the picture: that needs libass, fontconfig, freetype and harfbuzz, it means downloading a hardware picture and uploading it again for every frame, and the result could not afterwards be turned off, moved or restyled.

Threads

Any thread, and several at once, which is not a courtesy but a requirement: the pictures are pulled by a player's decode thread, the soundtrack by the audio engine's streaming thread, and a picture is released by whichever thread happened to finish with it. A read-write lock is what makes that safe: every call takes the read lock, so none of them waits for another, and close() takes the write lock, so it waits for the reads that are in flight and everything after it finds the handle gone.

  • Field Details

    • DEFAULT_SLOTS

      public static final int DEFAULT_SLOTS
      Pictures the decoder may have in flight at once.

      Two is the smallest that works (one being shown while the next is produced), and four leaves room for a player's ring without the decoder stalling every time the consumer is a frame behind. Each one costs a reference to a picture libavcodec already allocated, not a picture of its own, so the number buys latency rather than memory.

      See Also:

    • NO_SUBTITLES

      public static final int NO_SUBTITLES
      What selectSubtitles(int) takes to turn subtitles off, and what selectedSubtitleTrack() answers when none is on.
      See Also:

  • Method Details

    • open

      public static FfmpegMedia open(Path file)
      Opens file, taking its video track and its audio track if it has one.

      This blocks, and not briefly. It opens the input, then asks libavformat to probe the container (which reads and decodes real packets to fill in what a header does not state), then enumerates the tracks, opens a decoder, and on the hardware path creates a platform decode device. That is far longer than a frame on an ordinary file, and it grows with the container's stream count, with the file's bitrate, and on a cold or network volume with the disk. The first call in a process additionally links the native library, which is an extraction of tens of megabytes on a build that carries them in its jar.

      So: any thread except the UI thread, which is a freeze for the whole of it. There is nothing thread-affine about the container itself (it is opened wherever this is called and may be handed over afterwards), but each track it hands out then follows the usual rule and belongs to whichever thread reads it, the video stream to the thread that decodes it and the soundtrack to the audio engine.

      There is no asynchronous form, and the reason is ownership. What this returns is one container holding several tracks, and closing it closes all of them, so a job wrapping it needs a disposer that knows which tracks the caller took and in which order to let them go, which is a fact about the call site rather than about the open. A caller that wants the work off the calling thread writes that job itself: a body returning whatever record it needs (the container, the tracks it took, anything it derived from them) and one disposer on it that closes them in the caller's own order. Videos.openAsync is the ready-made form for the one case that has a single obvious owner, a container reduced to its video track; anything wanting the soundtrack or the subtitles is past what that shape can carry.

      Throws:
      FfmpegException - if the native library is not loaded, the input is not a container this build can demultiplex, or it holds no video
      NullPointerException - if file is null

    • open

      public static FfmpegMedia open(Path file, boolean withAudio, int slots)

      Blocks exactly as open(Path) does, and must not be called on the UI thread for the same reason.

      Parameters:
      slots - pictures in flight at once, in [1..16]
      Throws:
      IllegalArgumentException - if slots is outside that range

    • open

      public static FfmpegMedia open(Path file, boolean withAudio, int slots, FfmpegMedia.Hardware hardware)
      Opens file, choosing explicitly whether its pictures may be device handles.

      Blocks exactly as open(Path) does, and must not be called on the UI thread for the same reason, with one addition: FfmpegMedia.Hardware.PREFER also creates a platform decode device here, which is a driver call and is the slowest single step of an open on the machines that have one.

      Parameters:
      slots - pictures in flight at once, in [1..16]
      hardware - whether to attach a platform accelerator; see FfmpegMedia.Hardware, and isHardwareDecoding() for what was actually attached
      Throws:
      IllegalArgumentException - if slots is outside that range

    • file

      public Path file()
      Returns:
      the file this was opened from

    • video

      public VideoStreamSource video()
      Returns:
      the pictures; never null, because a container with no video never opens

    • isHardwareDecoding

      public boolean isHardwareDecoding()
      Returns:
      whether an accelerator is attached, and therefore whether this stream's pictures are VideoFrame.Kind.IO_SURFACE handles rather than samples. False for every software decode, for every codec with no accelerator, on every platform without one, and whenever FfmpegMedia.Hardware.OFF was asked for, so a caller that needs to know asks this rather than assuming what it requested

    • hasAudio

      public boolean hasAudio()
      Returns:
      whether a soundtrack is open; see audioTracks() for what the file holds

    • audio

      public AudioStreamSource audio()
      The container's default soundtrack (FfmpegMedia.AudioTrack.isDefault()), which is what a caller that does not care which track it gets has always been given.

      Called twice, this hands back the same source both times. Called after audio(int) took a different track, it takes the default back and supersedes that one, because that is what asking for the default means.

      A source that has been closed is not reopened by asking again: closing is the consumer saying it is done with that track. Selecting another track and coming back opens a fresh one.

      Returns:
      the soundtrack, or null: a container may have none, and one whose audio codec this build was not compiled with is opened without it rather than refused, because a film with no sound is better than no film

    • audioTracks

      public List<FfmpegMedia.AudioTrack> audioTracks()
      Every audio track the container holds, in the container's own order, including any this build has no decoder for, because a track that cannot be played is still a track the file has and hiding it would report a two-language film as having one language.
      Returns:
      an unmodifiable list, empty when the container carries no audio at all. Read once at open: a container's tracks do not change while it is open

    • audio

      public AudioStreamSource audio(int index)
      Opens audio track index and hands it over, ending whichever track was open.

      This is not a getter. The source it returns is the caller's to close, exactly as audio()'s is, and handing one to the audio engine transfers it, so a caller that takes a track and never gives it to a player must close it, and a caller that does give it to one must not.

      What happens to the track that was open is the part worth reading. It is ended, not invalidated: every subsequent readFrames on it reports 0, which is what the end of a track means to the audio engine, so an engine streaming it stops and closes it on its own. A close arriving from that superseded source afterwards does nothing; in particular it does not tell the demultiplexer that nobody is reading the track that is now open, which is the one way this could go wrong quietly. Asking for the track that is already open hands back the same source, closed or not, and disturbs nothing.

      Only one track is decoded at a time. Two at once would be two decoders, two packet queues and two positions to keep coherent across a seek, for samples no consumer reads: the audio engine mixes one source per player.

      A container whose default track this build has no decoder for opens with no soundtrack rather than refusing to open, and another of its tracks can still be asked for here, which is the point of the undecodable ones being listed.

      Parameters:
      index - a position in audioTracks()
      Throws:
      IndexOutOfBoundsException - if there is no such track
      IllegalStateException - if the container was opened without audio
      FfmpegException - if this build has no decoder for that track

    • selectedAudioTrack

      public int selectedAudioTrack()
      Returns:
      the position in audioTracks() of the track being decoded, or -1 when there is none: a container with no audio, or one opened without it

    • subtitleTracks

      public List<FfmpegMedia.SubtitleTrack> subtitleTracks()
      Every subtitle track the container holds, in the container's own order, including any this build cannot decode and any that is a bitmap format, because a track that cannot be shown is still a track the file has.
      Returns:
      an unmodifiable list, empty when the container carries no subtitles at all. Read once at open: a container's tracks do not change while it is open

    • selectedSubtitleTrack

      public int selectedSubtitleTrack()
      Returns:
      the position in subtitleTracks() being decoded, or NO_SUBTITLES. NO_SUBTITLES when a container opens, whatever the file marks as its default: whether a viewer wants subtitles is the application's question, and a track nobody selected costs nothing because its packets are freed as they are met

    • selectSubtitles

      public void selectSubtitles(int index)
      Chooses which subtitle track is decoded, or turns subtitles off with NO_SUBTITLES.

      The cues held by subtitles() are dropped either way: they belong to the track that was open, and showing them over another one is the failure this exists to avoid.

      Only one track is decoded at a time, for the reason audio(int) gives, and one more: a second track's packets would be queued for a consumer that never polls them.

      Parameters:
      index - a position in subtitleTracks(), or NO_SUBTITLES
      Throws:
      IndexOutOfBoundsException - if there is no such track
      FfmpegException - if the track is a bitmap format, which this SPI does not carry, or if this build has no decoder for it. Both are refused by name rather than opened to show nothing

    • subtitles

      public SubtitleCues subtitles()
      The cues of whichever subtitle track is selected.

      Never null, and empty while nothing is selected, so a paint loop can ask without a null check and without knowing whether the viewer turned subtitles on.

      Returns:
      the same object for this container's whole life; selecting another track changes what it answers rather than replacing it

    • queuedSubtitlePackets

      public long queuedSubtitlePackets()
      Returns:
      subtitle packets waiting to be turned into cues. Zero while no track is selected, which is the point: an unselected track's packets are freed as they are demultiplexed rather than queued, so a release carrying a dozen languages costs nothing for the eleven nobody is reading

    • droppedSubtitlePackets

      public long droppedSubtitlePackets()
      Returns:
      subtitle packets thrown away because the queue was full, which means the application selected a track and then stopped asking for cues while the pictures ran on. Zero throughout an ordinary playback

    • audioSourceChannels

      public int audioSourceChannels()
      Returns:
      how many channels the file actually holds in the selected track, before the fold to the one or two the audio engine will admit; 0 when there is no soundtrack. A 5.1 track reports 6 here and 2 from audio().channels().

    • droppedPackets

      public long[] droppedPackets()
      Packets thrown away because a track's queue was full, which only happens when that track's consumer has stopped reading, since the bound is far above any interleaving a muxer produces. Zero throughout an ordinary playback.
      Returns:
      dropped video packets, then dropped audio packets

    • queuedPackets

      public long[] queuedPackets()
      Packets waiting for a consumer that has not asked for them yet: the video track's, then the selected audio track's. A track nobody is reading at all queues nothing, because its packets are freed as they are demultiplexed rather than held; so is every audio track that is not the selected one, which is what keeps a film's other languages costing nothing.
      Returns:
      queued video packets, then queued audio packets

    • containerSeeks

      public long containerSeeks()
      Returns:
      how many times this container's demultiplexer was actually moved. Lower than the number of seeks asked for whenever both tracks asked for the same target: one position serves both, so a target each of them asks for is one move and not two

    • close

      public void close()
      Releases the decoder and the input. Idempotent, and safe while another thread is reading.
      Specified by:
      close in interface AutoCloseable

    • isOpen

      public boolean isOpen()
      Returns:
      whether the container is still open, for a diagnostic

    • canWriteClip

      public static boolean canWriteClip()
      Whether writeClip(java.nio.file.Path, limn.video.ffmpeg.FfmpegMedia.ClipCodec, int, int, int, int, int, int, int) can do anything on this build.

      False for the library that ships, which holds no encoder at all: a player does not encode, and an encoder is not merely bytes; MPEG-4 Visual and AVC are licensed separately for encoding and for decoding, so shipping one would buy patent surface for a capability nothing uses. The build that carries encoders is produced by the limn-ffmpeg-natives repository's scripts/build-ffmpeg.sh --profile full, never published, and picked up from a sibling clone by this module's tests and the Kitchen Sink so they can make a file to read rather than commit one (ADR 037).

      A field read, except possibly once: the first call in a process may be the one that links the native library, which on a build carrying the libraries in its jar extracts tens of megabytes under a global lock. Ask it on a worker, or after FfmpegVideoDecoder.warmUp() has paid for the link somewhere else.

    • writeClip

      public static void writeClip(Path path, FfmpegMedia.ClipCodec codec, int width, int height, int frames, int rateNum, int rateDen, int audioChannels, int sampleRate)
      Writes a real MP4 (a real encoded video track, and optionally a real AAC one) so that something exists to demultiplex. Nothing this build can generate is committed, which makes producing one the only honest way to have it.

      The picture is eight flat colour bars in the studio code table, shifting one bar per picture so that a stream which is not advancing is visible without a stopwatch. Flat on purpose: an encoder moves every sample, so an assertion can only be about an area's mean, and large flat areas are what make that tight enough to catch a swapped chroma pair.

      Blocks for the whole encode (every picture, every sample of every soundtrack, and the mux, all through the native library), which is far longer than a frame and grows with frames and the picture size. So it belongs off the UI thread. There is no asynchronous form of it: a caller that needs one wraps this call in Ui.work and owns the job that results.

      Parameters:
      audioChannels - channels in the soundtrack, or 0 for no soundtrack. Above 2 produces a track the audio engine will not admit as it stands, which is the case the fold to stereo exists for.
      Throws:
      FfmpegException - if this build has no encoder, or the file cannot be written

    • writeClip

      public static void writeClip(Path path, FfmpegMedia.ClipCodec codec, int width, int height, int frames, int rateNum, int rateDen, List<FfmpegMedia.ClipAudioTrack> audio, int sampleRate)
      Writes a clip with any number of audio tracks, so that there is something to select between.

      Every track sounds different, on purpose. The first track's first channel is 440 Hz, each further channel is an octave up, and each further track is an odd multiple, so no two of the frequencies coincide, and a test that asked for track 1 and is hearing track 0 sees a wrong tone rather than a level it has to interpret. That is what makes a wrong index and a downmix into failures instead of judgement calls.

      Blocks for the whole encode and has no asynchronous form, for the reason stated on the overload that takes a single soundtrack; more tracks make it longer.

      Parameters:
      audio - one entry per audio track, in the order they are written; empty for no sound
      Throws:
      FfmpegException - if this build has no encoder, if the file cannot be written, or if a track's tones would not fit below half the sample rate

    • writeClip

      public static void writeClip(Path path, FfmpegMedia.ClipCodec codec, int width, int height, int frames, int rateNum, int rateDen, List<FfmpegMedia.ClipAudioTrack> audio, int sampleRate, List<String> subtitleLanguages)
      Writes a clip that also carries subtitle tracks, so that there is something to read cues from.

      The cues are generated rather than supplied, the same way the tones are: each one's text names its track and its own index ("T0 C3"), so a reader holding the wrong track, or a cue from where the film used to be, sees which rather than having to infer it. They are contiguous, one per ten pictures, so at every instant of the clip exactly one cue is on screen and a seek assertion is about a string rather than about an interval.

      The first cue of each track carries ASS override tags and a hard line break and the rest do not, which is what makes the markup rule assertable in both directions from one file.

      Blocks for the whole encode and has no asynchronous form, for the reason stated on the overload that takes a single soundtrack; more tracks make it longer.

      Parameters:
      subtitleLanguages - one entry per subtitle track, each a tag or null to state none
      Throws:
      FfmpegException - if this build has no encoder, if the file cannot be written, or if a track's tones would not fit below half the sample rate

    • identity

      public static String identity()

      Reads a string out of the linked library, except possibly once: the first call in a process may be the one that links it, which on a build carrying the libraries in its jar extracts tens of megabytes under a global lock. Ask it on a worker, or after FfmpegVideoDecoder.warmUp() has paid for the link somewhere else.

      Returns:
      the licence, version and configure line of the linked FFmpeg, newline separated: what LicenceTest reads to assert that this build is still LGPL and still opens nothing but files
      Throws:
      FfmpegException - if the native library is not loaded

    • components

      public static String components()
      Every codec and container the linked libraries actually hold, read out of them rather than recited from the build script, one per line, as decoder:h264, encoder:mpeg4, demuxer:mov or muxer:mp4.

      A configure flag is a claim and a linked symbol is a fact. A build whose decoder list quietly lost an entry (dropped in an edit, or refused by configure because a dependency was switched off) would still advertise the codec and then fail to open the file, which is the one failure this answers.

      Enumerates out of the linked library, and carries the same first-call cost as identity(): the first consultation in a process may be the one that links it.

      Throws:
      FfmpegException - if the native library is not loaded