Images and media
Pictures
Section titled “Pictures”Images loads them and ImageView shows them:
Image logo = Images.fromResource("app/logo.png");ImageView view = new ImageView(logo);An image whose header claims more than 2^28 pixels is refused before it is decoded, rather than asking for gigabytes.
Images.load(Path), Images.fromResource(String) and Images.decode(byte[]) all read on
the thread that calls them, which is fine during startup and not fine in a click handler.
Anywhere a frame is already running, load on the worker pool and take the result on the UI
thread instead. A file or a resource has a name, so Images.loadShared and
Images.fromResourceShared cache it: every caller of the same path gets the same Image and
the GPU gets one texture. Bytes you already hold have no name, so Images.decodeAsync is a
job you start, and may cancel:
/** A picture from disk: read and decoded on the worker pool, set on the UI thread. */static void showPicture(ImageView view, Label status, Path file) { Images.loadShared(file) // cached: every caller shares one Image .thenAccept(view::setImage) // runs on the UI thread .exceptionally(error -> { status.setText("Could not open " + file.getFileName()); return null; });}
/** Bytes already in memory (a download, a database blob): a job you start, and may cancel. */static void showDownloaded(ImageView view, byte[] png) { Images.decodeAsync(png) .onSuccess(view::setImage) .start();}
/** A sound bundled with the application. */static void preloadClick(java.util.function.Consumer<AudioClip> ready) { Sounds.fromResourceShared("sounds/click.wav").thenAccept(ready);}Each window keeps the texture of every image it draws, and the Image behind it, until 300
frames go by without drawing it, or sooner, least recently drawn first, while it holds more than
Images.textureBudget(): 256 MiB by default, mipmaps included, about twenty-four photographs of
1920 × 1080. What a frame draws is never dropped, so a single picture larger than the budget
still shows. An application short of memory lowers it, and one that keeps coming back to a set
of pictures bigger than that raises it, with Images.setTextureBudget.
The shared loads have a budget of their own, on the heap: what loadShared and
fromResourceShared loaded is kept, so that asking again costs nothing, while it fits a quarter
of the most heap the process may use. Past that the picture asked for longest ago is let go, and
every file before any resource, since an application’s own resources are few and it comes back
to them. Letting go is not losing: a picture still on screen, or held anywhere, is the same
object when it is asked for again, and only one that nobody holds is read a second time.
Images.setSharedBudget moves the budget. For the pictures that must never be read twice,
whatever else is loaded, Images.loadPinned and Images.fromResourcePinned load the same way
and keep the result outside the budget until Images.unpin, once for each time it was pinned,
so two parts of an application can each pin one picture. Sounds has the same calls, with
an eighth of the heap.
Pictures that move
Section titled “Pictures that move”An animated GIF, PNG or WebP decodes to an Image that says isAnimated(). It is an image
like any other, and its own pixels are its first frame’s, so anything that draws it without
playing it shows a still picture. Give it to an ImageView and it plays, by itself and for as
many turns as the file asks for. A sheet of sprites is the same thing made
by hand, and Images.animation(frames, millis, turns) takes frames from anywhere:
/** An animated file, played by the view it is given to. */static ImageView spinner(String resource) { ImageView spinner = new ImageView(Images.fromResource(resource)); spinner.setSpeed(1.5f); // half again as fast as the file says spinner.setLoop(ImageAnimation.Loop.FOREVER); // whatever the file says return spinner;}
/** Holds the frame shown, and plays on from it. */static void pause(ImageView view, boolean paused) { view.setPlaying(!paused);}
/** A sheet of sprites: eight frames from one row, each shown for 90 ms. */static Image coin(Image sheet) { return Images.fromSheet(sheet, 8, 1, 90);}
/** Every frame without loss, as an animated PNG; ImageFormat.GIF writes 256 colours. */static void save(Image animation, Path file) { Images.save(animation, ImageFormat.PNG, file);}A picture playing costs your window one frame for each of its own: an image view asks for none
between two frames of its picture, and none at all while it is hidden. A widget of your own that draws frames itself uses
ImageAnimation, the clockwork an image view is built on: hand it the time from a scene’s
ticker and draw frame().
A small animation is decoded whole when it is loaded; one whose frames together pass
Images.setAnimationBudget, 32 MiB unless you say, keeps its file’s bytes instead and decodes
a few frames ahead of the one shown as it plays, so that a long clip costs its file and a
handful of frames. Loaded with loadShared, each frame is kept or let go by the shared budget
like any picture, and loadPinned keeps them all.
A moving picture is not stopped by the system’s setting for less motion, which is for the
motion an interface adds to itself. One that moves for more than a few seconds should be
something your user can stop, and setPlaying is what that control calls.
Going the other way, Images.encode(…) and Images.saveAsync(…) write PNG and GIF, still or
animated: an animated PNG keeps every frame without loss, and a GIF holds 256 colours.
view.setDragEnabled(true) lets the user drag the picture out of its view, to another part of
your window or to another program; Drag and drop
has the details, the clipboard among them.
The toolkit takes an Icon wherever a small vector mark belongs: a button, a text field’s
leading slot, a menu item. Icons are drawn as paths rather than loaded as bitmaps, so they
stay sharp at every control size and pick up the theme’s colour. An SvgIcon whose source the
renderer refuses draws nothing and says why in the log, once.
An SvgIcon draws any SVG: SvgIcon.of(text), fromResource(path), or load(path), whose
pictures beside it in its own folder are shown too. Gradients, patterns, clipping, masks,
markers, style sheets and text are drawn; text in the faces the toolkit bundles and in those the
system has. Of any aspect, it is fitted into the box it is given. An illustration that should fill
a space at any size goes in an ImageView, which draws an icon at the view’s own size and density
each time either changes:
ImageView empty = new ImageView(SvgIcon.fromResource("/app/illustrations/empty-inbox.svg"));Images.decode reads an SVG too, drawn at its own size, when you want its pixels.
The toolkit reads every SVG itself before drawing it, and nothing in one runs or reaches out: no
script, no event, no address on the network, no file outside the SVG’s own folder. A file that is
damaged, or larger than any picture should be, is refused with a MalformedContentException.
An SVG may move, by SMIL (<animate>, <animateTransform>, <set>) or by CSS (@keyframes).
It is then an animated icon like any other, played by itself in every component and in an
ImageView, at 24 frames a second unless you say otherwise: svgIcon.withFrameRate(60) for one,
or -Dlimn.svg.frameRate=60 for every SVG. A file never says, since neither SMIL nor CSS has a
frame rate; both measure time, and the toolkit draws that time as frames.
An icon may move. BitmapIcon.picture(spinner), or mask and themed, of an animated image
is an animated icon, and every component that shows it plays it by itself at the image’s own
times: a button saying it is busy, a tab whose content is loading, a row of a list that is
syncing. It costs its window one frame for each of its own, repaints only its own box, and
waits while nobody sees it. A tray plays one, and an ImageCursor of an animated image is a
cursor that plays. A menu shows the first frame, as every system’s menu does. A component
offers no control over its icon; to hold, speed or loop one, show it in an ImageView.
The limn-icons-tabler artifact packages the Tabler icon set if you would rather not draw your
own — one enum constant per icon, drawn through the same SvgIcon. It is versioned with Tabler
rather than with the toolkit (3.48.0.x is Tabler 3.48.0), so it stays in your cache across
Limn upgrades:
implementation("io.github.limn-toolkit:limn-icons-tabler:3.48.0.1")<dependency> <groupId>io.github.limn-toolkit</groupId> <artifactId>limn-icons-tabler</artifactId> <version>3.48.0.1</version></dependency>On the module path it is the module limn.icons.tabler, which an application that is a module
of its own requires beside the backend.
AudioClip click = Sounds.fromResource("app/click.wav");Sounds.play(click);WAV, Ogg Vorbis and MP3 decode out of the box, at sample rates up to 384 kHz: a file above that
is refused with an UnsupportedOperationException rather than played slow, which is what the
audio device would do with it. Sounds.isAvailable() reports whether an
audio device was found at all. On a machine with none, playback is a no-op rather than an
exception, so a game does not need a silent-mode branch.
Loading is I/O, so once the window is up prefer Sounds.loadShared(…) and
Sounds.fromResourceShared(…), which read and decode on the worker pool, complete on the UI
thread and share one clip between callers, as the image loaders above do; Sounds.decodeAsync
is the job for bytes in memory.
VideoView is a widget that shows a stream your application opened. It never opens a file,
never closes what it is given and knows nothing about decoders, so the rest of this section is
where the stream comes from:
VideoView view = new VideoView();view.setFit(VideoView.Fit.CONTAIN);view.setLooping(true);It is a widget in every sense: a scroll view clips it, a stack draws over it, and it takes part in layout like a label does. The decoder behind it is a separate concern:
-
The pure-Java decoders ship in
limn-toolkititself, with no native code and no third-party dependency. They read Y4M and generate test patterns, which is enough to drive the widget and not enough to play a film. -
limn-video-ffmpegdecodes H.264, HEVC, VP9 and VP8 video, with AAC, Opus or Vorbis audio, out of MP4 and Matroska (WebM included), through a trimmed FFmpeg. The FFmpeg libraries are a separate artifact,limn-ffmpeg-natives, versioned with FFmpeg rather than with the toolkit (so it stays in your cache across Limn upgrades), onenatives-<os>-<arch>classifier per desktop target, for the same platforms the backend covers (macOS, Windows and Linux, x64 and ARM64 alike); add the one your machine is, orlimn-video-ffmpeg-natives-all, a POM versioned with the toolkit that names all six at the payload version this release was tested with, for a bundle that ships everywhere. The JNI shim comes withlimn-video-ffmpegitself. Leave the libraries out and the decoder reports itself unavailable and everything else keeps working. They are LGPL (version 2.1 or later) and each jar carries the licence text and notice alongside them; Packaging has what shipping them means.build.gradle.kts implementation("io.github.limn-toolkit:limn-video-ffmpeg:1.0.0")// A build for one known machine takes the payload's classifier for it:runtimeOnly("io.github.limn-toolkit:limn-ffmpeg-natives:9.0.2.1:natives-macos-aarch64")// A bundle that ships everywhere takes the POM that names all six:runtimeOnly("io.github.limn-toolkit:limn-video-ffmpeg-natives-all:1.0.0")pom.xml <dependency><groupId>io.github.limn-toolkit</groupId><artifactId>limn-video-ffmpeg</artifactId><version>1.0.0</version></dependency><!-- A build for one known machine takes the payload's classifier for it: --><dependency><groupId>io.github.limn-toolkit</groupId><artifactId>limn-ffmpeg-natives</artifactId><version>9.0.2.1</version><classifier>natives-macos-aarch64</classifier><scope>runtime</scope></dependency><!-- A bundle that ships everywhere takes the POM that names all six: --><dependency><groupId>io.github.limn-toolkit</groupId><artifactId>limn-video-ffmpeg-natives-all</artifactId><version>1.0.0</version><type>pom</type><scope>runtime</scope></dependency>
Where the stream comes from
Section titled “Where the stream comes from”A backend brings an image decoder and an audio engine, and no video decoder: one has a size and
a licence, so which a process carries is the application’s choice. Install it once, where the
application starts, and open a file with Videos.openAsync, which does the slow part (the
header, the probe, the decoder, and the first time the native library) on the worker pool and
hands the stream over on the UI thread:
/** Once, where the application starts: the decoder that reads MP4 and Matroska. */static void installDecoders() { Videos.installDecoder(new FfmpegVideoDecoder()); Videos.warmUpAsync().start(); // links the native library off the UI thread}
/** The picture alone: a clip with no sound, or one whose sound the screen does not want. */static Job showClip(VideoView view, Label status, Path file) { return Videos.openAsync(file) // opened on the worker pool .onSuccess(stream -> view.setSource(stream).setPaused(false)) .onFailure(error -> status.setText(error.getMessage())) .start();}A video starts paused, showing its poster, until something asks it to run: setPaused(false) as
above, a play button, or setAutoplay(true) before the stream is set. Videos.open is the same
open on the calling thread, for setup code, and Videos.canOpen(file) answers cheaply whether an
installed decoder claims a file, for a control deciding whether to enable itself. The stream is
yours to close: the view leaves it open when it is replaced or taken out of the window, so that
one stream can be shown twice or handed on.
A film with its sound
Section titled “A film with its sound”A stream is pictures alone. A film with a soundtrack plays through a MediaPlayer, which
decodes ahead on a thread of its own and times the pictures by the sound. FfmpegMedia, from
limn-video-ffmpeg, opens the container and hands over both tracks:
/** A film and its soundtrack; closed by the screen once it no longer shows them. */record Film(FfmpegMedia media, MediaPlayer player) implements AutoCloseable { @Override public void close() { player.close(); // stops reading first, media.close(); // then frees the decoder }}
/** Opens a film with its sound, shows it with the built-in controls, and starts it. */static Job playFilm(VideoView view, Path file, Consumer<Film> opened) { return Ui.work(progress -> FfmpegMedia.open(file)) // a header, a probe, a decoder: slow .onSuccess(media -> { MediaPlayer player = new MediaPlayer(media.video()); if (media.hasAudio()) { player.setAudio(media.audio(), PlayOptions.DEFAULTS.withBus(AudioBus.MUSIC)); } view.setPlayer(player).setControlsVisible(true); player.start(); // a view never starts a player itself opened.accept(new Film(media, player)); }) .onDiscarded(FfmpegMedia::close) // withdrawn before it arrived .start();}
/** The subtitle lines to draw over the picture now, once a track is chosen. */static List<String> subtitleLines(Film film) { List<String> lines = new ArrayList<>(); for (SubtitleCues.Cue cue : film.media().subtitles().activeAt(film.player().positionMicros())) { lines.add(cue.text()); } return lines;}view.setPlayer(player) shows what the player decoded, and the view’s pause, seek and looping
then drive the player, so the sound pauses with the picture. setControlsVisible(true) lays the
built-in MediaControls over the picture: play and pause, a scrub bar, the position, and, while
the media can sound, a mute button and a volume. The bar is a widget of its own as well:
new MediaControls(view) puts it anywhere, under the picture say, and addLeading and
addTrailing take widgets of yours into its row, a track chooser or a subtitle button. From
code, player.setGain(0.5f) sets the soundtrack’s volume, from 0 to 1. Close the player before
the container, as Film does: closing the player joins its decode thread, and after it nothing
reads the container. A film in two languages has two soundtracks: media.audioTracks() lists
them and media.audio(index) opens the one asked for.
Before it plays. What a video shows before its first play is its poster: the first picture
unless you say otherwise, setPoster(Poster.at(Duration.ofSeconds(12))) for the picture at a
time, Poster.image(image) for a picture of yours, or Poster.none() for the bar colour alone.
Playback starts from the beginning whichever it is, and an image poster reads nothing from the
stream until then.
Subtitles come out of the container as text and timing, and drawing them is yours, so they can
be restyled, moved or turned off. media.subtitleTracks() lists them,
media.selectSubtitles(index) chooses one (none is chosen when a film opens, whatever the file
marks as its default), and media.subtitles().activeAt(time) answers the cues on screen at the
player’s position, which is what subtitleLines above reads. Draw them in a widget stacked over
the view.
Viewport3D renders a scene to a texture and composites it as a 2D layer, so it is another
ordinary widget: a dialog draws over it and a scroll view clips it. What it draws is its
renderer’s, a callback handed a render pass every frame, and the usual one draws a Scene3D,
which is what a glTF model loads into:
/** A glTF model from disk: read and decoded on the worker pool, uploaded by the first frame after. */static Viewport3D modelViewer(Path file) { Viewport3D viewport = new Viewport3D().setPreferredSize(640, 420); viewport.setAnimated(false); // a still model: redraw on orbit and zoom only viewport.setController(new OrbitController(viewport.camera())); // drag orbits, wheel zooms GltfModel[] model = {null}; GltfModel.DecodedTextures[] textures = {null}; Scene3D[] shown = {null};
viewport.setRenderer((pass, seconds) -> { if (shown[0] == null && textures[0] != null) { Scene3D scene = model[0].toScene3D(textures[0]) // the upload: it needs the frame .background(new Vec4(0.08f, 0.08f, 0.1f, 1f)) .exposure(1.1f); scene.root().add(new LightNode(Light.Directional.of( new Vec3(0.4f, 0.9f, 0.5f), new Vec3(1f, 0.96f, 0.9f)))); shown[0] = scene; Ui.post(() -> viewport.frameContent(scene.bounds())); // the camera moves after this frame } if (shown[0] != null) { float aspect = viewport.height() > 0 ? viewport.width() / viewport.height() : 1f; shown[0].render(pass, viewport.camera(), aspect); } });
GltfLoader.loadAsync(file) .onSuccess(loaded -> loaded.decodeTexturesAsync() .onSuccess(decoded -> { model[0] = loaded; textures[0] = decoded; viewport.invalidate(); // the next frame uploads and draws it }) .start()) .start();
viewport.observeDispose(() -> { // taken out of its window: free the GPU copy if (shown[0] != null) { shown[0].dispose(); shown[0] = null; } }); return viewport;}Read that in order:
GltfLoaderreads the model off the UI thread:loadAsync(path),fromResourceAsyncfor one bundled with the application, or the synchronousloadfor setup code. It reads.gltfwith its buffers embedded and.glb: the node hierarchy, the meshes and the metallic-roughness materials with their textures; skinning and animation are not read.decodeTexturesAsyncdecodes the pictures on the worker pool too.toScene3Duploads it, inside a frame, since it needs the graphics context: that is why it runs in the renderer, once, when the decoded model has arrived. TheScene3Dit returns owns what it uploaded, anddispose()frees it, whichobserveDisposeruns when the viewport leaves its window.- A scene is nodes.
scene.root()takes aMeshNode, a mesh with aMaterialand a place, and aLightNodewith aLight:Light.Directional,Light.PointorLight.Spot. A material isMaterial.Pbr, metallic-roughness, in most scenes;Material.Unlitignores the lights.background,ambient,exposure,castShadowsandenvironment, a sky the metals reflect, are the scene’s own. - The camera is
viewport.camera():eye,targetandfovyplace it, andviewport.frameContent(scene.bounds())backs it off until the whole model is in the frame. AnOrbitControllercircles the target as the pointer drags and zooms with the wheel and with a pinch; aFlyControllerturns the camera where it stands and flies forward and back.
It renders physically-based materials to a linear high-dynamic-range target and applies the display transform once, when the result is composited, which is why a bright highlight in the viewport does not clip differently from the rest of your window.
setAnimated(true) asks for a frame every tick; leave it off and the viewport only redraws
when something changes, which is what you want for a static model on a laptop battery.
onClick gives you a ray through the pointer for picking, and captureNext hands you the
next rendered frame as an Image.
A surface of your own. Material.Surface takes a shader written in Java rather than in a
shading language: limn.render3d.shader is a small typed language of Expr nodes, and a surface
is the SurfaceOutputs they compute, a base colour, a roughness, an emission and the rest, which
the backend turns into the shader its graphics API runs. StandardSurface.metallicRoughness() is
the built-in material written that way, and the model for one of your own. An Expr.Loop repeats
a step a bounded number of times, at most Expr.Loop.MAX_ITERATIONS (1024), with an optional
condition that ends it early, so a fractal or a ray marched through a volume is one node; a node
used in two places is computed once.
A file that is not what it claims to be
Section titled “A file that is not what it claims to be”A file a user picks may be cut short, damaged, or made on purpose to break whatever reads it.
Every loader refuses such a file with one exception, MalformedContentException, whose
message says what is wrong:
try { picture.setImage(Images.load(chosen));} catch (MalformedContentException broken) { status.setText("That file is not a picture we can read: " + broken.getMessage());} catch (UncheckedIOException unreadable) { status.setText("That file could not be opened.");}The other two things that can go wrong are kept apart from it. A file that cannot be read at
all is an I/O failure. A file that is whole but of a kind nothing here plays, a sound with
six channels, is an UnsupportedOperationException. broken.format() names the format, and
broken.position() the line or the offset where the reader knows one.
What may come from anywhere: a picture, a sound, a model and a theme. Their readers hold
every size a file claims to what the file can back, so a small file cannot ask for a large
heap, and a damaged one is refused. The one exception is video: what the video library
cannot open arrives as its own FfmpegException, since it does not tell a broken file from
one it does not play.
A file that crashes a reader, hangs it or makes it ask for all of memory is a security bug in Limn: report it privately, as SECURITY.md describes, and not in a public issue.
What should come only from you, or from a source you trust: a font. The library that draws a font’s outlines checks nothing a font says about itself, and a font made to break it can end the process. Load the fonts you ship; do not offer “choose a font file” to a user whose files you would not otherwise run. The same caution, more mildly, goes for an SVG: an icon is your resource, and one a user supplies is drawn by a native library.
A sound decoded whole stops at about fifty minutes of stereo; Sounds.stream plays anything
longer without holding it in memory.