Font collection files contain multiple faces, which can have different
variable data, such as different named styles. Both the C font selection
code and the Python variable data cache were keyed on the file path alone,
so once the data for one face of a collection was cached, other faces in
the same file got that data. For example, selecting a named style that
only exists in the second face of a collection failed if the first face
was looked at first, falling back to a different family.
Key the caches on the face index for fontconfig and the postscript name
for CoreText, which does not expose the face index. Do not share cache
entries for fonts without a path in the C code, matching the Python code.
Expose the face index on the FreeType Face object for the Python cache.
- Convert OpenType axis tags to Python as four Latin-1 characters instead
of UTF-8, so that fonts with axis tags that are not valid UTF-8 no longer
cause an exception. On macOS this made font selection fail entirely if
the selected variable font had such a tag. Convert axis_map keys back to
tags losslessly, validating them, instead of reading four bytes from the
UTF-8 encoding, which could read past the end of short keys.
- variable_data_from_font_file() now raises ValueError for files that are
not valid fonts, instead of an OSError with a stale or zero errno.
- Avoid quadratic work when reading a crafted font with many axes, named
instances and name records: de-duplicate axis tags once rather than per
instance, and binary search name records by name id. A 64MB crafted font
that took over a minute to parse now takes about a second.
list_fonts() and variable_fonts() returned pointers into ctx->lists, which
is reallocated when a new list is cached. find_best_match() holds such a
pointer across sorted_candidates() -> weight_range_for_family() ->
list_fonts(), which can grow the cache and leave the pointer dangling.
Allocate each cache entry separately so returned pointers stay valid. Do
the same for the variable data cache, which had the same pattern.
Fixes#10896
Cells containing a codepoint from the registration block that had a
registered glyph when they were printed are rendered with that glyph,
into the area the character occupies: its cell, the cells given to it
by the text sizing protocol, or two cells when it is followed by a space
and the width of its viewBox when fit to the cell height is more than
one cell.
The glyph is bound to the text when it is printed, by storing a
reference to it in the text cache entry of the cell. Changing or
clearing a registration affects only text printed afterwards, so
programs using the same codepoints do not change each other's output
and registering never requires re-rendering lines. Bound entries are
kept in a separate map so that other text is not slowed down. Glyphs
are interned by a seeded hash of their content.
Glyphs that use only the foreground color are alpha masks and glyphs
that use only fixed colors are colored sprites. Glyphs that mix them are
a colored sprite followed in the sprite map by a sprite with the weight
of the foreground color, which the cell shader combines with the
foreground color, so that selection, cursor and other foreground color
changes apply to them exactly.
Sprites of a glyph are freed when the glyph is, which happens only when
no cell uses it, and are reused. Each window can keep alive a limited
number and size of glyphs, beyond which registration fails with
no_space, so a program that keeps registering and printing new glyphs
cannot exhaust memory or sprites for other windows.
A dedicated renderer for vg glyphs. Paths are flattened to polygons in
device space and rasterized by a scanline rasterizer that computes exact
winding numbers on 16 sub-scanlines per pixel row with exact horizontal
coverage, so both fill rules and unions of overlapping polygons are
correct. Strokes are generated in layer space, where the pen is round,
as a union of quadrilaterals, joins and caps, giving the elliptical pen
of SVG under non-uniform matrices. Gradients are linear and two point
conical with pad, reflect and repeat.
The output is decomposed into a premultiplied color, an alpha and a
foreground weight, so that the foreground color can be applied when
drawing, which is exact as compositing is linear in the opaque
foreground color.
Parse vg payloads into a VGGlyph structure, normalizing path data to
absolute moves, lines, quadratic and cubic Béziers and closes, with
arcs converted to cubics as the spec requires. All limits of the spec
are enforced while parsing.
Registrations are stored per screen in a GlyphRegistry owned by the
text cache of the screen. r and c are implemented and RIS clears all
registrations.
Parse the APC escape codes of the glyph protocol (kitty#10872) into a
GlyphCommand structure and reply to the s and q verbs. The s verb
advertises no payload formats until rendering is implemented. q for
codepoints in the registration block is answered from the
registrations; for all other codepoints it is answered with
reason=unknown, as kitty does not perform font lookups while parsing
input. Grapheme queries are not supported.
While rendering is paused the cursor is sized and colored from the cell
under it in the paused snapshot, but the snapshot row was taken to be the
cursor's screen row. With pixel scroll the snapshot has an extra leading
row, so the cursor took the colors of the cell above it.
Map linebuf rows to snapshot rows in one place, accounting for both the
extra pixel scroll row and the scroll position captured at pause time, and
use it for the cursor, selections and the cursor row check when rendering
the snapshot, which was off by the scroll position when scrolled back.
The choice of the line under the cursor, including the IME overlay, moves
from the renderer to screen_line_for_cursor() so that it can be tested.
Closes#10892
HarfBuzz returns the glyphs of a backward (RTL) buffer in visual order,
with decreasing clusters, but cells were consumed from the logical front
of the run, so pointed Hebrew and Arabic lam-alef ligatures were grouped
with the wrong cells.
Replace the per codepoint cell cursor with a table, built while loading
the HarfBuzz buffer, mapping codepoint offsets to cells. The clusters
grouped so far always cover a prefix (LTR) or a suffix (RTL) of the text,
so the cells they complete come straight from the table, with no
direction specific code. The codepoint used to detect special glyphs is
now that of the glyph's own cluster.
Add DejaVu Sans Mono to the test fonts for Arabic shaping tests.
Closes#10891
Kittens can opt in with Loop.DetailedWheelEvents(), after which scroll
events arrive as MOUSE_WHEEL mouse events whose Wheel field has the
deltas, unit, multiplier, source, phase and momentum/inverted flags.
WheelEvent.Lines() converts the deltas to lines using the terminal's
multiplier when available.
The mouse-demo kitten uses this to move a block drawn with the graphics
protocol up and down with pixel precision, falling back to wheel button
presses in terminals without support, and shows the event details.
Mode 5523, while mouse tracking is on, makes kitty report each scroll event
as a single CSI > ... M escape code instead of as button 4-7 presses. It
carries integer deltas in v120 units or pixels, with fractions carried
over, the user's scroll multiplier as a hint, the kind of device, the
gesture phase, and flags for momentum and inverted (natural) scrolling,
and the full set of modifiers. Only the sign of the multipliers is applied.
To support this GLFW scroll events now carry the source device, the
gesture phase and whether the direction is inverted, on Wayland, X11 and
macOS. The momentum scroller now forwards the gesture end before starting
momentum and no longer reports a cancel for momentum that never began.
PyOS_AfterFork_Child() runs Python at-fork handlers, re-initializes thread
state and allocates memory in the forked child. On macOS this can get the
child killed with SIGKILL before it execs, which is still happening on CI in
the file transmission tests even after the previous fix. The Python docs only
require PyOS_AfterFork_Child() when the child will call back into the
interpreter, which ours never does, so do not call it. Also convert pass_fds
to a C array before forking, and use only async-signal-safe functions in the
child (_exit() instead of exit(), no snprintf()).
The child side of spawn() closed every inherited fd from 3 to 255. On macOS
system frameworks such as Metal and libdispatch create guarded file
descriptors and calling close() on one of them raises EXC_GUARD which kills
the process with SIGKILL. Instead mark the inherited fds close-on-exec, which
is always allowed for guarded fds, and let exec() close them.
The snapshot of the screen taken for a synchronized update is kept after
the update ends, so its cells prevented the style table from being
emptied on hard resets and clearing of the scrollback, and kept their
styles alive through garbage collection. It is unused when rendering is
not paused and is overwritten entirely when rendering is next paused, so
only consider it while rendering is paused.
Also detect use of the formatting of stale line copies when comparing
lines and in test_shape() and test_render_line().
Lines scrolled onto the screen no longer have their render data cleared,
instead they are marked dirty so that rendering recomputes it, except when
there is a marker, as marks are render data used without rendering. This
reduces the memory written to clear a line by 40%.
The ASCII fast path in draw_text no longer checks for multicell cells in
a line that was just cleared by a line feed scrolling the screen, until
something that can create multicell cells happens.
Together these make the ascii benchmark ~14% and the scrollback and
render benchmarks ~8% and ~14% faster. Also update the throughput
numbers in the performance docs, using the median of several runs.
Runs the benchmark kitten in all installed terminals from xterm, foot,
kitty, ghostty, alacritty, wezterm, konsole, ptyxis and alacritty+tmux,
with default configuration except for font, font size and window size.
Terminals are pinned to the fast cores on machines with heterogeneous
cores and the performance governor is set if root access is available.
Emptying it whenever the whole screen was erased meant that full screen
programs, which erase the screen on every redraw, had all their styles
interned again and the style texture uploaded again every frame. Also do
not empty tables with few styles, as they are not worth it. Empty it when
the scrollback is cleared by the clear_terminal action as well.
Improves the truecolor text and half blocks benchmarks by 50% and 33%.
The history buffer is much larger than the CPU cache and lines added to
it are not read again soon. Normal stores to it stall acquiring
ownership of the destination cache lines, filling the store buffer and
stalling the drawing of the next line. Write the whole cache lines of
the cells with non-temporal stores instead, using normal stores for the
partial cache lines at either end, as partial non-temporal writes are
slow. Fence once per parse pass rather than after every line.
benchmark.py scrollback: +32%, in-process parsing to the main screen
+26% to +62% depending on the number of columns. Builds without SSE2
keep using prefetching.
Replace the --with-scrollback and --render flags with two new
benchmarks, scrollback and render, that use the ASCII text with the
main screen and with rendering allowed, respectively. All other
benchmarks use the alternate screen with rendering suppressed. Add an
--output flag to also write the results to a file as JSON and list the
available benchmarks with descriptions in the help text.
The two benchmarks now use text with the same structure: the same lines,
of the same widths in cells, ending in CRLF and with tabs at the same
positions, differing only in the characters used to fill them. Before,
the ASCII text had a bare newline about every 90 bytes, while the
Unicode text was long lines of mostly CJK text, so they were dominated
by different costs. Also report the rate at which cells are drawn for
them, since non-ASCII characters are encoded in more bytes.
Remove the temp_flag bit from CPUCell, which was used only to mark the
cells whose text is extracted for a selection, and use a temporary bitmap
for that instead. The freed bit is used for the style index, doubling
the maximum number of styles to 262144.
Copies of lines made with LineBuf.create_line_copy() are not remapped
when the style table of their screen is garbage collected, so their
style indices become invalid. Record the generation of the style table
in copies and raise ValueError when the formatting of a copy made
before a collection is used. Copying a whole line into a copy makes
its formatting valid again.
On a hard reset or when erasing the whole display, if the scrollback is
empty and no cell on any screen buffer has a style, empty the style
table, freeing the memory used by styles from, for example, a burst of
output with many colors. Checking is cheap as only the screen buffers
need to be scanned.
Upload only the sprite index and the style index and mark of each cell,
8 bytes instead of 20, and have the cell and padding shaders get the
style from a per screen texture holding the style table. Styles are
uploaded incrementally, all of them when the table is garbage collected.
Also only build the attributes and sprites of cells when rendering
lines, as nothing else is used. Together these bring the CPU cost of
preparing cells for the GPU back to what it was before styles were
interned.
The formatting of cells (colors and SGR attributes) is now interned per
screen in a StyleTable and cells store a 17 bit index into it in the
previously unused bits of CPUCell. Render data (sprite and mark) is kept
in a separate RenderCell array for screen lines and in a small render
cache for history lines being displayed, so the scrollback stores only
CPUCells, 12 bytes per cell instead of 32. GPUCell is now only the upload
format, built from the CPUCell, its style and its RenderCell.
When the table fills up, it is compacted at the start of drawing text,
reducing the color precision of cells in the scrollback if needed, cells
on screen always keep their exact styles. The map from styles to indices
is a verstable set of indices with a small cache of recent styles in
front of it.
benchmark.py throughput: ascii +48%, unicode +29%, csi +10%, truecolor
text -22%, truecolor half blocks -9%.
With --debug-rendering, the "No render frame received" message was
logged on every main loop wakeup while frames were stalled, which can be
indefinitely, for example, while a Wayland window is hidden. Report each
stall only once, till a render frame is next received.
Also report stalls on X11, when the X server stops reporting vertical
blanks and frames are rendered without them. A single missed frame while
frames are arriving is a glitch and is not reported.
Errors during startup that are not caused by invalid config lines, such
as failing to listen on the remote control socket, were shown under the
title "Errors parsing configuration" with the heading "In final effective
configuration". The title now reflects whether the errors are config
parse errors, startup errors or both.
For a second after a resize, render frames are not used, so that the
resized buffer is filled immediately. This meant that continuous output
was rendered as fast as possible, about 900 FPS, during that second.
Now only the first render after the resize is immediate, subsequent
renders are limited to 120 FPS, unless a render frame arrives sooner.
If the socket file for --listen-on was left behind by a kitty instance
that was killed, binding to it failed, so remote control was disabled
and an error was shown over the program. Now, if no process is listening
on the socket, it is removed and binding is retried. A socket that a
process is listening on and files that are not sockets are never
removed, and the error shown includes the reason for the failure.
The new render_interval option either syncs redraws to the monitor
(the default), optionally with a minimum interval between frames, or
redraws on a schedule with a minimum interval between frames. The
interval is now enforced per OS window, including when output is being
parsed, which the old repaint_delay was not.
On X11, render frames are now provided by the Present extension, with
libxcb-present loaded at runtime. Frames are requested lazily on X11,
since a pending NotifyMSC request delays buffer presentation by the GPU
driver under Xwayland. Xwayland completing requests early and failing to
report vertical blanks for long periods are handled. Without Present,
kitty falls back to blocking buffer swaps as before.
Render frames now only wake up the main loop when something is waiting
for them, so animations such as the eased cursor blink are no longer
sampled at the monitor refresh rate. On Wayland, frame callbacks are
also used when not syncing to the monitor, to stop rendering hidden
windows.
The removed options are converted to render_interval if it is not
specified anywhere, including in other config files and overrides,
otherwise they are ignored with a warning.
Also fix gen config failing on options whose type is a builtin.
The SIMD UTF-8 decoders read up to a vector past the end of their input,
which is fine for real callers that have slack after their data, but
test_utf8_decode_to_sentinel passed the Python buffer directly, so the
new long input test triggered heap-buffer-overflow reports under ASAN.
Copy the input into a padded buffer, as the other SIMD test helpers do.
Also fix clang-format violations and broken references to the removed
input_delay option in the changelog that failed the docs build.
The C font selection code read font tables only from sfnt files, so
variable fonts in WOFF/WOFF2 files had no variable data, causing wrong
faces to be selected, for example, for FiraCode-VF.woff2.
Read WOFF tables using zlib and WOFF2 tables using brotli, decompressing
only the prefix of the stream containing the needed tables, in fixed
size chunks so that memory use does not depend on possibly corrupt
offsets. kitty now links against libbrotlidec on platforms other than
macOS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Add runs of plain combining chars to a cell with a single lookup in
the text cache, rather than one for each char, which also added the
intermediate texts of the cell to the cache.
- Also collect garbage in the text cache when starting a new line, not
just when starting to draw text, as a single draw can add a great many
entries, making the cache slower to use the larger it grows.
- Size the hash map of the text cache for the expected number of entries
after garbage collection, to avoid repeatedly rehashing it as it grows.
This makes the unique multi-codepoint cells benchmark ~80% faster.
The differential test of batched drawing now compares cells by content,
since cells with multiple codepoints store indices into the text cache,
which change as intermediate texts are no longer added to it.
VBMI2 was only used by the AVX-512 UTF-8 decoder, which has been
removed, so CPUs with AVX-512 F, BW and VL but not VBMI2, such as
Skylake-X, can now use the AVX-512 implementations of the remaining
functions.
Decode 64 byte blocks of valid UTF-8 containing no ESC using the
approach of simdutf: blocks are validated with the lookup tables of
Keiser and Lemire and converted twelve bytes at a time using shuffles
looked up from a mask of the bytes that end code points, with fast
paths for runs of ASCII, two byte and three byte sequences. Anything
else, ESC, invalid UTF-8 and the end of the input, is decoded as
before.
The shuffle tables are generated by gen/utf8_decode_tables.py.
This decodes the text of the unicode benchmark 2.6 times faster, faster
than simdutf's own conversion with error checking, and speeds up the
unicode benchmark by ~8%.
The new AVX2 decoder is faster than the AVX-512 one, so remove the
AVX-512 decoder and use the AVX2 one on CPUs with AVX-512 as well.
Also add a test comparing the SIMD decoders with the scalar one on long
inputs, which exercise the decoding of blocks, as the existing tests use
short inputs.
Building the cells of wide chars with bitfield initializers made
compilers write parts of them to the stack and reload them with wider
loads, which stalls on store forwarding on every call of the batched
non-ASCII draw path and for every wide char drawn by the scalar path.
Build the cells from constant words in registers instead, using helpers
shared by both paths.
This speeds up the unicode benchmark by ~6%.
Decoding the packed bitfields of CharProps took about half the
instructions of the batched non-ASCII draw loop. Precompute, for each
distinct CharProps value, the number of cells a char occupies and its
grapheme segmentation property, and use that in the loop instead.
This speeds up the unicode benchmark by ~8%.
Also add test cases for a batched run that stops at an existing
multicell char after an unpaired regional indicator, which the random
test cases did not cover.
- Check for multicell chars in the erased range before looking for them
cell by cell, with a branch free check that reads only the word of each
cell containing the multicell flag. The common case is a line with no
multicell chars.
- Add a fast path that parses CSI codes of the common form, an optional
private marker, numeric params separated by ; or : and the final byte,
in a single pass, falling back to the generic CSI parser for anything
else.
- Do not call the UTF-8 decoder for no text when an escape code directly
follows another one, and start parsing CSI codes right after ESC [
rather than in the next iteration of the parse loop.
This speeds up the CSI benchmark by 25% to 30% and the truecolor
benchmarks by ~10%.
A differential test compares the fast path and the generic CSI parser on
random input, and a new test covers erasing parts of multicell chars,
which no existing test did.
Draw runs of non-ASCII chars that each start a new cell of width one or
two in a tight loop, with the run invariant checks done once and state
kept in locals, falling back to the scalar path for anything that needs
special handling: combining chars, zero width chars, the image
placeholder, existing multicell cells, wrapping, insert mode and charset
translation.
Cells are written as 64 bit values built in registers, since assembling
them as structs makes compilers go through the stack with partial stores
that stall on store forwarding. The grapheme segmentation step is cached
as it depends only on the state and the segmentation property of the
char, which repeat in a run of text, breaking the chain of dependent
table lookups between consecutive chars.
This speeds up the unicode benchmark by ~30%, drawing CJK text by ~70%.
A differential test compares the batched and scalar paths on random
operations.
Add truecolor_text, where every char has a random 24-bit foreground,
palette16_text, the same but with only 16 distinct colors, and
halfblocks, rows of half blocks with random 24-bit foreground and
background as written by terminal image viewers.
The existing benchmarks use either no colors or a handful of fixed
SGR codes, which favors terminals that store styles in a per page
deduplicated table rather than in each cell. These benchmarks measure
the cost of output with many distinct styles, which is common in
practice.