kitty/docs/content-rendering-protocol.rst
2026-09-30 18:56:54 +01:00

944 lines
45 KiB
ReStructuredText
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Terminal content rendering protocol
===================================
This protocol allows a program running in the terminal, hereafter called the
*client*, to render structured text as graphics. The terminal uses a locally
configured renderer and keeps the original source with the image. Changing
the font size renders the source again; copying the image returns the source,
even after the client exits.
It extends the :doc:`graphics protocol <graphics-protocol>`, reusing images,
virtual placements and Unicode placeholders. Clients send source and a kind,
such as ``math`` or ``barcode/qr``, without needing a rendering engine.
Terminals need not ship renderers: the user chooses them in local
configuration. If a kind or input is unsupported, the client displays the
source as ordinary text.
Some examples of rendered content:
.. figure:: screenshots/content-math.png
:alt: Math example showing a displayed equation alongside formulas in ordinary text.
:align: center
Math example.
.. figure:: screenshots/content-music.png
:alt: One bar of music in treble clef, with eight notes, a triplet, a slur, a grace note, a trill and a fermata.
:width: 540px
:align: center
Music from :download:`ABC source <screenshots/content-music.abc>`.
Both barcodes below encode `https://github.com/kovidgoyal/kitty
<https://github.com/kovidgoyal/kitty>`__.
.. grid:: 1 2 2 2
:gutter: 2
.. grid-item::
.. figure:: screenshots/content-qr.png
:alt: QR code for https://github.com/kovidgoyal/kitty.
:width: 180px
:align: center
QR code (``barcode/qr``).
.. grid-item::
.. figure:: screenshots/content-pdf417.png
:alt: PDF417 barcode for https://github.com/kovidgoyal/kitty.
:width: 324px
:align: center
PDF417 (``barcode/pdf417``).
.. grid-item::
.. figure:: screenshots/content-caffeine.png
:alt: Chemical structure of caffeine.
:align: center
Caffeine (:download:`SMILES source <screenshots/content-caffeine.smi>`).
.. grid-item::
.. figure:: screenshots/content-aspirin.png
:alt: Chemical structure of aspirin.
:align: center
Aspirin (:download:`SMILES source <screenshots/content-aspirin.smi>`).
The terminal's responsibilities are discovery, bounded render requests, image
placement and the lifetime of retained source. Renderer installation, APIs,
isolation and scheduling are implementation choices.
A minimal example
------------------
To display a QR code for ``https://github.com/kovidgoyal/kitty``, the client
queries support, requests a render and writes its placeholder cells. Here,
``<ESC>`` stands for the escape byte.
#. Query the kind, followed by a primary device attributes query (DA1)::
client → terminal: <ESC>]8776;a=q:t=barcode/qr:R=101<ESC>\<ESC>[c
terminal → client: <ESC>]8776;a=q:R=101:s=OK:B=4096:D=500<ESC>\
terminal → client: <DA1 reply>
Read both replies with terminal echo disabled. If DA1 arrives without the
content reply, support was not established; display the URL as text.
#. Ask for a render within 40 columns and 12 rows, with the URL in base64::
client → terminal: <ESC>]8776;a=d:t=barcode/qr:R=102:M=40:N=12;aHR0cHM6Ly9naXRodWIuY29tL2tvdmlkZ295YWwva2l0dHk=<ESC>\
terminal → client: <ESC>]8776;a=d:R=102:s=OK:i=42:c=8:r=4:w=80:h=80<ESC>\
These example dimensions assume 10 by 20 pixel cells. Use the image ID
and dimensions returned by the terminal, and match the reply using ``R``.
On an error or timeout, display the URL as text.
#. Write an 8 by 4 rectangle of :ref:`Unicode placeholders
<graphics_unicode_placeholders>` for image 42. Give each cell its row and
column diacritics, and end each image row with a hard line break. Reset
the text colors afterward. Only these cells make the image visible.
The :ref:`shell example <content_example>` below shows the full placeholder
loop. The following sections describe discovery, rendering, display and
copying; :ref:`content_reference` gives the wire format and request lifecycle.
Protocol overview
-----------------
This protocol covers discovering support, sizing, placement, replies and the
retention of source. How renderers are installed and configured, and how the
terminal talks to them, is left to each terminal. Implementing this protocol
does not require a particular helper or renderer API. The protocol uses its
own OSC code and reuses the virtual placements, deletion commands and Unicode
placeholders of the graphics protocol. There is no compression, file transfer
or shared-memory transport.
Throughout this document, a *connection* is one terminal session: the byte
stream between a terminal instance and the processes attached to its pty.
*Reset* means a full reset (RIS). A soft reset (DECSTR) does not affect the
state of this protocol.
Querying support
----------------
To detect support for the protocol, send ``a=q`` without ``t``. The terminal
must report support even when no renderers are configured, and return ``B``,
its source size limit in bytes, and ``D``, its :ref:`render deadline
<content_deadlines>` in milliseconds. Queries and listings have no payload
in the request; a non-empty one fails with ``EINVAL``::
<ESC>]8776;a=q:R=1<ESC>\
<ESC>]8776;a=q:R=1:s=OK:B=131072:D=500<ESC>\
With ``t``, the terminal reports success only for a kind that is configured.
The reply does not include ``t``; the client matches it to the query using
``R``. The ``B`` in this reply may be lower than the terminal-wide limit. With
no renderers configured, every kind query returns ``ENOTSUP``. Note that
queries do not parse any source, and success does not guarantee that a render
will succeed::
<ESC>]8776;a=q:t=math:R=2<ESC>\
<ESC>]8776;a=q:R=2:s=OK:B=1024:D=500<ESC>\
<ESC>]8776;a=q:t=unknown:R=3<ESC>\
<ESC>]8776;a=q:R=3:s=ENOTSUP;VW5zdXBwb3J0ZWQga2luZA==<ESC>\
Clients can also :ref:`list the available kinds <content_listing>`.
The terminal must answer queries and listings from its local metadata, as soon
as it parses them, without invoking renderers or waiting for queued renders.
It must queue these replies before the reply to any subsequent primary device
attributes (DA1, ``CSI c``) query. Where both escape codes pass through,
clients can use this ordering to detect an OSC query that was ignored: if the
DA1 reply arrives first, the query was ignored somewhere along the way.
When this protocol is disabled in the configuration, the terminal must ignore
OSC 8776 escape codes without replying, and must not re-render retained
content. Retained images keep their last pixels, scaled to their cells as for
a failed re-render, and copying and exporting still substitute the retained
source.
Falling back to text
^^^^^^^^^^^^^^^^^^^^
For a reliable fallback, the client queries the kind, requests rendering, and
writes placeholders only after the render succeeds. A successful kind query
alone is not enough. On failure or timeout, the client prints the source as
ordinary text; a successful render that arrives late stays invisible, since
there are no placeholders for it. Applications must print the fallback
themselves, because terminals that do not support this protocol do not
display OSC payloads. Clients should print text when their output is
redirected, or goes through intermediaries that cannot carry the protocol.
Content kinds
-------------
Kind identifiers contain 1 to 64 ASCII letters, digits or the characters
``-_.+/``, and are matched exactly. The terminal routes them to configured
renderers without inspecting the source or guessing its format.
A kind names an input format and its meaning, independently of the renderer.
Examples include ``math``, ``smiles``, ``abc``, ``barcode/qr`` and
``barcode/ean13``. Supporting these kinds is optional. A successful kind
query reports availability, not support for every extension of its format.
Additional kinds need no change to the transport. Their authors should
publish the grammar or format version, semantic examples and error cases.
Incompatible interpretations need different identifiers; private extensions
should use a project prefix, such as ``example/diagram``.
Renderers must reject unsupported constructs rather than silently change
meaning. Fonts, colors and internal layout may differ, while preserving
meaning and the requested cell dimensions. Pixel-identical output is not
required.
.. _content_sizing:
Rendering content
-----------------
The terminal passes the :ref:`captured rendition and cell metrics
<content_lifecycle>` to the renderer; other SGR attributes are unspecified.
Unless the kind itself specifies colors, the renderer uses that foreground
color over transparency or the supplied background, as it chooses.
The renderer works out the natural layout of the content from the source and
the cell metrics, including any padding the content needs. It must use a
consistent natural scale for all inputs of the same kind; what that scale
means is up to the kind. ``c`` and ``r`` specify an exact number of columns
and rows, ``M`` and ``N`` specify maximums, and zero leaves them unspecified.
The renderer must follow these sizing rules:
#. With no exact dimension, keep the natural scale, shrinking the content
uniformly to fit within ``M``. ``N`` is checked after fitting the width and
rounding to whole cells, and exceeding it is ``EFBIG``. Bounds never
enlarge content. To scale the height, the client specifies ``r``.
#. With one exact dimension, scale uniformly to fit it, and derive the other.
With both exact, use the largest uniform scale that fits.
#. ``F=1`` caps the scale at the natural size. Exact dimensions still set the
size of the canvas, and unused space is padded.
#. Round unspecified canvas dimensions up to whole cells. The renderer chooses
how to align the content within the padding. Never crop the content to
meet the constraints.
#. A kind with discrete scales, such as a whole number of pixels per barcode
module, uses the largest scale it can represent that does not exceed the
scale given by these rules. It derives unspecified dimensions from the scale
it actually used, and pads the rest of the canvas, including any exact
dimensions it cannot fill. If it can represent no scale that fits, the
render fails with ``EFBIG``.
The terminal must enforce these limits:
* The terminal must reject exact dimensions larger than their maximums with
``EINVAL``, and derived dimensions larger than their maximums with
``EFBIG``. The terminal never changes exact dimensions to make them fit.
* The terminal must reject a size of more than 297 columns or 297 rows with
``EFBIG``, whether it was requested exactly or computed, because the
:ref:`placeholder diacritics <graphics_unicode_placeholders>` can only
address rows and columns 0 to 296. The limit does not implicitly shrink
content that is too large.
Renderers may also reject boxes that are too small for valid or legible output
with ``EFBIG``, and the terminal reports arithmetic overflow as ``EFBIG`` too.
A successful reply includes ``i``, positive ``c`` and ``r``, and positive
pixel dimensions ``w`` and ``h`` covering the whole canvas, which is a whole
number of cells. When the request specified a nonzero ``p``, the reply
includes the ID of the installed placement, as a canonical decimal number;
otherwise the reply omits ``p``. The optional signed ``b`` gives the baseline
in pixels, down from the top of the canvas. Negative values are above the
canvas, and ``b=0`` is a baseline at its top. The value may lie outside the
canvas. The terminal omits ``b`` when no baseline is meaningful.
The geometry in the reply describes the render at the captured cell metrics.
If the cell size changes before the result is installed, the terminal must
still install it into the cells it reported, and then re-render it as for any
later font change. The terminal does not update ``w``, ``h`` or ``b`` after
replying, so clients must not assume they are still current after a font
change.
Measuring and releasing renders
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Rendering without writing placeholders is how you measure content; there is
no separate operation for it. Note that rendering allocates storage in the
terminal, even when no placeholders are ever displayed. Clients must release
successful renders they do not use with the graphics protocol's
``a=d,d=I,i=<id>`` command, when they know the image ID and have not reset the
terminal or switched screens since sending the render request. This includes
images used only for measurement, results the client rejected for its own
reasons, and successful replies that arrived late. Otherwise, clients must
leave reclaiming the storage to the terminal. Specifying an image ID in the
request allows cleaning up without a reply, under the same conditions.
Displaying content
------------------
The terminal creates an invisible virtual placement, with ``z`` as its
stacking order. Clients display it by writing :ref:`U+10EEEE placeholders
<graphics_unicode_placeholders>`, using the cell dimensions from the reply.
These cells control the position, scrolling, erasing and reflow of the
content. There is no key to choose a placement mode or to move the cursor.
Clients must take care of the following:
* Clients manage line breaks, and reset or restore the text colors after
writing the image and placement IDs as colors.
* Clients must not send a final chunk while an image ID color is the current
foreground, since the terminal cannot tell a color that encodes an image ID
from a color intended for text.
* Rows of placeholders. Clients should keep each row within the width of the
terminal. For rows that could wrap, including after a resize, they must
specify both the row and column diacritics on every cell. This avoids
depending on the graphics protocol inferring columns across physical lines.
The rendered image always remains a single rectangle; these rules concern
only its placeholder cells.
When the request has no ``p``, or ``p=0``, the new virtual placement is
unnumbered; the terminal does not allocate a placement ID for it. Separately,
a placeholder with no underline color, or an underline color of zero, may
refer to any virtual placement of the image, as the graphics protocol
specifies. So images with a single placement need no underline color. In this
protocol, ``p`` must be between 0 and 16777215, inclusive, and the terminal
must reject larger values with ``EINVAL``. This range fits the 24-bit
underline color of the placeholder, and does not change the range of
placement IDs in the graphics protocol.
Clients that specify both the image ID and the exact cell dimensions can write
placeholders without waiting for the reply. However, suppressing replies means
giving up confirmation of success, and with it a reliable text fallback.
Image IDs
^^^^^^^^^
``R``, ``i`` and ``p`` are independent of each other. Images and placements
share the per-screen namespaces of the graphics protocol. For ``i=0``, the
terminal allocates a nonzero image ID that fits in 24 bits, and is neither
used by an existing image nor reserved by a pending render. It must not reuse
an ID it previously allocated for this protocol while old placeholders could
still remain, on screen or in the scrollback, even after the image data was
deleted or evicted. When no IDs are left, the request fails with ``EFBIG``.
These rules do not change the range of image IDs, or how they are allocated,
in the graphics protocol. Clients that specify image IDs themselves are
responsible for choosing them.
The placement and deletion commands of the graphics protocol apply to these
images; there is no need to transmit an image with the graphics protocol. A
successful ``a=d`` render to an existing image ID follows the graphics
protocol's :ref:`replacement rules <graphics_display_images>`, removing the
old image and all its placements before installing the new pixels, source and
virtual placement. Retransmitting the whole image with the graphics protocol
also replaces it, and removes its association with the source.
Other successful changes to pixels with the graphics protocol, including
editing frames (``a=f``) and composition (``a=c``), clear the association
with the source and stop re-rendering from it. They otherwise behave as they
always have. Failed changes, and changes to placements only, do not clear the
source. Clients that need old scrollback to keep its meaning must use fresh
image IDs for new content.
Interaction with other terminal actions
---------------------------------------
Images follow the graphics protocol's :ref:`rules for reset, the alternate
screen, erasing the display and scrolling <graphics_interaction>`.
Placeholders wrap and reflow like ordinary text.
Pending renders follow the :ref:`cancellation rules <content_cancellation>`.
Full-screen programs that redraw by erasing the display should erase before
requesting renders, or wait for outstanding replies first.
Retaining and copying source
----------------------------
The terminal must keep each image's kind, exact source bytes, sizing policy,
rendition and final cell size, even after the program that sent it exits. The
source lives as long as the image data, unless a pixel change with the
graphics protocol clears the association. Deleting placements while keeping
the image data also keeps the source. Freeing or evicting the image data
releases its source, leaving nothing to substitute when copying or exporting.
When the font or cell size changes, the terminal must re-render the content
into the cells it already occupies, keeping the image ID, placements and
association with the source. The re-render follows the sizing rules, with the
existing cells as both exact dimensions and the original ``F``. A render that
was originally requested without exact dimensions uses ``F=1``, keeping its
natural scale rather than growing to fill the cells. Content whose natural
size at the new cell size is larger than the cells shrinks to fit; it is
never cropped. If the re-render fails, the terminal must keep the previous
pixels, scaled to the same cells.
Terminals should remember when a rendition uses default or indexed colors,
and re-render when those colors change. Explicit RGB colors stay fixed. The
terminal must never take the foreground color of the content from the image
ID color of a placeholder cell.
When copying or exporting text, as plain text or with ANSI styling, including
selections, buffer exports and viewing the scrollback in a pager, the
terminal must substitute the original source, never a description from the
renderer:
* It emits the complete source once for each selected *occurrence* of the
image, at its first selected cell, even when only part of the occurrence is
selected, and emits nothing for the remaining cells. Two selected
occurrences produce two copies of the source.
* It must preserve the order of other text and the newlines in the source,
even if the geometry of the export differs.
* Exporting needs no renderer, and ANSI exports may add ordinary text
styling. There is no such compatibility guarantee for raw recordings of the
bytes the program sent.
Roughly, an occurrence is one printed copy of the image, even if its rows
wrapped. Clients that need copying to be stable must end each row of
placeholders with a hard line break; see :ref:`content_occurrences` for
the precise rules.
.. _content_reference:
Protocol reference
------------------
The following sections specify the wire format and the less common cases.
They are part of the protocol, including for terminals that use their own
renderer API. Discovery, rendering and copying must behave the same way
regardless of how the terminal schedules work.
The content rendering escape code
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The protocol uses OSC **8776** with the following form::
<ESC>] 8776 ; <key>=<value>[:<key>=<value>]* [;<payload>] <ESC>\
Spaces in this definition are for clarity only. ``ESC`` is the byte ``0x1b``.
Senders, whether client or terminal, use the seven-bit forms ``ESC ]`` and
``ESC \``, and receivers also accept ``BEL (0x07)`` as the terminator. The
eight-bit forms need not be supported.
The metadata is a colon-separated list of ``key=value`` pairs:
* Keys and values are case-sensitive ASCII. Keys contain letters, digits and
``_``. Values must not be empty, and contain visible ASCII (``0x21`` to
``0x7e``) other than ``:`` and ``;``.
* Unsigned integers contain only decimal digits and fit in 32 bits. ``z`` and
``b`` are signed 32-bit integers, with an optional leading minus sign. A
plus sign is not allowed. Leading zeros do not matter, so ``R=007`` and
``R=7`` are the same token, and ``-0`` is zero.
* Numeric fields that are not specified default to zero, unless stated
otherwise. Flags accept only the values listed for them.
* Both clients and terminals must ignore keys they do not know. The terminal
also ignores keys that this specification defines only for other actions,
except on continuation chunks, as described under `Errors in chunks`_.
Ignored keys must still follow the key and value syntax and the rule
against duplicate keys, but their numbers and flags are not validated.
Extensions preserve existing behavior and provide a capability query.
Clients must use that query before relying on the extension.
The payload carries the source. A missing payload is the same as an empty
payload after the ``;``. The client must encode each chunk of source
independently in standard :rfc:`base64 <4648>`, with the padding and without
whitespace. Whoever receives a message may either reject or ignore tab, LF, CR
and space inside the payload, but must not tolerate them anywhere else in the
escape code. Once all chunks are put together, the source must be valid
UTF-8.
The terminal must reject source containing C0 controls (``U+0000`` to
``U+001F``, including ``ESC``) other than tab, LF and CR, as well as
``DEL (U+007F)`` and the C1 controls (``U+0080`` to ``U+009F``). It must pass
the source on unchanged, without normalizing whitespace, line endings or
Unicode. Ordinary backslashes need no extra escaping.
The following limits do not count the OSC framing:
.. csv-table:: Size limits
:header: "What", "Limit"
"Metadata", "1,024 bytes"
"Payload per chunk", "4,096 encoded bytes, which is 3,072 decoded bytes"
"Total source", "131,072 decoded bytes; the ``B`` key reports the terminal's own limit, which may be lower"
.. _content_listing:
Listing available kinds
^^^^^^^^^^^^^^^^^^^^^^^
To list the available kinds, send ``a=l``. The terminal returns up to 16
kinds in ascending ASCII order, separated by LF without a trailing LF, and
base64-encoded. The reply has ``m=1`` if there are more pages, or ``m=0`` for
the last page. A page with ``m=1`` must contain at least one kind. To get the
next page, the client sends ``after=<last-kind>``, which returns only names
that sort strictly after it. The ``after`` value uses kind syntax and is
omitted for the first page. Listings expose no paths or renderer
configuration, and do not reserve any renderer.
When no renderers are configured, or no kinds follow the cursor, the terminal
returns an empty successful listing with ``m=0``, and a trailing ``;`` for the
empty payload::
<ESC>]8776;a=l:R=6<ESC>\
<ESC>]8776;a=l:R=6:s=OK:m=0;<ESC>\
Transferring source in chunks
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Source can be sent in a single message, or in multiple chunks that share a
nonzero ``R``. For example, here is ``x^2`` followed by ``+ y^2``, with a
leading space, sent in two chunks::
<ESC>]8776;a=d:t=math:R=5:M=80:N=12:m=1;eF4y<ESC>\
<ESC>]8776;a=d:R=5:m=0;ICsgeV4y<ESC>\
<ESC>]8776;a=d:R=5:s=OK:i=43:c=7:r=2:w=70:h=40:b=24<ESC>\
* The first chunk contains ``t`` and all the rendering metadata.
* Continuation chunks omit ``t`` and contain only ``a=d``, ``R``, ``m`` and the
payload.
* Every chunk of a multipart upload must carry the same nonzero ``R``.
* ``m=1`` means more chunks follow, and ``m=0``, the default, completes the
upload.
* Every chunk except the last must be non-empty. The last chunk may be empty,
but a complete source that is empty fails with ``EINVAL``.
* Uploads with different tokens may be interleaved, but only as whole OSC
escape codes, never as bytes within one. Chunks are put together in the
order they arrive; there is no chunk index.
The terminal must decode each chunk separately, put the results together, and
only then validate the UTF-8, since characters may be split across chunks. The
terminal must not start parsing the source as its kind, or rendering it,
before the final chunk has been validated. The first chunk supplies the
reply policy for the whole request.
Note that chunking does not guarantee that intermediaries such as terminal
multiplexers will support or deliver the upload.
.. _content_lifecycle:
Request lifecycle
^^^^^^^^^^^^^^^^^
Terminals may render synchronously or continue processing input while a render
is pending. Clients match replies using ``R``: a render reply can arrive after
replies to later requests, including cursor position reports and DA1. Queries
and listings retain the ordering required by `Querying support`_.
The reply policy is ``q=0`` for all replies (the default), ``q=1`` for errors
only, or ``q=2`` for no replies.
A request is *outstanding* from its first message until it completes or is
silently discarded. It *completes* when its final outcome has been committed
and any reply allowed by ``q`` has been queued. Completion releases its token
even when replies are suppressed. Finishing an upload does not complete its
pending render.
On the final chunk, the terminal must capture the active screen, its cell
metrics, and the colors for ordinary text under the current SGR attributes
and screen-wide reverse video state. This color state is the *rendition*.
Rendering uses these captured values, even if the terminal state changes
before completion.
The terminal sends one final reply per request, unless ``q`` suppresses it or
the protocol requires silence. A render can return ``OK`` only after
installing its pixels, source and virtual placement. There are no unsolicited
rendering notifications. The terminal must not move the cursor for these
requests. On error, the terminal must not create placements or replace
retained content, and must discard later results from that request.
.. _content_deadlines:
Deadlines and retries
~~~~~~~~~~~~~~~~~~~~~
``D`` bounds the time from parsing the final chunk to completion, including
time queued inside the terminal. It applies even when ``q`` suppresses the
reply. The terminal must fail a render it cannot finish in time with
``ETIMEDOUT`` or ``EBUSY``. Transport and time spent waiting unparsed in the
terminal's input are outside this deadline.
A synchronous terminal can leave later requests, including queries, unparsed
behind earlier renders from any client on the connection. With ``N`` requests
outstanding, clients should allow about ``N × D`` for the last reply, plus
transport time. A client-side timeout leaves the outcome unknown; a query
timeout does not prove that support is absent.
``EBUSY`` must include a positive ``retry`` delay in milliseconds approximating
the remaining wait, including failure backoff. This is advice, not a
reservation. Clients may retry with a fresh token or fall back to text.
Permanent unavailability must become ``ENOTSUP`` or ``EIO``, rather than
``EBUSY`` forever.
Incomplete uploads may expire silently. When the terminal cannot accept a
new upload, it replies with ``EBUSY`` or ``ENOMEM``.
.. _content_cancellation:
Cancellation
~~~~~~~~~~~~
After the final chunk, a pending render targets the requested image ID or one
reserved by the terminal. These events must cancel pending work, with an
``ECANCELED`` reply subject to ``q`` unless the table specifies silence:
.. list-table:: Cancellation rules
:header-rows: 1
:widths: 60 40
* - Event
- Effect
* - A screen is cleared, including clearing the alternate screen on entry.
- Cancel renders captured for that screen, even if no image has been
installed.
* - A graphics command directed at the target ID frees or replaces image
data, or successfully changes pixels: deletion with an uppercase
selector, retransmission, frame editing or composition.
- Cancel pending renders for that ID.
* - A render submitted later for the same target ID succeeds.
- Cancel older pending renders for that ID.
* - The terminal resets or the connection closes.
- Discard uploads and pending renders silently. This takes precedence
over cancellation by screen clearing.
Switching screens while preserving the original screen leaves its pending
renders running; they complete on that screen. Placement-only deletions and
deletions by position or visibility do not cancel pending renders. A later
render that fails also leaves earlier renders running.
Cancellation produces no further reply after completion. The terminal must
discard cancelled work; late results must never restore it or appear in
another screen or tab.
Requests and replies
^^^^^^^^^^^^^^^^^^^^
There are three actions. ``a=q`` queries support, ``a=l`` lists the available
kinds, and ``a=d``, the default, renders content into a virtual placement. The
:ref:`key reference <content_keys>` below lists all the keys. For timing,
completion and cancellation, see :ref:`content_lifecycle`.
The ``R`` key identifies a request, not the content. A nonzero token must be
unique among the outstanding requests on a connection, and is required for
uploads that span multiple chunks. ``R=0``, the default, is for independent
single-message requests whose replies need not be told apart. Such a request
never invalidates another request through token reuse. Clients must use
nonzero tokens when they need to tell concurrent replies apart. Clients should
never reuse a nonzero token on a connection, for example by counting upward
from a random starting point; the 32-bit space is ample.
The terminal replies using the same escape code, always terminated by
``ESC \``. Every reply carries ``a``, ``R`` and ``s=<status>``, where ``a`` is
the reply type, one of ``q``, ``l`` or ``d``. Only ``s=OK`` means success;
any status code the client does not recognize is an error.
Replies are built only from fixed protocol tokens, numbers written by the
terminal and base64 payloads; they never copy raw bytes from the request.
Numeric values may be the same as those the client sent, including ``i``,
``p``, ``c`` and ``r``. The terminal writes all numbers in canonical decimal,
without leading zeros, and zero as ``0``. ``R`` is the parsed request token,
or ``0`` when it was omitted, so clients compare tokens as numbers. Apart from
the OSC framing, replies contain only ASCII letters, digits and the characters
``_=:;+/-``.
Successful queries and renders have no payload; listings carry kind names.
Errors may carry a base64-encoded UTF-8 diagnostic, of at most 512 decoded
bytes, following the same control character restrictions as source. Clients
act on the status code, not on the diagnostic text. Terminals must not forward
raw diagnostics from renderers or quote values from the request in them.
.. csv-table:: Reply status codes
:header: "Status", "Meaning"
"``OK``", "Completed."
"``ENOTSUP``", "Unsupported action, kind or kind feature."
"``EINVAL``", "Invalid envelope, source or conflicting constraints."
"``EFBIG``", "Source/output limit or automatic image-ID space exhausted, or minimum valid rendering cannot fit the box."
"``ENOMEM``", "Insufficient storage."
"``EBUSY``", "Retryable temporary unavailability. ``retry`` gives a suggested delay in milliseconds."
"``ETIMEDOUT``", "Render deadline exceeded."
"``ECANCELED``", "Pending render invalidated by screen clearing, image deletion, replacement or pixel mutation."
"``EIO``", "Renderer failed or returned invalid output."
.. _content_keys:
Control data reference
^^^^^^^^^^^^^^^^^^^^^^
The table below lists the defaults for requests and the requirements for
replies.
.. csv-table:: Protocol keys
:header: "Key", "Used in", "Default", "Meaning"
"``a``", "Requests/replies", "``d``", "``q`` query, ``l`` list, ``d`` render. In replies, the reply type; ``d`` for an invalid or unknown action."
"``R``", "Requests/replies", "``0``", "Request token. Nonzero for multipart; returned in canonical decimal in every reply."
"``t``", "Kind queries, first render chunk", "Unset", "Kind. Required on first render chunk, forbidden on continuations. Never returned in replies."
"``after``", "List requests", "Unset", "Pagination cursor. Names strictly after this kind; unset for first page."
"``q``", "Requests", "``0``", "``0`` all replies, ``1`` errors only, ``2`` none. On render continuations, ``EINVAL``. A duplicated or invalid ``q`` supplies no policy."
"``m``", "Render chunks, list replies", "``0`` on requests", "``1`` more, ``0`` final. Mandatory on list replies."
"``c``, ``r``", "First render chunk, render replies", "``0``", "Exact columns/rows; zero unspecified. Successful replies give dimensions from 1 to 297."
"``M``, ``N``", "First render chunk", "``0``", "Maximum columns/rows; zero unbounded by client. ``M`` bounds width fitting; ``N`` bounds resulting height."
"``F``", "First render chunk", "``0``", "``0`` scales to exact box; ``1`` prevents enlargement above natural size. Bounds alone never enlarge."
"``i``", "First render chunk, render replies", "``0``", "Image ID; zero requests terminal allocation. Successful replies give nonzero ID."
"``p``", "First render chunk, successful render replies", "``0``", "Graphics protocol placement ID of the virtual placement, 0–16777215 in this protocol; larger values are ``EINVAL``. Zero leaves the placement unnumbered. Successful replies return ``p`` only when nonzero."
"``z``", "First render chunk", "``0``", "Signed 32-bit graphics protocol stacking order of the virtual placement."
"``s``", "All replies", "Required", "``OK`` or error code."
"``B``", "Successful query replies", "Required", "Terminal/kind source-byte limit, at most 131,072."
"``D``", "Successful query replies", "Required", "Positive render deadline in milliseconds; see :ref:`content_deadlines`."
"``retry``", "``EBUSY`` replies", "Required", "Positive delay in milliseconds approximating the remaining wait, including failure backoff."
"``w``, ``h``", "Successful render replies", "Required", "Positive canvas dimensions in device pixels."
"``b``", "Successful render replies", "Unset", "Signed 32-bit baseline from canvas top. Zero is valid; negative is above it. Omitted if meaningless."
.. _content_occurrences:
Identifying image occurrences
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
An *occurrence* consists of placeholder cells with the same image ID,
placement ID and *logical origin*. Decode each cell's zero-based image row
and column ``(r, c)`` as in the graphics protocol. Soft-wrapped physical
lines form one logical line; a hard line break starts the next one.
For a cell on logical line ``L`` at cell offset ``X``, the logical origin
is ``(L - r, X - c)``. Cells with equal origins belong to the same occurrence,
even if some cells are erased or unselected. No physical-origin links or
transitive grouping are needed. For example, a four-column image in a
three-column terminal has these coordinates (each entry is ``r,c``)::
logical line 0: [0,0] [0,1] [0,2] ← soft wrap
[0,3] ← hard line break
logical line 1: [1,0] [1,1] [1,2] ← soft wrap
[1,3]
Every cell has origin ``(0, 0)``. Selecting any part of either row emits the
source once. Printing a second copy at another origin emits it again.
Clients must end each image row with a hard line break and start the next at
the same logical offset for it to be one occurrence. Advancing the image row
across an automatic wrap, without a hard line break, changes the logical
origin: those rows are separate occurrences and can copy the source more
than once. This rule continues to apply after a terminal resize.
Selection boundaries do not redefine logical lines. When a selection starts
inside a soft-wrapped line, account for its retained prefix to recover the
cell offsets and any inferred diacritics. There is no need to inspect
unrelated scrollback or follow links through unselected cells. During a
forward export, origins more than 296 logical lines behind the current line
can be forgotten, since no valid placeholder row can refer to them.
Placeholders that refer to no retained source fall into two cases:
* If a live image has the ID but no source, because of retransmission or a
pixel change with the graphics protocol, it is just a graphics protocol
image. Its placeholders are copied the way the terminal already copies
placeholders of any other image.
* If no live image has the ID, because its data was deleted or evicted, the
terminal must omit the placeholders while it still remembers that this
protocol rendered that ID. The terminal may limit how much of this it
remembers; beyond that limit, these placeholders are also copied the way the
terminal already copies them.
.. _content_errors:
Handling errors
^^^^^^^^^^^^^^^
This section collects what happens when messages are malformed, tokens
collide, or work has to be abandoned. Many of these rules share one concern: a
reply that no client reads can reach a shell as input, so when the terminal
cannot be sure who a reply is for, it stays silent.
Malformed and oversized messages
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* The terminal rejects duplicate keys, empty values, malformed numbers and
invalid encodings with ``EINVAL``.
* Actions must be a single character: other lengths fail with ``EINVAL``, and
unknown single-character actions fail with ``ENOTSUP``. Both kinds of
failure are answered as ``a=d``.
* The terminal must skip an oversized escape code up to and including its
terminator, without acting on it. When the request token can be validated
from the complete metadata, within the metadata limit, the terminal must
treat the oversized message according to the rules for chunks below, with
the status ``EFBIG``. That is, it aborts the matching upload, rejects a new
request, or silently discards a continuation that matches no upload.
* Duplicate or malformed ``R`` fields, and truncated metadata, must never
supply a token.
* Only ``ESC \`` or ``BEL`` completes a message. An OSC 8776 escape code that
ends any other way, for example because of a CAN or SUB, or an ESC that
starts another escape code, is malformed and never supplies a token, even if
the bytes received so far would parse.
* The terminal must never turn malformed bytes into a valid request by
removing control characters from any part of the message, including the OSC
number. It must reject or silently discard such a message instead. So if a
terminal's parser drops control bytes from OSC strings, it must record that
it did so, and where, so that the protocol handling forgives nothing except
tab, LF and CR inside the payload.
If the terminal cannot tell which request a message belongs to, it must
silently discard that message, and also every incomplete upload on that
connection, since a later continuation could otherwise complete a source with
a missing chunk. The same applies to malformed messages that are discarded
without a trustworthy token, including escape codes ended by anything other
than ``ESC \`` or ``BEL``. Note that on a shared connection this also
abandons the incomplete uploads of other clients. That is intentional, since
the terminal has no way to tell which upload lost a chunk.
Who gets an error reply
~~~~~~~~~~~~~~~~~~~~~~~
An error reply follows the reply policy of the outstanding request whose token
the message matches, and otherwise the message's own ``q``. A message whose
own ``q`` is duplicated or invalid has no policy of its own. If no outstanding
request supplies one, the terminal must discard the message silently, so that
an unwanted reply can never become input to a shell.
Errors in chunks
~~~~~~~~~~~~~~~~
The terminal classifies each message that has a trustworthy token before
acting on it:
* A message with no ``t`` and no action other than ``d`` is a continuation.
* Any other message is a new request: ``t`` starts a new render, and any
other action, valid or not, starts a new request even without ``t``.
``R=0`` with ``m=1``, or on a continuation, fails with ``EINVAL`` under the
message's own ``q``, without affecting other requests.
A continuation applies only to an incomplete upload with the same token:
* If there is no such upload, the terminal must discard the continuation
silently, whether or not it is otherwise valid. That includes continuations
that arrive after the upload has failed or expired, and while the render of
a completed upload is pending.
* If there is such an upload, a malformed continuation aborts it with
``EINVAL``. That covers every ``EINVAL`` condition of
`The content rendering escape code`_
and `Malformed and oversized messages`_, a ``q`` or any other known key
that belongs only on the first chunk, invalid base64 and an empty chunk
that is not the last.
* An oversized chunk, or too much source in total, aborts the upload with
``EFBIG``.
* Unknown keys are ignored, as usual.
Token reuse
~~~~~~~~~~~
A new request whose nonzero ``R`` is already outstanding invalidates the
outstanding request, and is itself rejected. The terminal replies exactly once
for both, under the outstanding request's reply policy, with the reply type
taken from the new message's action, or ``a=d`` if that is invalid or unknown.
The status is the new message's own error (``EFBIG`` when oversized,
``ENOTSUP`` for an unknown action, ``EINVAL`` when malformed), or ``EINVAL``
if the new message is otherwise valid. A new request whose token is not
outstanding receives its normal error, even without a kind.
So clients replacing an abandoned upload must use a fresh token. A client that
abandons a request must not display a late result, and must avoid reusing its
token while old chunks or replies could still arrive. Note that with ``q=2`` a
client cannot tell when a request has completed, and reusing a token whose
render is still pending silently cancels both requests. Clients release late
successful renders as described under `Measuring and releasing renders`_.
Multiplexers that implement this protocol should route replies to the right
client and prevent token collisions.
Security considerations
^^^^^^^^^^^^^^^^^^^^^^^
Only the user's local configuration chooses renderers. Requests cannot name
executable paths, install code or change renderer settings. Both the source
and the output of renderers are untrusted. Receiving source does not give
access to any local resources. How renderers are isolated, and what they are
allowed to run, is outside the scope of this protocol.
Copied or exported source may contain syntax and newlines that are not visible
in the rendered image. The terminal must treat it as untrusted text, and keep
its normal protections for the clipboard and pasting.
A reply that no client reads can reach a shell as input. That is why replies
are limited to the characters listed under `Requests and replies`_, and
contain nothing from the request other than decimal numbers. Silently
discarding expired uploads, and messages without a trustworthy token or reply
policy, serves the same goal.
Rendering spends local CPU time and memory on behalf of untrusted output.
Terminals limit that work according to their own policy, and report it with
``EBUSY``, ``ENOMEM`` and ``ETIMEDOUT``. The protocol promises clients a
fallback to text, not a render.
Queries and listings reveal which kinds the user has configured to any program
that writes to the terminal, including programs running on remote machines.
Like other queries that identify the terminal, this can be used for
fingerprinting. Listings expose no paths or renderer configuration, and users
can disable the protocol.
.. _content_example:
A shell example
---------------
With a QR renderer configured, paste this into zsh or bash 4.2 or newer,
using a UTF-8 locale and a terminal at least 24 columns wide. It displays
a QR code for ``https://github.com/kovidgoyal/kitty`` at a fixed size.
Older shells, including macOS's ``/bin/sh``, do not understand the
``\U`` escapes.
This demonstration uses ``q=2`` because the shell does not read replies.
It therefore cannot detect failure or provide a reliable text fallback.
See `A minimal example`_ for the request/reply flow applications should use.
.. code-block:: sh
printf '\e]8776;a=d:t=barcode/qr:R=102:i=42:c=24:r=12:q=2;aHR0cHM6Ly9naXRodWIuY29tL2tvdmlkZ295YWwva2l0dHk=\e\\'
d=0305030D030E03100312033D033E033F0346034A034B034C0350035103520357035B0363036403650366036703680369
printf '\e[38;2;0;0;42m'
for r in {0..11}; do
for c in {0..23}; do
printf "\U10EEEE\U${d:$((r*4)):4}\U${d:$((c*4)):4}"
done
printf '\n'
done
printf '\e[39m'
The first command asks the terminal to render the base64-encoded URL with
the ``barcode/qr`` kind, as image 42 (``i=42``),
exactly 24 columns by 12 rows (``c=24:r=12``).
The rest writes a 24 by 12 rectangle of :ref:`placeholder cells
<graphics_unicode_placeholders>`, which is where the image appears. The
foreground color ``38;2;0;0;42`` carries the image ID. Each cell is
``U+10EEEE`` followed by a row diacritic and a column diacritic, so that the
coordinates of every cell survive wrapping or clipping. ``d`` holds the
diacritics for the numbers 0 to 23, four hex digits each, from
:download:`rowcolumn-diacritics.txt <../gen/rowcolumn-diacritics.txt>`.
.. _content_renderer_conventions:
Renderer conventions
--------------------
Renderer authors could maintain a shared registry of kind names and format
references, with a corpus of input text paired with example PNG files for
comparing output. These conventions are optional; terminals do not interpret
the content format or validate renderer output against the corpus.