diff --git a/docs/content-rendering-protocol.rst b/docs/content-rendering-protocol.rst new file mode 100644 index 000000000..b75f160ac --- /dev/null +++ b/docs/content-rendering-protocol.rst @@ -0,0 +1,944 @@ +Terminal content rendering protocol +=================================== + +This protocol allows a program running in the terminal, hereafter called the +*client*, to render structured text as graphics. The terminal uses a locally +configured renderer and keeps the original source with the image. Changing +the font size renders the source again; copying the image returns the source, +even after the client exits. + +It extends the :doc:`graphics protocol `, reusing images, +virtual placements and Unicode placeholders. Clients send source and a kind, +such as ``math`` or ``barcode/qr``, without needing a rendering engine. +Terminals need not ship renderers: the user chooses them in local +configuration. If a kind or input is unsupported, the client displays the +source as ordinary text. + +Some examples of rendered content: + +.. figure:: screenshots/content-math.png + :alt: Math example showing a displayed equation alongside formulas in ordinary text. + :align: center + + Math example. + +.. figure:: screenshots/content-music.png + :alt: One bar of music in treble clef, with eight notes, a triplet, a slur, a grace note, a trill and a fermata. + :width: 540px + :align: center + + Music from :download:`ABC source `. + +Both barcodes below encode `https://github.com/kovidgoyal/kitty +`__. + +.. grid:: 1 2 2 2 + :gutter: 2 + + .. grid-item:: + + .. figure:: screenshots/content-qr.png + :alt: QR code for https://github.com/kovidgoyal/kitty. + :width: 180px + :align: center + + QR code (``barcode/qr``). + + .. grid-item:: + + .. figure:: screenshots/content-pdf417.png + :alt: PDF417 barcode for https://github.com/kovidgoyal/kitty. + :width: 324px + :align: center + + PDF417 (``barcode/pdf417``). + + .. grid-item:: + + .. figure:: screenshots/content-caffeine.png + :alt: Chemical structure of caffeine. + :align: center + + Caffeine (:download:`SMILES source `). + + .. grid-item:: + + .. figure:: screenshots/content-aspirin.png + :alt: Chemical structure of aspirin. + :align: center + + Aspirin (:download:`SMILES source `). + +The terminal's responsibilities are discovery, bounded render requests, image +placement and the lifetime of retained source. Renderer installation, APIs, +isolation and scheduling are implementation choices. + + +A minimal example +------------------ + +To display a QR code for ``https://github.com/kovidgoyal/kitty``, the client +queries support, requests a render and writes its placeholder cells. Here, +```` stands for the escape byte. + +#. Query the kind, followed by a primary device attributes query (DA1):: + + client → terminal: ]8776;a=q:t=barcode/qr:R=101\[c + terminal → client: ]8776;a=q:R=101:s=OK:B=4096:D=500\ + terminal → client: + + Read both replies with terminal echo disabled. If DA1 arrives without the + content reply, support was not established; display the URL as text. + +#. Ask for a render within 40 columns and 12 rows, with the URL in base64:: + + client → terminal: ]8776;a=d:t=barcode/qr:R=102:M=40:N=12;aHR0cHM6Ly9naXRodWIuY29tL2tvdmlkZ295YWwva2l0dHk=\ + terminal → client: ]8776;a=d:R=102:s=OK:i=42:c=8:r=4:w=80:h=80\ + + These example dimensions assume 10 by 20 pixel cells. Use the image ID + and dimensions returned by the terminal, and match the reply using ``R``. + On an error or timeout, display the URL as text. + +#. Write an 8 by 4 rectangle of :ref:`Unicode placeholders + ` for image 42. Give each cell its row and + column diacritics, and end each image row with a hard line break. Reset + the text colors afterward. Only these cells make the image visible. + +The :ref:`shell example ` below shows the full placeholder +loop. The following sections describe discovery, rendering, display and +copying; :ref:`content_reference` gives the wire format and request lifecycle. + + +Protocol overview +----------------- + +This protocol covers discovering support, sizing, placement, replies and the +retention of source. How renderers are installed and configured, and how the +terminal talks to them, is left to each terminal. Implementing this protocol +does not require a particular helper or renderer API. The protocol uses its +own OSC code and reuses the virtual placements, deletion commands and Unicode +placeholders of the graphics protocol. There is no compression, file transfer +or shared-memory transport. + +Throughout this document, a *connection* is one terminal session: the byte +stream between a terminal instance and the processes attached to its pty. +*Reset* means a full reset (RIS). A soft reset (DECSTR) does not affect the +state of this protocol. + + +Querying support +---------------- + +To detect support for the protocol, send ``a=q`` without ``t``. The terminal +must report support even when no renderers are configured, and return ``B``, +its source size limit in bytes, and ``D``, its :ref:`render deadline +` in milliseconds. Queries and listings have no payload +in the request; a non-empty one fails with ``EINVAL``:: + + ]8776;a=q:R=1\ + ]8776;a=q:R=1:s=OK:B=131072:D=500\ + +With ``t``, the terminal reports success only for a kind that is configured. +The reply does not include ``t``; the client matches it to the query using +``R``. The ``B`` in this reply may be lower than the terminal-wide limit. With +no renderers configured, every kind query returns ``ENOTSUP``. Note that +queries do not parse any source, and success does not guarantee that a render +will succeed:: + + ]8776;a=q:t=math:R=2\ + ]8776;a=q:R=2:s=OK:B=1024:D=500\ + ]8776;a=q:t=unknown:R=3\ + ]8776;a=q:R=3:s=ENOTSUP;VW5zdXBwb3J0ZWQga2luZA==\ + +Clients can also :ref:`list the available kinds `. + +The terminal must answer queries and listings from its local metadata, as soon +as it parses them, without invoking renderers or waiting for queued renders. +It must queue these replies before the reply to any subsequent primary device +attributes (DA1, ``CSI c``) query. Where both escape codes pass through, +clients can use this ordering to detect an OSC query that was ignored: if the +DA1 reply arrives first, the query was ignored somewhere along the way. + +When this protocol is disabled in the configuration, the terminal must ignore +OSC 8776 escape codes without replying, and must not re-render retained +content. Retained images keep their last pixels, scaled to their cells as for +a failed re-render, and copying and exporting still substitute the retained +source. + +Falling back to text +^^^^^^^^^^^^^^^^^^^^ + +For a reliable fallback, the client queries the kind, requests rendering, and +writes placeholders only after the render succeeds. A successful kind query +alone is not enough. On failure or timeout, the client prints the source as +ordinary text; a successful render that arrives late stays invisible, since +there are no placeholders for it. Applications must print the fallback +themselves, because terminals that do not support this protocol do not +display OSC payloads. Clients should print text when their output is +redirected, or goes through intermediaries that cannot carry the protocol. + + +Content kinds +------------- + +Kind identifiers contain 1 to 64 ASCII letters, digits or the characters +``-_.+/``, and are matched exactly. The terminal routes them to configured +renderers without inspecting the source or guessing its format. + +A kind names an input format and its meaning, independently of the renderer. +Examples include ``math``, ``smiles``, ``abc``, ``barcode/qr`` and +``barcode/ean13``. Supporting these kinds is optional. A successful kind +query reports availability, not support for every extension of its format. + +Additional kinds need no change to the transport. Their authors should +publish the grammar or format version, semantic examples and error cases. +Incompatible interpretations need different identifiers; private extensions +should use a project prefix, such as ``example/diagram``. + +Renderers must reject unsupported constructs rather than silently change +meaning. Fonts, colors and internal layout may differ, while preserving +meaning and the requested cell dimensions. Pixel-identical output is not +required. + + +.. _content_sizing: + +Rendering content +----------------- + +The terminal passes the :ref:`captured rendition and cell metrics +` to the renderer; other SGR attributes are unspecified. +Unless the kind itself specifies colors, the renderer uses that foreground +color over transparency or the supplied background, as it chooses. + +The renderer works out the natural layout of the content from the source and +the cell metrics, including any padding the content needs. It must use a +consistent natural scale for all inputs of the same kind; what that scale +means is up to the kind. ``c`` and ``r`` specify an exact number of columns +and rows, ``M`` and ``N`` specify maximums, and zero leaves them unspecified. +The renderer must follow these sizing rules: + +#. With no exact dimension, keep the natural scale, shrinking the content + uniformly to fit within ``M``. ``N`` is checked after fitting the width and + rounding to whole cells, and exceeding it is ``EFBIG``. Bounds never + enlarge content. To scale the height, the client specifies ``r``. +#. With one exact dimension, scale uniformly to fit it, and derive the other. + With both exact, use the largest uniform scale that fits. +#. ``F=1`` caps the scale at the natural size. Exact dimensions still set the + size of the canvas, and unused space is padded. +#. Round unspecified canvas dimensions up to whole cells. The renderer chooses + how to align the content within the padding. Never crop the content to + meet the constraints. +#. A kind with discrete scales, such as a whole number of pixels per barcode + module, uses the largest scale it can represent that does not exceed the + scale given by these rules. It derives unspecified dimensions from the scale + it actually used, and pads the rest of the canvas, including any exact + dimensions it cannot fill. If it can represent no scale that fits, the + render fails with ``EFBIG``. + +The terminal must enforce these limits: + +* The terminal must reject exact dimensions larger than their maximums with + ``EINVAL``, and derived dimensions larger than their maximums with + ``EFBIG``. The terminal never changes exact dimensions to make them fit. +* The terminal must reject a size of more than 297 columns or 297 rows with + ``EFBIG``, whether it was requested exactly or computed, because the + :ref:`placeholder diacritics ` can only + address rows and columns 0 to 296. The limit does not implicitly shrink + content that is too large. + +Renderers may also reject boxes that are too small for valid or legible output +with ``EFBIG``, and the terminal reports arithmetic overflow as ``EFBIG`` too. + +A successful reply includes ``i``, positive ``c`` and ``r``, and positive +pixel dimensions ``w`` and ``h`` covering the whole canvas, which is a whole +number of cells. When the request specified a nonzero ``p``, the reply +includes the ID of the installed placement, as a canonical decimal number; +otherwise the reply omits ``p``. The optional signed ``b`` gives the baseline +in pixels, down from the top of the canvas. Negative values are above the +canvas, and ``b=0`` is a baseline at its top. The value may lie outside the +canvas. The terminal omits ``b`` when no baseline is meaningful. + +The geometry in the reply describes the render at the captured cell metrics. +If the cell size changes before the result is installed, the terminal must +still install it into the cells it reported, and then re-render it as for any +later font change. The terminal does not update ``w``, ``h`` or ``b`` after +replying, so clients must not assume they are still current after a font +change. + +Measuring and releasing renders +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Rendering without writing placeholders is how you measure content; there is +no separate operation for it. Note that rendering allocates storage in the +terminal, even when no placeholders are ever displayed. Clients must release +successful renders they do not use with the graphics protocol's +``a=d,d=I,i=`` command, when they know the image ID and have not reset the +terminal or switched screens since sending the render request. This includes +images used only for measurement, results the client rejected for its own +reasons, and successful replies that arrived late. Otherwise, clients must +leave reclaiming the storage to the terminal. Specifying an image ID in the +request allows cleaning up without a reply, under the same conditions. + + +Displaying content +------------------ + +The terminal creates an invisible virtual placement, with ``z`` as its +stacking order. Clients display it by writing :ref:`U+10EEEE placeholders +`, using the cell dimensions from the reply. +These cells control the position, scrolling, erasing and reflow of the +content. There is no key to choose a placement mode or to move the cursor. +Clients must take care of the following: + +* Clients manage line breaks, and reset or restore the text colors after + writing the image and placement IDs as colors. +* Clients must not send a final chunk while an image ID color is the current + foreground, since the terminal cannot tell a color that encodes an image ID + from a color intended for text. +* Rows of placeholders. Clients should keep each row within the width of the + terminal. For rows that could wrap, including after a resize, they must + specify both the row and column diacritics on every cell. This avoids + depending on the graphics protocol inferring columns across physical lines. + The rendered image always remains a single rectangle; these rules concern + only its placeholder cells. + +When the request has no ``p``, or ``p=0``, the new virtual placement is +unnumbered; the terminal does not allocate a placement ID for it. Separately, +a placeholder with no underline color, or an underline color of zero, may +refer to any virtual placement of the image, as the graphics protocol +specifies. So images with a single placement need no underline color. In this +protocol, ``p`` must be between 0 and 16777215, inclusive, and the terminal +must reject larger values with ``EINVAL``. This range fits the 24-bit +underline color of the placeholder, and does not change the range of +placement IDs in the graphics protocol. + +Clients that specify both the image ID and the exact cell dimensions can write +placeholders without waiting for the reply. However, suppressing replies means +giving up confirmation of success, and with it a reliable text fallback. + +Image IDs +^^^^^^^^^ + +``R``, ``i`` and ``p`` are independent of each other. Images and placements +share the per-screen namespaces of the graphics protocol. For ``i=0``, the +terminal allocates a nonzero image ID that fits in 24 bits, and is neither +used by an existing image nor reserved by a pending render. It must not reuse +an ID it previously allocated for this protocol while old placeholders could +still remain, on screen or in the scrollback, even after the image data was +deleted or evicted. When no IDs are left, the request fails with ``EFBIG``. +These rules do not change the range of image IDs, or how they are allocated, +in the graphics protocol. Clients that specify image IDs themselves are +responsible for choosing them. + +The placement and deletion commands of the graphics protocol apply to these +images; there is no need to transmit an image with the graphics protocol. A +successful ``a=d`` render to an existing image ID follows the graphics +protocol's :ref:`replacement rules `, removing the +old image and all its placements before installing the new pixels, source and +virtual placement. Retransmitting the whole image with the graphics protocol +also replaces it, and removes its association with the source. + +Other successful changes to pixels with the graphics protocol, including +editing frames (``a=f``) and composition (``a=c``), clear the association +with the source and stop re-rendering from it. They otherwise behave as they +always have. Failed changes, and changes to placements only, do not clear the +source. Clients that need old scrollback to keep its meaning must use fresh +image IDs for new content. + + +Interaction with other terminal actions +--------------------------------------- + +Images follow the graphics protocol's :ref:`rules for reset, the alternate +screen, erasing the display and scrolling `. +Placeholders wrap and reflow like ordinary text. + +Pending renders follow the :ref:`cancellation rules `. +Full-screen programs that redraw by erasing the display should erase before +requesting renders, or wait for outstanding replies first. + + +Retaining and copying source +---------------------------- + +The terminal must keep each image's kind, exact source bytes, sizing policy, +rendition and final cell size, even after the program that sent it exits. The +source lives as long as the image data, unless a pixel change with the +graphics protocol clears the association. Deleting placements while keeping +the image data also keeps the source. Freeing or evicting the image data +releases its source, leaving nothing to substitute when copying or exporting. + +When the font or cell size changes, the terminal must re-render the content +into the cells it already occupies, keeping the image ID, placements and +association with the source. The re-render follows the sizing rules, with the +existing cells as both exact dimensions and the original ``F``. A render that +was originally requested without exact dimensions uses ``F=1``, keeping its +natural scale rather than growing to fill the cells. Content whose natural +size at the new cell size is larger than the cells shrinks to fit; it is +never cropped. If the re-render fails, the terminal must keep the previous +pixels, scaled to the same cells. + +Terminals should remember when a rendition uses default or indexed colors, +and re-render when those colors change. Explicit RGB colors stay fixed. The +terminal must never take the foreground color of the content from the image +ID color of a placeholder cell. + +When copying or exporting text, as plain text or with ANSI styling, including +selections, buffer exports and viewing the scrollback in a pager, the +terminal must substitute the original source, never a description from the +renderer: + +* It emits the complete source once for each selected *occurrence* of the + image, at its first selected cell, even when only part of the occurrence is + selected, and emits nothing for the remaining cells. Two selected + occurrences produce two copies of the source. +* It must preserve the order of other text and the newlines in the source, + even if the geometry of the export differs. +* Exporting needs no renderer, and ANSI exports may add ordinary text + styling. There is no such compatibility guarantee for raw recordings of the + bytes the program sent. + +Roughly, an occurrence is one printed copy of the image, even if its rows +wrapped. Clients that need copying to be stable must end each row of +placeholders with a hard line break; see :ref:`content_occurrences` for +the precise rules. + + +.. _content_reference: + +Protocol reference +------------------ + +The following sections specify the wire format and the less common cases. +They are part of the protocol, including for terminals that use their own +renderer API. Discovery, rendering and copying must behave the same way +regardless of how the terminal schedules work. + + +The content rendering escape code +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The protocol uses OSC **8776** with the following form:: + + ] 8776 ; =[:=]* [;] \ + +Spaces in this definition are for clarity only. ``ESC`` is the byte ``0x1b``. +Senders, whether client or terminal, use the seven-bit forms ``ESC ]`` and +``ESC \``, and receivers also accept ``BEL (0x07)`` as the terminator. The +eight-bit forms need not be supported. + +The metadata is a colon-separated list of ``key=value`` pairs: + +* Keys and values are case-sensitive ASCII. Keys contain letters, digits and + ``_``. Values must not be empty, and contain visible ASCII (``0x21`` to + ``0x7e``) other than ``:`` and ``;``. +* Unsigned integers contain only decimal digits and fit in 32 bits. ``z`` and + ``b`` are signed 32-bit integers, with an optional leading minus sign. A + plus sign is not allowed. Leading zeros do not matter, so ``R=007`` and + ``R=7`` are the same token, and ``-0`` is zero. +* Numeric fields that are not specified default to zero, unless stated + otherwise. Flags accept only the values listed for them. +* Both clients and terminals must ignore keys they do not know. The terminal + also ignores keys that this specification defines only for other actions, + except on continuation chunks, as described under `Errors in chunks`_. + Ignored keys must still follow the key and value syntax and the rule + against duplicate keys, but their numbers and flags are not validated. + +Extensions preserve existing behavior and provide a capability query. +Clients must use that query before relying on the extension. + +The payload carries the source. A missing payload is the same as an empty +payload after the ``;``. The client must encode each chunk of source +independently in standard :rfc:`base64 <4648>`, with the padding and without +whitespace. Whoever receives a message may either reject or ignore tab, LF, CR +and space inside the payload, but must not tolerate them anywhere else in the +escape code. Once all chunks are put together, the source must be valid +UTF-8. + +The terminal must reject source containing C0 controls (``U+0000`` to +``U+001F``, including ``ESC``) other than tab, LF and CR, as well as +``DEL (U+007F)`` and the C1 controls (``U+0080`` to ``U+009F``). It must pass +the source on unchanged, without normalizing whitespace, line endings or +Unicode. Ordinary backslashes need no extra escaping. + +The following limits do not count the OSC framing: + +.. csv-table:: Size limits + :header: "What", "Limit" + + "Metadata", "1,024 bytes" + "Payload per chunk", "4,096 encoded bytes, which is 3,072 decoded bytes" + "Total source", "131,072 decoded bytes; the ``B`` key reports the terminal's own limit, which may be lower" + + +.. _content_listing: + +Listing available kinds +^^^^^^^^^^^^^^^^^^^^^^^ + +To list the available kinds, send ``a=l``. The terminal returns up to 16 +kinds in ascending ASCII order, separated by LF without a trailing LF, and +base64-encoded. The reply has ``m=1`` if there are more pages, or ``m=0`` for +the last page. A page with ``m=1`` must contain at least one kind. To get the +next page, the client sends ``after=``, which returns only names +that sort strictly after it. The ``after`` value uses kind syntax and is +omitted for the first page. Listings expose no paths or renderer +configuration, and do not reserve any renderer. + +When no renderers are configured, or no kinds follow the cursor, the terminal +returns an empty successful listing with ``m=0``, and a trailing ``;`` for the +empty payload:: + + ]8776;a=l:R=6\ + ]8776;a=l:R=6:s=OK:m=0;\ + + +Transferring source in chunks +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Source can be sent in a single message, or in multiple chunks that share a +nonzero ``R``. For example, here is ``x^2`` followed by ``+ y^2``, with a +leading space, sent in two chunks:: + + ]8776;a=d:t=math:R=5:M=80:N=12:m=1;eF4y\ + ]8776;a=d:R=5:m=0;ICsgeV4y\ + ]8776;a=d:R=5:s=OK:i=43:c=7:r=2:w=70:h=40:b=24\ + +* The first chunk contains ``t`` and all the rendering metadata. +* Continuation chunks omit ``t`` and contain only ``a=d``, ``R``, ``m`` and the + payload. +* Every chunk of a multipart upload must carry the same nonzero ``R``. +* ``m=1`` means more chunks follow, and ``m=0``, the default, completes the + upload. +* Every chunk except the last must be non-empty. The last chunk may be empty, + but a complete source that is empty fails with ``EINVAL``. +* Uploads with different tokens may be interleaved, but only as whole OSC + escape codes, never as bytes within one. Chunks are put together in the + order they arrive; there is no chunk index. + +The terminal must decode each chunk separately, put the results together, and +only then validate the UTF-8, since characters may be split across chunks. The +terminal must not start parsing the source as its kind, or rendering it, +before the final chunk has been validated. The first chunk supplies the +reply policy for the whole request. + +Note that chunking does not guarantee that intermediaries such as terminal +multiplexers will support or deliver the upload. + + +.. _content_lifecycle: + +Request lifecycle +^^^^^^^^^^^^^^^^^ + +Terminals may render synchronously or continue processing input while a render +is pending. Clients match replies using ``R``: a render reply can arrive after +replies to later requests, including cursor position reports and DA1. Queries +and listings retain the ordering required by `Querying support`_. + +The reply policy is ``q=0`` for all replies (the default), ``q=1`` for errors +only, or ``q=2`` for no replies. + +A request is *outstanding* from its first message until it completes or is +silently discarded. It *completes* when its final outcome has been committed +and any reply allowed by ``q`` has been queued. Completion releases its token +even when replies are suppressed. Finishing an upload does not complete its +pending render. + +On the final chunk, the terminal must capture the active screen, its cell +metrics, and the colors for ordinary text under the current SGR attributes +and screen-wide reverse video state. This color state is the *rendition*. +Rendering uses these captured values, even if the terminal state changes +before completion. + +The terminal sends one final reply per request, unless ``q`` suppresses it or +the protocol requires silence. A render can return ``OK`` only after +installing its pixels, source and virtual placement. There are no unsolicited +rendering notifications. The terminal must not move the cursor for these +requests. On error, the terminal must not create placements or replace +retained content, and must discard later results from that request. + +.. _content_deadlines: + +Deadlines and retries +~~~~~~~~~~~~~~~~~~~~~ + +``D`` bounds the time from parsing the final chunk to completion, including +time queued inside the terminal. It applies even when ``q`` suppresses the +reply. The terminal must fail a render it cannot finish in time with +``ETIMEDOUT`` or ``EBUSY``. Transport and time spent waiting unparsed in the +terminal's input are outside this deadline. + +A synchronous terminal can leave later requests, including queries, unparsed +behind earlier renders from any client on the connection. With ``N`` requests +outstanding, clients should allow about ``N × D`` for the last reply, plus +transport time. A client-side timeout leaves the outcome unknown; a query +timeout does not prove that support is absent. + +``EBUSY`` must include a positive ``retry`` delay in milliseconds approximating +the remaining wait, including failure backoff. This is advice, not a +reservation. Clients may retry with a fresh token or fall back to text. +Permanent unavailability must become ``ENOTSUP`` or ``EIO``, rather than +``EBUSY`` forever. + +Incomplete uploads may expire silently. When the terminal cannot accept a +new upload, it replies with ``EBUSY`` or ``ENOMEM``. + +.. _content_cancellation: + +Cancellation +~~~~~~~~~~~~ + +After the final chunk, a pending render targets the requested image ID or one +reserved by the terminal. These events must cancel pending work, with an +``ECANCELED`` reply subject to ``q`` unless the table specifies silence: + +.. list-table:: Cancellation rules + :header-rows: 1 + :widths: 60 40 + + * - Event + - Effect + * - A screen is cleared, including clearing the alternate screen on entry. + - Cancel renders captured for that screen, even if no image has been + installed. + * - A graphics command directed at the target ID frees or replaces image + data, or successfully changes pixels: deletion with an uppercase + selector, retransmission, frame editing or composition. + - Cancel pending renders for that ID. + * - A render submitted later for the same target ID succeeds. + - Cancel older pending renders for that ID. + * - The terminal resets or the connection closes. + - Discard uploads and pending renders silently. This takes precedence + over cancellation by screen clearing. + +Switching screens while preserving the original screen leaves its pending +renders running; they complete on that screen. Placement-only deletions and +deletions by position or visibility do not cancel pending renders. A later +render that fails also leaves earlier renders running. + +Cancellation produces no further reply after completion. The terminal must +discard cancelled work; late results must never restore it or appear in +another screen or tab. + + +Requests and replies +^^^^^^^^^^^^^^^^^^^^ + +There are three actions. ``a=q`` queries support, ``a=l`` lists the available +kinds, and ``a=d``, the default, renders content into a virtual placement. The +:ref:`key reference ` below lists all the keys. For timing, +completion and cancellation, see :ref:`content_lifecycle`. + +The ``R`` key identifies a request, not the content. A nonzero token must be +unique among the outstanding requests on a connection, and is required for +uploads that span multiple chunks. ``R=0``, the default, is for independent +single-message requests whose replies need not be told apart. Such a request +never invalidates another request through token reuse. Clients must use +nonzero tokens when they need to tell concurrent replies apart. Clients should +never reuse a nonzero token on a connection, for example by counting upward +from a random starting point; the 32-bit space is ample. + +The terminal replies using the same escape code, always terminated by +``ESC \``. Every reply carries ``a``, ``R`` and ``s=``, where ``a`` is +the reply type, one of ``q``, ``l`` or ``d``. Only ``s=OK`` means success; +any status code the client does not recognize is an error. + +Replies are built only from fixed protocol tokens, numbers written by the +terminal and base64 payloads; they never copy raw bytes from the request. +Numeric values may be the same as those the client sent, including ``i``, +``p``, ``c`` and ``r``. The terminal writes all numbers in canonical decimal, +without leading zeros, and zero as ``0``. ``R`` is the parsed request token, +or ``0`` when it was omitted, so clients compare tokens as numbers. Apart from +the OSC framing, replies contain only ASCII letters, digits and the characters +``_=:;+/-``. + +Successful queries and renders have no payload; listings carry kind names. +Errors may carry a base64-encoded UTF-8 diagnostic, of at most 512 decoded +bytes, following the same control character restrictions as source. Clients +act on the status code, not on the diagnostic text. Terminals must not forward +raw diagnostics from renderers or quote values from the request in them. + +.. csv-table:: Reply status codes + :header: "Status", "Meaning" + + "``OK``", "Completed." + "``ENOTSUP``", "Unsupported action, kind or kind feature." + "``EINVAL``", "Invalid envelope, source or conflicting constraints." + "``EFBIG``", "Source/output limit or automatic image-ID space exhausted, or minimum valid rendering cannot fit the box." + "``ENOMEM``", "Insufficient storage." + "``EBUSY``", "Retryable temporary unavailability. ``retry`` gives a suggested delay in milliseconds." + "``ETIMEDOUT``", "Render deadline exceeded." + "``ECANCELED``", "Pending render invalidated by screen clearing, image deletion, replacement or pixel mutation." + "``EIO``", "Renderer failed or returned invalid output." + + +.. _content_keys: + +Control data reference +^^^^^^^^^^^^^^^^^^^^^^ + +The table below lists the defaults for requests and the requirements for +replies. + +.. csv-table:: Protocol keys + :header: "Key", "Used in", "Default", "Meaning" + + "``a``", "Requests/replies", "``d``", "``q`` query, ``l`` list, ``d`` render. In replies, the reply type; ``d`` for an invalid or unknown action." + "``R``", "Requests/replies", "``0``", "Request token. Nonzero for multipart; returned in canonical decimal in every reply." + "``t``", "Kind queries, first render chunk", "Unset", "Kind. Required on first render chunk, forbidden on continuations. Never returned in replies." + "``after``", "List requests", "Unset", "Pagination cursor. Names strictly after this kind; unset for first page." + "``q``", "Requests", "``0``", "``0`` all replies, ``1`` errors only, ``2`` none. On render continuations, ``EINVAL``. A duplicated or invalid ``q`` supplies no policy." + "``m``", "Render chunks, list replies", "``0`` on requests", "``1`` more, ``0`` final. Mandatory on list replies." + "``c``, ``r``", "First render chunk, render replies", "``0``", "Exact columns/rows; zero unspecified. Successful replies give dimensions from 1 to 297." + "``M``, ``N``", "First render chunk", "``0``", "Maximum columns/rows; zero unbounded by client. ``M`` bounds width fitting; ``N`` bounds resulting height." + "``F``", "First render chunk", "``0``", "``0`` scales to exact box; ``1`` prevents enlargement above natural size. Bounds alone never enlarge." + "``i``", "First render chunk, render replies", "``0``", "Image ID; zero requests terminal allocation. Successful replies give nonzero ID." + "``p``", "First render chunk, successful render replies", "``0``", "Graphics protocol placement ID of the virtual placement, 0–16777215 in this protocol; larger values are ``EINVAL``. Zero leaves the placement unnumbered. Successful replies return ``p`` only when nonzero." + "``z``", "First render chunk", "``0``", "Signed 32-bit graphics protocol stacking order of the virtual placement." + "``s``", "All replies", "Required", "``OK`` or error code." + "``B``", "Successful query replies", "Required", "Terminal/kind source-byte limit, at most 131,072." + "``D``", "Successful query replies", "Required", "Positive render deadline in milliseconds; see :ref:`content_deadlines`." + "``retry``", "``EBUSY`` replies", "Required", "Positive delay in milliseconds approximating the remaining wait, including failure backoff." + "``w``, ``h``", "Successful render replies", "Required", "Positive canvas dimensions in device pixels." + "``b``", "Successful render replies", "Unset", "Signed 32-bit baseline from canvas top. Zero is valid; negative is above it. Omitted if meaningless." + + +.. _content_occurrences: + +Identifying image occurrences +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +An *occurrence* consists of placeholder cells with the same image ID, +placement ID and *logical origin*. Decode each cell's zero-based image row +and column ``(r, c)`` as in the graphics protocol. Soft-wrapped physical +lines form one logical line; a hard line break starts the next one. + +For a cell on logical line ``L`` at cell offset ``X``, the logical origin +is ``(L - r, X - c)``. Cells with equal origins belong to the same occurrence, +even if some cells are erased or unselected. No physical-origin links or +transitive grouping are needed. For example, a four-column image in a +three-column terminal has these coordinates (each entry is ``r,c``):: + + logical line 0: [0,0] [0,1] [0,2] ← soft wrap + [0,3] ← hard line break + logical line 1: [1,0] [1,1] [1,2] ← soft wrap + [1,3] + +Every cell has origin ``(0, 0)``. Selecting any part of either row emits the +source once. Printing a second copy at another origin emits it again. + +Clients must end each image row with a hard line break and start the next at +the same logical offset for it to be one occurrence. Advancing the image row +across an automatic wrap, without a hard line break, changes the logical +origin: those rows are separate occurrences and can copy the source more +than once. This rule continues to apply after a terminal resize. + +Selection boundaries do not redefine logical lines. When a selection starts +inside a soft-wrapped line, account for its retained prefix to recover the +cell offsets and any inferred diacritics. There is no need to inspect +unrelated scrollback or follow links through unselected cells. During a +forward export, origins more than 296 logical lines behind the current line +can be forgotten, since no valid placeholder row can refer to them. + +Placeholders that refer to no retained source fall into two cases: + +* If a live image has the ID but no source, because of retransmission or a + pixel change with the graphics protocol, it is just a graphics protocol + image. Its placeholders are copied the way the terminal already copies + placeholders of any other image. +* If no live image has the ID, because its data was deleted or evicted, the + terminal must omit the placeholders while it still remembers that this + protocol rendered that ID. The terminal may limit how much of this it + remembers; beyond that limit, these placeholders are also copied the way the + terminal already copies them. + + +.. _content_errors: + +Handling errors +^^^^^^^^^^^^^^^ + +This section collects what happens when messages are malformed, tokens +collide, or work has to be abandoned. Many of these rules share one concern: a +reply that no client reads can reach a shell as input, so when the terminal +cannot be sure who a reply is for, it stays silent. + +Malformed and oversized messages +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +* The terminal rejects duplicate keys, empty values, malformed numbers and + invalid encodings with ``EINVAL``. +* Actions must be a single character: other lengths fail with ``EINVAL``, and + unknown single-character actions fail with ``ENOTSUP``. Both kinds of + failure are answered as ``a=d``. +* The terminal must skip an oversized escape code up to and including its + terminator, without acting on it. When the request token can be validated + from the complete metadata, within the metadata limit, the terminal must + treat the oversized message according to the rules for chunks below, with + the status ``EFBIG``. That is, it aborts the matching upload, rejects a new + request, or silently discards a continuation that matches no upload. +* Duplicate or malformed ``R`` fields, and truncated metadata, must never + supply a token. +* Only ``ESC \`` or ``BEL`` completes a message. An OSC 8776 escape code that + ends any other way, for example because of a CAN or SUB, or an ESC that + starts another escape code, is malformed and never supplies a token, even if + the bytes received so far would parse. +* The terminal must never turn malformed bytes into a valid request by + removing control characters from any part of the message, including the OSC + number. It must reject or silently discard such a message instead. So if a + terminal's parser drops control bytes from OSC strings, it must record that + it did so, and where, so that the protocol handling forgives nothing except + tab, LF and CR inside the payload. + +If the terminal cannot tell which request a message belongs to, it must +silently discard that message, and also every incomplete upload on that +connection, since a later continuation could otherwise complete a source with +a missing chunk. The same applies to malformed messages that are discarded +without a trustworthy token, including escape codes ended by anything other +than ``ESC \`` or ``BEL``. Note that on a shared connection this also +abandons the incomplete uploads of other clients. That is intentional, since +the terminal has no way to tell which upload lost a chunk. + +Who gets an error reply +~~~~~~~~~~~~~~~~~~~~~~~ + +An error reply follows the reply policy of the outstanding request whose token +the message matches, and otherwise the message's own ``q``. A message whose +own ``q`` is duplicated or invalid has no policy of its own. If no outstanding +request supplies one, the terminal must discard the message silently, so that +an unwanted reply can never become input to a shell. + +Errors in chunks +~~~~~~~~~~~~~~~~ + +The terminal classifies each message that has a trustworthy token before +acting on it: + +* A message with no ``t`` and no action other than ``d`` is a continuation. +* Any other message is a new request: ``t`` starts a new render, and any + other action, valid or not, starts a new request even without ``t``. + +``R=0`` with ``m=1``, or on a continuation, fails with ``EINVAL`` under the +message's own ``q``, without affecting other requests. + +A continuation applies only to an incomplete upload with the same token: + +* If there is no such upload, the terminal must discard the continuation + silently, whether or not it is otherwise valid. That includes continuations + that arrive after the upload has failed or expired, and while the render of + a completed upload is pending. +* If there is such an upload, a malformed continuation aborts it with + ``EINVAL``. That covers every ``EINVAL`` condition of + `The content rendering escape code`_ + and `Malformed and oversized messages`_, a ``q`` or any other known key + that belongs only on the first chunk, invalid base64 and an empty chunk + that is not the last. +* An oversized chunk, or too much source in total, aborts the upload with + ``EFBIG``. +* Unknown keys are ignored, as usual. + +Token reuse +~~~~~~~~~~~ + +A new request whose nonzero ``R`` is already outstanding invalidates the +outstanding request, and is itself rejected. The terminal replies exactly once +for both, under the outstanding request's reply policy, with the reply type +taken from the new message's action, or ``a=d`` if that is invalid or unknown. +The status is the new message's own error (``EFBIG`` when oversized, +``ENOTSUP`` for an unknown action, ``EINVAL`` when malformed), or ``EINVAL`` +if the new message is otherwise valid. A new request whose token is not +outstanding receives its normal error, even without a kind. + +So clients replacing an abandoned upload must use a fresh token. A client that +abandons a request must not display a late result, and must avoid reusing its +token while old chunks or replies could still arrive. Note that with ``q=2`` a +client cannot tell when a request has completed, and reusing a token whose +render is still pending silently cancels both requests. Clients release late +successful renders as described under `Measuring and releasing renders`_. +Multiplexers that implement this protocol should route replies to the right +client and prevent token collisions. + + +Security considerations +^^^^^^^^^^^^^^^^^^^^^^^ + +Only the user's local configuration chooses renderers. Requests cannot name +executable paths, install code or change renderer settings. Both the source +and the output of renderers are untrusted. Receiving source does not give +access to any local resources. How renderers are isolated, and what they are +allowed to run, is outside the scope of this protocol. + +Copied or exported source may contain syntax and newlines that are not visible +in the rendered image. The terminal must treat it as untrusted text, and keep +its normal protections for the clipboard and pasting. + +A reply that no client reads can reach a shell as input. That is why replies +are limited to the characters listed under `Requests and replies`_, and +contain nothing from the request other than decimal numbers. Silently +discarding expired uploads, and messages without a trustworthy token or reply +policy, serves the same goal. + +Rendering spends local CPU time and memory on behalf of untrusted output. +Terminals limit that work according to their own policy, and report it with +``EBUSY``, ``ENOMEM`` and ``ETIMEDOUT``. The protocol promises clients a +fallback to text, not a render. + +Queries and listings reveal which kinds the user has configured to any program +that writes to the terminal, including programs running on remote machines. +Like other queries that identify the terminal, this can be used for +fingerprinting. Listings expose no paths or renderer configuration, and users +can disable the protocol. + + +.. _content_example: + +A shell example +--------------- + +With a QR renderer configured, paste this into zsh or bash 4.2 or newer, +using a UTF-8 locale and a terminal at least 24 columns wide. It displays +a QR code for ``https://github.com/kovidgoyal/kitty`` at a fixed size. +Older shells, including macOS's ``/bin/sh``, do not understand the +``\U`` escapes. + +This demonstration uses ``q=2`` because the shell does not read replies. +It therefore cannot detect failure or provide a reliable text fallback. +See `A minimal example`_ for the request/reply flow applications should use. + +.. code-block:: sh + + printf '\e]8776;a=d:t=barcode/qr:R=102:i=42:c=24:r=12:q=2;aHR0cHM6Ly9naXRodWIuY29tL2tvdmlkZ295YWwva2l0dHk=\e\\' + d=0305030D030E03100312033D033E033F0346034A034B034C0350035103520357035B0363036403650366036703680369 + printf '\e[38;2;0;0;42m' + for r in {0..11}; do + for c in {0..23}; do + printf "\U10EEEE\U${d:$((r*4)):4}\U${d:$((c*4)):4}" + done + printf '\n' + done + printf '\e[39m' + +The first command asks the terminal to render the base64-encoded URL with +the ``barcode/qr`` kind, as image 42 (``i=42``), +exactly 24 columns by 12 rows (``c=24:r=12``). + +The rest writes a 24 by 12 rectangle of :ref:`placeholder cells +`, which is where the image appears. The +foreground color ``38;2;0;0;42`` carries the image ID. Each cell is +``U+10EEEE`` followed by a row diacritic and a column diacritic, so that the +coordinates of every cell survive wrapping or clipping. ``d`` holds the +diacritics for the numbers 0 to 23, four hex digits each, from +:download:`rowcolumn-diacritics.txt <../gen/rowcolumn-diacritics.txt>`. + + +.. _content_renderer_conventions: + +Renderer conventions +-------------------- + +Renderer authors could maintain a shared registry of kind names and format +references, with a corpus of input text paired with example PNG files for +comparing output. These conventions are optional; terminals do not interpret +the content format or validate renderer output against the corpus. diff --git a/docs/graphics-protocol.rst b/docs/graphics-protocol.rst index 67e1d2428..7267271f5 100644 --- a/docs/graphics-protocol.rst +++ b/docs/graphics-protocol.rst @@ -609,6 +609,8 @@ does not support the queried action, or does not support feature queries at all, and the client has to fall back to other means of detecting features. +.. _graphics_display_images: + Display images on screen ----------------------------- @@ -1317,6 +1319,8 @@ Key Value Default Description ======= ==================== ========= ================= +.. _graphics_interaction: + Interaction with other terminal actions -------------------------------------------- diff --git a/docs/protocol-extensions.rst b/docs/protocol-extensions.rst index 6fed2911c..552a19c49 100644 --- a/docs/protocol-extensions.rst +++ b/docs/protocol-extensions.rst @@ -26,6 +26,7 @@ please do so by opening issues in the `GitHub bug tracker underlines graphics-protocol + content-rendering-protocol keyboard-protocol text-sizing-protocol dnd-protocol diff --git a/docs/screenshots/content-aspirin.png b/docs/screenshots/content-aspirin.png new file mode 100644 index 000000000..cddc602ca Binary files /dev/null and b/docs/screenshots/content-aspirin.png differ diff --git a/docs/screenshots/content-aspirin.smi b/docs/screenshots/content-aspirin.smi new file mode 100644 index 000000000..9471bcfeb --- /dev/null +++ b/docs/screenshots/content-aspirin.smi @@ -0,0 +1 @@ +CC(=O)Oc1ccccc1C(=O)O \ No newline at end of file diff --git a/docs/screenshots/content-caffeine.png b/docs/screenshots/content-caffeine.png new file mode 100644 index 000000000..6d35d9eb9 Binary files /dev/null and b/docs/screenshots/content-caffeine.png differ diff --git a/docs/screenshots/content-caffeine.smi b/docs/screenshots/content-caffeine.smi new file mode 100644 index 000000000..fbd159fd7 --- /dev/null +++ b/docs/screenshots/content-caffeine.smi @@ -0,0 +1 @@ +Cn1cnc2c1c(=O)n(C)c(=O)n2C \ No newline at end of file diff --git a/docs/screenshots/content-math.png b/docs/screenshots/content-math.png new file mode 100644 index 000000000..bf18c74c1 Binary files /dev/null and b/docs/screenshots/content-math.png differ diff --git a/docs/screenshots/content-music.abc b/docs/screenshots/content-music.abc new file mode 100644 index 000000000..8c3ca7c13 --- /dev/null +++ b/docs/screenshots/content-music.abc @@ -0,0 +1,6 @@ +X:1 +M:4/4 +L:1/8 +%%stretchlast true +K:D clef=treble +!mf!(3DEF (G>A) {c}B !trill!^A !fermata!d2 |] diff --git a/docs/screenshots/content-music.png b/docs/screenshots/content-music.png new file mode 100644 index 000000000..75ed57074 Binary files /dev/null and b/docs/screenshots/content-music.png differ diff --git a/docs/screenshots/content-pdf417.png b/docs/screenshots/content-pdf417.png new file mode 100644 index 000000000..ed13948ac Binary files /dev/null and b/docs/screenshots/content-pdf417.png differ diff --git a/docs/screenshots/content-qr.png b/docs/screenshots/content-qr.png new file mode 100644 index 000000000..d049b6648 Binary files /dev/null and b/docs/screenshots/content-qr.png differ