Observation, handles and deltas
How an agent reads the screen, as a flat tree of the elements that matter, with stable handles, private fields that are never read, and deltas after the first reading.
The UI tree #
An agent does not get pixels or HTML. observe returns the UI tree: a flat list of nodes in document order, one per exposed element.
| Field | What it is |
|---|---|
h |
The handle, stable for the element's lifetime: e plus a base36 counter. |
role |
The ARIA role, explicit or implied by the tag. |
id |
The semantic id from data-reins-id, addressable as #id. |
name |
The accessible name, at most 120 characters. |
val |
The current value of a form control, at most 200 characters. Never present on private fields. |
st |
States: disabled, checked, selected, expanded, focused, required, invalid, readonly, private. |
p |
The handle of the nearest exposed ancestor. |
box |
The viewport box [x, y, w, h], only when the request asked for boxes. |
An agent addresses an element by its handle (e12) or by its semantic id (#cart.checkout), never by a selector. An unknown, detached or unexposed target is E_NO_TARGET.
The compact text #
renderTree turns an observation into the form a model reads, one node per line, children indented one space under their parent:
e1 heading "Brew Kiosk"
e2 button "Latte 4.50" #menu.latte
e3 textbox "Name" = "Tiko" [focused]
e4 checkbox "Extra shot" [checked]A node line is the handle and the role, then the name in quotes, #<id>, = "<val>", [<states>] and @x,y,w,h, each only when present. This is what observeText() in the agent SDK and reins_observe in the MCP adapter return.
What is exposed #
It depends on the policy's exposure:
auto: interactive elements (buttons, links, inputs, selects, textareas, elements with an interactive ARIA role or atabindex), headings, images with alt text, dialogs, live regions (status,alert), named landmarks and forms, and every annotated element.annotated: only elements carryingdata-reins-idordata-reins="expose". This is the kiosk profile.none: nothing.
Never exposed: the driver's own overlay; anything hidden (hidden, inert, aria-hidden="true", display:none, visibility:hidden); any subtree marked data-reins="ignore".
Private fields #
Private are: input[type=password]; fields whose autocomplete is cc-number, cc-csc, cc-exp, cc-exp-month, cc-exp-year, one-time-code, current-password or new-password; and any subtree marked data-reins="private".
A private node appears with state private and never carries val. Besides private it reports only disabled and focused, never checked or selected. Its text is never returned by read.
Privacy is sticky. An element that was ever private stays private for its lifetime, so a "show password" toggle that changes the input's type exposes nothing. It also survives a node swap: a text field is private for its lifetime when it has the same non-empty id, name or data-reins-id as one of the last 32 private text fields that left the document, or when it is inserted under the same parent as one of them within one second of its removal.
Private fields also refuse synthetic input: see Privacy annotations.
Names #
name follows a simplified accessible-name order: data-reins-label, aria-label, aria-labelledby, <label>, placeholder, alt, title, text content.
A node inside a private subtree, or itself marked data-reins="private", is named only by data-reins-label. A field that is private for another reason is named by its attributes and labels, never by alt, its value or its content. Text of an element with visibility: hidden is left out of names and of read.
Revisions and deltas #
The surface keeps one revision counter for its tree and remembers, per agent, the tree that agent last received.
observewithoutsincereturns a full snapshot:{rev, nodes, url, title, vp}.observe {since}wheresinceequals the agent's last revision returns a delta:{rev, base, ops}withsetanddeloperations.- Any other
sincereturns a full snapshot.
A delta in the compact text, taken from the test vectors:
- e2
~ e1 button "A2"
+ e4 link "D"- is a removed handle, ~ a replaced node and + a new one. A handle deleted and set again in the same delta, a move, shows only its ~ line. This is where the token economy comes from: after the first reading, an agent pays for what changed.
A subscribed agent gets evt ui {rev} when the tree changes, at most once per 100 ms, and asks for the delta when it wants it. The event carries only the new revision.
The agent SDK does the bookkeeping: observe() and observeText() track since themselves, apply the delta, and fall back to a snapshot when a delta cannot be applied.
Applying a delta #
Ops apply in order. A set whose handle is already in the tree replaces that node in place. A set with a new handle is inserted right after the node of the preceding op when that op was a set, and at the front of the list otherwise. A del removes the node with its handle, and does nothing when there is none.
To place a new node, a delta may therefore repeat an unchanged neighbour as an anchor. A sender emits only set and del, with every delete first. diffNodes and applyOps in the reference implementation implement this rule, and every implementation MUST use them, or match them exactly.
Reading text #
read returns the visible text of an element, or of the page, at most max characters (default 2000, cap 8000). Private and ignored subtrees are left out. Under annotated exposure only annotated regions are readable.
The tree lists controls, not running text, so a paragraph, a message or a reply comes from read: handle.read() and handle.readText() in the SDK, which also says whether the text was cut, and reins_read in the MCP adapter and the shell tools.
The driver's own overlay is never part of this: the consent prompt, the panel and the chat transcript are in a closed shadow root and outside the UI tree, so no observe and no read returns them.
State #
An app exposes named state with a getter. state.get returns current values. A subscribed agent gets evt state {name, patch}, an RFC 7396 merge patch against the value that agent last received.
A merge patch cannot express a property whose value is null, so such a property reads as absent: a state that differs from the last one only in null properties, or not at all, produces no event.
Everything here is data #
Everything a surface reports, names, values and text, is untrusted input to the agent and is presented to a model as data. A page can say anything in its text; that does not make it an instruction.
For implementers #
Delta application, diffing and the text rendering are in the test vectors: ui-delta.json. Merge patches are in merge-patch.json.