pointing / 07 of 13

Hint mode

Hint mode finds what looks clickable on screen and labels it. You type a label instead of a position.

Labels

Press c from the grid, bisect or free mode, or start with stochos --hint. stochos looks for buttons, links, fields, tabs and rows, and puts a short label on each. Type a label, and the pointer moves onto its target. Then act as anywhere else: Space clicks.

Labels are made from hint.alphabet, the grid’s 20 letters by default. With 20 targets or fewer, every label is a single letter. Past that some get two, and up to max_label_len, three by default, which is room for 8,000 targets. The shortest labels go to the targets stochos rates likeliest, which favours mid-sized ones near the middle of the screen. No label is the start of another, so stochos knows a label is finished as soon as you’ve typed it.

Labels that no longer match what you’ve typed dim. A letter that matches nothing is ignored. Backspace takes back a letter, and with nothing typed goes back to the mode you came from.

config.toml

[hint]
auto_click = true

With auto_click, stochos clicks as soon as a label is complete, without waiting for Space.

From a selected target, / starts a drag, and v switches to free mode there.

The arrow keys don’t scroll in hint mode, since a scroll would move the targets out from under their labels.

Try it here

Click Send by its label

keys·Focus the screen, then type a label.Type the label on Send.Press Space to click. Backspace undoes.Tap a label to pick its target.Press Space to click, or tap another.Pick a target first, by its label.Tap a label first, then press Space.Press Backspace first to pick another.Sent. Now try it on your own desktop.Missed Send. Each click starts over.No label is “m”.Focus the screen, then type a label.

Focus the screen and type. stochos finds targets in the accessibility tree or a screenshot. Here the window's five controls are marked by hand.

Finding targets

hint.detector says where the targets come from:

detector	what it does
`auto`	The default. Reads the accessibility tree, and adds what a screenshot shows in the active window that the tree missed. If the tree has fewer than five targets, it uses the screenshot alone.
`atspi`	Linux only. Reads the AT-SPI accessibility tree, nothing else.
`cv`	Looks for controls in a screenshot, on every system.

On macOS, auto reads the macOS accessibility tree of the app in front, and keeps a screenshot’s targets inside its window.

The accessibility tree

Toolkits publish their buttons and fields to the accessibility tree, with names, roles and exact positions, which makes it the better source. GTK, Qt and Firefox apps fill it in well. Chromium and Electron apps may need their accessibility support switched on, and games and apps that draw on a canvas publish little or nothing. stochos asks the session to turn accessibility on the first time it looks, and waits two seconds for an answer at most.

On Wayland, programs can’t see where their windows are. stochos asks the compositor instead, and labels only the active window. Hyprland and Sway report them. On X11, the tree’s positions are already on screen.

The screenshot

The cv detector traces edges in a screenshot, joins them into shapes, and keeps the shapes sized and proportioned like controls. There’s no model and no network. It sees outlines rather than meaning. Expect it to label the odd border or divider and miss a faint button. Three settings tune it:

key	default	what it does
`downscale_longest`	`1280`	Shrinks the screenshot to this many pixels on its longest side first. Lower is faster and coarser, and `0` keeps every pixel.
`edge_low_threshold`	`50`	The faintest edge that can continue a line.
`edge_high_threshold`	`150`	The faintest edge that can start one.

Raise the two thresholds for fewer labels on a busy screen, and lower them to catch faint controls.

The screenshot needs wlr-screencopy on Wayland, and Screen Recording permission on macOS. When hint mode can’t look, on a compositor without wlr-screencopy for instance, switching to it prints why and leaves you in the mode you were in. Started with --hint, stochos exits with the error.