Tiiny Bench Updates & Release Notes

TiinyBench 0.1.11

key() gains one more fallback, after every existing source (env var, farm config, saved config, TiinyOS scrape) comes up empty: it asks for the box's password and calls the device's own account API to get a working key directly, no TiinyOS desktop app needed. Only runs at an interactive terminal, never in a cron job.

Also carries the fixes since 0.1.10:

• Two numbers on a public page that could not be true
• Results survive an update now
• A rate divided by two tokens is not a rate
• A 404 that meant four different things
• The alternation was two backends, not a fussy validator
• A predicate that could raise instead of answering

Everything in this release came out of one thing: a sweep of all 52 installed models that came back with eleven measurements empty. Reading what the device actually said in those failures turned four of them into real numbers and four more into a diagnosis.

Numbers that did not exist before

• Music generation is measured. Foundation-1 runs at 0.29x real time, about seventy seconds of wall clock for twenty seconds of audio. It also ignores the duration you ask for and returns twenty seconds regardless, which the record now shows because it keeps both the ask and the result.
• Speech for Supertone-3, 2.10x real time.
• Both 0.6B embedding models, around 22 vectors a second at 1024 wide.

Why they were missing

• Music came back as a JSON envelope with the audio base64 inside it. The benchmark only understood a raw file or a job to poll, so it recorded a failure for a model that had just produced audio.
• Two music models disagree about their own request shape: one requires a duration, the other refuses it. One refuses the prompt and then says a prompt is required.
• Supertone's refusal listed the ten speakers it does have. Nobody read it.
• Two embedding models answered 502 because the device lists a model as running before its runtime accepts connections.

What changed

Refusals are now read rather than discarded, everywhere. The benchmark parses the field a 400 objects to, the speakers a 500 enumerates, and the …

Everything in 0.1.8, plus the one thing a sweep of unmeasured model classes needs. 0.1.8 was tagged and released before this landed and never reached the farm catalog: install this one.

A missing model, and a reply this could not read

Both looked like a null in a result file, and the second costs a model load to reproduce once a sweep has moved on. Four of the nine model classes are being measured against real hardware for the first time, having only ever run against a fake built from the device's own documentation, so a response shape guessed wrong is the likely failure and a silent null is the worst way to find out about it.

A failed call now keeps the device's own words. The transport used to flatten an HTTP error to its status line, which threw away the only thing that separates the two cases: the gateway answers 503 with an envelope that says "No suitable model is currently running", and nothing else on the device uses that envelope.

So a test whose class of model is not resident returns a reason rather than nothing, and both the report and the Markdown export print "not measured because the model was not resident" instead of leaving a blank that reads as a failure. It is not a failed measurement: the box holds one accelerator and a hundred NPU units, and a class that was not loaded when the sweep reached it has nothing to report.

A reply that did come back and could not be used is captured instead: the …

Four more kinds of model can be measured, the report is rebuilt, and every run now records what makes it comparable to somebody else's box.

Every run carries its provenance

A run taken without this is a run that cannot be compared tomorrow. Each file now has a provenance block with a schema number on it.

Per run: the host that drove the benchmark (os, release, version, arch, python, logical CPUs, physical RAM, machine model), the transport (plane, gateway port, vhost if used, and the round trip measured before any work starts), the device (name, model, serial, TiinyOS, service version, NPU unit budget), and the tool (bench version, suite version, envelope schema, git commit). A benchmark driven from a Mac over a cable and one driven from a Windows box over Wi-Fi are not the same measurement, and until now nothing in the file said which.

Per model rather than per run, because a sweep loads and unloads as it goes and the conditions the fifth model met are not the ones the first met: the resident set with each model's NPU units at the moment that model's tests begin, units used and free, whether another process held the device lock, the sampling parameters actually sent, which tests ran and which were skipped, and the served context window. The lock is read and never taken, because a benchmark that fought for it would change the thing it measures.

Temperature and power are recorded as null with a reason rather than omitted. …

Four more kinds of model can be measured, and the run page will offer them.

Four new test types

Twenty-nine models are installed across nine classes and the suite had tests for five of them, so nine models wore a type with nothing behind it.

Class	Test	The figure
ASR	asr	seconds of audio transcribed per second of wall clock
Image-to-Text	ocr	seconds per page, and whether the digits came back
Music Generation	music	seconds of audio produced per second of wall clock
Text Reranking	rerank	query-document pairs scored per second

The audio clip and the page of digits are generated in code rather than committed: a fixture you cannot diff is one you cannot trust when a number moves. Both are byte for byte identical every run, and the tests decode them back and read the digits off the pixels.

OCR takes whichever of two paths answers and records which, because a box whose OCR model is a vision language model has no gateway route and has to be read through chat completions, and those two numbers are not comparable. Music handles both a blocking call and a polled job for the same reason.

Transcription accuracy is deliberately not scored: it needs a clip of known speech, which cannot be generated without a text-to-speech model loaded, and a benchmark that depends on a second model is measuring two things. Reranking does get a free sense check, because one of its four passages actually answers the query. …

TiinyBench 0.1.6

Problem. 0.1.5 folded the sidebar to icons, but the main column was capped at 1500 pixels and left-aligned, so on a wide window the space the fold gave back sat empty on the right and the chat stayed the same width. Jason, looking at the folded chat: "the chat section stayed exactly the same width left to right."

Fix. The cap is gone. The main column now fills whatever the sidebar leaves, folded or wide, on every page. Prose keeps its own readable measure (the page blurbs and the empty-chat note are capped in characters); tables and the chat grow with the window.

Tests. The 100 offline tests pass unchanged; the chat page rendered at 2000 wide with the sidebar open runs to the right edge (grid: 236 px sidebar, main the rest).

TiinyBench 0.1.5

Problem. The Chat page is the first page in TiinyBench that wants the whole width of the window: a model card, the conversation and the loaded models side by side. The sidebar took 236 pixels of it for eleven labels the eye already knows by their glyphs. Jason, looking at a chat: "if I could collapse the left-hand nav to just icons, would I have a little bit more space?"

Fix. The sidebar folds to icons. A button in the brand row folds it to 64 pixels and back; the choice is remembered per browser and applied before the first paint, so a folded sidebar never flashes wide on load. Folded, each button carries its label as a tooltip, section headings become hairlines, and the footer keeps only the Tiiny mark. On a phone the sidebar is already a strip and the button is not offered. Nothing about the pages, the routes or the chat changes.

Tests. The 100 offline tests from 0.1.4 pass unchanged (the page's script order, the em dash rule, the chat and the catalogue); both states rendered at 1440 wide: 236 px and 64 px, the document exactly the viewport in each.

TiinyBench 0.1.4

Problem. TiinyBench could tell you what a model costs and never let you talk to one. Its sibling on the same hardware, AINode Pocket, has a chat tab that reports every reply's cost under the reply and the device beside the conversation, and asked for it here. The two apps also derived the same numbers from the same two gateway blocks in two places, which is the arrangement that drifts the first time the gateway renames a field and then has a chat bar and a saved benchmark disagreeing about the same request.

Fix. A Chat page, Pocket's three columns adapted to one Tiiny. Under every reply: time to first token, decode tok/s, total wall time, tokens in with the cache hits, tokens out, which device and model answered, and why generation stopped, all read off the gateway's own timings and usage on the stream's final chunk plus this machine's clock; the numbers are saved with the message. Left: a model card for whatever is selected (type, params, size, NPU units, input and output, where it is loaded, the vendor's description, a catalogue link) and the last twenty conversations, kept in the browser. Right: every model loaded on the box with its units and status and an Unload button, the NPU budget bar, and a Load panel that refuses in a sentence when a model is not installed, cannot chat, or would not fit, because the device accepts a load that does not fit and rolls it back without saying so. Replies render as markdown with highlighted code blocks a…

Running 0.1.2 from the launcher at about 2000px wide, the Connection page showed REACHED AT as 172.17.7.1 on a box that is at 172.17.7.177, and HOW as given, port 80 as p8800.api. with the rest gone. The tiles were drawn in the face the dashboard numbers use, at one size whether the value was three characters or twenty.

• The connection tile values now get a face sized for a short line of text, they wrap instead of running off the card, and hovering one shows the whole value.
• The tiles take the width their own value needs, so 1 TB and given, port 80 as p8800.api.tiiny are no longer made the same size.
• The discovery file on :39218 is asked once per address and fills in whatever the management API could not answer, so a box handed over by the launcher stops reading "found, unnamed" over a column of dashes.
• The tiles say what found the box, "the launcher" or "the farm", rather than naming this suite's own environment variables.

The browser job in CI now loads the page at 2000 as well as 1440 and 390, and fails if any tile value is wider than the box holding it.

Two testers reported the same thing in different words, and it was one cause: the page never ran.

The confirm dialog added in 0.1.1 sat above the script's own scope and called $ on its first line, so the whole <script> stopped on a ReferenceError before it asked for anything. Nothing below it executed. No navigation, no request for the device, no find button. The app came up looking finished with every control dead, on Windows and on a Mac alike.

Fixed in this release

• The page runs. Connection opens, the find button finds, the dashboard fills in.
• Discovery works on Windows. It used to shell out to ip and then ifconfig, and Windows has neither, so the candidate list was empty and a Tiiny on the same network could not be found. It is stdlib now and the same on every platform.
• The scan asks the responder the farm CLI asks, a GADGET_DISCOVER_V1 datagram on port 39217, which is the only probe that finds a box whose address moved.
• The Connection panel says what it is doing. It paints "Looking for your Tiiny" before the first request leaves, says so plainly when nothing answers, and offers the address box and the key box either way.
• The dashboard no longer dies on a fresh install. Asking for the leaderboard without a key was dropping the connection.
• tiiny-bench --where prints what it looked for and exits cleanly when there is no Tiiny attached.
• At 390 pixels wide the navigation used to push every panel off the right edge behind a sideways scroll. …

Fixes TiinyBench on Tiiny 1.0 firmware (TiinyOS 0.1.34, service 0.1.30), which moved the AI gateway off port 8800 and onto the port 80 vhost router, and whose DHCP lease moves a box's address.

• Transport negotiation. The service's own port is tried first; only ECONNREFUSED moves that service onto http://<addr>:80 with the Host header for it (p8800.api.tiiny, openai.api.tiiny). A timeout, DNS failure or reset does not. Old firmware keeps working on the direct port.
• A resolver. TIINY_BASE (TIINY_HOST still honoured), then ~/.tiinyapps/device.json, then --host, then the saved config, then a scan of every USB /30 peer and this host's own /24 on :39218. The scan identifies a box by serial, dedupes a box that answers on both Wi-Fi and USB, prefers the USB address, and refuses to guess when two serials answer.
• New flags: --host, --serial, --rescan, --where, --version.
• Fixed: the device's nginx 301 on /api/v1/models sent urllib off the box to a name that answers 502. Redirects are now kept on the address we are talking to.
• Fixed: the TiinyOS key scrape read only the .ldb half of the app's LevelDB and could pick up a stale key. It now reads the .log half too and ranks candidates by frequency.
• Result files record the address, plane, transport and both firmware versions.

Every measurement and the report format are unchanged. Standard library only. The suite is read-only on the device except for inference. …