v0.2.1: shorter card, storyboard, base-vs-fine-tuned table, latency profiling

.gitattributes

@@ -34,3 +34,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+figures/overview.png filter=lfs diff=lfs merge=lfs -text
+figures/replay_mixed.png filter=lfs diff=lfs merge=lfs -text
+figures/replay_success.png filter=lfs diff=lfs merge=lfs -text

NOTICE

@@ -23,7 +23,8 @@ This project builds on the following third-party work:
    Copyright 2024 The Google Research Authors.
    Licensed under the Apache License, Version 2.0.
    Li et al., "On the Effects of Data Scale on Computer Control Agents", arXiv:2406.03679.
-   The dataset is not redistributed in this repository.
+   The dataset is not redistributed in this repository, except for the recorded screenshots of two
+   validation episodes shown in figures/replay_*.png (unmodified apart from resizing and overlays).
 
 4. Accessibility proto field layout follows google-deepmind/android_env
    (Apache License, Version 2.0). No android_env code is included.

README.md

@@ -37,40 +37,21 @@ model-index:
 
 # Laya-Android
 
-A 322M non-generative typed-decision policy for Android GUI action selection, fine-tuned on AndroidControl using accessibility-tree input only.
+A 322M typed-decision policy for Android GUI control, fine-tuned from [Laya](https://huggingface.co/convaiinnovations/laya) on AndroidControl. Accessibility tree only · no vision · no autoregressive generation.
 
-[Code (GitHub)](https://github.com/Byun11/laya-android) · [Evaluation protocol](https://github.com/Byun11/laya-android/blob/main/docs/eval_protocol.md)
+[Code, docs and full results on GitHub](https://github.com/Byun11/laya-android)
 
-**322M · no vision · 76.7 Type / 61.7 Grounding on AndroidControl-High · ~38 ms per decision (RTX 4090)**
+## Highlights
 
-| | Laya-Android |
-|---|---:|
-| Parameters | **322M** |
-| Visual input | **None** (accessibility tree only) |
-| Training data | AndroidControl train split only |
-| AndroidControl-High Type | **76.7** |
-| AndroidControl-High Grounding | **61.7** |
-| Full Step SR | N/A (see below) |
-| Median latency | **38.3 ms** (RTX 4090, batch 1) |
+| Parameters | AndroidControl-High Type | AndroidControl-High Grounding | Latency (RTX 4090, bs1) |
+|:---:|:---:|:---:|:---:|
+| **322M** | **76.7** | **61.7** | **~40 ms** |
 
-## What does it do?
+![Laya-Android on a validation episode](figures/replay_success.png)
 
-At each step it reads the user goal, the last three actions and the actionable UI elements from the Android accessibility tree, and answers two questions in one forward pass: *which operation* and *which element*. It does not generate text and does not look at the screen image. It is a fine-tune of [Laya](https://huggingface.co/convaiinnovations/laya) by Convai Innovations; the architecture is Laya's, unchanged.
+Every step of a recorded validation episode with the model's predictions (blue = predicted, green = gold). The model never sees the screenshots; it reads the accessibility tree. Gold history, not a live run.
 
-A real validation step (fixed selection rule, not hand-picked):
-
-```text
-Goal     find a nike casual shoes for women on the kicks crew app
-History  INPUT_TEXT "casual shoes for women" → CLICK (WMNS) PUMA Oslo Maja … → CLICK Filter
-UI       [0] Dismiss (View)  [1] FILTER Clear All … scroll  [2] - (Button)  [3] Clear All (View)
-         [4] Brand (ImageView)  [5] Size And Type (ImageView)  [6] Release Year (ImageView)  … 16 total
-
-operation   CLICK 0.935   WAIT 0.039   SCROLL_DOWN 0.012
-target      [4] Brand 0.931   [7] Product Types 0.033   [5] Size And Type 0.008
-gold        CLICK [4] Brand  ✓
-```
-
-## Quick start
+## Usage
 
 ```python
 # pip install laya==0.3.28
@@ -98,25 +79,19 @@ out = agent.system_one(state, questions, max_len=3072, head_max_len=1536)
 print(out["answers"]["operation"]["choice"], out["answers"]["target"]["choice"])
 ```
 
-Candidate labels must be built as in the repository's `src/candidates.py` / `src/serialization.py` (`"<text> (<class>) <flags>"`); other formats are out of distribution.
-
-## Results
-
-One evaluation of the frozen checkpoint (epoch 2, chosen on validation). Type and Grounding were computed from the saved predictions after the protocol was committed.
-
-### AndroidControl-High (InfiGUI-R1 evaluator, 8444 steps / 1543 episodes)
+Candidate labels must follow the repository's `src/candidates.py` / `src/serialization.py` (`"<text> (<class>) <flags>"`).
 
-| Model | Params | Input | Type | Grounding | Full SR |
-|---|---:|---|---:|---:|---:|
-| InfiGUI-R1-3B† | 3B | Screenshot | 82.7 | 74.4 | 71.1 |
-| Laya base (zero-shot) | 322M | Accessibility tree | 29.0 | 11.8 | N/A |
-| **Laya-Android** | **322M** | **Accessibility tree** | **76.7** | **61.7** | **N/A** |
+## Evaluation
 
-† Reported by the authors ([InfiGUI-R1](https://github.com/InfiXAI/InfiGUI-R1), arXiv:2504.14239); not reproduced by us. Same evaluator and test set as ours.
+AndroidControl-High, 8,444 test steps, public InfiGUI-R1 evaluator. Same model before and after fine-tuning:
 
-Laya-Android selects actions from accessibility-derived UI candidates rather than predicting arbitrary screen coordinates; the gold boxes also come from the accessibility tree. 6.4% of test click/long-press targets (327 / 5,083) are outside our candidate set and count as misses. Full Step SR is not reported because v0 does not generate `INPUT_TEXT` text or `OPEN_APP` app names. AndroidControl-Low is not evaluated: v0 was trained with the high-level goal only.
+| Model | Params | Type | Grounding | Full SR |
+|---|---:|---:|---:|---:|
+| Laya base (zero-shot) | 322M | 29.0 | 11.8 | N/A |
+| **Laya-Android** | **322M** | **76.7** | **61.7** | **N/A** |
+| Δ | | +47.6 | +49.9 | |
 
-### Base Laya → Laya-Android (Policy Joint Accuracy, test)
+Full SR is N/A (v0 does not generate text or app names). Screenshot VLMs on the same evaluator report higher numbers (e.g. InfiGUI-R1-3B 82.7 / 74.4) but solve a different task (pixel coordinates vs. choosing accessibility candidates).
 
 | Split | Base Laya | Laya-Android | Δ |
 |---|---:|---:|---:|
@@ -125,85 +100,26 @@ Laya-Android selects actions from accessibility-derived UI candidates rather tha
 | Task-Unseen | 0.108 | **0.567** | +45.9pt |
 | Category-Unseen | 0.108 | **0.545** | +43.7pt |
 
-![Base vs Laya-Android across test splits](figures/androidcontrol_generalization.png)
-
-Policy Joint = operation correct and, for clicks, the chosen element is the gold element. It is an internal metric, not AndroidControl SR.
-
-### Target top-1 by candidate count (union of test steps)
-
-| Candidates | n | Base Laya | Laya-Android |
-|---|---:|---:|---:|
-| 1-5 | 547 | 0.397 | 0.839 |
-| 6-10 | 860 | 0.138 | 0.677 |
-| 11-20 | 2025 | 0.107 | 0.665 |
-| 21-50 | 1228 | 0.067 | 0.543 |
-| >50 | 96 | 0.021 | 0.542 |
-
-![Target top-1 by candidate count](figures/candidate_scaling.png)
-
-### Latency (NVIDIA GeForce RTX 4090)
-
-| Params | dtype | Batch | Median | p90 | p95 | Mean | End-to-end median | Peak VRAM (bs1) | Throughput (bs32) |
-|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|
-| 322M | bf16 autocast | 1 | 38.3 ms | 41.2 ms | 42.7 ms | 38.0 ms | 42.8 ms | 1.66 GB | 64 steps/s |
-
-### Training progression (validation Policy Joint)
-
-| Checkpoint | Val Policy Joint |
-|---|---:|
-| Base | 0.098 |
-| Smoke-15k | 0.457 |
-| Epoch 1 | 0.605 |
-| Epoch 2 (selected) | 0.669 |
-| Epoch 3 | 0.665 |
-
-![Training progression on validation](figures/training_progression.png)
-
-Per-split Type / Grounding, operation recall and failure examples: see the [GitHub README](https://github.com/Byun11/laya-android#failure-analysis).
+Policy Joint = operation correct and, for clicks, the gold element chosen (internal metric, not SR). One test pass of the frozen checkpoint; protocol fixed before scoring ([eval_protocol.md](https://github.com/Byun11/laya-android/blob/main/docs/eval_protocol.md)).
 
 ## Model details
 
-| | |
-|---|---|
-| Developed by | Jaeyeon Byun (GitHub: Byun11) |
-| Model type | Non-generative typed-decision model (bidirectional encoder + option-marker decision head) |
-| Base model | `convaiinnovations/laya`, subfolder `multilingual`, revision `7b928d828b7b0e022f929d9bd2e44165aa270148` (mmBERT-base encoder) |
-| Parameters | ~322M |
-| Sequence budget | `max_len` 3072, `head_max_len` 1536 |
-| License | Apache-2.0 |
-
-## Intended use
-
-- Research on lightweight Android GUI action-selection policies
-- Accessibility-based mobile agents, e.g. a fast step selector under a planner
-- Fast System-1 style decision models
-
-## Out-of-scope use
-
-- Operating real devices or accounts unsupervised, especially payments, messaging, account or security settings, or other irreversible actions
-- Tasks that need visual understanding (images, canvases, games, custom-drawn UI)
-- Generating text: `INPUT_TEXT` content and `OPEN_APP` names are not produced
-- Claims of end-to-end task success: v0 is evaluated offline, step by step
+- Base: `convaiinnovations/laya` / `multilingual` @ `7b928d8` (mmBERT-base encoder), ~322M parameters, Apache-2.0
+- Input: goal, last 3 actions, accessibility-derived UI candidates; `max_len` 3072, `head_max_len` 1536
+- Output: probabilities over 12 operations and over the candidates
+- Developed by Jaeyeon Byun (GitHub: Byun11); not affiliated with Convai Innovations
 
 ## Training
 
-AndroidControl only, official splits: 13,594 train episodes / 74,714 actions (9 zero-action episodes excluded), plus a synthetic `DONE` per episode → 88,308 steps, 132,448 training items. Supervised fine-tuning (cross-entropy over options) with `laya.train`, bf16, effective batch 32, encoder lr 2.5e-5, head lr 1e-4, option order shuffled, 3 epochs on one RTX 4090 (2.7 h). No RL, no oversampling, no class weights, no candidate truncation. Checkpoint chosen by validation Policy Joint; temperatures fit on validation.
+AndroidControl train split only (13,594 episodes, 74,714 actions). Supervised fine-tuning with `laya.train`, bf16, 3 epochs on one RTX 4090; checkpoint and temperatures chosen on validation. No RL, no oversampling, no screenshots. Details: [training.md](https://github.com/Byun11/laya-android/blob/main/docs/training.md).
 
 ## Limitations
 
-- **No visual input.** Unlabeled icons, canvases and custom-drawn UI are hard or impossible to distinguish.
-- **Candidate coverage.** 92.3–95.2% of test click targets are covered by the candidate extractor; the rest are unrecoverable.
-- **Generalization gap.** Policy Joint drops 9–12 points from IDD to unseen apps, tasks and categories.
-- **No text or app-name generation**, hence no Full Step SR.
-- **Weak rare actions.** WAIT, BACK, horizontal scroll and LONG_PRESS have low recall.
-- **Offline benchmark only.** Step-by-step with gold history; not a measured task success rate.
-
-## Ethical and safety considerations
-
-- A GUI action model can trigger real side effects. Keep a human in the loop and gate sensitive or irreversible actions.
-- Accessibility trees can contain personal data shown on screen. Process them locally and avoid logging them.
-- Calibrated confidence can support abstention, but it does not guarantee correctness.
-- Training data covers a limited set of apps and locales.
+- No visual input: unlabeled icons and custom-drawn UI are hard to tell apart.
+- No text or app-name generation, so no Full Step SR; AndroidControl-Low not evaluated.
+- Policy Joint drops 9–12 points on unseen apps, tasks and categories.
+- Weak on rare actions (WAIT, BACK, horizontal scroll, LONG_PRESS).
+- Offline, step-level benchmark with gold history. Do not let it operate real devices unsupervised; gate irreversible actions and keep on-screen personal data local.
 
 ## Citation
 
@@ -212,16 +128,9 @@ AndroidControl only, official splits: 13,594 train episodes / 74,714 actions (9
   title   = {Laya-Android},
   author  = {Byun, Jaeyeon},
   year    = {2026},
-  version = {0.1.0},
+  version = {0.2.1},
   url     = {https://github.com/Byun11/laya-android}
 }
 ```
 
-Please also cite AndroidControl (Li et al., 2024, arXiv:2406.03679) and Laya.
-
-## Acknowledgements
-
-- Built on top of [Laya](https://github.com/NandhaKishorM/laya) by Convai Innovations (Apache-2.0). Not affiliated with or endorsed by Convai Innovations.
-- Encoder: [mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base) (MIT).
-- Data: [AndroidControl](https://github.com/google-research/google-research/tree/master/android_control) by Google Research (Apache-2.0).
-- Evaluator: [InfiGUI-R1](https://github.com/InfiXAI/InfiGUI-R1) (Apache-2.0).
+Built on [Laya](https://github.com/NandhaKishorM/laya) (Convai Innovations), [mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base) and [AndroidControl](https://github.com/google-research/google-research/tree/master/android_control) (Google Research). Evaluator: [InfiGUI-R1](https://github.com/InfiXAI/InfiGUI-R1).

eval/rtx4090.json

@@ -6,124 +6,162 @@
  "warmup": 100,
  "measured": 1000,
  "inputs": "test_idd, seed 0 (labels unused)",
+ "software": {
+  "laya": "0.3.28",
+  "transformers": "4.57.6",
+  "cuda": "13.0"
+ },
  "questions_per_step": "operation + target (all candidates), one forward",
  "max_len": 3072,
  "head_max_len": 1536,
  "models": {
-  "base_laya": {
+  "laya_android": {
    "forward_ms": {
     "n": 1000,
-    "median": 38.6,
-    "p90": 42.75,
-    "p95": 44.67,
-    "mean": 38.51
+    "median": 39.74,
+    "p90": 41.95,
+    "p95": 43.51,
+    "mean": 40.26
    },
    "e2e_ms": {
     "n": 1000,
-    "median": 42.91,
-    "p90": 48.49,
-    "p95": 51.52,
-    "mean": 43.22
+    "median": 44.46,
+    "p90": 48.63,
+    "p95": 51.13,
+    "mean": 45.34
    },
    "forward_ms_by_candidates": {
     "1-5": {
      "n": 102,
-     "median": 38.71,
-     "p90": 41.06,
-     "p95": 43.27,
-     "mean": 37.92
+     "median": 39.66,
+     "p90": 41.49,
+     "p95": 42.07,
+     "mean": 39.85
     },
     "6-10": {
      "n": 171,
-     "median": 38.38,
-     "p90": 41.95,
-     "p95": 43.82,
-     "mean": 38.22
+     "median": 39.49,
+     "p90": 41.12,
+     "p95": 42.08,
+     "mean": 39.72
     },
     "11-20": {
      "n": 406,
-     "median": 38.27,
-     "p90": 41.51,
-     "p95": 42.75,
-     "mean": 37.99
+     "median": 39.62,
+     "p90": 40.93,
+     "p95": 41.55,
+     "mean": 39.71
     },
     "21-50": {
      "n": 295,
-     "median": 38.41,
-     "p90": 43.65,
-     "p95": 44.96,
-     "mean": 38.62
+     "median": 40.27,
+     "p90": 42.85,
+     "p95": 43.91,
+     "mean": 40.47
     },
     ">50": {
      "n": 21,
-     "median": 46.85,
-     "p90": 65.68,
-     "p95": 68.65,
-     "mean": 51.96
+     "median": 48.97,
+     "p90": 69.71,
+     "p95": 78.59,
+     "mean": 54.25
     }
    },
-   "throughput_steps_per_s_batch32": 64.2,
-   "peak_vram_gb_bs1": 1.66,
-   "peak_vram_gb_batch32": 8.34,
-   "params_m": 321.9
-  },
-  "laya_android": {
-   "forward_ms": {
-    "n": 1000,
-    "median": 38.27,
-    "p90": 41.2,
-    "p95": 42.66,
-    "mean": 37.95
-   },
-   "e2e_ms": {
-    "n": 1000,
-    "median": 42.82,
-    "p90": 47.36,
-    "p95": 49.52,
-    "mean": 42.68
+   "forward_ms_by_input_tokens": {
+    "<=512": {
+     "n": 649,
+     "median": 39.6,
+     "p90": 41.14,
+     "p95": 41.81,
+     "mean": 39.78
+    },
+    "513-1024": {
+     "n": 292,
+     "median": 39.85,
+     "p90": 41.68,
+     "p95": 42.49,
+     "mean": 39.92
+    },
+    "1025-1536": {
+     "n": 49,
+     "median": 42.88,
+     "p90": 46.74,
+     "p95": 48.44,
+     "mean": 43.75
+    },
+    ">1536": {
+     "n": 10,
+     "median": 53.1,
+     "p90": 83.47,
+     "p95": 105.45,
+     "mean": 64.05
+    }
    },
-   "forward_ms_by_candidates": {
-    "1-5": {
-     "n": 102,
-     "median": 38.1,
-     "p90": 40.7,
-     "p95": 41.78,
-     "mean": 37.72
+   "peak_vram_allocated_gb_bs1": 1.66,
+   "peak_vram_reserved_gb_bs1": 1.91,
+   "batch_sweep": {
+    "1": {
+     "batch_ms": {
+      "n": 1000,
+      "median": 39.06,
+      "p90": 41.12,
+      "p95": 42.26,
+      "mean": 39.48
+     },
+     "throughput_steps_per_s": 25.3,
+     "peak_vram_allocated_gb": 1.66,
+     "peak_vram_reserved_gb": 1.91
     },
-    "6-10": {
-     "n": 171,
-     "median": 38.32,
-     "p90": 40.17,
-     "p95": 41.66,
-     "mean": 37.76
+    "4": {
+     "batch_ms": {
+      "n": 250,
+      "median": 39.99,
+      "p90": 51.89,
+      "p95": 67.4,
+      "mean": 44.25
+     },
+     "throughput_steps_per_s": 90.4,
+     "peak_vram_allocated_gb": 2.3,
+     "peak_vram_reserved_gb": 3.45
     },
-    "11-20": {
-     "n": 406,
-     "median": 38.18,
-     "p90": 40.48,
-     "p95": 41.79,
-     "mean": 37.56
+    "8": {
+     "batch_ms": {
+      "n": 125,
+      "median": 71.35,
+      "p90": 134.79,
+      "p95": 158.07,
+      "mean": 81.02
+     },
+     "throughput_steps_per_s": 98.7,
+     "peak_vram_allocated_gb": 3.16,
+     "peak_vram_reserved_gb": 6.3
     },
-    "21-50": {
-     "n": 295,
-     "median": 38.33,
-     "p90": 41.43,
-     "p95": 42.37,
-     "mean": 37.96
+    "16": {
+     "batch_ms": {
+      "n": 63,
+      "median": 177.91,
+      "p90": 293.32,
+      "p95": 371.29,
+      "mean": 198.02
+     },
+     "throughput_steps_per_s": 80.2,
+     "peak_vram_allocated_gb": 4.89,
+     "peak_vram_reserved_gb": 11.99
     },
-    ">50": {
-     "n": 21,
-     "median": 44.33,
-     "p90": 63.6,
-     "p95": 66.23,
-     "mean": 48.22
+    "32": {
+     "batch_ms": {
+      "n": 32,
+      "median": 431.11,
+      "p90": 778.52,
+      "p95": 827.76,
+      "mean": 535.89
+     },
+     "throughput_steps_per_s": 58.3,
+     "peak_vram_allocated_gb": 8.34,
+     "peak_vram_reserved_gb": 23.35
     }
    },
-   "throughput_steps_per_s_batch32": 63.6,
-   "peak_vram_gb_bs1": 1.66,
-   "peak_vram_gb_batch32": 8.34,
    "params_m": 321.9
   }
- },
- "model": "ByunByun/laya-android"
+ }
 }