diff --git a/.gitattributes b/.gitattributes index a6344aac8c09253b3b630fb776ae94478aa0275b..c3113eb055ab551382f5bcf57347161a8c4bf16f 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1,35 +1,12 @@ -*.7z filter=lfs diff=lfs merge=lfs -text -*.arrow filter=lfs diff=lfs merge=lfs -text -*.bin filter=lfs diff=lfs merge=lfs -text -*.bz2 filter=lfs diff=lfs merge=lfs -text -*.ckpt filter=lfs diff=lfs merge=lfs -text -*.ftz filter=lfs diff=lfs merge=lfs -text -*.gz filter=lfs diff=lfs merge=lfs -text -*.h5 filter=lfs diff=lfs merge=lfs -text -*.joblib filter=lfs diff=lfs merge=lfs -text -*.lfs.* filter=lfs diff=lfs merge=lfs -text -*.mlmodel filter=lfs diff=lfs merge=lfs -text -*.model filter=lfs diff=lfs merge=lfs -text -*.msgpack filter=lfs diff=lfs merge=lfs -text -*.npy filter=lfs diff=lfs merge=lfs -text -*.npz filter=lfs diff=lfs merge=lfs -text -*.onnx filter=lfs diff=lfs merge=lfs -text -*.ot filter=lfs diff=lfs merge=lfs -text -*.parquet filter=lfs diff=lfs merge=lfs -text -*.pb filter=lfs diff=lfs merge=lfs -text -*.pickle filter=lfs diff=lfs merge=lfs -text -*.pkl filter=lfs diff=lfs merge=lfs -text -*.pt filter=lfs diff=lfs merge=lfs -text -*.pth filter=lfs diff=lfs merge=lfs -text -*.rar filter=lfs diff=lfs merge=lfs -text *.safetensors filter=lfs diff=lfs merge=lfs -text -saved_model/**/* filter=lfs diff=lfs merge=lfs -text -*.tar.* filter=lfs diff=lfs merge=lfs -text -*.tar filter=lfs diff=lfs merge=lfs -text -*.tflite filter=lfs diff=lfs merge=lfs -text -*.tgz filter=lfs diff=lfs merge=lfs -text -*.wasm filter=lfs diff=lfs merge=lfs -text -*.xz filter=lfs diff=lfs merge=lfs -text -*.zip filter=lfs diff=lfs merge=lfs -text -*.zst filter=lfs diff=lfs merge=lfs -text -*tfevents* filter=lfs diff=lfs merge=lfs -text +native/tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text +assets/architecture-atlas.pdf filter=lfs diff=lfs merge=lfs -text +assets/architecture.pdf filter=lfs diff=lfs merge=lfs -text +assets/architecture.png filter=lfs diff=lfs merge=lfs -text +assets/attention-geglu.pdf filter=lfs diff=lfs merge=lfs -text +assets/attention-geglu.png filter=lfs diff=lfs merge=lfs -text +assets/candidate-readout.pdf filter=lfs diff=lfs merge=lfs -text +assets/candidate-readout.png filter=lfs diff=lfs merge=lfs -text +assets/decision-lex-header.png filter=lfs diff=lfs merge=lfs -text +assets/residual-layers.pdf filter=lfs diff=lfs merge=lfs -text +assets/residual-layers.png filter=lfs diff=lfs merge=lfs -text diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000000000000000000000000000000000000..767f11a05f4879c5b18be6d09c89f64c26834df8 --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,15 @@ +# Lex architecture + +Lex fine-tunes Kai's 571,909,635-parameter encoder architecture. Three independently parameterized 22-layer paths serve Choice, Noul and Score. Each path has its own final normalization, two interaction layers and a candidate scorer; token embeddings and embedding normalization are shared. + +![Lex architecture](assets/architecture.png) + +A question and all of its candidate descriptions are encoded jointly with the context. Marker states supply scalar candidate scores, normalized over the valid candidates. A shared type embedding is added after each path's final normalization. Each needed encoder path processes the full padded batch; interaction heads gather rows of that type. + +![Candidate readout](assets/candidate-readout.png) + +Choice returns a candidate and probabilities; Noul returns the probability of yes; Score returns the ordered distribution and its expected value. Candidate IDs preserve associations outside the token stream. Multiple questions use batches, with repeated context encoding. + +The public API accepts complete packed inputs of at most 1,024 tokens per question. No field is silently truncated. + +[Main SVG](assets/architecture.svg) · [Residual layers](assets/residual-layers.svg) · [Attention and GEGLU](assets/attention-geglu.svg) · [PDF atlas](assets/architecture-atlas.pdf) diff --git a/DISTRIBUTION_TERMS.md b/DISTRIBUTION_TERMS.md new file mode 100644 index 0000000000000000000000000000000000000000..0542a5f5964d15614008b5c52671f3eeb6b79158 --- /dev/null +++ b/DISTRIBUTION_TERMS.md @@ -0,0 +1,7 @@ +# Decision distribution terms + +Decision's new code and model-weight contributions are provided under the included [MIT license](LICENSE). Retained third-party material remains subject to its original license and notices; the MIT grant does not replace those conditions. + +The inherited tokenizer is distributed with the [Gemma Terms of Use](LICENSES/gemma/GEMMA_TERMS.html), including the incorporated [Gemma Prohibited Use Policy](LICENSES/gemma/GEMMA_PROHIBITED_USE_POLICY.html). By accessing, using or distributing the covered tokenizer material, you agree to comply with those terms and restrictions. Those documents are incorporated into this distribution agreement for that material. Redistributors must pass on the applicable agreement, restrictions and required notices, and identify their modifications as required by the upstream terms. + +This provision concerns the inherited tokenizer material. It does not describe Decision's encoder weights as Gemma weights or replace the separate MIT and Apache-2.0 grants for other components. See [license scope](LICENSING_STATUS.md) and [NOTICE](NOTICE) for attribution and the component mapping. diff --git a/EVALUATION.md b/EVALUATION.md new file mode 100644 index 0000000000000000000000000000000000000000..4d6359996f66718e17f978a98182c54f86887693 --- /dev/null +++ b/EVALUATION.md @@ -0,0 +1,30 @@ +# Official typed-decisions TEST + +The fixed final Lex checkpoint and the official `laya-typed-decisions` specialist were each evaluated once on the original 2,000 TEST decisions, from 400 state components. Both models support all 2,000 complete inputs within 1,024 tokens, their native windows and their untruncated official-default support. No TRAIN state component or full input overlaps this TEST support. The test contains four English workflows and 20 question schemas. This is not evidence of multilingual or unseen-schema generalization. + +| Metric | Lex | Official specialist (shipped temperature) | +|---|---:|---:| +| Original-label hard accuracy | **78.15%** (1,563/2,000) | 76.60% (1,532/2,000) | +| Soft NLL | 0.961902 | **0.884390** | +| Mean soft Brier | 0.030858 | **0.018452** | +| Score RPS | 0.022994 | **0.015602** | +| Score expected-value MAE | 0.251408 | **0.242408** | + +Lex's accuracy gain is 1.55 percentage points (31 decisions). The paired 95% interval is **[-0.20, +3.15125] percentage points**, which includes zero. The interval uses 2,000 state-component bootstrap samples, seed 20260921, stratified by each component's workflow set; complete components are retained. This supports a higher observed hard-accuracy point, not statistically significant or uniformly better performance. Probability metrics are worse for Lex in this comparison. No calibration was fitted. + +| Type | Decisions | Lex accuracy | Official accuracy | +|---|---:|---:|---:| +| Choice | 600 | 74.00% | 73.33% | +| Noul | 600 | 84.67% | 85.67% | +| Score | 800 | 76.375% | 72.25% | + +| Workflow | Decisions | Lex accuracy | Official accuracy | +|---|---:|---:|---:| +| Agent trace observability | 500 | 72.4% | 73.0% | +| Customer service | 500 | 79.6% | 76.4% | +| Invoice processing | 500 | 84.4% | 80.4% | +| Security incidents | 500 | 76.2% | 76.6% | + +Hard accuracy uses the source's original semantic hard label. Choice and Score use the first maximum in supplied order; Noul uses `p_yes >= 0.5`. Soft NLL uses the explicitly normalized original soft label distribution and natural logarithms (probability floor 1e-12). Brier averages squared probability error over the valid candidates; ordinal Score RPS averages cumulative-distribution squared error over K−1 boundaries. Score MAE compares the predicted expected value against the original source score. The official raw-temperature accuracy is also 76.60%; raw soft NLL is 0.869598. + +The frozen TEST parquet SHA256 is `4f294f218ea1da27f3efef936359389c62ea4d3973a41457732990f1d31b647c`. The actual saved-summary SHA256 is `1a21955b24be16cefa2963779e71edf4d6b5adcd14679fc438e6a87e3e604b88`. No TEST text, labels or predictions are redistributed. The annotations are synthetic/teacher-derived; accuracy does not certify operational safety or calibrated business utility. diff --git a/FINETUNING.md b/FINETUNING.md new file mode 100644 index 0000000000000000000000000000000000000000..04d9f974f8ff1c310b403caf35975556b73cefdd --- /dev/null +++ b/FINETUNING.md @@ -0,0 +1,30 @@ +# Fine-tune on your data + +The included CLI trains the existing Choice, Noul and Score paths without adding new parameters. It supports hard or soft labels, deterministic scheduling, full-input admission, DEV checkpoint selection and optimizer/RNG resume. Token embedding, embedding normalization and type embedding remain frozen. The other 486 parameter tensors are trainable when their type is present. + +Provide your own TRAIN and DEV JSONL. The CLI rejects shared IDs, shared normalized full inputs and same-source components where supplied. It does not prove semantic independence; use an appropriate development split and do not train on a release/test set. + +| Type | Target | +|---|---| +| Choice | `{"choice_id":"candidate-id"}` or `{"probabilities":[0.2,0.8]}` | +| Noul | `{"probability":0.8}`; hard labels are 0 or 1 | +| Score | `{"probabilities":[0.1,0.2,0.7]}` in increasing-value level order; hard labels are one-hot | + +Every row retains `id`, full `state_text` and a complete typed `question`. Probability vectors sum to one. A scalar Score is not silently converted into a distribution. + +```bash +TRAIN=/data/train.jsonl DEV=/data/dev.jsonl OUTPUT=/runs/my-decision \ +ROCR_VISIBLE_DEVICES=0 bash examples/finetune.sh +``` + +`examples/finetune.sh` is the editable example configuration; it uses actual supported flags, not a nonexistent `--config` option. Its four-epoch recipe uses logical batch 64 / microbatch 8, BF16 autocast with FP32 parameters/loss, AdamW, encoder LR 2.5e-5 / head LR 1e-4, clip 1, CE plus 0.1 ordinal RPS for Score, and a cosine floor of 1e-6 after 10% warmup. Logical tails use their actual row count. These are example defaults, not a guarantee of improvement. + +The default selector minimizes equal-type macro soft NLL at step 0 and epoch ends. The separate `--selection hard-accuracy` policy requires an explicit `hard_target_id` on every DEV row, uses row hard accuracy then soft NLL then the earlier step, and never derives hard gold from soft targets. Noul hard selection uses `p_yes >= .5`. Record the selector and cadence used in any model comparison; these defaults are starting points rather than a guarantee of task quality. + +Successful runs export the selected native bundle and report its manifest in `COMPLETE.json`. Selecting step 0 means no adopted adaptation. Immutable checkpoints include model, AdamW and RNG state; allow substantial disk space. For an interrupted run, use exactly the same original arguments and output plus `--resume --resume-sha256 `. Read `LATEST.json` for the last completely saved boundary. `--stop-after-step N` is an optional clean pause; it does not restart or shorten the LR horizon. + +The included source underwent a bounded AMD continuous-versus-resume check and independent native reload. That interface evidence is separate from dataset quality or long training stability. Validate the quality of your selected model on an appropriately isolated evaluation set; no automatic release or adoption is performed. + +## Lex checkpoint provenance + +The editable example above is a downstream fine-tuning recipe, not the exact Lex release recipe. Lex was refit from Kai on all 6,000 original typed-decisions TRAIN decisions using eight epochs, fixed final step 750 and blended original hard/soft cross-entropy. It used no final-refit DEV selector, no warmup and no RPS term. See [METHODS.md](METHODS.md) and [TRAINING_PROVENANCE.json](TRAINING_PROVENANCE.json). The public CLI does not provide a named `blend50` switch; explicitly form normalized blended targets if using that objective. Reusing Lex as an initializer is a new experiment. diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000000000000000000000000000000000000..f6ee7616ed458eda488479d393542d25365053f1 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Decision Nano contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/LICENSES/Transformers-Apache-2.0.txt b/LICENSES/Transformers-Apache-2.0.txt new file mode 100644 index 0000000000000000000000000000000000000000..68b7d66c97d66c58de883ed0c451af2b3183e6f3 --- /dev/null +++ b/LICENSES/Transformers-Apache-2.0.txt @@ -0,0 +1,203 @@ +Copyright 2018- The Hugging Face team. All rights reserved. + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/LICENSES/gemma/GEMMA_PROHIBITED_USE_POLICY.html b/LICENSES/gemma/GEMMA_PROHIBITED_USE_POLICY.html new file mode 100644 index 0000000000000000000000000000000000000000..9f6448af09d469cfc3e3a983f39e2b57c72add4f --- /dev/null +++ b/LICENSES/gemma/GEMMA_PROHIBITED_USE_POLICY.html @@ -0,0 +1,84 @@ + +Gemma Prohibited Use Policy

Gemma Prohibited Use Policy

Official Google legal text, retrieved 2026-09-21T11:09:44.489495+00:00. Source: https://ai.google.dev/gemma/prohibited_use_policy. Article markup preserved; website navigation omitted. This document is not the MIT license.

+

+

+

Google reserves the right to update this Gemma Prohibited Use Policy from time +to time.

+

Last modified: February 21, 2024

+

You may not use nor allow others to use Gemma or Model Derivatives to:

+
    +
  1. Generate any content, including the outputs or results generated by Gemma or +Model Derivatives, that infringes, misappropriates, or otherwise violates any +individual's or entity's rights (including, but not limited to rights in +copyrighted content).
    2. +
  2. Perform or facilitate dangerous, illegal, or malicious activities, including: +
      +
    1. Facilitation or promotion of illegal activities or violations of law, +such as: +
        +
      1. Promoting or generating content related to child sexual abuse or +exploitation;
        2. +
      2. Promoting or facilitating sale of, or providing instructions for +synthesizing or accessing, illegal substances, goods, or services;
        3. +
      3. Facilitating or encouraging users to commit any type of crimes; or
        4. +
      4. Promoting or generating violent extremism or terrorist content.
        5. +
      2. +
    2. Engagement in the illegal or unlicensed practice of any vocation or +profession including, but not limited to, legal, medical, accounting, or +financial professional practices.
      3. +
    3. Abuse, harm, interference, or disruption of services (or enable others to +do the same), such as: +
        +
      1. Promoting or facilitating the generation or distribution of spam; or
        2. +
      2. Generating content for deceptive or fraudulent activities, scams, +phishing, or malware.
        3. +
      4. +
    4. Attempts to override or circumvent safety filters or intentionally drive +Gemma or Model Derivatives to act in a manner that contravenes this Gemma +Prohibited Use Policy.
      5. +
    5. Generation of content that may harm or promote the harm of individuals or +a group, such as: +
        +
      1. Generating content that promotes or encourages hatred;
        2. +
      2. Facilitating methods of harassment or bullying to intimidate, abuse, +or insult others;
        3. +
      3. Generating content that facilitates, promotes, or incites violence;
        4. +
      4. Generating content that facilitates, promotes, or encourages self +harm;
        5. +
      5. Generating personally identifying information for distribution or +other harms;
        6. +
      6. Tracking or monitoring people without their consent;
        7. +
      7. Generating content that may have unfair or adverse impacts on people, +particularly impacts related to sensitive or protected +characteristics; or
        8. +
      8. Generating, gathering, processing, or inferring sensitive personal or +private information about individuals without obtaining all rights, +authorizations, and consents required by applicable laws.
        9. +
      6. +
    3. +
  3. Generate and distribute content intended to misinform, misrepresent or +mislead, including: +
      +
    1. Misrepresentation of the provenance of generated content by claiming +content was created by a human, or represent generated content as +original works, in order to deceive;
      2. +
    2. Generation of content that impersonates an individual (living or dead) +without explicit disclosure, in order to deceive;
      3. +
    3. Misleading claims of expertise or capability made particularly in +sensitive areas (e.g. health, finance, government services, or legal);
      4. +
    4. Making automated decisions in domains that affect material or individual +rights or well-being (e.g., finance, legal, employment, healthcare, +housing, insurance, and social welfare);
      5. +
    5. Generation of defamatory content, including defamatory statements, +images, or audio content; or
      6. +
    6. Engaging in the unauthorized or unlicensed practice of any profession +including, but not limited to, financial, legal, medical/health, or +related professional practices.
      7. +
    4. +
  4. Generate sexually explicit content, including content created for the +purposes of pornography or sexual gratification (e.g. sexual chatbots). Note +that this does not include content created for scientific, educational, +documentary, or artistic purposes.
    5. +
+ +
diff --git a/LICENSES/gemma/GEMMA_TERMS.html b/LICENSES/gemma/GEMMA_TERMS.html new file mode 100644 index 0000000000000000000000000000000000000000..61b2f45ccbdaf04dc8a1159bdf6c3aff080ffbc6 --- /dev/null +++ b/LICENSES/gemma/GEMMA_TERMS.html @@ -0,0 +1,176 @@ + +Gemma Terms of Use

Gemma Terms of Use

Official Google legal text, retrieved 2026-09-21T11:09:43.368595+00:00. Source: https://ai.google.dev/gemma/terms. Article markup preserved; website navigation omitted. This document is not the MIT license.

+

+

+

The terms below apply to Gemma models listed in the Appendix at bottom of this page. For Gemma 4 terms, see the Gemma 4 license.

+

Last modified: April 1, 2026

+

By using, reproducing, modifying, distributing, performing or displaying any +portion or element of Gemma, Model Derivatives including via any Hosted Service, +(each as defined below) (collectively, the "Gemma Services") or otherwise +accepting the terms of this Agreement, you agree to be bound by this Agreement.

+

Section 1: DEFINITIONS

+

1.1 Definitions

+

(a) "Agreement" or "Gemma Terms of Use" means these terms and conditions +that govern the use, reproduction, Distribution or modification of the Gemma +Services and any terms and conditions incorporated by reference.

+

(b) "Distribution" or "Distribute" means any transmission, publication, +or other sharing of Gemma or Model Derivatives to a third party, including by +providing or making Gemma or its functionality available as a hosted service via +API, web access, or any other electronic or remote means ("Hosted Service").

+

(c) "Gemma" means the set of machine learning language models, trained model +weights and parameters identified in the Appendix, +regardless of the source that you obtained it from.

+

(d) "Google" means Google LLC.

+

(e) "Model Derivatives" means all (i) modifications to Gemma, (ii) works based +on Gemma, or (iii) any other machine learning model which is created by transfer +of patterns of the weights, parameters, operations, or Output of Gemma, to that +model in order to cause that model to perform similarly to Gemma, including +distillation methods that use intermediate data representations or methods based +on the generation of synthetic data Outputs by Gemma for training that model. +For clarity, Outputs are not deemed Model Derivatives.

+

(f) "Output" means the information content output of Gemma or a Model +Derivative that results from operating or otherwise using Gemma or the Model +Derivative, including via a Hosted Service.

+

1.2

+

As used in this Agreement, "including" means +"including without limitation".

+

Section 2: ELIGIBILITY AND USAGE

+

2.1 Eligibility

+

You represent and warrant that you have the legal capacity to enter into this +Agreement (including being of sufficient age of consent). If you are accessing +or using any of the Gemma Services for or on behalf of a legal entity, (a) you +are entering into this Agreement on behalf of yourself and that legal entity, +(b) you represent and warrant that you have the authority to act on behalf of +and bind that entity to this Agreement and (c) references to "you" or +"your" in the remainder of this Agreement refers to both you (as an +individual) and that entity.

+

2.2 Use

+

You may use, reproduce, modify, Distribute, perform or display any of the Gemma +Services only in accordance with the terms of this Agreement, and must not +violate (or encourage or permit anyone else to violate) any term of this +Agreement.

+

Section 3: DISTRIBUTION AND RESTRICTIONS

+

3.1 Distribution and Redistribution

+

You may reproduce or Distribute copies of Gemma or Model Derivatives if you meet +all of the following conditions:

+
    +
  1. You must include the use restrictions referenced in Section 3.2 as an +enforceable provision in any agreement (e.g., license agreement, terms of use, +etc.) governing the use and/or distribution of Gemma or Model Derivatives and +you must provide notice to subsequent users you Distribute to that Gemma or +Model Derivatives are subject to the use restrictions in Section 3.2.
    2. +
  2. You must provide all third party recipients of Gemma or Model Derivatives a +copy of this Agreement.
    3. +
  3. You must cause any modified files to carry prominent notices stating that you +modified the files.
    4. +
  4. All Distributions (other than through a Hosted Service) must be accompanied +by a "Notice" text file that contains the following notice: +"Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms".
    5. +
+

You may add your own intellectual property statement to your modifications and, +except as set forth in this Section, may provide additional or different terms +and conditions for use, reproduction, or Distribution of your modifications, or +for any such Model Derivatives as a whole, provided your use, reproduction, +modification, Distribution, performance, and display of Gemma otherwise complies +with the terms and conditions of this Agreement. Any additional or different +terms and conditions you impose must not conflict with the terms of this +Agreement.

+

3.2 Use Restrictions

+

You must not use any of the Gemma Services:

+
    +
  1. for the restricted uses set forth in the Gemma Prohibited Use Policy at +ai.google.dev/gemma/prohibited_use_policy +("Prohibited Use Policy"), which is hereby incorporated by reference into +this Agreement; or
    2. +
  2. in violation of applicable laws and regulations.
    3. +
+

To the maximum extent permitted by law, Google reserves the right to restrict +(remotely or otherwise) usage of any of the Gemma Services that Google +reasonably believes are in violation of this Agreement.

+

3.3 Generated Output

+

Google claims no rights in Outputs you generate using Gemma. You and your users +are solely responsible for Outputs and their subsequent uses.

+

Section 4: ADDITIONAL PROVISIONS

+

4.1 Updates

+

Google may update Gemma from time to time.

+

4.2 Trademarks

+

Nothing in this Agreement grants you any rights to use Google's trademarks, +trade names, logos or to otherwise suggest endorsement or misrepresent the +relationship between you and Google. Google reserves any rights not expressly +granted herein.

+

4.3 DISCLAIMER OF WARRANTY

+

UNLESS REQUIRED BY APPLICABLE LAW, THE GEMMA SERVICES, AND OUTPUTS, ARE PROVIDED +ON AN "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER +EXPRESS OR IMPLIED, INCLUDING ANY WARRANTIES OR CONDITIONS OF TITLE, +NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. YOU ARE +SOLELY RESPONSIBLE FOR DETERMINING THE APPROPRIATENESS OF USING, REPRODUCING, +MODIFYING, PERFORMING, DISPLAYING OR DISTRIBUTING ANY OF THE GEMMA SERVICES +OR OUTPUTS AND ASSUME ANY AND ALL RISKS ASSOCIATED WITH YOUR USE OR DISTRIBUTION +OF ANY OF THE GEMMA SERVICES OR OUTPUTS AND YOUR EXERCISE OF RIGHTS AND +PERMISSIONS UNDER THIS AGREEMENT.

+

4.4 LIMITATION OF LIABILITY

+

TO THE FULLEST EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT AND UNDER NO +LEGAL THEORY, WHETHER IN TORT (INCLUDING NEGLIGENCE), PRODUCT LIABILITY, +CONTRACT, OR OTHERWISE, UNLESS REQUIRED BY APPLICABLE LAW, SHALL GOOGLE OR ITS +AFFILIATES BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY DIRECT, INDIRECT, +SPECIAL, INCIDENTAL, EXEMPLARY, CONSEQUENTIAL, OR PUNITIVE DAMAGES, OR LOST +PROFITS OF ANY KIND ARISING FROM THIS AGREEMENT OR RELATED TO, ANY OF THE GEMMA +SERVICES OR OUTPUTS EVEN IF GOOGLE OR ITS AFFILIATES HAVE BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES.

+

4.5 Term, Termination, and Survival

+

The term of this Agreement will commence upon your acceptance of this Agreement +(including acceptance by your use, modification, or Distribution, reproduction, +performance or display of any portion or element of the Gemma Services) and will +continue in full force and effect until terminated in accordance with the terms +of this Agreement. Google may terminate this Agreement if you are in breach of +any term of this Agreement. Upon termination of this Agreement, you must delete +and cease use and Distribution of all copies of Gemma and Model Derivatives in +your possession or control. Sections 1, 2.1, 3.3, 4.2 to 4.9 shall survive the +termination of this Agreement.

+

4.6 Governing Law and Jurisdiction

+

This Agreement will be governed by the laws of the State of California without +regard to choice of law principles. The UN Convention on Contracts for the +International Sale of Goods does not apply to this Agreement. The state and +federal courts of Santa Clara County, California shall have exclusive +jurisdiction of any dispute arising out of this Agreement.

+

4.7 Severability

+

If any provision of this Agreement is held to be invalid, illegal or +unenforceable, the remaining provisions shall be unaffected thereby and remain +valid as if such provision had not been set forth herein.

+

4.8 Entire Agreement

+

This Agreement states all the terms agreed between the parties and supersedes +all other agreements between the parties as of the date of acceptance relating +to its subject matter.

+

4.9 No Waiver

+

Google will not be treated as having waived any rights by not exercising (or +delaying the exercise of) any rights under this Agreement.

+

Appendix

+ + + +
diff --git a/LICENSES/gemma/Notice b/LICENSES/gemma/Notice new file mode 100644 index 0000000000000000000000000000000000000000..253d6675dbd069e2f259b6d7d5ac416826979eb0 --- /dev/null +++ b/LICENSES/gemma/Notice @@ -0,0 +1,19 @@ +Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms + +Tokenizer provenance for Decision-1.0-Kai + +The tokenizer payload is inherited from Vela-1.0-Encoder-307M, whose base +model is mmBERT. The mmBERT author configuration identifies google/gemma-2-9b +as its tokenizer source. The upstream tokenizer terms are preserved for +applicable inherited material; the project's MIT grant does not replace them. + +These tokenizer files are redistributed unchanged from the pinned Vela +payload. This statement does not assert that Vela's files are byte-identical +to Google's original files, or that Kai's encoder weights are Gemma weights. +No independent tokenizer-specific MIT grant or Google exception was located. +If any tokenizer file is subsequently modified, prominently identify that +modification and preserve the original terms and notices. + +See GEMMA_TERMS.html, GEMMA_PROHIBITED_USE_POLICY.html and TOKENIZER_TERMS.md. +This supplement does not replace the package's existing MIT LICENSE, +third-party NOTICE, or Transformers Apache-2.0 license text. diff --git a/LICENSES/gemma/TOKENIZER_TERMS.md b/LICENSES/gemma/TOKENIZER_TERMS.md new file mode 100644 index 0000000000000000000000000000000000000000..0ed949aecf23a5043f7c219f46a026964eb5076a --- /dev/null +++ b/LICENSES/gemma/TOKENIZER_TERMS.md @@ -0,0 +1,7 @@ +# Inherited tokenizer terms + +This package contains tokenizer material inherited through Vela and mmBERT from the Gemma tokenizer. This file preserves the applicable upstream conditions; it does not relicense Google's material under MIT or assert that Kai's separately trained encoder weights are Gemma weights. + +As a condition of using or redistributing the included Gemma-origin material, you must comply with the [Gemma Terms of Use](GEMMA_TERMS.html), including Section 3.2 and the [Gemma Prohibited Use Policy](GEMMA_PROHIBITED_USE_POLICY.html), which are incorporated into these terms. You must not use, or permit others to use, that material in violation of those restrictions or applicable law. On further distribution, provide recipients these terms and Google's complete agreement and preserve the accompanying `Notice`. Prominently identify any modifications to covered files. Nothing in the project's MIT license removes these obligations or grants rights to Google trademarks. + +The authoritative sources are [Google's Terms](https://ai.google.dev/gemma/terms) and [Google's policy](https://ai.google.dev/gemma/prohibited_use_policy). The accompanying copies identify their retrieval date. The current agreement still includes Gemma 2; the separate Gemma 4 license does not replace it for this inherited tokenizer. diff --git a/LICENSING_STATUS.md b/LICENSING_STATUS.md new file mode 100644 index 0000000000000000000000000000000000000000..dd949b13bd539bb6ab3598bf70cb4a273b818b5b --- /dev/null +++ b/LICENSING_STATUS.md @@ -0,0 +1,17 @@ +# Decision-1.0-Lex licensing + +**Lex's new code and model-weight contributions use MIT.** Third-party material retains its original terms. The [MIT LICENSE](LICENSE) and existing [NOTICE](NOTICE) remain unchanged. + +| Material | Applicable terms and attribution | +|---|---| +| Lex's new contributions; inherited Vela/mmBERT model contributions | Project MIT grant and upstream attribution in [NOTICE](NOTICE) | +| Adapted ModernBERT/Transformers code | [Apache-2.0](LICENSES/Transformers-Apache-2.0.txt); original copyright and modification attribution are retained | +| Inherited tokenizer material | [Tokenizer terms](LICENSES/gemma/TOKENIZER_TERMS.md), the [Gemma agreement](LICENSES/gemma/GEMMA_TERMS.html), [use policy](LICENSES/gemma/GEMMA_PROHIBITED_USE_POLICY.html), and required [Notice](LICENSES/gemma/Notice) | + +The tokenizer comes through Vela and mmBERT; mmBERT's author configuration identifies `google/gemma-2-9b` as its source. The included materials preserve the applicable upstream conditions. They do not assert a separate tokenizer MIT exemption, byte identity with Google's original files, or that Lex's encoder weights are Gemma weights. The original MIT and Apache grants do not replace those tokenizer conditions. + +Vela's fixed model card (`fe9ccc074b781bc0e2e13c2c8d26f2640410636a`) and mmBERT's publisher card (`c5955035435e2bf121cde7f3c8863ef52ff35d82`) declare MIT. Transformers 4.57.6 (`753d61104116eefc8ffc977327b441ee0c8d599f`) supplies the Apache-2.0 code attribution. Public source links, revisions and file identities are recorded in [NOTICE_SOURCES.json](NOTICE_SOURCES.json). The Gemma agreement copy is dated April 1, 2026 and continues to cover Gemma 2; the separate Gemma 4 license is not substituted. + +Downloads use automatic access after acknowledgment of the [distribution terms](DISTRIBUTION_TERMS.md), including the retained tokenizer agreement and use restrictions. This mechanism does not change the component-specific license scope or establish an unrestricted all-files-MIT grant. The distribution contains unchanged native weights and inherited tokenizer payload; the bundled notices and upstream texts remain part of the package. + +Training/evaluation data, Laya weights and Laya SDK source are not redistributed in this release. Installed dependencies retain their own licenses. diff --git a/METHODS.md b/METHODS.md new file mode 100644 index 0000000000000000000000000000000000000000..cbac6e0b637d31b35c235f9967ff50cc788ef973 --- /dev/null +++ b/METHODS.md @@ -0,0 +1,11 @@ +# Lex training method + +Decision-1.0-Lex is an English typed-decisions specialist fine-tuned from the fixed Kai native `da603662bc57e89ccfb51c972ed9c1f2825f267597353cf1337df9117a3dfabe`. Its 571,909,635-parameter geometry is unchanged. All three type paths are trained: 486 tensors across three encoders, interaction heads and scorers; the shared token embedding, embedding normalization and type embedding remain frozen. + +Development used a state-component split of the original official TRAIN: 4,800 decisions for training and 1,200 for development. The fixed selected recipe was blend50 at the higher learning-rate setting. Those development scores are distinct from the final TEST result. The final refit then combined all 6,000 original TRAIN decisions, trained from the same Kai parent for eight epochs, and exported fixed step 750 without checkpoint selection. Every decision appears exactly eight times; epochs are concatenated before dividing into logical batches, so no tail is dropped or repeated. + +For the original normalized soft label vector q and original semantic hard-label one-hot h, the target is 0.5q + 0.5h. The loss is cross-entropy summed over 64 decisions and divided by 64, with no RPS, consistency, distillation or calibration fitting. Candidate order and semantic descriptions are retained. Each logical batch uses eight physical batches of eight. Training uses BF16 autocast with FP32 parameters and loss, AdamW (weight decay 0.01), gradient clipping at 1, and seed 20260921. There is no warmup. Encoder/head starting learning rates are 2.5e-5/1e-4 and each follows `1e-6 + (base - 1e-6) * (1 + cos(pi * (step - 1) / 749)) / 2` for steps 1–750. + +All complete packed inputs fit 1,024 tokens. The final native manifest is `f288d873999832a3f37c6a7c4268c2ab309691e621794dbf7acab891acbbb7e6`. Native metadata retains the original research export provenance; its native Score training-policy metadata is an inference/export compatibility contract, not a claim that only Score was trained. The actual all-types update is recorded in the export provenance and [TRAINING_PROVENANCE.json](TRAINING_PROVENANCE.json). + +The final TEST was opened only after this checkpoint and recipe were frozen. It was evaluated once against the official typed specialist on identical complete inputs. Neither TEST checkpoint selection nor temperature fitting was performed. [EVALUATION.md](EVALUATION.md) records the point improvement and its limits. This package does not transfer Kai's multilingual quality or latency measurements to Lex. diff --git a/NOTICE b/NOTICE new file mode 100644 index 0000000000000000000000000000000000000000..292735b9bcae88a312833c7ad36025d5830c59c9 --- /dev/null +++ b/NOTICE @@ -0,0 +1,82 @@ +Decision Nano +Copyright (c) 2026 Decision Nano contributors + +The accompanying MIT LICENSE applies to the project's original code, +documentation, and new model-weight contributions. It does not replace +third-party copyrights, licenses, or applicable source-material terms. +Upstream components and remaining provenance boundaries are identified below. + +BASE ENCODER AND MODEL WEIGHTS + +Decision Nano was developed from Vela-1.0-Encoder-307M, published by the +vLLM Semantic Router / Vela contributors: +https://huggingface.co/llm-semantic-router/Vela-1.0-Encoder-307M +Revision: fe9ccc074b781bc0e2e13c2c8d26f2640410636a +The pinned publisher model card declares the MIT license and identifies +jhu-clsp/mmBERT-base as its base model. Nano adds type-specific encoder +paths, candidate interaction/readout layers, packing and training changes. +No Vela, mmBERT, or other third-party copyright is transferred to Nano. + +mmBERT is by Marc Marone, Orion Weller, William Fleshman, Eugene Yang, +Dawn Lawrie, and Benjamin Van Durme. Its official model card declares MIT: +https://huggingface.co/jhu-clsp/mmBERT-base +Model-card revision inspected: c5955035435e2bf121cde7f3c8863ef52ff35d82 +This revision identifies the license evidence; it is not a claim that Vela's +historical training ancestry was reconstructed to that mmBERT checkpoint. +Reference: mmBERT: A Modern Multilingual Encoder with Annealed Language +Learning (2025), https://arxiv.org/abs/2509.06888 + +The original publisher cards and their hashes are retained in +source-metadata/MODEL_LICENSE_METADATA.json. Neither inspected model +snapshot supplies a separate upstream LICENSE file or copyright notice. + +MODERNBERT / TRANSFORMERS CODE + +Copyright 2018- The Hugging Face team. All rights reserved. +Copyright 2024 Answer.AI, LightOn, and contributors, and the HuggingFace Inc. +team. All rights reserved. + +The ModernBERT implementation used by Nano is from Transformers 4.57.6, +commit 753d61104116eefc8ffc977327b441ee0c8d599f, under Apache License 2.0. +The complete upstream license is retained in +LICENSES/Transformers-Apache-2.0.txt. The original ModernBERT copyright +header and exact source references are retained under source-metadata/. + +native/policy/reference/modernbert_sdpa_layout.py adapts the upstream +ModernBERT attention computation: it makes Q/K/V contiguous on the guarded +ROCm efficient-SDPA path and binds the wrapper locally. The original path +is retained outside that guard. This is a project modification, not an +unmodified upstream Transformers file. The adapted upstream portions +remain subject to Apache-2.0; the root MIT LICENSE does not override them. + +TOKENIZER PROVENANCE BOUNDARY + +Nano retains the tokenizer payload inherited through Vela/mmBERT. +The mmBERT authors identify its tokenizer as Gemma 2; their training +configuration names google/gemma-2-9b. Google publishes that original +repository under Gemma terms, whereas the inspected mmBERT and Vela model +cards declare MIT for their repositories without a tokenizer-specific +notice or exception. The exact applicable scope for these inherited +tokenizer files is not established by a model-card tag alone. + +This notice therefore does not independently relicense tokenizer.json, +tokenizer_config.json, or special_tokens_map.json as Nano-created MIT +material, and does not assert that Gemma terms govern Nano's independently +trained encoder weights. See LICENSING_STATUS.md for the exact affected +files and the remaining clarification needed before an unrestricted +all-files-MIT distribution claim. + +TRAINING SOURCES AND RUNTIME DEPENDENCIES + +Training-source credits and the changes made in constructing typed tasks +are listed in TRAINING_ATTRIBUTION.md. No training or evaluation examples, +Laya model weights, or Laya SDK source files are included in this package. +The root LICENSE does not relicense the underlying datasets or waive their +attribution, share-alike, or other applicable conditions. + +PyTorch, Transformers, tokenizers, safetensors, NumPy and their dependencies +are installed separately. Their distributions retain their own licenses +and third-party notices. This package does not substitute its MIT LICENSE +for those dependency licenses. + +This NOTICE is informational and does not alter any applicable license. diff --git a/NOTICE_SOURCES.json b/NOTICE_SOURCES.json new file mode 100644 index 0000000000000000000000000000000000000000..7dd0659a7efb024c18c99448dc20ab115a6a8299 --- /dev/null +++ b/NOTICE_SOURCES.json @@ -0,0 +1,229 @@ +{ + "schema": "decision-nano-license-sources-v1", + "date": "2026-09-21", + "project_license": { + "spdx": "MIT", + "scope": "Original Decision Nano code, documentation and new model-weight contributions; third-party terms preserved", + "publisher_choice_explicit": true + }, + "base_models": { + "vela": { + "api_url": "https://huggingface.co/api/models/llm-semantic-router/Vela-1.0-Encoder-307M/revision/fe9ccc074b781bc0e2e13c2c8d26f2640410636a", + "revision": "fe9ccc074b781bc0e2e13c2c8d26f2640410636a", + "license": "mit", + "base_model": "jhu-clsp/mmBERT-base", + "files": [ + ".gitattributes", + "README.md", + "config.json", + "model.safetensors", + "special_tokens_map.json", + "tokenizer.json", + "tokenizer_config.json" + ], + "readme": { + "url": "https://huggingface.co/llm-semantic-router/Vela-1.0-Encoder-307M/resolve/fe9ccc074b781bc0e2e13c2c8d26f2640410636a/README.md", + "sha256": "817df5015d443e6a9e6bbc870d8bd4f0f5c0d48e29b8f50f665319609838f7ef", + "bytes": 2046, + "local_file": "source-metadata/VELA_LICENSE_SOURCE_README.md" + } + }, + "mmbert": { + "api_url": "https://huggingface.co/api/models/jhu-clsp/mmBERT-base", + "revision": "c5955035435e2bf121cde7f3c8863ef52ff35d82", + "license": "mit", + "base_model": null, + "files": [ + ".gitattributes", + "README.md", + "config.json", + "pytorch_model.bin", + "special_tokens_map.json", + "tokenizer.json", + "tokenizer_config.json" + ], + "readme": { + "url": "https://huggingface.co/jhu-clsp/mmBERT-base/resolve/c5955035435e2bf121cde7f3c8863ef52ff35d82/README.md", + "sha256": "35724415037ee3159aa733c16050620cc19a76219112a6ceeb3949572e0b37e9", + "bytes": 29650, + "local_file": "source-metadata/MMBERT_LICENSE_SOURCE_README.md" + } + } + }, + "transformers": [ + { + "url": "https://raw.githubusercontent.com/huggingface/transformers/753d61104116eefc8ffc977327b441ee0c8d599f/LICENSE", + "repository": "huggingface/transformers", + "revision": "753d61104116eefc8ffc977327b441ee0c8d599f", + "tag": "v4.57.6", + "upstream_file_sha256": "77fd4710def9ec3c0f6225800e0235f15a425abd4a8b03559127fcd782612049", + "saved_file": "LICENSES/Transformers-Apache-2.0.txt", + "saved_sha256": "77fd4710def9ec3c0f6225800e0235f15a425abd4a8b03559127fcd782612049", + "saved_bytes": 11418, + "full_license": true + }, + { + "url": "https://raw.githubusercontent.com/huggingface/transformers/753d61104116eefc8ffc977327b441ee0c8d599f/src/transformers/models/modernbert/modeling_modernbert.py", + "repository": "huggingface/transformers", + "revision": "753d61104116eefc8ffc977327b441ee0c8d599f", + "tag": "v4.57.6", + "upstream_file_sha256": "1b23db2f59d3f80bd9af89fb6df3ce15affd90b5ffc09925e732450e694868ff", + "saved_file": "source-metadata/TRANSFORMERS_MODERNBERT_SOURCE_HEADER.txt", + "saved_sha256": "4577f1803f9dbe8b43f5da709f8750c798def2c33f7eb055d62baf286ed472c8", + "saved_bytes": 1497, + "full_license": false + } + ], + "local_modernbert_adaptation": { + "package_file": "native/policy/reference/modernbert_sdpa_layout.py", + "sha256": "0fb3a22db93ad76e30dfbfb3011de442149d97c565e139a3a55738287a1fbbc8", + "bytes": 3365, + "source_verified_against_planned_native_payload": true, + "terms": "Apache-2.0 for adapted upstream portions; project modifications separately identified in NOTICE" + }, + "tokenizer_lineage": { + "configuration_url": "https://raw.githubusercontent.com/JHU-CLSP/mmBERT/108ffb376ad65c2512801bb88931933dcd286baf/configs/mmBERT-base.yaml", + "configuration": { + "file": "source-metadata/MMBERT_TOKENIZER_LINEAGE.yaml", + "sha256": "0ffa0f1c8388868a42cdbb8ec4b709db6eab4b3e33d8495f4e5f0c32bd15afd3", + "bytes": 2768 + }, + "declared_tokenizer_name": "google/gemma-2-9b", + "original_publisher_repository": "https://huggingface.co/google/gemma-2-9b", + "original_terms_url": "https://ai.google.dev/gemma/terms", + "scope_status": "UPSTREAM_TERMS_SUPPLEMENT_PACKAGED_DOWNLOAD_USE_AGREEMENT_NOT_YET_INTEGRATED", + "affected_planned_files": { + "native/tokenizer/special_tokens_map.json": { + "sha256": "6e3204c5a4004719c185007a745d1940471a6fb02521cfa0a00306dbb6bc5903", + "bytes": 1051 + }, + "native/tokenizer/tokenizer.json": { + "sha256": "609d8f4c067cd3950f88594c5a802616cea245823836ef5848ee4fc40aab5b6f", + "bytes": 34363188 + }, + "native/tokenizer/tokenizer_config.json": { + "sha256": "74a259bb1a3811a7e3028adcd07a65765d866d5e66e0e883f0994ccfa67e8455", + "bytes": 46470 + } + }, + "no_gated_source_fetched": true + }, + "distribution": { + "native_manifest_sha256": "25642383af8af8b24ac8a482a5b1697bd59cbcac02978cd6c6fd571bd3e6148f", + "training_or_evaluation_data_included": false, + "laya_weights_or_sdk_source_included": false, + "installed_dependency_distributions_vendored": false + }, + "superseded_draft": { + "sha256": "d9bc648b19d435e2173aeabbed6728b5fe4c1e766ac67c20faa934e98218c46f", + "bytes": 2419, + "preserved_at": "previous package-preview attribution draft" + }, + "files": [ + { + "file": "LICENSE", + "sha256": "fa0d68dc88b1e5f5511e7d0f21744acff46782693ea453326bd5b12c8a756f3f", + "bytes": 1083 + }, + { + "file": "NOTICE", + "sha256": "f154da77b74087d1e000dc6bd4ef6b9e2cc169195da78eff416abdb18d3fa191", + "bytes": 4245 + }, + { + "file": "NOTICE_DRAFT.md", + "sha256": "626249b6e003b5e199ce1bba549f92f7fb0074a8c9935f791f896105312d1ce0", + "bytes": 440 + }, + { + "file": "TRAINING_ATTRIBUTION.md", + "sha256": "19571d6d1844cee1e546d37b38b12ea4e4298a780bd427e53b6236d24fb1044f", + "bytes": 4625 + }, + { + "file": "LICENSING_STATUS.md", + "sha256": "46c4964c3a01be179608b3902ac0275460e8fcbd951421372f03fea5024fbde0", + "bytes": 2258 + }, + { + "file": "LICENSES/Transformers-Apache-2.0.txt", + "sha256": "77fd4710def9ec3c0f6225800e0235f15a425abd4a8b03559127fcd782612049", + "bytes": 11418 + }, + { + "file": "source-metadata/MODEL_LICENSE_METADATA.json", + "sha256": "696395f97428b851f136d50d850978af8c2a3b278c82620f63246a9312b74a30", + "bytes": 1538 + }, + { + "file": "source-metadata/TRANSFORMERS_LICENSE_SOURCE.json", + "sha256": "b60e54814b6c70791318475ebd4737c29fe5816f3141df392144dd31ac316d77", + "bytes": 1178 + }, + { + "file": "source-metadata/TRANSFORMERS_MODERNBERT_SOURCE_HEADER.txt", + "sha256": "4577f1803f9dbe8b43f5da709f8750c798def2c33f7eb055d62baf286ed472c8", + "bytes": 1497 + }, + { + "file": "LICENSES/gemma/GEMMA_PROHIBITED_USE_POLICY.html", + "sha256": "7e50ae0c7a6386ab51d36aa868e1d20a4fb28cbcab38778a5dbdac6a445f5a96", + "bytes": 4636 + }, + { + "file": "LICENSES/gemma/GEMMA_TERMS.html", + "sha256": "01ccbe2f6504a6a1364db0e5d6d91ed37309435229c322a29b20ee5daebebc1a", + "bytes": 13510 + }, + { + "file": "LICENSES/gemma/Notice", + "sha256": "a7b8b1b625d4a140b4dd55ee4479db7d64b5cda047f2cbc1be9d21516b470921", + "bytes": 1088 + }, + { + "file": "LICENSES/gemma/TOKENIZER_TERMS.md", + "sha256": "3929fe0a22a189b5ed3f29db13aec7b11f8a88feba727c8029a1e11ef5f8d70d", + "bytes": 1326 + } + ], + "license_resolution_supplement": { + "product": "Decision-1.0-Kai", + "source_review": { + "file": "release/license-resolution-v1/REVIEW.md", + "sha256": "dce1b3a1ae3a536033898c8f514b151d56ed48bfe328a42d0d712c923da6db9f", + "bytes": 3932 + }, + "official_fetch_receipt": { + "file": "release/license-resolution-v1/OFFICIAL_FETCH_RECEIPT.json", + "sha256": "34a017f07218e98b8fbef4a738421f9d944ffff751edbe3be7419b272363d20b", + "bytes": 2223 + }, + "copies": [ + { + "file": "LICENSES/gemma/GEMMA_PROHIBITED_USE_POLICY.html", + "sha256": "7e50ae0c7a6386ab51d36aa868e1d20a4fb28cbcab38778a5dbdac6a445f5a96", + "bytes": 4636 + }, + { + "file": "LICENSES/gemma/GEMMA_TERMS.html", + "sha256": "01ccbe2f6504a6a1364db0e5d6d91ed37309435229c322a29b20ee5daebebc1a", + "bytes": 13510 + }, + { + "file": "LICENSES/gemma/Notice", + "sha256": "a7b8b1b625d4a140b4dd55ee4479db7d64b5cda047f2cbc1be9d21516b470921", + "bytes": 1088 + }, + { + "file": "LICENSES/gemma/TOKENIZER_TERMS.md", + "sha256": "3929fe0a22a189b5ed3f29db13aec7b11f8a88feba727c8029a1e11ef5f8d70d", + "bytes": 1326 + } + ], + "download_use_agreement_integrated": false, + "hf_gate_configured": false, + "legal_enforceability_determined": false, + "all_files_mit_claim": false, + "original_LICENSE_NOTICE_unchanged": true + } +} diff --git a/PACKAGE_MANIFEST.json b/PACKAGE_MANIFEST.json new file mode 100644 index 0000000000000000000000000000000000000000..107257f54f80bde8fa27148d6914a6131e8e9c81 --- /dev/null +++ b/PACKAGE_MANIFEST.json @@ -0,0 +1,376 @@ +{ + "files": { + ".gitattributes": { + "bytes": 753, + "sha256": "d401c47d3da9ae93830e327b487cde895438378a1f90620a70b5c7b3e5d3ed92" + }, + "ARCHITECTURE.md": { + "bytes": 1330, + "sha256": "82ceb4a1b1d6372ec15cce9b2902dc52477383f5bc52ba4206aad93efce097ff" + }, + "DISTRIBUTION_TERMS.md": { + "bytes": 1147, + "sha256": "f1d3806ecf7c89046e6a422ad3cad9f85a589f28c3936b78b41806e92b297b7f" + }, + "EVALUATION.md": { + "bytes": 2813, + "sha256": "00862d0a49c69c406b7dbfa890f95f09a34680568309d1cc0571733abe7b933c" + }, + "FINETUNING.md": { + "bytes": 3629, + "sha256": "33c76f3490f283d9e189eeb4190c0297795329b56546271a52b80cd503dd2ec0" + }, + "LICENSE": { + "bytes": 1083, + "sha256": "fa0d68dc88b1e5f5511e7d0f21744acff46782693ea453326bd5b12c8a756f3f" + }, + "LICENSES/Transformers-Apache-2.0.txt": { + "bytes": 11418, + "sha256": "77fd4710def9ec3c0f6225800e0235f15a425abd4a8b03559127fcd782612049" + }, + "LICENSES/gemma/GEMMA_PROHIBITED_USE_POLICY.html": { + "bytes": 4636, + "sha256": "7e50ae0c7a6386ab51d36aa868e1d20a4fb28cbcab38778a5dbdac6a445f5a96" + }, + "LICENSES/gemma/GEMMA_TERMS.html": { + "bytes": 13510, + "sha256": "01ccbe2f6504a6a1364db0e5d6d91ed37309435229c322a29b20ee5daebebc1a" + }, + "LICENSES/gemma/Notice": { + "bytes": 1088, + "sha256": "a7b8b1b625d4a140b4dd55ee4479db7d64b5cda047f2cbc1be9d21516b470921" + }, + "LICENSES/gemma/TOKENIZER_TERMS.md": { + "bytes": 1326, + "sha256": "3929fe0a22a189b5ed3f29db13aec7b11f8a88feba727c8029a1e11ef5f8d70d" + }, + "LICENSING_STATUS.md": { + "bytes": 2324, + "sha256": "fdc2482eb5a734e2390058b673aca9d38374703ad6fc13fc9794ef1d89fd07ba" + }, + "METHODS.md": { + "bytes": 2543, + "sha256": "a7dd6dec55dfe463cc0211c3dffa114b88bb95db90816b70c18b62b5d7be7ad1" + }, + "NOTICE": { + "bytes": 4245, + "sha256": "f154da77b74087d1e000dc6bd4ef6b9e2cc169195da78eff416abdb18d3fa191" + }, + "NOTICE_SOURCES.json": { + "bytes": 8858, + "sha256": "55bd38eb79306965774c0531717177b1ad8f168c886414ffd421f8bbec7097b5" + }, + "README.md": { + "bytes": 4683, + "sha256": "76b5b4529cdb7e09d4a236e55ff5672c7f1bac9408780cda19bf4ed9dfe30dbe" + }, + "TECHNICAL_VALIDATION.json": { + "bytes": 1547, + "sha256": "fcfd2ce54f623325f268a702fc0c06044447e651dad665d6430e92d48f8e7353" + }, + "TRAINING_ATTRIBUTION.md": { + "bytes": 5808, + "sha256": "7d0689d8c5e342e2dce4c7f2ee5e7f5d8b69c43bb3ea262601a12156bf3bff41" + }, + "TRAINING_PROVENANCE.json": { + "bytes": 2310, + "sha256": "5c95cf851a9480e6037bc190d0deee87da35576489362262ad4151bafbb96990" + }, + "USAGE.md": { + "bytes": 3245, + "sha256": "289c507fcb507328511a88b41db3eb3adada16e8c0644152ce782617d4f4f074" + }, + "VALIDATION.md": { + "bytes": 2120, + "sha256": "85e1f45ddf0d5e2650490eefdbd50f81ccccd7f414d78a8075206da49ef0043d" + }, + "assets/architecture-atlas.pdf": { + "bytes": 635270, + "sha256": "ff64805a39d16be56d401166ef4d310da86305058bb2c532751754c8c34d49ee" + }, + "assets/architecture.pdf": { + "bytes": 173099, + "sha256": "6aa6895f5708cda7d935991cf9c1179d9e54e22a32459968a052acf05f62a14a" + }, + "assets/architecture.png": { + "bytes": 432419, + "sha256": "daeed79f92a66a2655eb0594f59af4e90424a84ba9519eeaf2ef5bb8b7949fa2" + }, + "assets/architecture.svg": { + "bytes": 23430, + "sha256": "7c3ae44a25e7d9035e6a9635608f7e491a22d9be6991243b85d85f30874e563c" + }, + "assets/attention-geglu.pdf": { + "bytes": 172797, + "sha256": "43b3485510971d29e66bc17b30b4bffe934ad66078598cc3e2efd63ae9b9bc7c" + }, + "assets/attention-geglu.png": { + "bytes": 265587, + "sha256": "de7d88e3cf036d3b8cba38df47d5b3182faecb9233e675104cee7c75dd7fd8c5" + }, + "assets/attention-geglu.svg": { + "bytes": 10256, + "sha256": "0ae02a36ebe97d0b6eed7b23e140c1fdc85e710c00015c518cae67ed5ec06cea" + }, + "assets/candidate-readout.pdf": { + "bytes": 156814, + "sha256": "56b9e06a343e9044c38cfb8d1ba5ba121bdc7d763db5ce45b927aacdcb00c995" + }, + "assets/candidate-readout.png": { + "bytes": 309736, + "sha256": "ad4be86a2aa0136cd7f528c317340c38988b932aab653f25a5a57a9a3cbad834" + }, + "assets/candidate-readout.svg": { + "bytes": 9825, + "sha256": "1d3709818d5b08014ba63042e5287f84f1c97a735a85ec49108ec6e9e77151ee" + }, + "assets/decision-lex-header.png": { + "bytes": 2237372, + "sha256": "064a4d5ad3ae8b25e973bcf6abdc4117b3d8883bcb87d11a95a8bf9eadd89673" + }, + "assets/residual-layers.pdf": { + "bytes": 131144, + "sha256": "2c5b50bb840ae115a1264b66d535a1048172e891a33d02d507c193525cc45630" + }, + "assets/residual-layers.png": { + "bytes": 309369, + "sha256": "f3ee6441fe1016e2435256644675b08bff99cec72d521e49a78f6cbe5b2dc3bf" + }, + "assets/residual-layers.svg": { + "bytes": 10990, + "sha256": "cdf5cfb263b50eef0171eafead3062ee75eb805961517f313490fdca03b390c3" + }, + "decision_finetune/__init__.py": { + "bytes": 81, + "sha256": "51ab50ae9a598c7b09970161be3551f9dff0851a209ec692f2e296b1bd44566d" + }, + "decision_finetune/__main__.py": { + "bytes": 5410, + "sha256": "d5cabebd24c823d5dc03a4b528dae8b8b70a63dd0692eb6516a59bd4bd64d6a7" + }, + "decision_finetune/data.py": { + "bytes": 6679, + "sha256": "9fae31cb39512952a1dd78fb0b6735434da6f6b94606341fc0200ab9cdaf447e" + }, + "decision_finetune/metrics.py": { + "bytes": 4046, + "sha256": "c3de487e9294d5278064b7e1768915bf68f593c9d7fb067addc52c4c9ac3b07e" + }, + "decision_finetune/run.py": { + "bytes": 16750, + "sha256": "045ff99dd95c69e2ccdf4b1f8af84797c2929a27b16a9c1a3208bba10b41361b" + }, + "decision_finetune/state.py": { + "bytes": 5751, + "sha256": "87dd485127fea7b80ad19300c0d6d95a92b7c2b6bd5b62c99bdc9201b681c9fe" + }, + "decision_inference/__init__.py": { + "bytes": 138, + "sha256": "4b5452be7f24be5accb72df1b2d74d9247a39a48173a79f3f5fd884225be0bbd" + }, + "decision_inference/profile.py": { + "bytes": 2375, + "sha256": "a634a52654efbc378fc564c4fa1cd1a1fdd866f8dd44d6d8f63a2bf4243649e3" + }, + "decision_runtime/__init__.py": { + "bytes": 723, + "sha256": "b0f9db94cfbcc73cab2f09e0269b8bd2eb87c0cc533b34ea5c9a8bfbb1ee48ca" + }, + "decision_runtime/_compat.py": { + "bytes": 1847, + "sha256": "a0dd42a4e3b20eabbaf0d0d62ffc290ef4d9d20bc2fb2ba39d980771a8d0744c" + }, + "decision_runtime/native.py": { + "bytes": 6903, + "sha256": "21c674a0d8a1406156504390974505ba5f981e7ecc9f0c3b29985876ef78511c" + }, + "decision_runtime/training.py": { + "bytes": 15998, + "sha256": "166ea54629d5c90e0e0972b82c0b96e008c799f384400886ffd64312ab972177" + }, + "examples/decisions.jsonl": { + "bytes": 867, + "sha256": "d4b89583ccc9aa35e74dd28c46e01884103ea7a0fa018de8c34d4f88a94287f1" + }, + "examples/finetune.sh": { + "bytes": 1242, + "sha256": "9b72d89237808e0019521ab4e42635007c39d20ea200d98afa9fdca6e49ae400" + }, + "infer.py": { + "bytes": 2373, + "sha256": "6313bfd5e295d5dc3d9b6ded5f85cda75cf1da8ed4b2dbae32f619a76021e410" + }, + "native/INVENTORY.json": { + "bytes": 31007, + "sha256": "06aed00ac4571582b992a995a793875804a0546bdae1461c4161b30fba2143a9" + }, + "native/MANIFEST.json": { + "bytes": 4326, + "sha256": "f288d873999832a3f37c6a7c4268c2ab309691e621794dbf7acab891acbbb7e6" + }, + "native/STATE_LAYOUT.json": { + "bytes": 10971, + "sha256": "a6b24716e2b21240d327ba8763a73f1b54862bc86d5e1f92bb9e670be968e494" + }, + "native/__init__.py": { + "bytes": 78, + "sha256": "1afb9dbcfc379f28049486fe6acb7acfdaa8d16b6773d6084ca4b82ead26f010" + }, + "native/artifacts.py": { + "bytes": 7958, + "sha256": "c98adaf6d782e9fc592cd78b1307d63faffa9ba622b9d508c63337c29156e4d3" + }, + "native/choice_encoder.safetensors": { + "bytes": 441337216, + "sha256": "9516cc841c485c98b27b4f63d2ea8e604fe8064121173064b113da8a6bf57ef6" + }, + "native/contract.py": { + "bytes": 10886, + "sha256": "51a24800792bb3e5bf2a11f50f7bc384770641f4f44ed46277dc01e891bf4726" + }, + "native/decision_config.json": { + "bytes": 14308, + "sha256": "1fefb4ad7dede00c4633bb2ce5fc44fb04237207b9e9ce90fcf8ea8b01a1328d" + }, + "native/decision_heads.safetensors": { + "bytes": 177241884, + "sha256": "bce3ee658a978a19c48c605b921cff994f892e7fc17abd6f8645f07428fbc36f" + }, + "native/encoder/config.json": { + "bytes": 2769, + "sha256": "7aff915e9f159305e0bef3eb0206416f99b8560260b8969b35f1dcd54aaad1a5" + }, + "native/encoder/model.safetensors": { + "bytes": 1227771752, + "sha256": "daaafd81c4ed767d203e24226d78e792800068a7dac344aa043844b82a6306c7" + }, + "native/infer.py": { + "bytes": 1095, + "sha256": "9d14841935c836a0765c705d5a24e8fa97437ae35fff65bc6e690b397f0d0315" + }, + "native/model.py": { + "bytes": 11098, + "sha256": "8fe91e2a77f372d32117e62281ccb58a054c144228b73b4987f1e1a3fca9ee5b" + }, + "native/packing.py": { + "bytes": 6748, + "sha256": "f1dab7f032f4aaab27118606397ca67d71b1f4a2ba49e65bebcfd83a64b27819" + }, + "native/policy/__init__.py": { + "bytes": 131, + "sha256": "0cb0bad7d3d0f258ecf95f52297ee8733a75f8504cabb86b9aea4b8256885114" + }, + "native/policy/artifacts.py": { + "bytes": 7877, + "sha256": "5838d2ea747d912789f3fc813a212af919cbc24ee185073576d9fbe32ed31cf2" + }, + "native/policy/contract.py": { + "bytes": 4334, + "sha256": "0b8eeeeebe9e367564a3c57f48364c5e94ae060f759e220b1326caf152276b67" + }, + "native/policy/infer.py": { + "bytes": 1545, + "sha256": "672bfae798060d51fcccac03143ac881fe7512893ffdacdb36b0618d4b160896" + }, + "native/policy/model.py": { + "bytes": 7678, + "sha256": "eaeebac8fd6bd96243a5c4b6225c86ee3359c067784f1595979ee603cba9acca" + }, + "native/policy/packing.py": { + "bytes": 6748, + "sha256": "f1dab7f032f4aaab27118606397ca67d71b1f4a2ba49e65bebcfd83a64b27819" + }, + "native/policy/reference/__init__.py": { + "bytes": 192, + "sha256": "c8d6fd86207752407f94437544a97266b9529f88d2b341d35aafcd911c942b28" + }, + "native/policy/reference/artifacts.py": { + "bytes": 9342, + "sha256": "3f59428b7a05608ac01c73c7db89c38fcc3e2c5030219df8d053af92b32e6159" + }, + "native/policy/reference/infer.py": { + "bytes": 1545, + "sha256": "672bfae798060d51fcccac03143ac881fe7512893ffdacdb36b0618d4b160896" + }, + "native/policy/reference/model.py": { + "bytes": 5060, + "sha256": "733ea4a48ea03089dac8c3e6a175920704fac50f93313b0214cec4052f25bd26" + }, + "native/policy/reference/modernbert_sdpa_layout.py": { + "bytes": 3365, + "sha256": "0fb3a22db93ad76e30dfbfb3011de442149d97c565e139a3a55738287a1fbbc8" + }, + "native/policy/reference/packing.py": { + "bytes": 6748, + "sha256": "f1dab7f032f4aaab27118606397ca67d71b1f4a2ba49e65bebcfd83a64b27819" + }, + "native/score_encoder.safetensors": { + "bytes": 441337080, + "sha256": "4f45795977846ef4c31b34e4f35bf95dabf5951d71358bab32c9a94814a84a53" + }, + "native/tokenizer/special_tokens_map.json": { + "bytes": 1051, + "sha256": "6e3204c5a4004719c185007a745d1940471a6fb02521cfa0a00306dbb6bc5903" + }, + "native/tokenizer/tokenizer.json": { + "bytes": 34363188, + "sha256": "609d8f4c067cd3950f88594c5a802616cea245823836ef5848ee4fc40aab5b6f" + }, + "native/tokenizer/tokenizer_config.json": { + "bytes": 46470, + "sha256": "74a259bb1a3811a7e3028adcd07a65765d866d5e66e0e883f0994ccfa67e8455" + }, + "native/training_policy.py": { + "bytes": 4761, + "sha256": "6f7fad91c5089b304a1d637b594027a9379a63600792766ce0abca6b24bd2ec5" + }, + "requirements.txt": { + "bytes": 296, + "sha256": "001dd433614bcbacee15858a38f432b97f44b01ad404714f442738f4a8ce2428" + }, + "source-metadata/ENCODER_README.md": { + "bytes": 2046, + "sha256": "817df5015d443e6a9e6bbc870d8bd4f0f5c0d48e29b8f50f665319609838f7ef" + }, + "source-metadata/ENCODER_SOURCE.json": { + "bytes": 425, + "sha256": "9bb1a753d9979117520152860a599064d92a8eaf7f23a0e0d458c66007c8fdf9" + }, + "source-metadata/MMBERT_LICENSE_SOURCE_README.md": { + "bytes": 29650, + "sha256": "35724415037ee3159aa733c16050620cc19a76219112a6ceeb3949572e0b37e9" + }, + "source-metadata/MMBERT_TOKENIZER_LINEAGE.yaml": { + "bytes": 2768, + "sha256": "0ffa0f1c8388868a42cdbb8ec4b709db6eab4b3e33d8495f4e5f0c32bd15afd3" + }, + "source-metadata/MODEL_LICENSE_METADATA.json": { + "bytes": 1538, + "sha256": "696395f97428b851f136d50d850978af8c2a3b278c82620f63246a9312b74a30" + }, + "source-metadata/TRANSFORMERS_LICENSE_SOURCE.json": { + "bytes": 1178, + "sha256": "b60e54814b6c70791318475ebd4737c29fe5816f3141df392144dd31ac316d77" + }, + "source-metadata/TRANSFORMERS_MODERNBERT_SOURCE_HEADER.txt": { + "bytes": 1497, + "sha256": "4577f1803f9dbe8b43f5da709f8750c798def2c33f7eb055d62baf286ed472c8" + }, + "source-metadata/VELA_LICENSE_SOURCE_README.md": { + "bytes": 2046, + "sha256": "817df5015d443e6a9e6bbc870d8bd4f0f5c0d48e29b8f50f665319609838f7ef" + } + }, + "language_scope": "English typed-decisions specialist", + "manifest_excludes_itself": true, + "name": "Decision-1.0-Lex", + "native_file_count": 31, + "native_manifest": "native/MANIFEST.json", + "parameter_tensors": 489, + "parameters": 571909635, + "schema": "decision.public-distribution.v1", + "status": "READY_FOR_ROOT_PUBLICATION", + "subject_manifest_sha256": "f288d873999832a3f37c6a7c4268c2ab309691e621794dbf7acab891acbbb7e6", + "technical_validation": "TECHNICAL_VALIDATION.json", + "training_provenance": "TRAINING_PROVENANCE.json" +} diff --git a/README.md b/README.md new file mode 100644 index 0000000000000000000000000000000000000000..8d7713a8990a0c31714b0162ccb29f190f2799af --- /dev/null +++ b/README.md @@ -0,0 +1,73 @@ +--- +base_model: llm-semantic-router/Decision-1.0-Kai +language: +- en +tags: +- decision-making +- text-classification +- ordinal-regression +- fine-tuned +- safetensors +license: other +license_name: decision-with-upstream-terms +license_link: https://huggingface.co/llm-semantic-router/Decision-1.0-Lex/blob/main/LICENSING_STATUS.md +extra_gated_heading: "Decision distribution terms" +extra_gated_description: "Decision's new code and weights use MIT; retained third-party material keeps its original terms. The inherited tokenizer is supplied under the Gemma Terms of Use and incorporated Prohibited Use Policy. The bundled distribution agreement incorporates the tokenizer terms at https://ai.google.dev/gemma/terms and use restrictions at https://ai.google.dev/gemma/prohibited_use_policy. Access is automatically granted after acknowledgment." +extra_gated_button_content: "Agree and access" +extra_gated_fields: {"I have read and agree to the distribution terms, including the Gemma Terms of Use and incorporated use restrictions for the inherited tokenizer": "checkbox"} +--- + +![Decision 1.0 — Lex](https://gist.githubusercontent.com/Xunzhuo/4020f574e3d38e5e5eae00063bdf9dce/raw/cd0352e12b789629cb83618aad73a6eff492c041/decision-lex-header.png) + +# Decision-1.0-Lex + +*Lex — Latin for law. Give your decisions a rule.* + +**Your move.** + +Turn operational context into structured decisions. Lex is a specialist fine-tuned from Kai to select actions, test conditions and apply scoring rubrics across four English workflows. + +**572M parameters · Three decision types · Complete 1K · Fine-tunable** + +Initial release · [Try Decision Studio](https://llm-semantic-router-decision-studio.hf.space) · [Decision collection](https://huggingface.co/collections/llm-semantic-router/decision-10-6ab12177bd0002394d8409f9). + +## A sharper specialist + +**78.15% accuracy — 1.55 points above Laya Typed Decisions.** + +Both models were evaluated on the original 2,000-decision typed-decisions test split, with complete inputs and the official fine-tuned Laya checkpoint. Lex answers **31 more decisions correctly**. + +| Test accuracy | Lex | Laya Typed Decisions | +|---|---:|---:| +| **Overall** | **78.15%** | 76.60% | +| Choice · 600 decisions | **74.00%** | 73.33% | +| Noul · 600 decisions | 84.67% | **85.67%** | +| Score · 800 decisions | **76.38%** | 72.25% | + +Observed accuracy gain; the paired 95% interval is −0.20 to +3.15 points. Laya leads on Noul and probability-quality metrics. [Full evaluation](https://gist.github.com/Xunzhuo/76b59cb158ce5069a53b946c6f0ee656#file-01-lex-evaluation-md). + +## Built for operational decisions + +| Customer service | Invoice processing | Security incidents | Agent traces | +|---|---|---|---| +| Select an action or escalation. | Check conditions and route exceptions. | Assess a signal against a rubric. | Classify outcomes and flag issues. | + +Supply the context, question and candidate descriptions at runtime. **Choice** returns candidate probabilities, **Noul** estimates whether a condition holds, and **Score** returns an ordered distribution and expected value. Applications choose how to act on those outputs. + +Lex is evaluated as an English specialist on these four workflows. For broader multilingual tasks, explore the general [Kai model](https://huggingface.co/llm-semantic-router/Decision-1.0-Kai). + +## Make it yours + +Run the [Python examples](USAGE.md), or adapt Lex to your own labels and rubrics with the included [fine-tuning CLI](FINETUNING.md). Both hard and soft training labels are supported, with checkpoint resume. + +The complete 1,024-token budget includes context, instructions, all candidates and special tokens. Overlength requests return an error. Native Choice and Score support 2–255 candidates or ordered levels; the Studio adapter uses 2–10 Score levels. + +## Architecture + +![Lex architecture](https://gist.githubusercontent.com/Xunzhuo/4020f574e3d38e5e5eae00063bdf9dce/raw/34876f9fd939829a3026353d18a5c58c7bb233fd/lex-architecture.png) + +Three 22-layer bidirectional encoder paths share multilingual input embeddings, with separate interaction layers and candidate readouts for Choice, Noul and Score. Lex retains Kai's architecture and specializes its weights through supervised fine-tuning. + +[Architecture details](ARCHITECTURE.md) · [Training and methods](METHODS.md) + +Built on [Kai](https://huggingface.co/llm-semantic-router/Decision-1.0-Kai) and [Vela Encoder](https://huggingface.co/llm-semantic-router/Vela-1.0-Encoder-307M). Probabilities are not calibrated confidence; candidate order and task wording can affect outputs. [Attribution and retained third-party terms](NOTICE) · [License scope](LICENSING_STATUS.md). diff --git a/TECHNICAL_VALIDATION.json b/TECHNICAL_VALIDATION.json new file mode 100644 index 0000000000000000000000000000000000000000..cd0bde2c3fe04f503088ab94c3ab88e7e2ff782e --- /dev/null +++ b/TECHNICAL_VALIDATION.json @@ -0,0 +1,55 @@ +{ + "TEST_rows": 0, + "actual_output_closure_sha256": "7ab577af71b775ea78529f40e29c02cb4dabc104b7a7c520a34cf7260a12691e", + "all_parameter_content_and_versions_unchanged": true, + "application_imports_confined_to_package": true, + "batch_sizes": [ + 8, + 8, + 8, + 8, + 8, + 8, + 8, + 8 + ], + "calls": 8, + "complete1K": true, + "device_family": "AMD ROCm", + "gradients_none": true, + "initial_diagnostic_attempt": { + "forward_calls": 0, + "model_loads": 0, + "model_or_package_change": false, + "result": "mechanical argument-path reader error before model load; corrected in separate attempt" + }, + "is_latency_benchmark": false, + "is_quality_evaluation": false, + "native_manifest_sha256": "f288d873999832a3f37c6a7c4268c2ab309691e621794dbf7acab891acbbb7e6", + "parameter_tensors": 489, + "parameters": 571909635, + "parity_against_independent_final_refit_fresh": { + "hard_flips": 0, + "logit_atol": 0.0001, + "max_logit_difference": 0.0, + "max_probability_difference": 0.0, + "pass": true, + "probability_atol": 2e-05 + }, + "peak_allocated_bytes": 2642393088, + "peak_reserved_bytes": 2768240640, + "precision": "FP32 SDPA", + "python_isolated": true, + "rows": 64, + "runtime_versions": { + "numpy": "2.5.3", + "safetensors": "0.8.0", + "tokenizers": "0.22.2", + "torch": "2.12.0+git6bbd260", + "transformers": "4.57.6" + }, + "schema": "decision.lex.public-package-validation.v1", + "source_split": "original official TRAIN", + "status": "PASS_LEX_COMPLETE_PACKAGE_AMD", + "updates": 0 +} diff --git a/TRAINING_ATTRIBUTION.md b/TRAINING_ATTRIBUTION.md new file mode 100644 index 0000000000000000000000000000000000000000..d93f8db90c1fa2850f0ef628947a9fc2e992c646 --- /dev/null +++ b/TRAINING_ATTRIBUTION.md @@ -0,0 +1,33 @@ +# Training-source attribution + +Lex inherits the following training-source lineage through its fixed Kai parent. This package redistributes model artifacts and project code, not the source datasets. Dataset licenses remain with their authors; the model's MIT notice does not grant new rights to those texts or annotations. + +| Source | Attribution and source terms | +|---|---| +| SNLI | Bowman, Angeli, Potts and Manning (2015), [Stanford Natural Language Inference](https://nlp.stanford.edu/projects/snli/); CC-BY-SA-4.0. | +| BoolQ | Clark et al. (2019), [Boolean Questions](https://github.com/google-research-datasets/boolean-questions); CC-BY-SA-3.0, with underlying Wikipedia attribution retained in source records. | +| ARC | Clark et al. (2018), [AI2 Reasoning Challenge](https://huggingface.co/datasets/allenai/ai2_arc); publisher metadata identifies CC-BY-SA-4.0. | +| MASSIVE | FitzGerald et al. (2022), [Amazon MASSIVE](https://github.com/alexa/massive), multilingual intent labels. | +| HelpSteer2 | Wang et al. (2024), [NVIDIA HelpSteer2](https://huggingface.co/datasets/nvidia/HelpSteer2), human ordinal response ratings. | +| KLUE-STS | Park et al. (2021), [KLUE](https://github.com/KLUE-benchmark/KLUE); retain the dataset's CC-BY-SA-4.0 terms. | +| JSTS and JCommonsenseQA | Kurihara, Kawahara and Shibata (2022), [JGLUE](https://github.com/yahoojapan/JGLUE/tree/6f071c09316baae89c3d083a90985b4b1cb9968c); CC-BY-SA-4.0. | +| Natural Instructions | Wang et al. (2022), [Super-NaturalInstructions](https://github.com/allenai/natural-instructions/tree/55a365637381ce7f3748fa2eac7aef1a113bbb82). Instruction definitions and dataset instances have distinct terms; the collection's code license does not replace instance-source licenses. | +| SocialIQA | Sap et al. (2019), [Social IQa](https://allenai.org/data/socialiqa); CC-BY-4.0. | +| QASC | Khot, Clark, Guerquin, Jansen and Sabharwal (2020), [QASC](https://github.com/allenai/qasc/blob/c44a320c2bb682d614a675d2ba4be072150db4eb/README.md); CC-BY-4.0. | +| Cosmos QA | Huang, Le Bras, Bhagavatula and Choi (2019), [Cosmos QA](https://github.com/wilburOne/cosmosqa/tree/b6eb99cca4e2a51dd28a9a6f562534872d851639); [official publisher card](https://huggingface.co/datasets/allenai/cosmos_qa) records CC-BY-4.0 and author confirmation. | +| TyDi QA | Clark et al. (2020), [TyDi QA](https://github.com/google-research-datasets/tydiqa); Apache-2.0 dataset/repository terms do not remove underlying Wikipedia attribution obligations. | +| OpenAssistant | Köpf et al. (2023), [OpenAssistant Conversations](https://huggingface.co/datasets/OpenAssistant/oasst1); Apache-2.0. Human quality/helpfulness annotations were used in training. | +| AfriSenti | Muhammad et al. (2023), [AfriSenti and SemEval-2023 Task 12](https://github.com/afrisenti-semeval/afrisent-semeval-2023/tree/5aec3cfcf87b59f885e98641f72e52182776429d); CC-BY-4.0. The upstream README also provides the NaijaSenti and ASAB citations. | +| SentiMix Hinglish | Patwa, Aguilar, Kar, Pandey, PYKL, Gambäck, Chakraborty, Solorio and Das (2020), [SemEval-2020 Task 9](https://aclanthology.org/2020.semeval-1.100/), [dataset DOI](https://doi.org/10.5281/zenodo.3974927); CC-BY-4.0. | + +The Natural Instructions training families were query well-formedness (task 673), CaSiNo (353), ATOMIC (1216/1211), poem sentiment (833), HateXplain (1502/1503), WIQA (1548/1727), and ProPara (1568). Their recorded instance terms are respectively CC-BY-SA-4.0, CC-BY-4.0, CC-BY-4.0, CC-BY-4.0, MIT, Apache-2.0 and Apache-2.0. For WIQA and ProPara, this records the published NI instance declarations; it does not claim a separate re-audit of every original data archive. Qasper was a development family, not a training family. + +Project transformations included explicit Choice/Noul/Score schemas, candidate identifiers, deterministic splits and component isolation, adjacent-level distributions for original continuous means, and one-hot representations of original hard labels. Sentiment categories remain coarse ordered sentiment, not fabricated continuous human quality ratings. TyDi uses the annotated supporting passage for Boolean prediction; it is not a retrieval task. OASST Spanish/Chinese rubrics and schema paraphrases were agent-authored changes, not new human annotations. Controlled rule/state examples were generated by the project. No original dataset is relicensed by these transformations. + +Training-source attribution is distinct from evaluation-source attribution and from claims about what may have appeared in upstream encoder pretraining. Benchmark-only datasets and excluded candidates are not represented here as training sources. + +## Lex specialization + +Lex additionally uses the original TRAIN split of [LocalLLaMA/typed-decisions](https://huggingface.co/datasets/LocalLLaMA/typed-decisions), fixed revision `ea9306458d6e9563628369a3d1e72e362fb381d2`. The publisher identifies this English synthetic dataset as Apache-2.0. The 1,200 original cases contain 6,000 decisions across four workflows and 20 question schemas. The dataset's generated states and teacher-derived hard/soft targets measure agreement with that annotation process, not human ground truth or validated business outcomes. + +The original TRAIN parquet SHA256 is `46a58d63edfd86e23229c78afe8b72307bb4ca9fb0e8df180cabb3c67ec9dcd5` (598,824 bytes). Lex preserves complete states, semantic candidates, question criteria, original hard labels, and explicitly normalized soft targets. The final training target is an equal mixture of the normalized soft distribution and the one-hot original hard label. All original TRAIN decisions appear exactly eight times. TEST was excluded from training and recipe selection. Dataset instances, gold labels, latent factors, and third-party model/SDK files are not included in this distribution. diff --git a/TRAINING_PROVENANCE.json b/TRAINING_PROVENANCE.json new file mode 100644 index 0000000000000000000000000000000000000000..0d26cdf5a802c42834eaed7385db8688dae9229c --- /dev/null +++ b/TRAINING_PROVENANCE.json @@ -0,0 +1,65 @@ +{ + "active_path_tensors": 486, + "dataset": { + "TEST_used_for_training_or_selection": false, + "cases": 1200, + "decision_exposures": 48000, + "decisions": 6000, + "epochs": 8, + "original_split": "train", + "parquet_bytes": 598824, + "parquet_sha256": "46a58d63edfd86e23229c78afe8b72307bb4ca9fb0e8df180cabb3c67ec9dcd5", + "repo": "LocalLLaMA/typed-decisions", + "revision": "ea9306458d6e9563628369a3d1e72e362fb381d2" + }, + "fresh_native_parity": { + "hard_flip_atol": 0, + "logit_atol": 0.0001, + "max_logit_difference": 0.0, + "max_probability_difference": 0.0, + "official_hard_flips": 0, + "pass": true, + "probability_atol": 2e-05, + "rows": 64 + }, + "frozen_training_plan_sha256": "d15c3cafd15d0ae14cf45e196d987016b304702f6f3930081b2ef30bbeabc638", + "language_scope": "English typed-decisions specialist", + "model": "Decision-1.0-Lex", + "native_files_unchanged": true, + "native_manifest_sha256": "f288d873999832a3f37c6a7c4268c2ab309691e621794dbf7acab891acbbb7e6", + "parameter_tensors": 489, + "parameters": 571909635, + "parent_native_manifest_sha256": "da603662bc57e89ccfb51c972ed9c1f2825f267597353cf1337df9117a3dfabe", + "provenance_scope": "Lex refit; not the Kai tensor-composition ledger or Kai evaluation/performance", + "python_version": "3.12.13", + "recipe": { + "DEV_evaluation_during_final_refit": false, + "bf16_autocast": true, + "clip_norm": 1.0, + "cosine_end_lr": 1e-06, + "encoder_lr": 2.5e-05, + "fixed_export_step": 750, + "head_lr": 0.0001, + "logical_batch_size": 64, + "loss": "cross_entropy(0.5*normalized_original_soft + 0.5*onehot_original_hard)", + "optimizer": "AdamW", + "parameter_precision": "fp32", + "physical_batch_size": 8, + "seed": 20260921, + "steps": 750, + "temperature_fit": false, + "warmup_steps": 0, + "weight_decay": 0.01 + }, + "runtime_versions": { + "numpy": "2.5.3", + "safetensors": "0.8.0", + "tokenizers": "0.22.2", + "torch": "2.12.0+git6bbd260", + "transformers": "4.57.6" + }, + "schema": "decision.lex.training-provenance.v1", + "shared_frozen_tensors": 3, + "training_complete_sha256": "d0cc6eba0cc698cfc6538351ce0f7f73ccd7143be6a9431ad0dd3ae773ba2645", + "training_output_closure_sha256": "d3525627110ea2331d4e32d40303c61f2fd576b8222adfc69394177bc6e28142" +} diff --git a/USAGE.md b/USAGE.md new file mode 100644 index 0000000000000000000000000000000000000000..7aa9a55d571258d893d30a7b3bb0dcc4cf7c065a --- /dev/null +++ b/USAGE.md @@ -0,0 +1,31 @@ +# Run typed decisions + +Open the [Lex model page](https://huggingface.co/llm-semantic-router/Decision-1.0-Lex), accept the distribution terms, then download the complete release with an authenticated Hugging Face CLI: + +```bash +hf auth login +hf download llm-semantic-router/Decision-1.0-Lex --local-dir Decision-1.0-Lex +cd Decision-1.0-Lex +``` + +Use `--local-dir` to materialize ordinary files for the native loader. Install the pinned Python dependencies from `requirements.txt` in your compatible ROCm environment before running the examples. + +This initial release bundles the verified native under `native/`, the public Python API. Run the following commands from the complete distribution root in a compatible environment. `PACKAGE_MANIFEST.json` records the exact payload; do not add files inside `native/` because its loader checks the complete file roster. + +Use an existing compatible AMD ROCm environment. The verified source pins Transformers 4.57.6; tested companion versions were Python 3.12.13, tokenizers 0.22.2 and safetensors 0.8.0. Actual validation used a ROCm PyTorch 2.12 development build, not a promised generic wheel installation. Choose a matching supported ROCm/PyTorch installation for your host; this release does not supply an installer or a CPU/NVIDIA inference path. Do not upgrade an active environment in place. + +```bash +export PYTHONPATH="$PWD${PYTHONPATH:+:$PYTHONPATH}" +export PYTHONDONTWRITEBYTECODE=1 +export HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 TOKENIZERS_PARALLELISM=false +ROCR_VISIBLE_DEVICES=0 python infer.py \ + --input examples/decisions.jsonl --output predictions.jsonl --batch-size 8 +``` + +The three examples illustrate the Choice, Noul and Score interfaces. They are synthetic interface examples, not evaluated model predictions. Each input line has exactly `id`, `state_text`, and `question`; do not attach labels. Choice uses `options`, Score uses ordered `levels`, and Noul uses the fixed no/yes pair with optional `false_criterion`/`true_criterion`. IDs identify outputs and do not become model tokens. + +The Python API returns original record/candidate identities, logits and probabilities in the supplied order. Choice includes the chosen candidate ID. Noul includes the yes probability; it does not emit a boolean decision. An application can choose first-argmax (an exact no/yes tie selects no) or `p_yes >= 0.5` (tie selects yes), but should declare that choice. Score includes `expected_value` over the supplied values and `score`, the expected ordinal index from 0 to K−1. Neither is automatically a calibrated business utility. + +All state/question/candidate text and markers count toward the complete 1024 limit. `predict_1k` checks every input before the first forward and rejects overflow without truncation. Batch sizes 1–8 are supported by this wrapper. Use it rather than directly invoking the native research-capacity entrypoint. + +A compatible fine-tuned export can be used with `--native --manifest-sha256 `. Only the pinned runtime revision and three-path architecture are accepted. Downloaded HF symlinks must be materialized into real files before loading. The manifest hash is an integrity check, not a reason to execute arbitrary untrusted Python. diff --git a/VALIDATION.md b/VALIDATION.md new file mode 100644 index 0000000000000000000000000000000000000000..90c422c2f9e09697ec7ca7a82b544ff0396f1a62 --- /dev/null +++ b/VALIDATION.md @@ -0,0 +1,9 @@ +# Technical validation + +The final refit completed 750 updates and exported a native bundle with 571,909,635 parameters in 489 tensors. The initial 64 TRAIN decisions passed same-B8 native/all-types-wrapper parity. The first update had finite gradients across all 486 active tensors and actual parameter-content changes in each of the nine encoder/head/scorer groups. The shared three tensors remained unchanged. + +After releasing the training model and optimizer, a separate process loaded the final native and repeated those 64 TRAIN decisions in eight FP32 batches. Logits and probabilities matched the saved final predictions exactly, with zero hard-label flips. This was a technical reload check, not another quality evaluation. + +The public Python inference and fine-tuning modules are copied unchanged from the tested Kai distribution. Their strict loader verifies every native file and the runtime source allowlist before loading. The native runtime/tokenizer/geometry match the supported architecture; no research-directory import is required. The completed Lex package-isolation check loaded the bundled native through public `load_native` and evaluated the same 64 TRAIN decisions through `predict_1k` in eight FP32 batches. It ran with isolated Python from an external working directory, with all application imports confined to this package. Maximum logit/probability differences from the independent final-refit fresh reference were 0.0/0.0, with zero hard flips. All 489 parameter contents and versions were unchanged; no gradients, updates or TEST evaluation occurred. See [TECHNICAL_VALIDATION.json](TECHNICAL_VALIDATION.json). + +The included fine-tuning CLI's earlier bounded continuous-versus-resume and independent-reload checks are interface evidence, not new Lex training results. Lex's final training used the fixed research recipe described in [METHODS.md](METHODS.md). The supported product profile is complete packed length ≤1,024 and FP32 inference on compatible AMD ROCm. No browser end-to-end latency, CPU/NVIDIA inference, multilingual-specialist quality or larger-window quality is claimed. diff --git a/assets/architecture-atlas.pdf b/assets/architecture-atlas.pdf new file mode 100644 index 0000000000000000000000000000000000000000..85ad801a1212dea2984e88747f464dc66f84465d --- /dev/null +++ b/assets/architecture-atlas.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ff64805a39d16be56d401166ef4d310da86305058bb2c532751754c8c34d49ee +size 635270 diff --git a/assets/architecture.pdf b/assets/architecture.pdf new file mode 100644 index 0000000000000000000000000000000000000000..93f616c8ed197b1aaf594621a3e7ed2958129e26 --- /dev/null +++ b/assets/architecture.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6aa6895f5708cda7d935991cf9c1179d9e54e22a32459968a052acf05f62a14a +size 173099 diff --git a/assets/architecture.png b/assets/architecture.png new file mode 100644 index 0000000000000000000000000000000000000000..804b4d572a37f70289386e446dd59ecd3ba17beb --- /dev/null +++ b/assets/architecture.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:daeed79f92a66a2655eb0594f59af4e90424a84ba9519eeaf2ef5bb8b7949fa2 +size 432419 diff --git a/assets/architecture.svg b/assets/architecture.svg new file mode 100644 index 0000000000000000000000000000000000000000..ec9bec9e7e57b6997cb11c28478caa78c2d5beb3 --- /dev/null +++ b/assets/architecture.svg @@ -0,0 +1,259 @@ +Decision Lex: three independent encoder pathsLex final native f288d873: 571909635 parameters, 489 tensors. Shared embeddings; three independently parameterized22-layer paths and type-specific interaction heads. Public complete-input profile1024; native architecture unchanged. + + +Choice +argmax p → candidate ID + + +Valid-candidate softmax +raw probabilities + + + + +Scalar scorer, specific to type +LN → Linear768 → GELU → Linear1 + + + + +Gather candidate [MASK] states + + + +2× + + +Pre-LN global attention + ReLU FFN +complete residual layer: Fig. 2b + + + + +Gather rows of this type +keep original padded L + + + ++ + +eₜ + +shared type table + + +Path-specific final LayerNorm + + + +22× + ++ + + + +GEGLU feed forward +768 → 2 × 1152 → 768 + + + + +LayerNorm + + + ++ + + + +Self-attention · 12 × 64 +Q/K RoPE · local / global + + + + +LayerNorm* + + + + + + +Noul +K = 2 · P(yes) = p₁ + + +Valid-candidate softmax +raw probabilities + + + + +Scalar scorer, specific to type +LN → Linear768 → GELU → Linear1 + + + + +Gather candidate [MASK] states + + + +2× + + +Pre-LN global attention + ReLU FFN +complete residual layer: Fig. 2b + + + + +Gather rows of this type +keep original padded L + + + ++ + +eₜ + +shared type table + + +Path-specific final LayerNorm + + + +22× + ++ + + + +GEGLU feed forward +768 → 2 × 1152 → 768 + + + + +LayerNorm + + + ++ + + + +Self-attention · 12 × 64 +Q/K RoPE · local / global + + + + +LayerNorm* + + + + + + +Score +E[index], E[value] from p + + +Valid-candidate softmax +raw probabilities + + + + +Scalar scorer, specific to type +LN → Linear768 → GELU → Linear1 + + + + +Gather candidate [MASK] states + + + +2× + + +Pre-LN global attention + ReLU FFN +complete residual layer: Fig. 2b + + + + +Gather rows of this type +keep original padded L + + + ++ + +eₜ + +shared type table + + +Path-specific final LayerNorm + + + +22× + ++ + + + +GEGLU feed forward +768 → 2 × 1152 → 768 + + + + +LayerNorm + + + ++ + + + +Self-attention · 12 × 64 +Q/K RoPE · local / global + + + + +LayerNorm* + + + + + + + + + + + + +Shared embedding LayerNorm · 768 + + + +Shared token embedding · 256000 × 768 + + + + +Complete input · B × L · L ≤ 1024 + + +Each needed encoder keeps full B × L; only interaction heads gather type rows. +571,909,635 parameters · dynamic candidates · complete 1K input +* Layer 1 attention normalization is Identity in every encoder path. + diff --git a/assets/attention-geglu.pdf b/assets/attention-geglu.pdf new file mode 100644 index 0000000000000000000000000000000000000000..97608587f03ed8b55632740502a6d819f2905808 --- /dev/null +++ b/assets/attention-geglu.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:43b3485510971d29e66bc17b30b4bffe934ad66078598cc3e2efd63ae9b9bc7c +size 172797 diff --git a/assets/attention-geglu.png b/assets/attention-geglu.png new file mode 100644 index 0000000000000000000000000000000000000000..46958a275f3de6263ef48cf004c6e210c4ec69c1 --- /dev/null +++ b/assets/attention-geglu.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:de7d88e3cf036d3b8cba38df47d5b3182faecb9233e675104cee7c75dd7fd8c5 +size 265587 diff --git a/assets/attention-geglu.svg b/assets/attention-geglu.svg new file mode 100644 index 0000000000000000000000000000000000000000..9389f56989a3ada745318882f29da805eee4f26d --- /dev/null +++ b/assets/attention-geglu.svg @@ -0,0 +1,115 @@ +ModernBERT attention and GEGLU operatorsQ/K-only rotary position transform with YaRN factor 4; ungated value path; correct GELU first branch in the MLP. + + +(a) Backbone self-attention +(b) Backbone GEGLU +12 heads × 64 dimensions +Q / K: RoPE, YaRN factor 4 +d = 768 · dFF = 1152 +no linear bias + + +Concat heads → Wₒ (768 → 768) + +Attention output + + + +Attention weights × V + + + + +Softmax + + + ++ + + + +QKᵀ / √64 + + + + +Padding + +local/global +mask + + + + +RoPE(Q) + + + +RoPE(K) + + + + + +Q + + + + +K + + + + +V + + + + +Wqkv: 768 → 2304 +split + reshape into 12 heads + + + + +Normalized token states + +Normalized token states + + +Wᵢ: 768 → 2 × 1152 + + + + +Split into u, v + + + + +u ∈ ℝ¹¹⁵² + + + +v ∈ ℝ¹¹⁵² + + + + + +GELU + + + +⊙ + + +1152 features + + +Wₒ: 1152 → 768 + + +FFN output + +Zero backbone dropout omitted. Interaction layers instead use global attention and ReLU FFN (Fig. 2b). + diff --git a/assets/candidate-readout.pdf b/assets/candidate-readout.pdf new file mode 100644 index 0000000000000000000000000000000000000000..f9a01c996c06eca36ef3e84a759129ce99e2f7ea --- /dev/null +++ b/assets/candidate-readout.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:56b9e06a343e9044c38cfb8d1ba5ba121bdc7d763db5ce45b927aacdcb00c995 +size 156814 diff --git a/assets/candidate-readout.png b/assets/candidate-readout.png new file mode 100644 index 0000000000000000000000000000000000000000..df5c76bdecf0f52b62b9ca7e593930e7dd55e083 --- /dev/null +++ b/assets/candidate-readout.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ad4be86a2aa0136cd7f528c317340c38988b932aab653f25a5a57a9a3cbad834 +size 309736 diff --git a/assets/candidate-readout.svg b/assets/candidate-readout.svg new file mode 100644 index 0000000000000000000000000000000000000000..da90d86d4188da63ee4dc5e8da007e5691007dff --- /dev/null +++ b/assets/candidate-readout.svg @@ -0,0 +1,121 @@ +Dynamic candidate packing and marker scoringOne packed row per question; no candidate-independent tower, fixed-label head, mean pooling, or state cache. + + +Dynamic packing, type-specific scoring, and outputs + + +Choice +argmax p → external ID + + + +Noul · K = 2 +P(yes) = p₁ + + + +Score +E[index] = Σⱼ j pⱼ +E[value] = Σⱼ vⱼ pⱼ + + + + + + + +Softmax over valid K candidates + + + + +Linear 768 → 1 (within type) + + + +GELU + + + + +Linear 768 → 768 (within type) + + + + +LayerNorm (within type) + + + + + +Gather h[m0], …, h[mK−1] + +K × 768 + + +Selected encoder path → + type vector → type-specific interaction +full sequence computation from Fig. 1; no candidate-wise encoding + + + + +CLS + + + +type question: q + + + +SEP + + + +[MASK] + + + +candidate 0 + + + +SEP + + + +… + + + +[MASK] + + + +candidate K−1 + + + +SEP + + + +state + + + +SEP + + +One packed row · L tokens + +m0 + +mK−1 +Score descriptions include “level j:”; values vⱼ and opaque IDs stay outside tokens. +Public profile: 1024 total tokens; complete fields, overflow raises an error. +Same scorer at every marker +within one type; dynamic K +Separate scorer per type +No per-candidate encoder + diff --git a/assets/decision-lex-header.png b/assets/decision-lex-header.png new file mode 100644 index 0000000000000000000000000000000000000000..bb2861cd7fc272f83a51f0b2dbdeb1f4f463335c --- /dev/null +++ b/assets/decision-lex-header.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:064a4d5ad3ae8b25e973bcf6abdc4117b3d8883bcb87d11a95a8bf9eadd89673 +size 2237372 diff --git a/assets/residual-layers.pdf b/assets/residual-layers.pdf new file mode 100644 index 0000000000000000000000000000000000000000..fbed66843979e4f626e49407c9c7d2ec1678d599 --- /dev/null +++ b/assets/residual-layers.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2c5b50bb840ae115a1264b66d535a1048172e891a33d02d507c193525cc45630 +size 131144 diff --git a/assets/residual-layers.png b/assets/residual-layers.png new file mode 100644 index 0000000000000000000000000000000000000000..5492f7709c227e1d4570ce1114f6feb83032789f --- /dev/null +++ b/assets/residual-layers.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f3ee6441fe1016e2435256644675b08bff99cec72d521e49a78f6cbe5b2dc3bf +size 309369 diff --git a/assets/residual-layers.svg b/assets/residual-layers.svg new file mode 100644 index 0000000000000000000000000000000000000000..686fde6c1112fdd06c946b7997c5af73708f1a0e --- /dev/null +++ b/assets/residual-layers.svg @@ -0,0 +1,118 @@ +Complete residual layers for current LexModernBERT pre-norm residual layer and independent type-specific global interaction layer. All three paths use these geometries with separate weights. + + +(a) Encoder layer: 22 per path +(b) Interaction layer: 2 per type +110,330,880 parameters per encoder path +14,175,744 parameters per 2-layer head + +22× + ++ + + +Linear 1152 → 768 + + + +GELU(u) ⊙ v +u, v ∈ ℝ¹¹⁵² + + + +Linear 768 → 2304 + + + +LayerNorm + + ++ + + +Self-Attention · 12 heads +local / global · Q,K RoPE + + + +LayerNorm* + + + + + + + + + + + + + + + + +Final LayerNorm + + +Input token states · L × 768 + + +2× + ++ + + +Linear 3072 → 768 + + + +ReLU → Dropout +p = 0.1 during training + + + +Linear 768 → 3072 + + + +LayerNorm + + ++ + + +Global Self-Attention +12 heads · padding mask + + + +LayerNorm + + + + + + + + + + + + + + +Dropout +Dropout + + +Candidate-marker readout (Fig. 3) + + +Input token states · L × 768 + +* Encoder layer 1: attention LayerNorm is Identity. All residual paths start before normalization. +Encoder: 8 global layers (1, 4, …, 22); remaining layers local ±64. No causal mask. +Interaction: full-token attention; no added positional encoding. Dropout 0.1 only in training. +A scalar scorer follows each type head; encoder / head / scorer weights are not tied across types. + diff --git a/decision_finetune/__init__.py b/decision_finetune/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..25337e4b317870e886901f74d19d19a61682b4ed --- /dev/null +++ b/decision_finetune/__init__.py @@ -0,0 +1 @@ +"""Portable JSONL fine-tuning CLI; end-to-end AMD validation remains pending.""" diff --git a/decision_finetune/__main__.py b/decision_finetune/__main__.py new file mode 100644 index 0000000000000000000000000000000000000000..2082eac59480991dfe169a2c4a478132bbfe69d3 --- /dev/null +++ b/decision_finetune/__main__.py @@ -0,0 +1,80 @@ +"""Run with PYTHONPATH= python -m decision_finetune.""" +import argparse +import json +import math +import sys +from . import data + + +def parser(): + p = argparse.ArgumentParser(description="Single-AMD full1K native Choice/Noul/Score fine-tuning") + commands = p.add_subparsers(dest="command", required=True) + v = commands.add_parser("validate-jsonl", help="Structural/split checks only; does NOT establish token support") + v.add_argument("--train", required=True); v.add_argument("--dev", required=True) + v.add_argument("--selection", choices=("macro-nll", "hard-accuracy"), default="macro-nll") + t = commands.add_parser("train", help="Loads a real AMD model; complete admission precedes every forward") + for name in ("native", "manifest-sha256", "train", "dev", "output"): + t.add_argument("--" + name, required=True) + t.add_argument("--epochs", type=int, default=4) + t.add_argument("--logical-batch-size", type=int, default=64) + t.add_argument("--micro-batch-size", type=int, default=8) + t.add_argument("--seed", type=int, default=20260921) + t.add_argument("--encoder-lr", type=float, default=2.5e-5) + t.add_argument("--head-lr", type=float, default=1e-4) + t.add_argument("--lr-min", type=float, default=1e-6) + t.add_argument("--weight-decay", type=float, default=.01) + t.add_argument("--clip-norm", type=float, default=1.) + t.add_argument("--score-rps-weight", type=float, default=.1) + t.add_argument("--warmup-steps", type=int) + t.add_argument("--warmup-ratio", type=float, default=.1) + t.add_argument("--max-steps", type=int) + t.add_argument("--eval-steps", help="Explicit comma-separated update steps including0/final; default epoch ends") + t.add_argument("--selection", choices=("macro-nll", "hard-accuracy"), default="macro-nll") + t.add_argument("--selection-tolerance", type=float, default=1e-8) + t.add_argument("--save-every", type=int, default=0, help="Additional update-boundary checkpoints;0 means evaluation points only") + t.add_argument("--cpu-threads", type=int, default=2) + t.add_argument("--max-reserved-gib", type=float, default=64.) + t.add_argument("--allow-nondeterministic-kernels", action="store_true", help="Explicitly opt out of PyTorch deterministic algorithm enforcement") + t.add_argument("--resume", help="Existing immutable update-boundary checkpoint directory") + t.add_argument("--resume-sha256", help="Trusted manifest SHA for --resume") + t.add_argument("--stop-after-step", type=int, help="Clean scheduling pause after this original update; does not change the LR horizon or training config") + return p + + +def configuration(args): + fields = ("epochs", "logical_batch_size", "micro_batch_size", "seed", "encoder_lr", "head_lr", "lr_min", "weight_decay", + "clip_norm", "score_rps_weight", "warmup_steps", "warmup_ratio", "max_steps", "selection", "selection_tolerance", + "save_every", "cpu_threads", "max_reserved_gib") + c = {k: getattr(args, k) for k in fields} + data.need(all(c[k] > 0 for k in ("epochs", "logical_batch_size", "micro_batch_size", "cpu_threads")), "Positive counts required") + data.need(c["micro_batch_size"] <= c["logical_batch_size"] and c["micro_batch_size"] <= 8, "Micro batch must fit logical batch and ≤8") + data.need(0 <= c["seed"] < 2**32 and c["save_every"] >= 0, "Invalid seed/save cadence") + data.need(all(math.isfinite(c[k]) and c[k] > 0 for k in ("encoder_lr", "head_lr", "clip_norm", "max_reserved_gib")), "Positive finite optimizer/memory values") + data.need(c["max_reserved_gib"] <= 64 and all(math.isfinite(c[k]) and c[k] >= 0 for k in ("weight_decay", "score_rps_weight", "selection_tolerance")) + and 0 <= c["warmup_ratio"] < 1, "Invalid budget/objective/warmup") + data.need(math.isfinite(c["lr_min"]) and 0 <= c["lr_min"] <= min(c["encoder_lr"], c["head_lr"]), "Invalid LR floor") + data.need(bool(args.resume) == bool(args.resume_sha256), "Resume directory AND manifest SHA required") + c.update(eval_steps=None if args.eval_steps is None else [int(v) for v in args.eval_steps.split(",")], + deterministic_algorithms=not args.allow_nondeterministic_kernels, + train_precision="bf16_autocast_fp32_parameters_loss", eval_precision="fp32", eval_batch_size=8, + lr_convention="linear warmup to base, then cosine to lr_min at final; with no warmup step1=base; one-step schedule uses base", + input_limit=1024, truncation="error", selection_weighting=("equal_present_types" if args.selection == "macro-nll" else "row")) + return c + + +def main(): + sys.dont_write_bytecode = True + args = parser().parse_args() + if args.command == "validate-jsonl": + train, dev = data.load_splits(args.train, args.dev, args.selection) + print(json.dumps({"status": "PASS_JSONL_AND_EXACT_SPLIT_CHECKS", "train_rows": len(train), "dev_rows": len(dev), + "train_sha256": data.file_sha(args.train), "dev_sha256": data.file_sha(args.dev), + "token_admission_performed": False, "model_calls": 0})) + return + from .run import execute + result = execute(args, configuration(args)) + print(json.dumps({k: result[k] for k in ("status", "native", "native_manifest_sha256", "selected0_no_adaptation", "checkpoint") if k in result})) + + +if __name__ == "__main__": + main() diff --git a/decision_finetune/data.py b/decision_finetune/data.py new file mode 100644 index 0000000000000000000000000000000000000000..46073b96d0d641650f37df8588ae8e6d5ad4bdbc --- /dev/null +++ b/decision_finetune/data.py @@ -0,0 +1,139 @@ +"""JSONL, split and deterministic schedule contracts. No model imports.""" +import copy +import hashlib +import json +import math +from pathlib import Path +import unicodedata + +KINDS = ("choice", "noul", "score") +DEFAULT_NO = "No. The statement or question is not satisfied." +DEFAULT_YES = "Yes. The statement or question is satisfied." + + +def need(ok, message): + if not ok: + raise ValueError(message) + + +def digest(value): + return hashlib.sha256(json.dumps(value, ensure_ascii=False, sort_keys=True, + separators=(",", ":"), allow_nan=False).encode()).hexdigest() + + +def file_sha(path): + h = hashlib.sha256() + with Path(path).open("rb") as stream: + for chunk in iter(lambda: stream.read(1 << 20), b""): + h.update(chunk) + return h.hexdigest() + + +def text(value): + need(isinstance(value, str) and value.strip(), "Nonempty text required") + return " ".join(unicodedata.normalize("NFKC", value).casefold().split()) + + +def semantics(row): + need(isinstance(row, dict) and isinstance(row.get("question"), dict), "Record/question objects required") + text(row.get("id")); text(row.get("state_text")) + q = row["question"]; text(q.get("text")) + kind = str(q.get("type", "")).lower() + need(kind in KINDS, "Question type must be Choice, Noul or Score") + if kind == "noul": + options = [{"id": "no", "text": q.get("false_criterion", DEFAULT_NO)}, + {"id": "yes", "text": q.get("true_criterion", DEFAULT_YES)}] + else: + options = q.get("options" if kind == "choice" else "levels") + need(isinstance(options, list) and 2 <= len(options) <= 255, "Require 2..255 candidates") + for option in options: + need(isinstance(option, dict), "Candidate object required") + text(option.get("id")); text(option.get("text")) + ids = [o["id"] for o in options] + need(len(set(ids)) == len(ids), "Candidate IDs must be unique") + values = [float(o["value"]) for o in options] if kind == "score" else list(range(len(ids))) + need(all(math.isfinite(v) for v in values) and all(a < b for a, b in zip(values, values[1:])), "Increasing finite Score values") + gold = row.get("target") + need(isinstance(gold, dict), "Explicit target required") + if kind == "noul": + need(set(gold) == {"probability"}, "Noul target is probability of yes") + yes = float(gold["probability"]); target = [1 - yes, yes] + elif set(gold) == {"probabilities"}: + target = [float(p) for p in gold["probabilities"]] + else: + need(kind == "choice" and set(gold) == {"choice_id"} and gold["choice_id"] in ids, + "Choice needs choice_id/probabilities; Score needs probabilities (one-hot is hard)") + target = [float(cid == gold["choice_id"]) for cid in ids] + need(len(target) == len(ids) and all(math.isfinite(p) and 0 <= p <= 1 for p in target) + and abs(sum(target) - 1) <= 1e-6, "Normalized finite hard/soft target required") + hard = row.get("hard_target_id") + need(hard is None or hard in ids, "hard_target_id must be an original candidate ID") + return kind, options, values, target + + +def input_signature(row): + kind, options, values, _ = semantics(row) + descriptions = [text(o["text"]) for o in options] + if kind == "choice": + descriptions.sort() # Reordering and opaque IDs cannot hide exact split overlap. + return digest({"kind": kind, "state": text(row["state_text"]), + "question": text(row["question"]["text"]), "descriptions": descriptions, + "values": values if kind == "score" else None}) + + +def model_row(row): + # External hard evaluation labels are never passed to the collator/model. + return copy.deepcopy({k: v for k, v in row.items() if k != "hard_target_id"}) + + +def read_jsonl(path): + rows = [] + with Path(path).open(encoding="utf-8") as stream: + for number, line in enumerate(stream, 1): + need(bool(line.strip()), f"Blank JSONL line {number}") + row = json.loads(line); semantics(row); rows.append(row) + need(bool(rows) and len({r["id"] for r in rows}) == len(rows), "Nonempty split with unique IDs required") + return rows + + +def load_splits(train_path, dev_path, selection): + train, dev = read_jsonl(train_path), read_jsonl(dev_path) + need(not ({r["id"] for r in train} & {r["id"] for r in dev}), "TRAIN/DEV IDs overlap") + need(not ({input_signature(r) for r in train} & {input_signature(r) for r in dev}), + "TRAIN/DEV normalized complete inputs overlap") + # When both fields are supplied, also reject same-source dependency components. + components = lambda rows: {(r["source_id"], r["component_id"]) for r in rows + if r.get("source_id") and r.get("component_id")} + need(not (components(train) & components(dev)), "TRAIN/DEV source components overlap") + if selection == "hard-accuracy": + need(all(r.get("hard_target_id") is not None for r in dev), + "Hard-accuracy selection requires explicit hard_target_id on EVERY DEV row") + return train, dev + + +def make_schedule(rows, epochs, logical_batch, seed, max_steps=None): + steps, epoch_ends = [], [] + for epoch in range(epochs): + order = sorted(range(len(rows)), key=lambda i: (digest([seed, epoch, rows[i]["id"]]), rows[i]["id"])) + steps.extend([order[i:i + logical_batch] for i in range(0, len(order), logical_batch)]) + epoch_ends.append(len(steps)) + if max_steps is not None: + need(1 <= max_steps <= len(steps), "max-steps must fit the declared epochs; no implicit extra epochs") + steps = steps[:max_steps] + return steps, sorted(set([0, len(steps)] + [s for s in epoch_ends if s <= len(steps)])) + + +def admit(collator, rows): + need(collator.max_length == 1024 and collator.state_truncation == "error", "Full1K collator required") + evidence = {} + for row in rows: + e = collator.encode(model_row(row), labeled=True) + kind, options, values, target = semantics(row) + need(e["id"] == row["id"] and e["kind"] == kind and e["target"] == target + and e["candidate_ids"] == [o["id"] for o in options] and e["values"] == values, + "CLI schema and loaded native collator disagree") + need(e["state_tokens_original"] == e["state_tokens_kept"] and e["input_tokens"] == len(e["ids"]) <= 1024, + "Complete input exceeds1K or was truncated; no rows are silently dropped") + evidence[row["id"]] = {"encoded_sha256": digest(e), "input_tokens": len(e["ids"]), + "token_ids_sha256": digest(e["ids"]), "record_sha256": digest(row)} + return evidence diff --git a/decision_finetune/metrics.py b/decision_finetune/metrics.py new file mode 100644 index 0000000000000000000000000000000000000000..1d50e225345c400dcdf94bc4ed4f7ed9ef8ee2d5 --- /dev/null +++ b/decision_finetune/metrics.py @@ -0,0 +1,61 @@ +"""Explicit DEV metrics/selectors; no external benchmark or hidden retention gates.""" +import math +from collections import defaultdict +from .data import need, semantics + + +def summarize(rows, predictions): + need(len(rows) == len(predictions) > 0, "Complete DEV predictions required") + groups = defaultdict(list) + for row, p in zip(rows, predictions): + kind, options, values, target = semantics(row) + ids = [o["id"] for o in options]; z = p["logits"]; probs = p["probabilities"] + need(p["id"] == row["id"] and p["type"].lower() == kind and p["candidate_ids"] == ids, + "Prediction ID/type/order mismatch") + need(len(z) == len(probs) == len(ids) and all(math.isfinite(v) for v in z + probs) + and all(0 <= v <= 1 for v in probs) and abs(sum(probs) - 1) <= 1e-5, "Invalid probabilities/logits") + peak = max(z); logsum = peak + math.log(sum(math.exp(v - peak) for v in z)) + need(max(abs(math.exp(v - logsum) - p) for v, p in zip(z, probs)) <= 2e-5, + "Probabilities disagree with saved logits") + # NLL from logits avoids an arbitrary probability floor. + nll = sum(t * (logsum - v) for t, v in zip(target, z)) + native_hat = max(range(len(probs)), key=probs.__getitem__) + # Original typed evaluation convention, distinct from native first argmax at exact ties. + hat = int(probs[1] >= .5) if kind == "noul" else native_hat + item = {"nll": nll, "hard_accuracy": None if row.get("hard_target_id") is None else float(ids[hat] == row["hard_target_id"]), + "native_argmax_accuracy": None if row.get("hard_target_id") is None else float(ids[native_hat] == row["hard_target_id"])} + if kind == "score": + pc = tc = rps = 0. + for a, b in zip(probs[:-1], target[:-1]): + pc += a; tc += b; rps += (pc - tc) ** 2 + item["rps"] = rps / (len(ids) - 1) + item["expected_value_squared_error"] = (sum(a * v for a, v in zip(probs, values)) - sum(a * v for a, v in zip(target, values))) ** 2 + groups[kind].append(item) + def report(items): + result = {"rows": len(items), "soft_nll": sum(x["nll"] for x in items) / len(items)} + result["hard_accuracy"] = (sum(x["hard_accuracy"] for x in items) / len(items) + if all(x["hard_accuracy"] is not None for x in items) else None) + result["native_argmax_accuracy_diagnostic"] = (sum(x["native_argmax_accuracy"] for x in items) / len(items) + if all(x["native_argmax_accuracy"] is not None for x in items) else None) + if all("rps" in x for x in items): + result.update(rps=sum(x["rps"] for x in items) / len(items), + expected_value_rmse=math.sqrt(sum(x["expected_value_squared_error"] for x in items) / len(items))) + return result + by_type = {kind: report(items) for kind, items in groups.items()} + return {"by_type": by_type, "row": report([x for items in groups.values() for x in items]), + "macro_soft_nll": sum(x["soft_nll"] for x in by_type.values()) / len(by_type), + "macro_weighting": "equal weight for each type present in DEV; row mean within type", + "hard_rule": "Noul p_yes>=0.5; Choice/Score original-order first argmax", + "native_argmax_diagnostic": "original-order first argmax for every type; Noul exact tie no"} + + +def better(candidate, best, selection, tolerance=1e-8): + if best is None: + return True + a, b = candidate["metrics"], best["metrics"] + if selection == "macro-nll": + return a["macro_soft_nll"] < b["macro_soft_nll"] - tolerance + need(selection == "hard-accuracy" and a["row"]["hard_accuracy"] is not None + and b["row"]["hard_accuracy"] is not None, "Explicit complete hard labels required") + delta = a["row"]["hard_accuracy"] - b["row"]["hard_accuracy"] + return delta > tolerance or (abs(delta) <= tolerance and a["row"]["soft_nll"] < b["row"]["soft_nll"] - tolerance) diff --git a/decision_finetune/run.py b/decision_finetune/run.py new file mode 100644 index 0000000000000000000000000000000000000000..b549798684470a69289952557e74433b0d683e56 --- /dev/null +++ b/decision_finetune/run.py @@ -0,0 +1,237 @@ +"""Single-device training orchestration over the public API, with no research paths.""" +import gc +import importlib.metadata +import json +import os +from pathlib import Path +import random +import sys +import time + +from . import data, metrics, state + + +def source_identity(): + import decision_runtime + roots = {"decision_finetune": Path(__file__).parent, + "decision_runtime": Path(decision_runtime.__file__).parent} + return {package + "/" + p.name: data.file_sha(p) for package, root in roots.items() + for p in sorted(root.glob("*.py")) if p.name not in ("checks.py", "bridge_checks.py")} + + +def setup(config): + import numpy as np + import torch + data.need(torch.version.hip is not None and torch.cuda.is_available() and torch.cuda.device_count() == 1, + "Expose exactly one real AMD ROCm device; CPU/NVIDIA training is unsupported") + torch.cuda.set_device(0); torch.set_num_threads(config["cpu_threads"]) + torch.backends.cuda.matmul.allow_tf32 = False; torch.backends.cudnn.allow_tf32 = False + torch.backends.mha.set_fastpath_enabled(False) + torch.use_deterministic_algorithms(config["deterministic_algorithms"]) + torch.cuda.set_per_process_memory_fraction(min(1., config["max_reserved_gib"] * 2**30 / torch.cuda.get_device_properties(0).total_memory)) + random.seed(config["seed"]); np.random.seed(config["seed"]); torch.manual_seed(config["seed"]) + environment = {"python": sys.version.split()[0], "rocm": torch.version.hip, + "gpu": torch.cuda.get_device_name(0), + "versions": {n: importlib.metadata.version(n) for n in ("torch", "transformers", "tokenizers", "safetensors", "numpy")}} + return torch, environment + + +def check_memory(torch, config): + data.need(torch.cuda.max_memory_reserved() <= config["max_reserved_gib"] * 2**30, "Reserved GPU memory cap exceeded") + + +def evaluation(api, native, dev, admission, out, step, torch): + # Parameter objects and optimizer state survive policy restoration unchanged. + api.restore_native_policy_for_export(native) + predictions = api.predict(native, [data.model_row(r) for r in dev], batch_size=8) + for row, p in zip(dev, predictions): + data.need(p["input_tokens"] == admission[row["id"]]["input_tokens"] <= 1024 + and p["state_tokens_original"] == p["state_tokens_kept"], "DEV admission mismatch") + point = {"step": step, "metrics": metrics.summarize(dev, predictions)} + state.write_json(out / f"evaluation-{step:06d}.json", point) + with (out / f"predictions-{step:06d}.jsonl").open("x", encoding="utf-8") as stream: + for p in predictions: + stream.write(json.dumps(p, ensure_ascii=False, allow_nan=False) + "\n") + api.configure_training(native, max_input_tokens=1024) + return point + + +def update(api, native, optimizer, rows, admission, config, step, total, torch): + api.set_training_mode(native, True); optimizer.zero_grad(set_to_none=True) + scale = state.learning_rate_scale(step, total, config["warmup_steps"]) + for group in optimizer.param_groups: + base = config["encoder_lr"] if group["name"].endswith("encoder") else config["head_lr"] + group["lr"] = state.learning_rate(step, total, config["warmup_steps"], base, config["lr_min"]) + sums = [0., 0., 0.]; tokens = padded = 0 + for start in range(0, len(rows), config["micro_batch_size"]): + micro = rows[start:start + config["micro_batch_size"]] + batch, encoded = native.collator([data.model_row(r) for r in micro], labeled=True, device="cuda:0") + data.need(all(data.digest(e) == admission[r["id"]]["encoded_sha256"] for r, e in zip(micro, encoded)), + "Actual training tokens/labels changed after complete admission") + tokens += sum(e["input_tokens"] for e in encoded); padded += len(micro) * batch["input_ids"].shape[1] + with torch.autocast("cuda", dtype=torch.bfloat16): + logits = api.forward_for_training(native, batch) + losses = api.loss_for_training(native, logits, batch, score_rps_weight=config["score_rps_weight"]) + (losses[0].sum() / len(rows)).backward() + for i, loss in enumerate(losses): + sums[i] += float(loss.detach().sum()) + del logits, losses, batch + check_memory(torch, config) + present = {r["question"]["type"].lower() for r in rows} + proof = api.gradient_report(native, present_types=present) if step == 1 else None + # AdamW must never decay a path absent from the logical batch. + for group in optimizer.param_groups: + for p in group["params"]: + if group["name"].split(".")[0] not in present: + data.need(p.grad is None, "Absent type has stale gradient") + elif p.grad is not None: + data.need(torch.isfinite(p.grad).all().item(), "Nonfinite gradient") + active = [p for g in optimizer.param_groups for p in g["params"]] + norm = float(torch.nn.utils.clip_grad_norm_(active, config["clip_norm"], error_if_nonfinite=True)) + optimizer.step(); optimizer.zero_grad(set_to_none=True) + check_memory(torch, config) + return {"step": step, "record_ids": [r["id"] for r in rows], "logical_rows": len(rows), + "denominator": len(rows), "micro_batches": (len(rows) + config["micro_batch_size"] - 1) // config["micro_batch_size"], + "mean_loss": sums[0] / len(rows), "mean_ce": sums[1] / len(rows), "mean_rps_all_types_diagnostic": sums[2] / len(rows), + "effective_tokens": tokens, "padded_tokens": padded, "gradient_norm_before_clip": norm, + "clipped": norm > config["clip_norm"], "lr_scale": scale, + "learning_rates": {g["name"]: g["lr"] for g in optimizer.param_groups}, "first_step_gradient_proof": proof} + + +def execute(args, config): + import decision_runtime as api + train, dev = data.load_splits(args.train, args.dev, config["selection"]) + schedule, default_points = data.make_schedule(train, config["epochs"], config["logical_batch_size"], config["seed"], config["max_steps"]) + points = default_points if config["eval_steps"] is None else config["eval_steps"] + data.need(points == sorted(set(points)) and points[0] == 0 and points[-1] == len(schedule) + and all(0 <= s <= len(schedule) for s in points), "Evaluation steps must be ordered, unique, and include0/final") + config = {**config, "eval_steps": points, "total_steps": len(schedule)} + data.need(args.stop_after_step is None or 1 <= args.stop_after_step < len(schedule), + "A scheduling pause must be an original update before final; it does not shorten training") + config["warmup_steps"] = (int(len(schedule) * config["warmup_ratio"]) if config["warmup_steps"] is None else config["warmup_steps"]) + data.need(0 <= config["warmup_steps"] < len(schedule), "Warmup must be shorter than the original full schedule") + train_sha, dev_sha = data.file_sha(args.train), data.file_sha(args.dev) + torch, environment = setup(config) + identity = {"config": config, "sources": source_identity(), "native_manifest_sha256": args.manifest_sha256, + "train_sha256": train_sha, "dev_sha256": dev_sha, "schedule_sha256": data.digest(schedule), "environment": environment} + out = Path(args.output).resolve() + data.need(not (out / "COMPLETE.json").exists(), "Completed runs are immutable") + if args.resume: + data.need(out.is_dir() and state.read_json(out / "RUN.json")["identity"] == identity, + "Resume must use the original output/config/data/source/environment") + data.need(Path(args.resume).resolve().is_relative_to(out / "attempts"), "Resume checkpoint must belong to this run") + else: + data.need(not out.exists(), "Fresh output required; use explicit --resume for an interrupted run") + out.mkdir(parents=True); (out / "attempts").mkdir() + state.write_json(out / "RUN.json", {"identity": identity, "train_path": str(Path(args.train).resolve()), + "dev_path": str(Path(args.dev).resolve()), "native_path": str(Path(args.native).resolve()), + "model_validation": "CLI integration not yet externally validated"}) + state.write_json(out / "SCHEDULE.json", {"train_ids": [r["id"] for r in train], "row_indices": schedule}) + import fcntl + with (out / "LOCK").open("a") as lock: + fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + attempt = out / "attempts" / f"{len(list((out / 'attempts').iterdir())) + 1:04d}" + attempt.mkdir(); started = time.monotonic() + try: + native = api.load_native(args.native, expected_manifest_sha256=args.manifest_sha256) + # Same native packing; the user profile lowers only the admission cap. + native.collator = type(native.collator)(native.collator.tokenizer, max_length=1024, state_truncation="error") + admitted_train, admitted_dev = data.admit(native.collator, train), data.admit(native.collator, dev) + admission = {"train": admitted_train, "dev": admitted_dev, "truncated": 0, "dropped": 0, + "full_input_limit": 1024, "identity_sha256": data.digest(identity)} + state.write_json(attempt / "ADMISSION.json", admission) + api.configure_training(native, max_input_tokens=1024) + frozen = api.frozen_snapshot(native) + optimizer = torch.optim.AdamW(api.optimizer_groups(native, encoder_lr=config["encoder_lr"], head_lr=config["head_lr"]), + weight_decay=config["weight_decay"], fused=False, foreach=False) + layout = state.optimizer_layout(native.model, optimizer) + parameter_ids = [id(p) for g in optimizer.param_groups for p in g["params"]] + curve, best, step = [], None, 0 + if args.resume: + meta, rng = state.load_checkpoint(args.resume, args.resume_sha256, native, optimizer, identity, torch) + api.assert_frozen(native, frozen, content=True, versions=False) + data.need(meta["frozen_hashes"] == {n: x["sha256"] for n, x in frozen.items()}, "Checkpoint frozen content differs from parent") + frozen = api.frozen_snapshot(native) + curve, best, step = meta["curve"], meta["best"], meta["step"] + data.need(0 <= step <= len(schedule) and all(p["step"] <= step for p in curve), "Resume position invalid") + data.need(args.stop_after_step is None or args.stop_after_step > step, "Pause must follow the resumed checkpoint") + if best is not None and best.get("checkpoint_manifest_sha256") is None: + data.need(Path(best["checkpoint"]).resolve() == Path(args.resume).resolve() + and best["step"] == step, "Missing previous best checkpoint identity") + best = {**best, "checkpoint_manifest_sha256": args.resume_sha256} + state.restore_rng(torch, rng) # Last potentially RNG-sensitive setup operation. + def checkpoint(current): + nonlocal best + directory = attempt / f"checkpoint-{current:06d}" + if best is not None and best.get("checkpoint") is None: + best = {**best, "checkpoint": str(directory.resolve())} + api.assert_frozen(native, frozen, content=True) + receipt = state.save_checkpoint(directory, native, optimizer, + {"identity": identity, "step": current, "curve": curve, "best": best, + "frozen_hashes": {n: x["sha256"] for n, x in frozen.items()}}, torch) + if best is not None and best["checkpoint"] == str(directory.resolve()): + best = {**best, "checkpoint_manifest_sha256": receipt["manifest_sha256"]} + state.write_json(out / "LATEST.json", receipt) + return receipt + def evaluate(current): + nonlocal best + api.assert_frozen(native, frozen, content=True) + opt_states = {id(p): id(s) for p, s in optimizer.state.items()} + point = evaluation(api, native, dev, admitted_dev, attempt, current, torch) + data.need(parameter_ids == [id(p) for g in optimizer.param_groups for p in g["params"]] + and layout == state.optimizer_layout(native.model, optimizer) + and opt_states == {id(p): id(s) for p, s in optimizer.state.items()}, "Evaluation broke optimizer identity") + curve.append(point) + if metrics.better(point, best, config["selection"], config["selection_tolerance"]): + best = {**point, "checkpoint": None} + checkpoint(current) + if not args.resume: + evaluate(0) + with (attempt / "TRAIN_LOG.jsonl").open("x", encoding="utf-8") as log: + for current in range(step + 1, len(schedule) + 1): + report = update(api, native, optimizer, [train[i] for i in schedule[current - 1]], admitted_train, config, current, len(schedule), torch) + api.assert_frozen(native, frozen) + log.write(json.dumps(report, allow_nan=False) + "\n"); log.flush() + if current in points: + evaluate(current) + elif config["save_every"] and current % config["save_every"] == 0: + checkpoint(current) + if current == args.stop_after_step: + if current not in points and not (config["save_every"] and current % config["save_every"] == 0): + checkpoint(current) + paused = {"status": "PAUSED_DECISION_FINETUNE", "identity": identity, "step": current, + "checkpoint": state.read_json(out / "LATEST.json"), "total_steps": len(schedule), + "scheduling_pause_not_selection": True, "native_exported": False} + state.write_json(attempt / "PAUSED.json", paused) + return paused + data.need([p["step"] for p in curve] == points and best is not None, "Missing evaluation/selected checkpoint") + # A resume after final checkpoint can finish export without replaying training. + del optimizer; gc.collect(); torch.cuda.empty_cache() + from safetensors.torch import load_file + selected_dir = Path(best["checkpoint"]) + selected_sha = best["checkpoint_manifest_sha256"] + _, selected_meta = state.verify_checkpoint(selected_dir, selected_sha) + data.need(selected_meta["identity"] == identity and selected_meta["step"] == best["step"], "Selected checkpoint identity") + native.model.load_state_dict(load_file(str(selected_dir / "model.safetensors"), device="cpu"), strict=True) + api.assert_frozen(native, frozen, content=True, versions=False) + api.restore_native_policy_for_export(native) + export_path = attempt / "selected-native" + api.export_native(native, export_path, provenance={"cli": "decision_finetune", "identity": identity, + "selected_step": best["step"], "selection": best["metrics"], "total_steps": len(schedule), + "selected0_no_adaptation": best["step"] == 0, "checkpoint_manifest_sha256": selected_sha, + "release_or_quality_qualification": False}) + data.need(data.file_sha(args.train) == train_sha and data.file_sha(args.dev) == dev_sha + and source_identity() == identity["sources"], "Source/data changed during run") + check_memory(torch, config) + result = {"status": "COMPLETE_DECISION_FINETUNE", "identity": identity, "curve": curve, + "selected": best, "selected0_no_adaptation": best["step"] == 0, + "native": str(export_path), "native_manifest_sha256": data.file_sha(export_path / "MANIFEST.json"), + "attempt_elapsed_seconds": time.monotonic() - started, + "peak_reserved_bytes": torch.cuda.max_memory_reserved(), "resumed": bool(args.resume), + "independent_fresh_reload_performed": False, "release_qualified": False} + state.write_json(out / "COMPLETE.json", result) + return result + except BaseException as error: + state.write_json(attempt / "FAILURE.json", {"exception_type": type(error).__name__, "message": str(error), + "resume_requires_explicit_checkpoint": True}) + raise diff --git a/decision_finetune/state.py b/decision_finetune/state.py new file mode 100644 index 0000000000000000000000000000000000000000..bffd96178c98e43b037ecd8291dae40763ae8fa9 --- /dev/null +++ b/decision_finetune/state.py @@ -0,0 +1,110 @@ +"""Atomic update-boundary checkpoints with named optimizer mapping and RNG state.""" +import json +import os +from pathlib import Path +import random +import tempfile +from .data import need, file_sha + + +def write_json(path, value): + path = Path(path) + fd, temporary = tempfile.mkstemp(prefix="." + path.name + ".", suffix=".tmp", dir=path.parent) + try: + with os.fdopen(fd, "w", encoding="utf-8") as stream: + json.dump(value, stream, ensure_ascii=False, sort_keys=True, indent=2, allow_nan=False) + stream.write("\n"); stream.flush(); os.fsync(stream.fileno()) + os.replace(temporary, path) + finally: + if os.path.exists(temporary): + os.unlink(temporary) + + +def read_json(path): + return json.loads(Path(path).read_text(encoding="utf-8")) + + +def optimizer_layout(model, optimizer): + names = {id(p): name for name, p in model.named_parameters()} + layout = [{"name": g["name"], "parameters": [names[id(p)] for p in g["params"]]} + for g in optimizer.param_groups] + flattened = [n for group in layout for n in group["parameters"]] + need(len(set(flattened)) == len(flattened), "Optimizer parameter alias") + return layout + + +def rng_state(torch): + import numpy as np + n = np.random.get_state() + return {"python": random.getstate(), "numpy": [n[0], n[1].tolist(), n[2], n[3], n[4]], + "torch_cpu": torch.get_rng_state(), "torch_cuda": torch.cuda.get_rng_state_all()} + + +def restore_rng(torch, saved): + import numpy as np + need(len(saved["torch_cuda"]) == torch.cuda.device_count(), "CUDA RNG device count changed") + random.setstate(saved["python"]) + n = saved["numpy"]; np.random.set_state((n[0], np.asarray(n[1], dtype=np.uint32), n[2], n[3], n[4])) + torch.set_rng_state(saved["torch_cpu"]); torch.cuda.set_rng_state_all(saved["torch_cuda"]) + + +def save_checkpoint(path, native, optimizer, metadata, torch): + from safetensors.torch import save_file + path = Path(path); temporary = path.with_name(path.name + ".tmp") + need(not path.exists() and not temporary.exists(), "Checkpoint destinations are immutable") + temporary.mkdir() + # CPU copies are serialization only; no CPU model or forward is constructed. + save_file({n: t.detach().cpu().contiguous() for n, t in native.model.state_dict().items()}, str(temporary / "model.safetensors")) + torch.save({"optimizer": optimizer.state_dict(), "rng": rng_state(torch)}, temporary / "state.pt") + write_json(temporary / "metadata.json", {**metadata, "optimizer_layout": optimizer_layout(native.model, optimizer)}) + files = {p.name: {"sha256": file_sha(p), "bytes": p.stat().st_size} for p in sorted(temporary.iterdir())} + write_json(temporary / "MANIFEST.json", {"schema": "decision.finetune.checkpoint.v1", "files": files}) + os.replace(temporary, path) + return {"path": str(path.resolve()), "manifest_sha256": file_sha(path / "MANIFEST.json")} + + +def verify_checkpoint(path, expected_sha): + path = Path(path).resolve(strict=True) + need(file_sha(path / "MANIFEST.json") == expected_sha, "Explicit checkpoint manifest SHA mismatch") + manifest = read_json(path / "MANIFEST.json") + need(manifest["schema"] == "decision.finetune.checkpoint.v1" + and set(manifest["files"]) == {"model.safetensors", "state.pt", "metadata.json"}, "Checkpoint roster mismatch") + need({p.name for p in path.iterdir()} == set(manifest["files"]) | {"MANIFEST.json"}, "Unexpected checkpoint file") + for name, ref in manifest["files"].items(): + p = path / name + need(not p.is_symlink() and p.is_file() and p.stat().st_size == ref["bytes"] and file_sha(p) == ref["sha256"], "Checkpoint file changed") + return path, read_json(path / "metadata.json") + + +def load_checkpoint(path, expected_sha, native, optimizer, identity, torch): + from safetensors.torch import load_file + path, meta = verify_checkpoint(path, expected_sha) + need(meta["identity"] == identity, "Resume source/data/native/config/schedule/environment identity changed") + need(meta["optimizer_layout"] == optimizer_layout(native.model, optimizer), "Named optimizer group/order mismatch") + parameter_ids = [id(p) for p in native.model.parameters()] + native.model.load_state_dict(load_file(str(path / "model.safetensors"), device="cpu"), strict=True) + need([id(p) for p in native.model.parameters()] == parameter_ids, "Loading replaced parameters") + # Only trust locally produced, explicitly hash-verified checkpoints. No arbitrary pickle globals. + state = torch.load(path / "state.pt", map_location="cpu", weights_only=True) + optimizer.load_state_dict(state["optimizer"]) + need(meta["optimizer_layout"] == optimizer_layout(native.model, optimizer), "Restored optimizer mapping drift") + for p, item in optimizer.state.items(): + need(all(item[k].shape == p.shape and torch.isfinite(item[k]).all().item() for k in ("exp_avg", "exp_avg_sq")), "Invalid AdamW moments") + need(0 < float(item["step"]) <= meta["step"], "Invalid per-parameter AdamW step") + return meta, state["rng"] + + +def learning_rate_scale(step, total, warmup): + import math + need(1 <= step <= total and 0 <= warmup < total, "Invalid original LR horizon") + if warmup and step <= warmup: + return step / warmup + progress = ((step - warmup) / (total - warmup) if warmup + else (step - 1) / (total - 1) if total > 1 else 0.) + return .5 * (1 + math.cos(math.pi * progress)) + + +def learning_rate(step, total, warmup, base, minimum): + need(0 <= minimum <= base, "LR floor must fit every optimizer group") + scale = learning_rate_scale(step, total, warmup) + return base * scale if warmup and step <= warmup else minimum + (base - minimum) * scale diff --git a/decision_inference/__init__.py b/decision_inference/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..dffa07f800089f30db94e23733be833651cbc124 --- /dev/null +++ b/decision_inference/__init__.py @@ -0,0 +1,3 @@ +from .profile import Complete1KCollator, MAX_INPUT_TOKENS, predict_1k + +__all__ = ["Complete1KCollator", "MAX_INPUT_TOKENS", "predict_1k"] diff --git a/decision_inference/profile.py b/decision_inference/profile.py new file mode 100644 index 0000000000000000000000000000000000000000..6463cc63e7fcd25ac328bc528140fc04489827e5 --- /dev/null +++ b/decision_inference/profile.py @@ -0,0 +1,55 @@ +"""Complete-input 1K product profile around an unchanged native runtime.""" +from dataclasses import replace + +MAX_INPUT_TOKENS = 1024 + + +class Complete1KCollator: + """Use the native collator's actual length error, never truncate fields. + + Constructible with a tokenizer before any model is loaded. A whole request + is admitted before predict, and every physical batch before tensor assembly. + """ + def __init__(self, native_collator): + self.base = type(native_collator)(native_collator.tokenizer, + max_length=MAX_INPUT_TOKENS, + state_truncation="error") + self.tokenizer = self.base.tokenizer + self.marker, self.pad = self.base.marker, self.base.pad + self.tensor_batches = 0 + + def tokens(self, text): + return self.base.tokens(text) + + def encode(self, row, labeled=False): + encoded = self.base.encode(row, labeled=labeled) + if (encoded["input_tokens"] > MAX_INPUT_TOKENS + or encoded["state_tokens_original"] != encoded["state_tokens_kept"]): + raise ValueError("Native collator violated the complete-input 1K profile") + return encoded + + def admit(self, records): + return [self.encode(row, labeled=False) for row in records] + + def __call__(self, records, labeled=False, device="cpu"): + records = list(records) + # All encodes finish before delegating any tensor allocation. + for row in records: + self.encode(row, labeled=labeled) + self.tensor_batches += 1 + return self.base(records, labeled=labeled, device=device) + + +def predict_1k(native, records, *, batch_size=8): + """FP32 native inference, B1..8, complete state/question/all candidates ≤1K. + + The native object/config/runtime/weights are not mutated. No claim of + candidate permutation invariance or task quality is made by this wrapper. + """ + if type(batch_size) is not int or not 1 <= batch_size <= 8: + raise ValueError("The product profile permits integer batch sizes 1..8") + records = list(records) + guard = Complete1KCollator(native.collator) + guard.admit(records) # Reject any oversized request before the first forward. + from decision_runtime import predict + return predict(replace(native, collator=guard), records, batch_size=batch_size) diff --git a/decision_runtime/__init__.py b/decision_runtime/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..0825244b03410d51a974f4a713aec422e982bcfd --- /dev/null +++ b/decision_runtime/__init__.py @@ -0,0 +1,11 @@ +"""Source-only portable training adapter; real AMD validation is still pending.""" +from .native import load_native, export_native, predict +from .training import (configure_training, set_training_mode, forward_for_training, + loss_for_training, optimizer_groups, inventory, + frozen_snapshot, assert_frozen, gradient_report, + restore_native_policy_for_export) + +__all__ = ["load_native", "export_native", "predict", "configure_training", + "set_training_mode", "forward_for_training", "loss_for_training", + "optimizer_groups", "inventory", "frozen_snapshot", "assert_frozen", + "gradient_report", "restore_native_policy_for_export"] diff --git a/decision_runtime/_compat.py b/decision_runtime/_compat.py new file mode 100644 index 0000000000000000000000000000000000000000..9773445c18578fb05e86acfda7f0cc25748a48eb --- /dev/null +++ b/decision_runtime/_compat.py @@ -0,0 +1,20 @@ +"""Supported self-contained runtime revision; weights may differ. No path bindings.""" +RUNTIME_SHA256 = {'__init__.py': '1afb9dbcfc379f28049486fe6acb7acfdaa8d16b6773d6084ca4b82ead26f010', + 'artifacts.py': 'c98adaf6d782e9fc592cd78b1307d63faffa9ba622b9d508c63337c29156e4d3', + 'contract.py': '51a24800792bb3e5bf2a11f50f7bc384770641f4f44ed46277dc01e891bf4726', + 'infer.py': '9d14841935c836a0765c705d5a24e8fa97437ae35fff65bc6e690b397f0d0315', + 'model.py': '8fe91e2a77f372d32117e62281ccb58a054c144228b73b4987f1e1a3fca9ee5b', + 'packing.py': 'f1dab7f032f4aaab27118606397ca67d71b1f4a2ba49e65bebcfd83a64b27819', + 'policy/__init__.py': '0cb0bad7d3d0f258ecf95f52297ee8733a75f8504cabb86b9aea4b8256885114', + 'policy/artifacts.py': '5838d2ea747d912789f3fc813a212af919cbc24ee185073576d9fbe32ed31cf2', + 'policy/contract.py': '0b8eeeeebe9e367564a3c57f48364c5e94ae060f759e220b1326caf152276b67', + 'policy/infer.py': '672bfae798060d51fcccac03143ac881fe7512893ffdacdb36b0618d4b160896', + 'policy/model.py': 'eaeebac8fd6bd96243a5c4b6225c86ee3359c067784f1595979ee603cba9acca', + 'policy/packing.py': 'f1dab7f032f4aaab27118606397ca67d71b1f4a2ba49e65bebcfd83a64b27819', + 'policy/reference/__init__.py': 'c8d6fd86207752407f94437544a97266b9529f88d2b341d35aafcd911c942b28', + 'policy/reference/artifacts.py': '3f59428b7a05608ac01c73c7db89c38fcc3e2c5030219df8d053af92b32e6159', + 'policy/reference/infer.py': '672bfae798060d51fcccac03143ac881fe7512893ffdacdb36b0618d4b160896', + 'policy/reference/model.py': '733ea4a48ea03089dac8c3e6a175920704fac50f93313b0214cec4052f25bd26', + 'policy/reference/modernbert_sdpa_layout.py': '0fb3a22db93ad76e30dfbfb3011de442149d97c565e139a3a55738287a1fbbc8', + 'policy/reference/packing.py': 'f1dab7f032f4aaab27118606397ca67d71b1f4a2ba49e65bebcfd83a64b27819', + 'training_policy.py': '6f7fad91c5089b304a1d637b594027a9379a63600792766ce0abca6b24bd2ec5'} diff --git a/decision_runtime/native.py b/decision_runtime/native.py new file mode 100644 index 0000000000000000000000000000000000000000..64942923580c2a9dbe8eb778a9e90826be149463 --- /dev/null +++ b/decision_runtime/native.py @@ -0,0 +1,152 @@ +"""Load the export's own verified runtime; never import an experiment package.""" +from dataclasses import dataclass, field +from pathlib import Path, PurePosixPath +import copy +import hashlib +import importlib +import importlib.util +import json +import re +import sys +import uuid + +from ._compat import RUNTIME_SHA256 + + +def sha256(path): + digest = hashlib.sha256() + with Path(path).open("rb") as stream: + for chunk in iter(lambda: stream.read(1 << 20), b""): + digest.update(chunk) + return digest.hexdigest() + + +def verify_files(directory, expected_manifest_sha256): + """Stdlib pre-import verification, including every weight and runtime file.""" + root = Path(directory).resolve(strict=True) + if not re.fullmatch(r"[0-9a-f]{64}", expected_manifest_sha256 or ""): + raise ValueError("An explicit trusted MANIFEST SHA256 is required") + if sha256(root / "MANIFEST.json") != expected_manifest_sha256: + raise ValueError("Native manifest identity mismatch") + manifest = json.loads((root / "MANIFEST.json").read_text()) + files = manifest.get("files") + if manifest.get("schema") != "decision.files.v1" or not isinstance(files, dict): + raise ValueError("Unsupported native manifest") + for name, ref in files.items(): + if not isinstance(name, str) or not isinstance(ref, dict): + raise ValueError("Invalid manifest entry") + rel = PurePosixPath(name) + if (rel.is_absolute() or ".." in rel.parts + or rel.as_posix() != name or "\\" in name or not rel.parts): + raise ValueError("Unsafe manifest path") + path = root / name + if (not path.is_file() or path.is_symlink() + or set(ref) != {"bytes", "sha256"} + or path.stat().st_size != ref["bytes"] or sha256(path) != ref["sha256"]): + raise ValueError("Native file mismatch: " + name) + actual = set() + for path in root.rglob("*"): + if path.is_symlink(): + raise ValueError("Materialized native required; symlinks are not supported") + rel = path.relative_to(root) + if path.is_file() and rel.as_posix() != "MANIFEST.json": + if "__pycache__" in rel.parts and path.suffix == ".pyc": + continue + actual.add(rel.as_posix()) + if actual != set(files): + raise ValueError("Native file roster mismatch") + runtime = {name: ref["sha256"] for name, ref in files.items() if name.endswith(".py")} + if runtime != RUNTIME_SHA256: + raise ValueError("Unsupported runtime revision; architecture name alone is insufficient") + return root, manifest + + +@dataclass +class Native: + directory: Path + manifest_sha256: str + model: object + collator: object + config: dict + model_api: object + contract: object + artifacts: object + training_state: dict | None = None + last_training_provenance: dict | None = None + package_name: str = field(default="", repr=False) + + +def _amd_device(device): + import torch + selected = torch.device(device) + if selected.type != "cuda" or torch.version.hip is None or not torch.cuda.is_available(): + raise ValueError("This runtime requires a real ROCm CUDA device; no CPU fallback") + if selected.index is None: + selected = torch.device("cuda", torch.cuda.current_device()) + return selected + + +def load_native(directory, *, expected_manifest_sha256, device="cuda:0"): + """Load any compatible weight export with the pinned public runtime revision. + + Verification precedes executing bundled Python. This function loads a model; + importing decision_runtime or running its stdlib checks does not. + """ + root, manifest = verify_files(directory, expected_manifest_sha256) + selected = _amd_device(device) + namespace = "_decision_native_" + uuid.uuid4().hex + spec = importlib.util.spec_from_file_location(namespace, root / "__init__.py", + submodule_search_locations=[str(root)]) + package = importlib.util.module_from_spec(spec) + sys.modules[namespace] = package + try: + spec.loader.exec_module(package) + artifacts = importlib.import_module(namespace + ".artifacts") + contract = importlib.import_module(namespace + ".contract") + api = importlib.import_module(namespace + ".model") + declared = contract.validate_config(json.loads((root / "decision_config.json").read_text())) + if (declared["arm"], declared["training_arm"]) != ("all22", "S22"): + raise ValueError("Only the complete three-path all22/S22 architecture is supported") + model, collator, cfg = artifacts.load_export(root, device=str(selected)) + if (cfg["arm"], cfg["training_arm"]) != ("all22", "S22"): + raise ValueError("Only the complete three-path all22/S22 architecture is supported") + if type(model) is not api.DecisionModel or artifacts.verify_native(root) != (manifest, cfg): + raise ValueError("Loaded class or native identity mismatch") + return Native(root, expected_manifest_sha256, model, collator, cfg, api, + contract, artifacts, package_name=namespace) + except BaseException: + for name in tuple(sys.modules): + if name == namespace or name.startswith(namespace + "."): + del sys.modules[name] + raise + + +def _native_mode(native): + if native.training_state is not None: + raise ValueError("Restore native policy before native inference or export") + native.model.verify_inventory(check_values=False) + + +def predict(native, records, *, batch_size=8): + """Unchanged bundled inference, FP32; no training graph is used here.""" + _native_mode(native) + device = next(native.model.parameters()).device + _amd_device(device) + return native.model_api.predict(native.model, native.collator, records, + batch_size=batch_size, device=str(device), precision="fp32") + + +def export_native(native, directory, *, provenance): + """Use the original strict exporter after explicit policy restoration.""" + _native_mode(native) + if native.last_training_provenance is None or not isinstance(provenance, dict): + raise ValueError("Restored training provenance and an explicit user provenance dict required") + if Path(directory).exists(): + raise ValueError("Export destination must be fresh") + details = {"parent_native_manifest_sha256": native.manifest_sha256, + "parent_provenance": copy.deepcopy(native.config.get("provenance", {})), + "training_adapter": copy.deepcopy(native.last_training_provenance), + "user": copy.deepcopy(provenance)} + json.dumps(details, allow_nan=False) + return native.artifacts.export_model(native.model, native.collator.tokenizer, + directory, native.config, details) diff --git a/decision_runtime/training.py b/decision_runtime/training.py new file mode 100644 index 0000000000000000000000000000000000000000..f04d6b78a79749e541c1623d4648cc07497ccf92 --- /dev/null +++ b/decision_runtime/training.py @@ -0,0 +1,310 @@ +"""Explicit all-types training graph for the unchanged three-path native.""" +import math +from pathlib import Path +from .native import sha256, _amd_device + +KINDS = ("choice", "noul", "score") +FROZEN = ("encoder.embeddings.tok_embeddings.weight", + "encoder.embeddings.norm.weight", "type_embedding.weight") +COUNTS = {"tensors": 489, "parameters": 571909635, "active_tensors": 486, + "active_parameters": 375298563, "frozen_tensors": 3, + "frozen_parameters": 196611072} + + +def parameter_group(name): + if name in FROZEN: + return "frozen" + for kind, prefixes in {"choice": ("choice_blocks.", "choice_final_norm."), + "noul": ("encoder.layers.", "encoder.final_norm."), + "score": ("score_blocks.", "score_final_norm.")}.items(): + if name.startswith(prefixes): + return kind + ".encoder" + for kind in KINDS: + for suffix, prefix in (("head", "heads."), ("scorer", "scorers.")): + if name.startswith(prefix + kind + "."): + return kind + "." + suffix + raise ValueError("Unsupported parameter name: " + name) + + +def _roots(model): + return {"choice": (model.choice_blocks, model.choice_final_norm, model.heads["choice"], model.scorers["choice"]), + "noul": (model.encoder.layers, model.encoder.final_norm, model.heads["noul"], model.scorers["noul"]), + "score": (model.score_blocks, model.score_final_norm, model.heads["score"], model.scorers["score"])} + + +def _identities(model): + return {n: (id(p), p.untyped_storage().data_ptr()) for n, p in model.named_parameters()} + + +def _api(native): + m, c, api = native.model, native.contract, native.model_api + if (type(m) is not api.DecisionModel or m.arm != "all22" or m.training_arm != "S22" + or m.trainability_mode != c.training_policy("S22")): + raise ValueError("Original all22/S22 native class and metadata required") + api.runtime_gate() + return api + + +def inventory(native): + import torch + _api(native) + m = native.model + shapes = native.contract.all_shapes(m.arm, m.training_arm) + raw = list(m.named_parameters(remove_duplicate=False)) + names = {n for n, _ in raw} + if names != set(shapes) or set(m.state_dict()) != names or len(raw) != len(names): + raise ValueError("Exact native parameter/state roster required") + if len({id(p) for _, p in raw}) != len(raw) or len({p.untyped_storage().data_ptr() for _, p in raw}) != len(raw): + raise ValueError("Aliased parameters/storage are unsupported") + if any(tuple(p.shape) != tuple(shapes[n]) or p.dtype != torch.float32 + or p.requires_grad != (parameter_group(n) != "frozen") for n, p in raw): + raise ValueError("Training shape, FP32 parameter dtype or policy drift") + active = [p for n, p in raw if parameter_group(n) != "frozen"] + counts = {"tensors": len(raw), "parameters": sum(p.numel() for _, p in raw), + "active_tensors": len(active), "active_parameters": sum(p.numel() for p in active), + "frozen_tensors": len(raw) - len(active), + "frozen_parameters": sum(p.numel() for n, p in raw if parameter_group(n) == "frozen")} + if counts != COUNTS: + raise ValueError("Unsupported native geometry") + groups = {g: [n for n, _ in raw if parameter_group(n) == g] + for g in ("frozen", *(k + "." + part for k in KINDS for part in ("encoder", "head", "scorer")))} + return {"counts": counts, "groups": groups} + + +def configure_training(native, *, max_input_tokens=1024): + """Enable all three existing encoder/head/scorer paths; allocate no parameters.""" + api = _api(native) + if native.training_state is not None: + raise ValueError("Restore before configuring again") + if type(max_input_tokens) is not int or not 1 <= max_input_tokens <= native.config["packing"]["max_length"]: + raise ValueError("Training token limit must fit the native complete-input cap") + m = native.model + m.verify_inventory(check_values=False) + native.contract.validate_encoder_config(m.encoder.config) + for blocks, _, _, _ in _roots(m).values(): + if len(blocks) != 22: + raise ValueError("Three full 22-layer paths required") + for index, layer in enumerate(blocks): + api.layer_gate(layer, index) + for name, p in m.named_parameters(): + p.requires_grad_(parameter_group(name) != "frozen") + p.grad = None + native.training_state = {"identities": _identities(m), "training": False, + "max_input_tokens": max_input_tokens} + set_training_mode(native, False) + return inventory(native) + + +def _assert_modes(native): + m, state = native.model, native.training_state + if state is None: + raise ValueError("Call configure_training first") + selected = {id(child) for roots in _roots(m).values() for root in roots for child in root.modules()} + if m.training != state["training"] or _identities(m) != state["identities"]: + raise ValueError("Mode or parameter/optimizer identity changed") + for module in m.modules(): + if module is not m and module.training != (state["training"] and id(module) in selected): + raise ValueError("Active/frozen module mode drift; use set_training_mode") + if any(p.requires_grad != (parameter_group(n) != "frozen") for n, p in m.named_parameters()): + raise ValueError("Training requires_grad drift") + + +def set_training_mode(native, training=True): + if type(training) is not bool or native.training_state is None: + raise ValueError("Configured boolean training mode required") + m = native.model + m.eval() + m.training = training + for roots in _roots(m).values(): + for module in roots: + module.train(training) + native.training_state["training"] = training + _assert_modes(native) + + +def optimizer_groups(native, *, encoder_lr, head_lr): + """Return disjoint complete groups; no optimizer or scheduler is created.""" + if any(not math.isfinite(x) or x <= 0 for x in (encoder_lr, head_lr)): + raise ValueError("Positive finite learning rates required") + info = inventory(native) + params = dict(native.model.named_parameters()) + return [{"name": group, "params": [params[n] for n in names], + "lr": encoder_lr if group.endswith("encoder") else head_lr} + for group, names in info["groups"].items() if group != "frozen"] + + +def _validate_batch(native, batch): + import torch + required = {"input_ids", "attention_mask", "marker_positions", "valid_candidates", "targets", "values", "kind_ids"} + if set(batch) != required or any(not isinstance(t, torch.Tensor) for t in batch.values()): + raise ValueError("Use the native collator's complete batch dictionary") + ids, mask, kinds = batch["input_ids"], batch["attention_mask"], batch["kind_ids"] + pos, valid = batch["marker_positions"], batch["valid_candidates"] + _amd_device(ids.device) + if any(p.device != ids.device for p in native.model.parameters()) or any(t.device != ids.device for t in batch.values()): + raise ValueError("All parameters and batch tensors must share the actual AMD device") + if (ids.dtype != torch.long or ids.ndim != 2 or ids.shape[0] < 1 + or not 1 <= ids.shape[1] <= native.training_state["max_input_tokens"] + or mask.dtype != torch.bool or mask.shape != ids.shape + or kinds.dtype != torch.long or kinds.shape != (ids.shape[0],)): + raise ValueError("Invalid complete B x L input") + if (pos.dtype != torch.long or valid.dtype != torch.bool or pos.ndim != 2 + or pos.shape != valid.shape or pos.shape[0] != ids.shape[0] + or not 2 <= pos.shape[1] <= 255 or (pos < 0).any().item() + or (pos >= ids.shape[1]).any().item()): + raise ValueError("Invalid candidate markers") + counts = valid.sum(-1) + prefix = torch.arange(valid.shape[1], device=ids.device)[None] < counts[:, None] + if (counts < 2).any().item() or not torch.equal(prefix, valid): + raise ValueError("At least two candidates, then contiguous padding required") + if not mask.gather(1, pos)[valid].all().item() or not mask.any(-1).all().item(): + raise ValueError("Candidate marker points into padding") + if not torch.equal(mask, torch.arange(ids.shape[1], device=ids.device)[None] < mask.sum(-1)[:, None]): + raise ValueError("Native right-padding layout required") + if ((pos[:, 1:] <= pos[:, :-1]) & valid[:, 1:]).any().item(): + raise ValueError("Candidate markers must retain their original sequence order") + if ((kinds < 0) | (kinds > 2)).any().item() or (counts[kinds == 1] != 2).any().item(): + raise ValueError("Unknown type or non-binary Noul") + if batch["targets"].shape != valid.shape or batch["values"].shape != valid.shape: + raise ValueError("Target/value shape mismatch") + values = batch["values"] + if values.dtype != torch.float32 or not torch.isfinite(values).all().item(): + raise ValueError("Finite FP32 values required") + scored = (kinds == 2)[:, None] & valid[:, 1:] + if ((values[:, 1:] <= values[:, :-1]) & scored).any().item(): + raise ValueError("Score values must remain strictly increasing") + + +def forward_for_training(native, batch): + """Full-B×L suffixes, typed row gathering only at heads; no native monkeypatch.""" + import torch + from torch.utils.checkpoint import checkpoint + api = _api(native) + _assert_modes(native) + _validate_batch(native, batch) + if native.training_state["training"] and not torch.is_grad_enabled(): + raise ValueError("Training forward requires autograd") + m = native.model + ids, mask = batch["input_ids"], batch["attention_mask"] + kinds, valid, positions = batch["kind_ids"], batch["valid_candidates"], batch["marker_positions"] + routes = api.route_indices(kinds) + m.encoder._maybe_set_compile() + position_ids = torch.arange(ids.shape[1], device=ids.device).unsqueeze(0) + global_mask, local_mask = m.encoder._update_attention_mask(mask, output_attentions=False) + kwargs = dict(attention_mask=global_mask, sliding_window_mask=local_mask, + position_ids=position_ids, cu_seqlens=None, max_seqlen=None, output_attentions=False) + with torch.no_grad(): + embedded = m.encoder.embeddings(input_ids=ids, inputs_embeds=None) + hidden = {} + roots = _roots(m) + for name, indices in routes: + if indices.numel() == 0: + continue + blocks, norm, _, _ = roots[name] + h = m._suffix(embedded, blocks, norm, kwargs) + with torch.no_grad(): + type_value = m.type_embedding(kinds)[:, None, :] + hidden[name] = h + type_value.to(h.dtype) + pad = ~mask.bool() + out = torch.empty(positions.shape, device=ids.device, dtype=torch.float32) + for name, indices in routes: + if indices.numel() == 0: + continue + h = hidden[name].index_select(0, indices) + branch_pad = pad.index_select(0, indices) + for layer in m.heads[name]: + h = (checkpoint(layer, h, src_key_padding_mask=branch_pad, use_reentrant=False) + if layer.training and m.head_config["gradient_checkpointing"] + else layer(h, src_key_padding_mask=branch_pad)) + loc = positions.index_select(0, indices) + markers = torch.gather(h, 1, loc[:, :, None].expand(-1, -1, h.shape[-1])) + logits = m.scorers[name](markers).squeeze(-1).float() + out = api.scatter_rows(out, indices, logits) + return out.masked_fill(~valid, torch.finfo(torch.float32).min) + + +def loss_for_training(native, logits, batch, *, score_rps_weight=0.0): + """Return (per-row total, CE, RPS); caller chooses the logical-batch denominator.""" + import torch + if not math.isfinite(score_rps_weight) or score_rps_weight < 0: + raise ValueError("Nonnegative finite Score RPS weight required") + target, valid = batch["targets"], batch["valid_candidates"] + if (logits.shape != target.shape or target.shape != valid.shape + or target.dtype != torch.float32 or not torch.isfinite(logits).all().item() + or not torch.isfinite(target).all().item() or ((target < 0) | (target > 1)).any().item() + or target[~valid].count_nonzero().item() or not torch.allclose(target.sum(-1), torch.ones_like(target[:, 0]), atol=1e-6, rtol=0)): + raise ValueError("Explicit normalized hard/soft targets and finite masked logits required") + return native.model_api.typed_loss(logits, batch, score_rps_weight=score_rps_weight) + + +def _tensor_sha(tensor): + import hashlib + return hashlib.sha256(tensor.detach().cpu().contiguous().numpy().tobytes()).hexdigest() + + +def frozen_snapshot(native): + inventory(native) + _assert_modes(native) + ps = dict(native.model.named_parameters()) + return {n: {"version": int(ps[n]._version), "sha256": _tensor_sha(ps[n])} for n in FROZEN} + + +def assert_frozen(native, snapshot, *, content=False, versions=True): + _assert_modes(native) + ps = dict(native.model.named_parameters()) + if set(snapshot) != set(FROZEN): + raise ValueError("Expected exactly the three shared frozen parameters") + for name, saved in snapshot.items(): + p = ps[name] + if (p.requires_grad or p.grad is not None or (versions and int(p._version) != saved["version"]) + or (content and _tensor_sha(p) != saved["sha256"])): + raise ValueError("Frozen parameter changed: " + name) + return True + + +def gradient_report(native, *, present_types): + """Check groups actually present in the logical batch; absent grads must be None.""" + import torch + if not present_types or not set(present_types) <= set(KINDS): + raise ValueError("Explicit nonempty logical-batch type set required") + info = inventory(native) + ps = dict(native.model.named_parameters()) + report = {} + for group, names in info["groups"].items(): + active = group != "frozen" and group.split(".")[0] in present_types + gradients = [ps[n].grad for n in names] + if not active: + if any(g is not None for g in gradients): + raise ValueError("Frozen/absent group has stale gradients: " + group) + continue + if any(g is None or not torch.isfinite(g).all().item() for g in gradients): + raise ValueError("Missing or nonfinite gradient: " + group) + norms = [float(g.float().norm().item()) for g in gradients] + if not any(n > 0 for n in norms): + raise ValueError("No nonzero gradient: " + group) + report[group] = {"tensors": len(names), "nonzero_tensors": sum(n > 0 for n in norms), + "l2": math.sqrt(sum(n * n for n in norms))} + return report + + +def restore_native_policy_for_export(native): + """Clear gradients and restore the original strict metadata, without replacing parameters.""" + _assert_modes(native) + inventory(native) + m, state, contract = native.model, native.training_state, native.contract + policy = contract.training_policy(m.training_arm) + sources = {p.name: sha256(p) for p in sorted(Path(__file__).parent.glob("*.py")) if p.name != "checks.py"} + provenance = {"actual_training_policy": "all_three_full22_paths_and_heads", + "counts": dict(COUNTS), "complete_input_limit": state["max_input_tokens"], + "native_policy_is_not_actual_training_provenance": True, + "restored_native_trainability_mode": policy, "adapter_sources_sha256": sources} + for n, p in m.named_parameters(): + p.requires_grad_(contract.trainable_name(n, policy)) + p.grad = None + m.eval() + m.verify_inventory(check_values=False) + if _identities(m) != state["identities"]: + raise ValueError("Restoration replaced parameter objects") + native.training_state = None + native.last_training_provenance = provenance + return provenance diff --git a/examples/decisions.jsonl b/examples/decisions.jsonl new file mode 100644 index 0000000000000000000000000000000000000000..e2c571e23b38f812233506ff3e99b5504a1a69f8 --- /dev/null +++ b/examples/decisions.jsonl @@ -0,0 +1,3 @@ +{"id":"route","state_text":"The customer asks for a refund for a duplicate charge.","question":{"id":"route-q","type":"Choice","text":"Which team should handle this request?","options":[{"id":"billing","text":"Billing and payments"},{"id":"technical","text":"Technical support"},{"id":"sales","text":"Sales enquiries"}]}} +{"id":"evidence","state_text":"The package was delivered on Monday. A signed receipt is available.","question":{"id":"evidence-q","type":"Noul","text":"Does the evidence confirm that the package was delivered?"}} +{"id":"sentiment","state_text":"The replacement arrived quickly and works perfectly.","question":{"id":"sentiment-q","type":"Score","text":"How positive is the customer's sentiment?","levels":[{"id":"negative","text":"Negative","value":0},{"id":"neutral","text":"Neutral","value":1},{"id":"positive","text":"Positive","value":2}]}} diff --git a/examples/finetune.sh b/examples/finetune.sh new file mode 100644 index 0000000000000000000000000000000000000000..d5f78f3dea3e26172b36b0c5de27eeb60337c1ef --- /dev/null +++ b/examples/finetune.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +set -euo pipefail +# Run from the package root in an existing compatible AMD ROCm environment. +# Replace these paths with your own isolated TRAIN and DEV JSONL files. +: "${TRAIN:?Set TRAIN to your training JSONL}" +: "${DEV:?Set DEV to your development JSONL}" +: "${OUTPUT:?Set OUTPUT to a fresh run directory}" +NATIVE="${NATIVE:-./native}" +MANIFEST_SHA256="${MANIFEST_SHA256:-f288d873999832a3f37c6a7c4268c2ab309691e621794dbf7acab891acbbb7e6}" +export PYTHONPATH="${PWD}${PYTHONPATH:+:${PYTHONPATH}}" +export PYTHONDONTWRITEBYTECODE=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 TOKENIZERS_PARALLELISM=false +python -m decision_finetune validate-jsonl --train "$TRAIN" --dev "$DEV" --selection macro-nll +# Device visibility is supplied by the caller, e.g. ROCR_VISIBLE_DEVICES=0. +python -m decision_finetune train \ + --native "$NATIVE" --manifest-sha256 "$MANIFEST_SHA256" \ + --train "$TRAIN" --dev "$DEV" --output "$OUTPUT" \ + --epochs 4 --logical-batch-size 64 --micro-batch-size 8 --seed 20260921 \ + --encoder-lr 2.5e-5 --head-lr 1e-4 --lr-min 1e-6 \ + --weight-decay 0.01 --clip-norm 1 --score-rps-weight 0.1 \ + --warmup-ratio 0.1 --selection macro-nll --selection-tolerance 1e-8 \ + --cpu-threads 2 --max-reserved-gib 64 diff --git a/infer.py b/infer.py new file mode 100644 index 0000000000000000000000000000000000000000..9d65f1b1e44acccb76719e00bb312317343cb5ce --- /dev/null +++ b/infer.py @@ -0,0 +1,46 @@ +"""Complete-input 1K inference using the bundled, unchanged native runtime.""" +import argparse +import json +from pathlib import Path + +NATIVE_MANIFEST_SHA256 = 'f288d873999832a3f37c6a7c4268c2ab309691e621794dbf7acab891acbbb7e6' + + +def main(): + p = argparse.ArgumentParser(description='Decision Lex: complete1024, FP32 AMD inference') + p.add_argument('--native', default=str(Path(__file__).resolve().parent / 'native')) + p.add_argument('--manifest-sha256', default=NATIVE_MANIFEST_SHA256) + p.add_argument('--input', required=True, help='JSONL with id, state_text, question') + p.add_argument('--output', required=True, help='Fresh prediction JSONL; existing files are not overwritten') + p.add_argument('--batch-size', type=int, choices=range(1, 9), default=8) + args = p.parse_args() + records = [] + with Path(args.input).open(encoding='utf-8') as stream: + for line in stream: + row = json.loads(line) + if set(row) != {'id', 'state_text', 'question'}: + raise ValueError('Inference rows must contain exactly id, state_text and question') + records.append(row) + if not records or len({r['id'] for r in records}) != len(records): + raise ValueError('Nonempty input with unique record IDs required') + if Path(args.output).exists(): + raise ValueError('Prediction output must be fresh') + import torch + from decision_runtime import load_native + from decision_inference import predict_1k + if torch.version.hip is None or not torch.cuda.is_available() or torch.cuda.device_count() != 1: + raise RuntimeError('Expose exactly one AMD ROCm GPU; this package has no CPU/NVIDIA fallback') + torch.cuda.set_device(0); torch.set_num_threads(2) + torch.backends.cuda.matmul.allow_tf32 = False + torch.backends.cudnn.allow_tf32 = False + torch.backends.mha.set_fastpath_enabled(False) + native = load_native(args.native, expected_manifest_sha256=args.manifest_sha256, device='cuda:0') + answers = predict_1k(native, records, batch_size=args.batch_size) + if len(answers) != len(records): + raise RuntimeError('Incomplete inference result') + with Path(args.output).open('x', encoding='utf-8') as stream: + for answer in answers: + stream.write(json.dumps(answer, ensure_ascii=False, allow_nan=False) + '\n') + + +if __name__ == '__main__': main() diff --git a/native/INVENTORY.json b/native/INVENTORY.json new file mode 100644 index 0000000000000000000000000000000000000000..dda11dec0b49645b457e082c906133ff93cddf83 --- /dev/null +++ b/native/INVENTORY.json @@ -0,0 +1,1786 @@ +{ + "choice_suffix_shapes": { + "choice_blocks.0.attn.Wo.weight": [ + 768, + 768 + ], + "choice_blocks.0.attn.Wqkv.weight": [ + 2304, + 768 + ], + "choice_blocks.0.mlp.Wi.weight": [ + 2304, + 768 + ], + "choice_blocks.0.mlp.Wo.weight": [ + 768, + 1152 + ], + "choice_blocks.0.mlp_norm.weight": [ + 768 + ], + "choice_blocks.1.attn.Wo.weight": [ + 768, + 768 + ], + "choice_blocks.1.attn.Wqkv.weight": [ + 2304, + 768 + ], + "choice_blocks.1.attn_norm.weight": [ + 768 + ], + "choice_blocks.1.mlp.Wi.weight": [ + 2304, + 768 + ], + "choice_blocks.1.mlp.Wo.weight": [ + 768, + 1152 + ], + "choice_blocks.1.mlp_norm.weight": [ + 768 + ], + "choice_blocks.10.attn.Wo.weight": [ + 768, + 768 + ], + "choice_blocks.10.attn.Wqkv.weight": [ + 2304, + 768 + ], + "choice_blocks.10.attn_norm.weight": [ + 768 + ], + "choice_blocks.10.mlp.Wi.weight": [ + 2304, + 768 + ], + "choice_blocks.10.mlp.Wo.weight": [ + 768, + 1152 + ], + "choice_blocks.10.mlp_norm.weight": [ + 768 + ], + "choice_blocks.11.attn.Wo.weight": [ + 768, + 768 + ], + "choice_blocks.11.attn.Wqkv.weight": [ + 2304, + 768 + ], + "choice_blocks.11.attn_norm.weight": [ + 768 + ], + "choice_blocks.11.mlp.Wi.weight": [ + 2304, + 768 + ], + "choice_blocks.11.mlp.Wo.weight": [ + 768, + 1152 + ], + "choice_blocks.11.mlp_norm.weight": [ + 768 + ], + "choice_blocks.12.attn.Wo.weight": [ + 768, + 768 + ], + "choice_blocks.12.attn.Wqkv.weight": [ + 2304, + 768 + ], + "choice_blocks.12.attn_norm.weight": [ + 768 + ], + "choice_blocks.12.mlp.Wi.weight": [ + 2304, + 768 + ], + "choice_blocks.12.mlp.Wo.weight": [ + 768, + 1152 + ], + "choice_blocks.12.mlp_norm.weight": [ + 768 + ], + "choice_blocks.13.attn.Wo.weight": [ + 768, + 768 + ], + "choice_blocks.13.attn.Wqkv.weight": [ + 2304, + 768 + ], + "choice_blocks.13.attn_norm.weight": [ + 768 + ], + "choice_blocks.13.mlp.Wi.weight": [ + 2304, + 768 + ], + "choice_blocks.13.mlp.Wo.weight": [ + 768, + 1152 + ], + "choice_blocks.13.mlp_norm.weight": [ + 768 + ], + "choice_blocks.14.attn.Wo.weight": [ + 768, + 768 + ], + "choice_blocks.14.attn.Wqkv.weight": [ + 2304, + 768 + ], + "choice_blocks.14.attn_norm.weight": [ + 768 + ], + "choice_blocks.14.mlp.Wi.weight": [ + 2304, + 768 + ], + "choice_blocks.14.mlp.Wo.weight": [ + 768, + 1152 + ], + "choice_blocks.14.mlp_norm.weight": [ + 768 + ], + "choice_blocks.15.attn.Wo.weight": [ + 768, + 768 + ], + "choice_blocks.15.attn.Wqkv.weight": [ + 2304, + 768 + ], + "choice_blocks.15.attn_norm.weight": [ + 768 + ], + "choice_blocks.15.mlp.Wi.weight": [ + 2304, + 768 + ], + "choice_blocks.15.mlp.Wo.weight": [ + 768, + 1152 + ], + "choice_blocks.15.mlp_norm.weight": [ + 768 + ], + "choice_blocks.16.attn.Wo.weight": [ + 768, + 768 + ], + "choice_blocks.16.attn.Wqkv.weight": [ + 2304, + 768 + ], + "choice_blocks.16.attn_norm.weight": [ + 768 + ], + "choice_blocks.16.mlp.Wi.weight": [ + 2304, + 768 + ], + "choice_blocks.16.mlp.Wo.weight": [ + 768, + 1152 + ], + "choice_blocks.16.mlp_norm.weight": [ + 768 + ], + "choice_blocks.17.attn.Wo.weight": [ + 768, + 768 + ], + "choice_blocks.17.attn.Wqkv.weight": [ + 2304, + 768 + ], + "choice_blocks.17.attn_norm.weight": [ + 768 + ], + "choice_blocks.17.mlp.Wi.weight": [ + 2304, + 768 + ], + "choice_blocks.17.mlp.Wo.weight": [ + 768, + 1152 + ], + "choice_blocks.17.mlp_norm.weight": [ + 768 + ], + "choice_blocks.18.attn.Wo.weight": [ + 768, + 768 + ], + "choice_blocks.18.attn.Wqkv.weight": [ + 2304, + 768 + ], + "choice_blocks.18.attn_norm.weight": [ + 768 + ], + "choice_blocks.18.mlp.Wi.weight": [ + 2304, + 768 + ], + "choice_blocks.18.mlp.Wo.weight": [ + 768, + 1152 + ], + "choice_blocks.18.mlp_norm.weight": [ + 768 + ], + "choice_blocks.19.attn.Wo.weight": [ + 768, + 768 + ], + "choice_blocks.19.attn.Wqkv.weight": [ + 2304, + 768 + ], + "choice_blocks.19.attn_norm.weight": [ + 768 + ], + "choice_blocks.19.mlp.Wi.weight": [ + 2304, + 768 + ], + "choice_blocks.19.mlp.Wo.weight": [ + 768, + 1152 + ], + "choice_blocks.19.mlp_norm.weight": [ + 768 + ], + "choice_blocks.2.attn.Wo.weight": [ + 768, + 768 + ], + "choice_blocks.2.attn.Wqkv.weight": [ + 2304, + 768 + ], + "choice_blocks.2.attn_norm.weight": [ + 768 + ], + "choice_blocks.2.mlp.Wi.weight": [ + 2304, + 768 + ], + "choice_blocks.2.mlp.Wo.weight": [ + 768, + 1152 + ], + "choice_blocks.2.mlp_norm.weight": [ + 768 + ], + "choice_blocks.20.attn.Wo.weight": [ + 768, + 768 + ], + "choice_blocks.20.attn.Wqkv.weight": [ + 2304, + 768 + ], + "choice_blocks.20.attn_norm.weight": [ + 768 + ], + "choice_blocks.20.mlp.Wi.weight": [ + 2304, + 768 + ], + "choice_blocks.20.mlp.Wo.weight": [ + 768, + 1152 + ], + "choice_blocks.20.mlp_norm.weight": [ + 768 + ], + "choice_blocks.21.attn.Wo.weight": [ + 768, + 768 + ], + "choice_blocks.21.attn.Wqkv.weight": [ + 2304, + 768 + ], + "choice_blocks.21.attn_norm.weight": [ + 768 + ], + "choice_blocks.21.mlp.Wi.weight": [ + 2304, + 768 + ], + "choice_blocks.21.mlp.Wo.weight": [ + 768, + 1152 + ], + "choice_blocks.21.mlp_norm.weight": [ + 768 + ], + "choice_blocks.3.attn.Wo.weight": [ + 768, + 768 + ], + "choice_blocks.3.attn.Wqkv.weight": [ + 2304, + 768 + ], + "choice_blocks.3.attn_norm.weight": [ + 768 + ], + "choice_blocks.3.mlp.Wi.weight": [ + 2304, + 768 + ], + "choice_blocks.3.mlp.Wo.weight": [ + 768, + 1152 + ], + "choice_blocks.3.mlp_norm.weight": [ + 768 + ], + "choice_blocks.4.attn.Wo.weight": [ + 768, + 768 + ], + "choice_blocks.4.attn.Wqkv.weight": [ + 2304, + 768 + ], + "choice_blocks.4.attn_norm.weight": [ + 768 + ], + "choice_blocks.4.mlp.Wi.weight": [ + 2304, + 768 + ], + "choice_blocks.4.mlp.Wo.weight": [ + 768, + 1152 + ], + "choice_blocks.4.mlp_norm.weight": [ + 768 + ], + "choice_blocks.5.attn.Wo.weight": [ + 768, + 768 + ], + "choice_blocks.5.attn.Wqkv.weight": [ + 2304, + 768 + ], + "choice_blocks.5.attn_norm.weight": [ + 768 + ], + "choice_blocks.5.mlp.Wi.weight": [ + 2304, + 768 + ], + "choice_blocks.5.mlp.Wo.weight": [ + 768, + 1152 + ], + "choice_blocks.5.mlp_norm.weight": [ + 768 + ], + "choice_blocks.6.attn.Wo.weight": [ + 768, + 768 + ], + "choice_blocks.6.attn.Wqkv.weight": [ + 2304, + 768 + ], + "choice_blocks.6.attn_norm.weight": [ + 768 + ], + "choice_blocks.6.mlp.Wi.weight": [ + 2304, + 768 + ], + "choice_blocks.6.mlp.Wo.weight": [ + 768, + 1152 + ], + "choice_blocks.6.mlp_norm.weight": [ + 768 + ], + "choice_blocks.7.attn.Wo.weight": [ + 768, + 768 + ], + "choice_blocks.7.attn.Wqkv.weight": [ + 2304, + 768 + ], + "choice_blocks.7.attn_norm.weight": [ + 768 + ], + "choice_blocks.7.mlp.Wi.weight": [ + 2304, + 768 + ], + "choice_blocks.7.mlp.Wo.weight": [ + 768, + 1152 + ], + "choice_blocks.7.mlp_norm.weight": [ + 768 + ], + "choice_blocks.8.attn.Wo.weight": [ + 768, + 768 + ], + "choice_blocks.8.attn.Wqkv.weight": [ + 2304, + 768 + ], + "choice_blocks.8.attn_norm.weight": [ + 768 + ], + "choice_blocks.8.mlp.Wi.weight": [ + 2304, + 768 + ], + "choice_blocks.8.mlp.Wo.weight": [ + 768, + 1152 + ], + "choice_blocks.8.mlp_norm.weight": [ + 768 + ], + "choice_blocks.9.attn.Wo.weight": [ + 768, + 768 + ], + "choice_blocks.9.attn.Wqkv.weight": [ + 2304, + 768 + ], + "choice_blocks.9.attn_norm.weight": [ + 768 + ], + "choice_blocks.9.mlp.Wi.weight": [ + 2304, + 768 + ], + "choice_blocks.9.mlp.Wo.weight": [ + 768, + 1152 + ], + "choice_blocks.9.mlp_norm.weight": [ + 768 + ], + "choice_final_norm.weight": [ + 768 + ] + }, + "counts": { + "frozen_parameters": 446810114, + "frozen_tensors": 327, + "integrated_choice_suffix_tensors": 132, + "score_suffix_parameters": 110330880, + "score_suffix_tensors": 132, + "shared_encoder_tensors": 134, + "trainable_parameters": 125099521, + "trainable_tensors": 162, + "type_and_head_tensors": 91, + "unique_parameters": 571909635, + "unique_tensors": 489 + }, + "encoder_shapes": { + "embeddings.norm.weight": [ + 768 + ], + "embeddings.tok_embeddings.weight": [ + 256000, + 768 + ], + "final_norm.weight": [ + 768 + ], + "layers.0.attn.Wo.weight": [ + 768, + 768 + ], + "layers.0.attn.Wqkv.weight": [ + 2304, + 768 + ], + "layers.0.mlp.Wi.weight": [ + 2304, + 768 + ], + "layers.0.mlp.Wo.weight": [ + 768, + 1152 + ], + "layers.0.mlp_norm.weight": [ + 768 + ], + "layers.1.attn.Wo.weight": [ + 768, + 768 + ], + "layers.1.attn.Wqkv.weight": [ + 2304, + 768 + ], + "layers.1.attn_norm.weight": [ + 768 + ], + "layers.1.mlp.Wi.weight": [ + 2304, + 768 + ], + "layers.1.mlp.Wo.weight": [ + 768, + 1152 + ], + "layers.1.mlp_norm.weight": [ + 768 + ], + "layers.10.attn.Wo.weight": [ + 768, + 768 + ], + "layers.10.attn.Wqkv.weight": [ + 2304, + 768 + ], + "layers.10.attn_norm.weight": [ + 768 + ], + "layers.10.mlp.Wi.weight": [ + 2304, + 768 + ], + "layers.10.mlp.Wo.weight": [ + 768, + 1152 + ], + "layers.10.mlp_norm.weight": [ + 768 + ], + "layers.11.attn.Wo.weight": [ + 768, + 768 + ], + "layers.11.attn.Wqkv.weight": [ + 2304, + 768 + ], + "layers.11.attn_norm.weight": [ + 768 + ], + "layers.11.mlp.Wi.weight": [ + 2304, + 768 + ], + "layers.11.mlp.Wo.weight": [ + 768, + 1152 + ], + "layers.11.mlp_norm.weight": [ + 768 + ], + "layers.12.attn.Wo.weight": [ + 768, + 768 + ], + "layers.12.attn.Wqkv.weight": [ + 2304, + 768 + ], + "layers.12.attn_norm.weight": [ + 768 + ], + "layers.12.mlp.Wi.weight": [ + 2304, + 768 + ], + "layers.12.mlp.Wo.weight": [ + 768, + 1152 + ], + "layers.12.mlp_norm.weight": [ + 768 + ], + "layers.13.attn.Wo.weight": [ + 768, + 768 + ], + "layers.13.attn.Wqkv.weight": [ + 2304, + 768 + ], + "layers.13.attn_norm.weight": [ + 768 + ], + "layers.13.mlp.Wi.weight": [ + 2304, + 768 + ], + "layers.13.mlp.Wo.weight": [ + 768, + 1152 + ], + "layers.13.mlp_norm.weight": [ + 768 + ], + "layers.14.attn.Wo.weight": [ + 768, + 768 + ], + "layers.14.attn.Wqkv.weight": [ + 2304, + 768 + ], + "layers.14.attn_norm.weight": [ + 768 + ], + "layers.14.mlp.Wi.weight": [ + 2304, + 768 + ], + "layers.14.mlp.Wo.weight": [ + 768, + 1152 + ], + "layers.14.mlp_norm.weight": [ + 768 + ], + "layers.15.attn.Wo.weight": [ + 768, + 768 + ], + "layers.15.attn.Wqkv.weight": [ + 2304, + 768 + ], + "layers.15.attn_norm.weight": [ + 768 + ], + "layers.15.mlp.Wi.weight": [ + 2304, + 768 + ], + "layers.15.mlp.Wo.weight": [ + 768, + 1152 + ], + "layers.15.mlp_norm.weight": [ + 768 + ], + "layers.16.attn.Wo.weight": [ + 768, + 768 + ], + "layers.16.attn.Wqkv.weight": [ + 2304, + 768 + ], + "layers.16.attn_norm.weight": [ + 768 + ], + "layers.16.mlp.Wi.weight": [ + 2304, + 768 + ], + "layers.16.mlp.Wo.weight": [ + 768, + 1152 + ], + "layers.16.mlp_norm.weight": [ + 768 + ], + "layers.17.attn.Wo.weight": [ + 768, + 768 + ], + "layers.17.attn.Wqkv.weight": [ + 2304, + 768 + ], + "layers.17.attn_norm.weight": [ + 768 + ], + "layers.17.mlp.Wi.weight": [ + 2304, + 768 + ], + "layers.17.mlp.Wo.weight": [ + 768, + 1152 + ], + "layers.17.mlp_norm.weight": [ + 768 + ], + "layers.18.attn.Wo.weight": [ + 768, + 768 + ], + "layers.18.attn.Wqkv.weight": [ + 2304, + 768 + ], + "layers.18.attn_norm.weight": [ + 768 + ], + "layers.18.mlp.Wi.weight": [ + 2304, + 768 + ], + "layers.18.mlp.Wo.weight": [ + 768, + 1152 + ], + "layers.18.mlp_norm.weight": [ + 768 + ], + "layers.19.attn.Wo.weight": [ + 768, + 768 + ], + "layers.19.attn.Wqkv.weight": [ + 2304, + 768 + ], + "layers.19.attn_norm.weight": [ + 768 + ], + "layers.19.mlp.Wi.weight": [ + 2304, + 768 + ], + "layers.19.mlp.Wo.weight": [ + 768, + 1152 + ], + "layers.19.mlp_norm.weight": [ + 768 + ], + "layers.2.attn.Wo.weight": [ + 768, + 768 + ], + "layers.2.attn.Wqkv.weight": [ + 2304, + 768 + ], + "layers.2.attn_norm.weight": [ + 768 + ], + "layers.2.mlp.Wi.weight": [ + 2304, + 768 + ], + "layers.2.mlp.Wo.weight": [ + 768, + 1152 + ], + "layers.2.mlp_norm.weight": [ + 768 + ], + "layers.20.attn.Wo.weight": [ + 768, + 768 + ], + "layers.20.attn.Wqkv.weight": [ + 2304, + 768 + ], + "layers.20.attn_norm.weight": [ + 768 + ], + "layers.20.mlp.Wi.weight": [ + 2304, + 768 + ], + "layers.20.mlp.Wo.weight": [ + 768, + 1152 + ], + "layers.20.mlp_norm.weight": [ + 768 + ], + "layers.21.attn.Wo.weight": [ + 768, + 768 + ], + "layers.21.attn.Wqkv.weight": [ + 2304, + 768 + ], + "layers.21.attn_norm.weight": [ + 768 + ], + "layers.21.mlp.Wi.weight": [ + 2304, + 768 + ], + "layers.21.mlp.Wo.weight": [ + 768, + 1152 + ], + "layers.21.mlp_norm.weight": [ + 768 + ], + "layers.3.attn.Wo.weight": [ + 768, + 768 + ], + "layers.3.attn.Wqkv.weight": [ + 2304, + 768 + ], + "layers.3.attn_norm.weight": [ + 768 + ], + "layers.3.mlp.Wi.weight": [ + 2304, + 768 + ], + "layers.3.mlp.Wo.weight": [ + 768, + 1152 + ], + "layers.3.mlp_norm.weight": [ + 768 + ], + "layers.4.attn.Wo.weight": [ + 768, + 768 + ], + "layers.4.attn.Wqkv.weight": [ + 2304, + 768 + ], + "layers.4.attn_norm.weight": [ + 768 + ], + "layers.4.mlp.Wi.weight": [ + 2304, + 768 + ], + "layers.4.mlp.Wo.weight": [ + 768, + 1152 + ], + "layers.4.mlp_norm.weight": [ + 768 + ], + "layers.5.attn.Wo.weight": [ + 768, + 768 + ], + "layers.5.attn.Wqkv.weight": [ + 2304, + 768 + ], + "layers.5.attn_norm.weight": [ + 768 + ], + "layers.5.mlp.Wi.weight": [ + 2304, + 768 + ], + "layers.5.mlp.Wo.weight": [ + 768, + 1152 + ], + "layers.5.mlp_norm.weight": [ + 768 + ], + "layers.6.attn.Wo.weight": [ + 768, + 768 + ], + "layers.6.attn.Wqkv.weight": [ + 2304, + 768 + ], + "layers.6.attn_norm.weight": [ + 768 + ], + "layers.6.mlp.Wi.weight": [ + 2304, + 768 + ], + "layers.6.mlp.Wo.weight": [ + 768, + 1152 + ], + "layers.6.mlp_norm.weight": [ + 768 + ], + "layers.7.attn.Wo.weight": [ + 768, + 768 + ], + "layers.7.attn.Wqkv.weight": [ + 2304, + 768 + ], + "layers.7.attn_norm.weight": [ + 768 + ], + "layers.7.mlp.Wi.weight": [ + 2304, + 768 + ], + "layers.7.mlp.Wo.weight": [ + 768, + 1152 + ], + "layers.7.mlp_norm.weight": [ + 768 + ], + "layers.8.attn.Wo.weight": [ + 768, + 768 + ], + "layers.8.attn.Wqkv.weight": [ + 2304, + 768 + ], + "layers.8.attn_norm.weight": [ + 768 + ], + "layers.8.mlp.Wi.weight": [ + 2304, + 768 + ], + "layers.8.mlp.Wo.weight": [ + 768, + 1152 + ], + "layers.8.mlp_norm.weight": [ + 768 + ], + "layers.9.attn.Wo.weight": [ + 768, + 768 + ], + "layers.9.attn.Wqkv.weight": [ + 2304, + 768 + ], + "layers.9.attn_norm.weight": [ + 768 + ], + "layers.9.mlp.Wi.weight": [ + 2304, + 768 + ], + "layers.9.mlp.Wo.weight": [ + 768, + 1152 + ], + "layers.9.mlp_norm.weight": [ + 768 + ] + }, + "head_shapes": { + "heads.choice.0.linear1.bias": [ + 3072 + ], + "heads.choice.0.linear1.weight": [ + 3072, + 768 + ], + "heads.choice.0.linear2.bias": [ + 768 + ], + "heads.choice.0.linear2.weight": [ + 768, + 3072 + ], + "heads.choice.0.norm1.bias": [ + 768 + ], + "heads.choice.0.norm1.weight": [ + 768 + ], + "heads.choice.0.norm2.bias": [ + 768 + ], + "heads.choice.0.norm2.weight": [ + 768 + ], + "heads.choice.0.self_attn.in_proj_bias": [ + 2304 + ], + "heads.choice.0.self_attn.in_proj_weight": [ + 2304, + 768 + ], + "heads.choice.0.self_attn.out_proj.bias": [ + 768 + ], + "heads.choice.0.self_attn.out_proj.weight": [ + 768, + 768 + ], + "heads.choice.1.linear1.bias": [ + 3072 + ], + "heads.choice.1.linear1.weight": [ + 3072, + 768 + ], + "heads.choice.1.linear2.bias": [ + 768 + ], + "heads.choice.1.linear2.weight": [ + 768, + 3072 + ], + "heads.choice.1.norm1.bias": [ + 768 + ], + "heads.choice.1.norm1.weight": [ + 768 + ], + "heads.choice.1.norm2.bias": [ + 768 + ], + "heads.choice.1.norm2.weight": [ + 768 + ], + "heads.choice.1.self_attn.in_proj_bias": [ + 2304 + ], + "heads.choice.1.self_attn.in_proj_weight": [ + 2304, + 768 + ], + "heads.choice.1.self_attn.out_proj.bias": [ + 768 + ], + "heads.choice.1.self_attn.out_proj.weight": [ + 768, + 768 + ], + "heads.noul.0.linear1.bias": [ + 3072 + ], + "heads.noul.0.linear1.weight": [ + 3072, + 768 + ], + "heads.noul.0.linear2.bias": [ + 768 + ], + "heads.noul.0.linear2.weight": [ + 768, + 3072 + ], + "heads.noul.0.norm1.bias": [ + 768 + ], + "heads.noul.0.norm1.weight": [ + 768 + ], + "heads.noul.0.norm2.bias": [ + 768 + ], + "heads.noul.0.norm2.weight": [ + 768 + ], + "heads.noul.0.self_attn.in_proj_bias": [ + 2304 + ], + "heads.noul.0.self_attn.in_proj_weight": [ + 2304, + 768 + ], + "heads.noul.0.self_attn.out_proj.bias": [ + 768 + ], + "heads.noul.0.self_attn.out_proj.weight": [ + 768, + 768 + ], + "heads.noul.1.linear1.bias": [ + 3072 + ], + "heads.noul.1.linear1.weight": [ + 3072, + 768 + ], + "heads.noul.1.linear2.bias": [ + 768 + ], + "heads.noul.1.linear2.weight": [ + 768, + 3072 + ], + "heads.noul.1.norm1.bias": [ + 768 + ], + "heads.noul.1.norm1.weight": [ + 768 + ], + "heads.noul.1.norm2.bias": [ + 768 + ], + "heads.noul.1.norm2.weight": [ + 768 + ], + "heads.noul.1.self_attn.in_proj_bias": [ + 2304 + ], + "heads.noul.1.self_attn.in_proj_weight": [ + 2304, + 768 + ], + "heads.noul.1.self_attn.out_proj.bias": [ + 768 + ], + "heads.noul.1.self_attn.out_proj.weight": [ + 768, + 768 + ], + "heads.score.0.linear1.bias": [ + 3072 + ], + "heads.score.0.linear1.weight": [ + 3072, + 768 + ], + "heads.score.0.linear2.bias": [ + 768 + ], + "heads.score.0.linear2.weight": [ + 768, + 3072 + ], + "heads.score.0.norm1.bias": [ + 768 + ], + "heads.score.0.norm1.weight": [ + 768 + ], + "heads.score.0.norm2.bias": [ + 768 + ], + "heads.score.0.norm2.weight": [ + 768 + ], + "heads.score.0.self_attn.in_proj_bias": [ + 2304 + ], + "heads.score.0.self_attn.in_proj_weight": [ + 2304, + 768 + ], + "heads.score.0.self_attn.out_proj.bias": [ + 768 + ], + "heads.score.0.self_attn.out_proj.weight": [ + 768, + 768 + ], + "heads.score.1.linear1.bias": [ + 3072 + ], + "heads.score.1.linear1.weight": [ + 3072, + 768 + ], + "heads.score.1.linear2.bias": [ + 768 + ], + "heads.score.1.linear2.weight": [ + 768, + 3072 + ], + "heads.score.1.norm1.bias": [ + 768 + ], + "heads.score.1.norm1.weight": [ + 768 + ], + "heads.score.1.norm2.bias": [ + 768 + ], + "heads.score.1.norm2.weight": [ + 768 + ], + "heads.score.1.self_attn.in_proj_bias": [ + 2304 + ], + "heads.score.1.self_attn.in_proj_weight": [ + 2304, + 768 + ], + "heads.score.1.self_attn.out_proj.bias": [ + 768 + ], + "heads.score.1.self_attn.out_proj.weight": [ + 768, + 768 + ], + "scorers.choice.0.bias": [ + 768 + ], + "scorers.choice.0.weight": [ + 768 + ], + "scorers.choice.1.bias": [ + 768 + ], + "scorers.choice.1.weight": [ + 768, + 768 + ], + "scorers.choice.3.bias": [ + 1 + ], + "scorers.choice.3.weight": [ + 1, + 768 + ], + "scorers.noul.0.bias": [ + 768 + ], + "scorers.noul.0.weight": [ + 768 + ], + "scorers.noul.1.bias": [ + 768 + ], + "scorers.noul.1.weight": [ + 768, + 768 + ], + "scorers.noul.3.bias": [ + 1 + ], + "scorers.noul.3.weight": [ + 1, + 768 + ], + "scorers.score.0.bias": [ + 768 + ], + "scorers.score.0.weight": [ + 768 + ], + "scorers.score.1.bias": [ + 768 + ], + "scorers.score.1.weight": [ + 768, + 768 + ], + "scorers.score.3.bias": [ + 1 + ], + "scorers.score.3.weight": [ + 1, + 768 + ], + "type_embedding.weight": [ + 3, + 768 + ] + }, + "score_suffix_shapes": { + "score_blocks.0.attn.Wo.weight": [ + 768, + 768 + ], + "score_blocks.0.attn.Wqkv.weight": [ + 2304, + 768 + ], + "score_blocks.0.mlp.Wi.weight": [ + 2304, + 768 + ], + "score_blocks.0.mlp.Wo.weight": [ + 768, + 1152 + ], + "score_blocks.0.mlp_norm.weight": [ + 768 + ], + "score_blocks.1.attn.Wo.weight": [ + 768, + 768 + ], + "score_blocks.1.attn.Wqkv.weight": [ + 2304, + 768 + ], + "score_blocks.1.attn_norm.weight": [ + 768 + ], + "score_blocks.1.mlp.Wi.weight": [ + 2304, + 768 + ], + "score_blocks.1.mlp.Wo.weight": [ + 768, + 1152 + ], + "score_blocks.1.mlp_norm.weight": [ + 768 + ], + "score_blocks.10.attn.Wo.weight": [ + 768, + 768 + ], + "score_blocks.10.attn.Wqkv.weight": [ + 2304, + 768 + ], + "score_blocks.10.attn_norm.weight": [ + 768 + ], + "score_blocks.10.mlp.Wi.weight": [ + 2304, + 768 + ], + "score_blocks.10.mlp.Wo.weight": [ + 768, + 1152 + ], + "score_blocks.10.mlp_norm.weight": [ + 768 + ], + "score_blocks.11.attn.Wo.weight": [ + 768, + 768 + ], + "score_blocks.11.attn.Wqkv.weight": [ + 2304, + 768 + ], + "score_blocks.11.attn_norm.weight": [ + 768 + ], + "score_blocks.11.mlp.Wi.weight": [ + 2304, + 768 + ], + "score_blocks.11.mlp.Wo.weight": [ + 768, + 1152 + ], + "score_blocks.11.mlp_norm.weight": [ + 768 + ], + "score_blocks.12.attn.Wo.weight": [ + 768, + 768 + ], + "score_blocks.12.attn.Wqkv.weight": [ + 2304, + 768 + ], + "score_blocks.12.attn_norm.weight": [ + 768 + ], + "score_blocks.12.mlp.Wi.weight": [ + 2304, + 768 + ], + "score_blocks.12.mlp.Wo.weight": [ + 768, + 1152 + ], + "score_blocks.12.mlp_norm.weight": [ + 768 + ], + "score_blocks.13.attn.Wo.weight": [ + 768, + 768 + ], + "score_blocks.13.attn.Wqkv.weight": [ + 2304, + 768 + ], + "score_blocks.13.attn_norm.weight": [ + 768 + ], + "score_blocks.13.mlp.Wi.weight": [ + 2304, + 768 + ], + "score_blocks.13.mlp.Wo.weight": [ + 768, + 1152 + ], + "score_blocks.13.mlp_norm.weight": [ + 768 + ], + "score_blocks.14.attn.Wo.weight": [ + 768, + 768 + ], + "score_blocks.14.attn.Wqkv.weight": [ + 2304, + 768 + ], + "score_blocks.14.attn_norm.weight": [ + 768 + ], + "score_blocks.14.mlp.Wi.weight": [ + 2304, + 768 + ], + "score_blocks.14.mlp.Wo.weight": [ + 768, + 1152 + ], + "score_blocks.14.mlp_norm.weight": [ + 768 + ], + "score_blocks.15.attn.Wo.weight": [ + 768, + 768 + ], + "score_blocks.15.attn.Wqkv.weight": [ + 2304, + 768 + ], + "score_blocks.15.attn_norm.weight": [ + 768 + ], + "score_blocks.15.mlp.Wi.weight": [ + 2304, + 768 + ], + "score_blocks.15.mlp.Wo.weight": [ + 768, + 1152 + ], + "score_blocks.15.mlp_norm.weight": [ + 768 + ], + "score_blocks.16.attn.Wo.weight": [ + 768, + 768 + ], + "score_blocks.16.attn.Wqkv.weight": [ + 2304, + 768 + ], + "score_blocks.16.attn_norm.weight": [ + 768 + ], + "score_blocks.16.mlp.Wi.weight": [ + 2304, + 768 + ], + "score_blocks.16.mlp.Wo.weight": [ + 768, + 1152 + ], + "score_blocks.16.mlp_norm.weight": [ + 768 + ], + "score_blocks.17.attn.Wo.weight": [ + 768, + 768 + ], + "score_blocks.17.attn.Wqkv.weight": [ + 2304, + 768 + ], + "score_blocks.17.attn_norm.weight": [ + 768 + ], + "score_blocks.17.mlp.Wi.weight": [ + 2304, + 768 + ], + "score_blocks.17.mlp.Wo.weight": [ + 768, + 1152 + ], + "score_blocks.17.mlp_norm.weight": [ + 768 + ], + "score_blocks.18.attn.Wo.weight": [ + 768, + 768 + ], + "score_blocks.18.attn.Wqkv.weight": [ + 2304, + 768 + ], + "score_blocks.18.attn_norm.weight": [ + 768 + ], + "score_blocks.18.mlp.Wi.weight": [ + 2304, + 768 + ], + "score_blocks.18.mlp.Wo.weight": [ + 768, + 1152 + ], + "score_blocks.18.mlp_norm.weight": [ + 768 + ], + "score_blocks.19.attn.Wo.weight": [ + 768, + 768 + ], + "score_blocks.19.attn.Wqkv.weight": [ + 2304, + 768 + ], + "score_blocks.19.attn_norm.weight": [ + 768 + ], + "score_blocks.19.mlp.Wi.weight": [ + 2304, + 768 + ], + "score_blocks.19.mlp.Wo.weight": [ + 768, + 1152 + ], + "score_blocks.19.mlp_norm.weight": [ + 768 + ], + "score_blocks.2.attn.Wo.weight": [ + 768, + 768 + ], + "score_blocks.2.attn.Wqkv.weight": [ + 2304, + 768 + ], + "score_blocks.2.attn_norm.weight": [ + 768 + ], + "score_blocks.2.mlp.Wi.weight": [ + 2304, + 768 + ], + "score_blocks.2.mlp.Wo.weight": [ + 768, + 1152 + ], + "score_blocks.2.mlp_norm.weight": [ + 768 + ], + "score_blocks.20.attn.Wo.weight": [ + 768, + 768 + ], + "score_blocks.20.attn.Wqkv.weight": [ + 2304, + 768 + ], + "score_blocks.20.attn_norm.weight": [ + 768 + ], + "score_blocks.20.mlp.Wi.weight": [ + 2304, + 768 + ], + "score_blocks.20.mlp.Wo.weight": [ + 768, + 1152 + ], + "score_blocks.20.mlp_norm.weight": [ + 768 + ], + "score_blocks.21.attn.Wo.weight": [ + 768, + 768 + ], + "score_blocks.21.attn.Wqkv.weight": [ + 2304, + 768 + ], + "score_blocks.21.attn_norm.weight": [ + 768 + ], + "score_blocks.21.mlp.Wi.weight": [ + 2304, + 768 + ], + "score_blocks.21.mlp.Wo.weight": [ + 768, + 1152 + ], + "score_blocks.21.mlp_norm.weight": [ + 768 + ], + "score_blocks.3.attn.Wo.weight": [ + 768, + 768 + ], + "score_blocks.3.attn.Wqkv.weight": [ + 2304, + 768 + ], + "score_blocks.3.attn_norm.weight": [ + 768 + ], + "score_blocks.3.mlp.Wi.weight": [ + 2304, + 768 + ], + "score_blocks.3.mlp.Wo.weight": [ + 768, + 1152 + ], + "score_blocks.3.mlp_norm.weight": [ + 768 + ], + "score_blocks.4.attn.Wo.weight": [ + 768, + 768 + ], + "score_blocks.4.attn.Wqkv.weight": [ + 2304, + 768 + ], + "score_blocks.4.attn_norm.weight": [ + 768 + ], + "score_blocks.4.mlp.Wi.weight": [ + 2304, + 768 + ], + "score_blocks.4.mlp.Wo.weight": [ + 768, + 1152 + ], + "score_blocks.4.mlp_norm.weight": [ + 768 + ], + "score_blocks.5.attn.Wo.weight": [ + 768, + 768 + ], + "score_blocks.5.attn.Wqkv.weight": [ + 2304, + 768 + ], + "score_blocks.5.attn_norm.weight": [ + 768 + ], + "score_blocks.5.mlp.Wi.weight": [ + 2304, + 768 + ], + "score_blocks.5.mlp.Wo.weight": [ + 768, + 1152 + ], + "score_blocks.5.mlp_norm.weight": [ + 768 + ], + "score_blocks.6.attn.Wo.weight": [ + 768, + 768 + ], + "score_blocks.6.attn.Wqkv.weight": [ + 2304, + 768 + ], + "score_blocks.6.attn_norm.weight": [ + 768 + ], + "score_blocks.6.mlp.Wi.weight": [ + 2304, + 768 + ], + "score_blocks.6.mlp.Wo.weight": [ + 768, + 1152 + ], + "score_blocks.6.mlp_norm.weight": [ + 768 + ], + "score_blocks.7.attn.Wo.weight": [ + 768, + 768 + ], + "score_blocks.7.attn.Wqkv.weight": [ + 2304, + 768 + ], + "score_blocks.7.attn_norm.weight": [ + 768 + ], + "score_blocks.7.mlp.Wi.weight": [ + 2304, + 768 + ], + "score_blocks.7.mlp.Wo.weight": [ + 768, + 1152 + ], + "score_blocks.7.mlp_norm.weight": [ + 768 + ], + "score_blocks.8.attn.Wo.weight": [ + 768, + 768 + ], + "score_blocks.8.attn.Wqkv.weight": [ + 2304, + 768 + ], + "score_blocks.8.attn_norm.weight": [ + 768 + ], + "score_blocks.8.mlp.Wi.weight": [ + 2304, + 768 + ], + "score_blocks.8.mlp.Wo.weight": [ + 768, + 1152 + ], + "score_blocks.8.mlp_norm.weight": [ + 768 + ], + "score_blocks.9.attn.Wo.weight": [ + 768, + 768 + ], + "score_blocks.9.attn.Wqkv.weight": [ + 2304, + 768 + ], + "score_blocks.9.attn_norm.weight": [ + 768 + ], + "score_blocks.9.mlp.Wi.weight": [ + 2304, + 768 + ], + "score_blocks.9.mlp.Wo.weight": [ + 768, + 1152 + ], + "score_blocks.9.mlp_norm.weight": [ + 768 + ], + "score_final_norm.weight": [ + 768 + ] + } +} diff --git a/native/MANIFEST.json b/native/MANIFEST.json new file mode 100644 index 0000000000000000000000000000000000000000..f7d6e31495b30dafafdcad0b8741fc5f9902b16b --- /dev/null +++ b/native/MANIFEST.json @@ -0,0 +1,125 @@ +{ + "files": { + "INVENTORY.json": { + "bytes": 31007, + "sha256": "06aed00ac4571582b992a995a793875804a0546bdae1461c4161b30fba2143a9" + }, + "STATE_LAYOUT.json": { + "bytes": 10971, + "sha256": "a6b24716e2b21240d327ba8763a73f1b54862bc86d5e1f92bb9e670be968e494" + }, + "__init__.py": { + "bytes": 78, + "sha256": "1afb9dbcfc379f28049486fe6acb7acfdaa8d16b6773d6084ca4b82ead26f010" + }, + "artifacts.py": { + "bytes": 7958, + "sha256": "c98adaf6d782e9fc592cd78b1307d63faffa9ba622b9d508c63337c29156e4d3" + }, + "choice_encoder.safetensors": { + "bytes": 441337216, + "sha256": "9516cc841c485c98b27b4f63d2ea8e604fe8064121173064b113da8a6bf57ef6" + }, + "contract.py": { + "bytes": 10886, + "sha256": "51a24800792bb3e5bf2a11f50f7bc384770641f4f44ed46277dc01e891bf4726" + }, + "decision_config.json": { + "bytes": 14308, + "sha256": "1fefb4ad7dede00c4633bb2ce5fc44fb04237207b9e9ce90fcf8ea8b01a1328d" + }, + "decision_heads.safetensors": { + "bytes": 177241884, + "sha256": "bce3ee658a978a19c48c605b921cff994f892e7fc17abd6f8645f07428fbc36f" + }, + "encoder/config.json": { + "bytes": 2769, + "sha256": "7aff915e9f159305e0bef3eb0206416f99b8560260b8969b35f1dcd54aaad1a5" + }, + "encoder/model.safetensors": { + "bytes": 1227771752, + "sha256": "daaafd81c4ed767d203e24226d78e792800068a7dac344aa043844b82a6306c7" + }, + "infer.py": { + "bytes": 1095, + "sha256": "9d14841935c836a0765c705d5a24e8fa97437ae35fff65bc6e690b397f0d0315" + }, + "model.py": { + "bytes": 11098, + "sha256": "8fe91e2a77f372d32117e62281ccb58a054c144228b73b4987f1e1a3fca9ee5b" + }, + "packing.py": { + "bytes": 6748, + "sha256": "f1dab7f032f4aaab27118606397ca67d71b1f4a2ba49e65bebcfd83a64b27819" + }, + "policy/__init__.py": { + "bytes": 131, + "sha256": "0cb0bad7d3d0f258ecf95f52297ee8733a75f8504cabb86b9aea4b8256885114" + }, + "policy/artifacts.py": { + "bytes": 7877, + "sha256": "5838d2ea747d912789f3fc813a212af919cbc24ee185073576d9fbe32ed31cf2" + }, + "policy/contract.py": { + "bytes": 4334, + "sha256": "0b8eeeeebe9e367564a3c57f48364c5e94ae060f759e220b1326caf152276b67" + }, + "policy/infer.py": { + "bytes": 1545, + "sha256": "672bfae798060d51fcccac03143ac881fe7512893ffdacdb36b0618d4b160896" + }, + "policy/model.py": { + "bytes": 7678, + "sha256": "eaeebac8fd6bd96243a5c4b6225c86ee3359c067784f1595979ee603cba9acca" + }, + "policy/packing.py": { + "bytes": 6748, + "sha256": "f1dab7f032f4aaab27118606397ca67d71b1f4a2ba49e65bebcfd83a64b27819" + }, + "policy/reference/__init__.py": { + "bytes": 192, + "sha256": "c8d6fd86207752407f94437544a97266b9529f88d2b341d35aafcd911c942b28" + }, + "policy/reference/artifacts.py": { + "bytes": 9342, + "sha256": "3f59428b7a05608ac01c73c7db89c38fcc3e2c5030219df8d053af92b32e6159" + }, + "policy/reference/infer.py": { + "bytes": 1545, + "sha256": "672bfae798060d51fcccac03143ac881fe7512893ffdacdb36b0618d4b160896" + }, + "policy/reference/model.py": { + "bytes": 5060, + "sha256": "733ea4a48ea03089dac8c3e6a175920704fac50f93313b0214cec4052f25bd26" + }, + "policy/reference/modernbert_sdpa_layout.py": { + "bytes": 3365, + "sha256": "0fb3a22db93ad76e30dfbfb3011de442149d97c565e139a3a55738287a1fbbc8" + }, + "policy/reference/packing.py": { + "bytes": 6748, + "sha256": "f1dab7f032f4aaab27118606397ca67d71b1f4a2ba49e65bebcfd83a64b27819" + }, + "score_encoder.safetensors": { + "bytes": 441337080, + "sha256": "4f45795977846ef4c31b34e4f35bf95dabf5951d71358bab32c9a94814a84a53" + }, + "tokenizer/special_tokens_map.json": { + "bytes": 1051, + "sha256": "6e3204c5a4004719c185007a745d1940471a6fb02521cfa0a00306dbb6bc5903" + }, + "tokenizer/tokenizer.json": { + "bytes": 34363188, + "sha256": "609d8f4c067cd3950f88594c5a802616cea245823836ef5848ee4fc40aab5b6f" + }, + "tokenizer/tokenizer_config.json": { + "bytes": 46470, + "sha256": "74a259bb1a3811a7e3028adcd07a65765d866d5e66e0e883f0994ccfa67e8455" + }, + "training_policy.py": { + "bytes": 4761, + "sha256": "6f7fad91c5089b304a1d637b594027a9379a63600792766ce0abca6b24bd2ec5" + } + }, + "schema": "decision.files.v1" +} diff --git a/native/STATE_LAYOUT.json b/native/STATE_LAYOUT.json new file mode 100644 index 0000000000000000000000000000000000000000..ccd75398f35c7317a9ebdc12cfe8d37ddaa8ace2 --- /dev/null +++ b/native/STATE_LAYOUT.json @@ -0,0 +1,163 @@ +{ + "arm": "all22", + "choice_and_noul_heads_frozen": true, + "counts": { + "frozen_parameters": 446810114, + "frozen_tensors": 327, + "integrated_choice_suffix_tensors": 132, + "score_suffix_parameters": 110330880, + "score_suffix_tensors": 132, + "shared_encoder_tensors": 134, + "trainable_parameters": 125099521, + "trainable_tensors": 162, + "type_and_head_tensors": 91, + "unique_parameters": 571909635, + "unique_tensors": 489 + }, + "cross_record_or_request_cache": false, + "embedding_storage": "encoder.embeddings; exactly one frozen vocabulary", + "forward_batch_policy": "one embedding pass; all encoder paths keep full B x L; only heads gather type rows", + "integrated_choice_storage": "choice_blocks + choice_final_norm; original trained values, all frozen", + "merged_weights": false, + "score_shared_prefix_layers": 0, + "score_suffix_copy_map": { + "score_blocks.0.attn.Wo.weight": "encoder.layers.0.attn.Wo.weight", + "score_blocks.0.attn.Wqkv.weight": "encoder.layers.0.attn.Wqkv.weight", + "score_blocks.0.mlp.Wi.weight": "encoder.layers.0.mlp.Wi.weight", + "score_blocks.0.mlp.Wo.weight": "encoder.layers.0.mlp.Wo.weight", + "score_blocks.0.mlp_norm.weight": "encoder.layers.0.mlp_norm.weight", + "score_blocks.1.attn.Wo.weight": "encoder.layers.1.attn.Wo.weight", + "score_blocks.1.attn.Wqkv.weight": "encoder.layers.1.attn.Wqkv.weight", + "score_blocks.1.attn_norm.weight": "encoder.layers.1.attn_norm.weight", + "score_blocks.1.mlp.Wi.weight": "encoder.layers.1.mlp.Wi.weight", + "score_blocks.1.mlp.Wo.weight": "encoder.layers.1.mlp.Wo.weight", + "score_blocks.1.mlp_norm.weight": "encoder.layers.1.mlp_norm.weight", + "score_blocks.10.attn.Wo.weight": "encoder.layers.10.attn.Wo.weight", + "score_blocks.10.attn.Wqkv.weight": "encoder.layers.10.attn.Wqkv.weight", + "score_blocks.10.attn_norm.weight": "encoder.layers.10.attn_norm.weight", + "score_blocks.10.mlp.Wi.weight": "encoder.layers.10.mlp.Wi.weight", + "score_blocks.10.mlp.Wo.weight": "encoder.layers.10.mlp.Wo.weight", + "score_blocks.10.mlp_norm.weight": "encoder.layers.10.mlp_norm.weight", + "score_blocks.11.attn.Wo.weight": "encoder.layers.11.attn.Wo.weight", + "score_blocks.11.attn.Wqkv.weight": "encoder.layers.11.attn.Wqkv.weight", + "score_blocks.11.attn_norm.weight": "encoder.layers.11.attn_norm.weight", + "score_blocks.11.mlp.Wi.weight": "encoder.layers.11.mlp.Wi.weight", + "score_blocks.11.mlp.Wo.weight": "encoder.layers.11.mlp.Wo.weight", + "score_blocks.11.mlp_norm.weight": "encoder.layers.11.mlp_norm.weight", + "score_blocks.12.attn.Wo.weight": "encoder.layers.12.attn.Wo.weight", + "score_blocks.12.attn.Wqkv.weight": "encoder.layers.12.attn.Wqkv.weight", + "score_blocks.12.attn_norm.weight": "encoder.layers.12.attn_norm.weight", + "score_blocks.12.mlp.Wi.weight": "encoder.layers.12.mlp.Wi.weight", + "score_blocks.12.mlp.Wo.weight": "encoder.layers.12.mlp.Wo.weight", + "score_blocks.12.mlp_norm.weight": "encoder.layers.12.mlp_norm.weight", + "score_blocks.13.attn.Wo.weight": "encoder.layers.13.attn.Wo.weight", + "score_blocks.13.attn.Wqkv.weight": "encoder.layers.13.attn.Wqkv.weight", + "score_blocks.13.attn_norm.weight": "encoder.layers.13.attn_norm.weight", + "score_blocks.13.mlp.Wi.weight": "encoder.layers.13.mlp.Wi.weight", + "score_blocks.13.mlp.Wo.weight": "encoder.layers.13.mlp.Wo.weight", + "score_blocks.13.mlp_norm.weight": "encoder.layers.13.mlp_norm.weight", + "score_blocks.14.attn.Wo.weight": "encoder.layers.14.attn.Wo.weight", + "score_blocks.14.attn.Wqkv.weight": "encoder.layers.14.attn.Wqkv.weight", + "score_blocks.14.attn_norm.weight": "encoder.layers.14.attn_norm.weight", + "score_blocks.14.mlp.Wi.weight": "encoder.layers.14.mlp.Wi.weight", + "score_blocks.14.mlp.Wo.weight": "encoder.layers.14.mlp.Wo.weight", + "score_blocks.14.mlp_norm.weight": "encoder.layers.14.mlp_norm.weight", + "score_blocks.15.attn.Wo.weight": "encoder.layers.15.attn.Wo.weight", + "score_blocks.15.attn.Wqkv.weight": "encoder.layers.15.attn.Wqkv.weight", + "score_blocks.15.attn_norm.weight": "encoder.layers.15.attn_norm.weight", + "score_blocks.15.mlp.Wi.weight": "encoder.layers.15.mlp.Wi.weight", + "score_blocks.15.mlp.Wo.weight": "encoder.layers.15.mlp.Wo.weight", + "score_blocks.15.mlp_norm.weight": "encoder.layers.15.mlp_norm.weight", + "score_blocks.16.attn.Wo.weight": "encoder.layers.16.attn.Wo.weight", + "score_blocks.16.attn.Wqkv.weight": "encoder.layers.16.attn.Wqkv.weight", + "score_blocks.16.attn_norm.weight": "encoder.layers.16.attn_norm.weight", + "score_blocks.16.mlp.Wi.weight": "encoder.layers.16.mlp.Wi.weight", + "score_blocks.16.mlp.Wo.weight": "encoder.layers.16.mlp.Wo.weight", + "score_blocks.16.mlp_norm.weight": "encoder.layers.16.mlp_norm.weight", + "score_blocks.17.attn.Wo.weight": "encoder.layers.17.attn.Wo.weight", + "score_blocks.17.attn.Wqkv.weight": "encoder.layers.17.attn.Wqkv.weight", + "score_blocks.17.attn_norm.weight": "encoder.layers.17.attn_norm.weight", + "score_blocks.17.mlp.Wi.weight": "encoder.layers.17.mlp.Wi.weight", + "score_blocks.17.mlp.Wo.weight": "encoder.layers.17.mlp.Wo.weight", + "score_blocks.17.mlp_norm.weight": "encoder.layers.17.mlp_norm.weight", + "score_blocks.18.attn.Wo.weight": "encoder.layers.18.attn.Wo.weight", + "score_blocks.18.attn.Wqkv.weight": "encoder.layers.18.attn.Wqkv.weight", + "score_blocks.18.attn_norm.weight": "encoder.layers.18.attn_norm.weight", + "score_blocks.18.mlp.Wi.weight": "encoder.layers.18.mlp.Wi.weight", + "score_blocks.18.mlp.Wo.weight": "encoder.layers.18.mlp.Wo.weight", + "score_blocks.18.mlp_norm.weight": "encoder.layers.18.mlp_norm.weight", + "score_blocks.19.attn.Wo.weight": "encoder.layers.19.attn.Wo.weight", + "score_blocks.19.attn.Wqkv.weight": "encoder.layers.19.attn.Wqkv.weight", + "score_blocks.19.attn_norm.weight": "encoder.layers.19.attn_norm.weight", + "score_blocks.19.mlp.Wi.weight": "encoder.layers.19.mlp.Wi.weight", + "score_blocks.19.mlp.Wo.weight": "encoder.layers.19.mlp.Wo.weight", + "score_blocks.19.mlp_norm.weight": "encoder.layers.19.mlp_norm.weight", + "score_blocks.2.attn.Wo.weight": "encoder.layers.2.attn.Wo.weight", + "score_blocks.2.attn.Wqkv.weight": "encoder.layers.2.attn.Wqkv.weight", + "score_blocks.2.attn_norm.weight": "encoder.layers.2.attn_norm.weight", + "score_blocks.2.mlp.Wi.weight": "encoder.layers.2.mlp.Wi.weight", + "score_blocks.2.mlp.Wo.weight": "encoder.layers.2.mlp.Wo.weight", + "score_blocks.2.mlp_norm.weight": "encoder.layers.2.mlp_norm.weight", + "score_blocks.20.attn.Wo.weight": "encoder.layers.20.attn.Wo.weight", + "score_blocks.20.attn.Wqkv.weight": "encoder.layers.20.attn.Wqkv.weight", + "score_blocks.20.attn_norm.weight": "encoder.layers.20.attn_norm.weight", + "score_blocks.20.mlp.Wi.weight": "encoder.layers.20.mlp.Wi.weight", + "score_blocks.20.mlp.Wo.weight": "encoder.layers.20.mlp.Wo.weight", + "score_blocks.20.mlp_norm.weight": "encoder.layers.20.mlp_norm.weight", + "score_blocks.21.attn.Wo.weight": "encoder.layers.21.attn.Wo.weight", + "score_blocks.21.attn.Wqkv.weight": "encoder.layers.21.attn.Wqkv.weight", + "score_blocks.21.attn_norm.weight": "encoder.layers.21.attn_norm.weight", + "score_blocks.21.mlp.Wi.weight": "encoder.layers.21.mlp.Wi.weight", + "score_blocks.21.mlp.Wo.weight": "encoder.layers.21.mlp.Wo.weight", + "score_blocks.21.mlp_norm.weight": "encoder.layers.21.mlp_norm.weight", + "score_blocks.3.attn.Wo.weight": "encoder.layers.3.attn.Wo.weight", + "score_blocks.3.attn.Wqkv.weight": "encoder.layers.3.attn.Wqkv.weight", + "score_blocks.3.attn_norm.weight": "encoder.layers.3.attn_norm.weight", + "score_blocks.3.mlp.Wi.weight": "encoder.layers.3.mlp.Wi.weight", + "score_blocks.3.mlp.Wo.weight": "encoder.layers.3.mlp.Wo.weight", + "score_blocks.3.mlp_norm.weight": "encoder.layers.3.mlp_norm.weight", + "score_blocks.4.attn.Wo.weight": "encoder.layers.4.attn.Wo.weight", + "score_blocks.4.attn.Wqkv.weight": "encoder.layers.4.attn.Wqkv.weight", + "score_blocks.4.attn_norm.weight": "encoder.layers.4.attn_norm.weight", + "score_blocks.4.mlp.Wi.weight": "encoder.layers.4.mlp.Wi.weight", + "score_blocks.4.mlp.Wo.weight": "encoder.layers.4.mlp.Wo.weight", + "score_blocks.4.mlp_norm.weight": "encoder.layers.4.mlp_norm.weight", + "score_blocks.5.attn.Wo.weight": "encoder.layers.5.attn.Wo.weight", + "score_blocks.5.attn.Wqkv.weight": "encoder.layers.5.attn.Wqkv.weight", + "score_blocks.5.attn_norm.weight": "encoder.layers.5.attn_norm.weight", + "score_blocks.5.mlp.Wi.weight": "encoder.layers.5.mlp.Wi.weight", + "score_blocks.5.mlp.Wo.weight": "encoder.layers.5.mlp.Wo.weight", + "score_blocks.5.mlp_norm.weight": "encoder.layers.5.mlp_norm.weight", + "score_blocks.6.attn.Wo.weight": "encoder.layers.6.attn.Wo.weight", + "score_blocks.6.attn.Wqkv.weight": "encoder.layers.6.attn.Wqkv.weight", + "score_blocks.6.attn_norm.weight": "encoder.layers.6.attn_norm.weight", + "score_blocks.6.mlp.Wi.weight": "encoder.layers.6.mlp.Wi.weight", + "score_blocks.6.mlp.Wo.weight": "encoder.layers.6.mlp.Wo.weight", + "score_blocks.6.mlp_norm.weight": "encoder.layers.6.mlp_norm.weight", + "score_blocks.7.attn.Wo.weight": "encoder.layers.7.attn.Wo.weight", + "score_blocks.7.attn.Wqkv.weight": "encoder.layers.7.attn.Wqkv.weight", + "score_blocks.7.attn_norm.weight": "encoder.layers.7.attn_norm.weight", + "score_blocks.7.mlp.Wi.weight": "encoder.layers.7.mlp.Wi.weight", + "score_blocks.7.mlp.Wo.weight": "encoder.layers.7.mlp.Wo.weight", + "score_blocks.7.mlp_norm.weight": "encoder.layers.7.mlp_norm.weight", + "score_blocks.8.attn.Wo.weight": "encoder.layers.8.attn.Wo.weight", + "score_blocks.8.attn.Wqkv.weight": "encoder.layers.8.attn.Wqkv.weight", + "score_blocks.8.attn_norm.weight": "encoder.layers.8.attn_norm.weight", + "score_blocks.8.mlp.Wi.weight": "encoder.layers.8.mlp.Wi.weight", + "score_blocks.8.mlp.Wo.weight": "encoder.layers.8.mlp.Wo.weight", + "score_blocks.8.mlp_norm.weight": "encoder.layers.8.mlp_norm.weight", + "score_blocks.9.attn.Wo.weight": "encoder.layers.9.attn.Wo.weight", + "score_blocks.9.attn.Wqkv.weight": "encoder.layers.9.attn.Wqkv.weight", + "score_blocks.9.attn_norm.weight": "encoder.layers.9.attn_norm.weight", + "score_blocks.9.mlp.Wi.weight": "encoder.layers.9.mlp.Wi.weight", + "score_blocks.9.mlp.Wo.weight": "encoder.layers.9.mlp.Wo.weight", + "score_blocks.9.mlp_norm.weight": "encoder.layers.9.mlp_norm.weight", + "score_final_norm.weight": "encoder.final_norm.weight" + }, + "score_suffix_copy_source": "original shared encoder, never trained Choice suffix", + "shared_encoder_storage": "encoder.layers + encoder.final_norm; all frozen", + "trainability_policy": "score_private_all22_and_head", + "training_arm": "S22", + "type_embedding_frozen": true, + "unique_registered_state": true +} diff --git a/native/__init__.py b/native/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..a55caa63ad8fac8a1e615977d7e1d98de866d07d --- /dev/null +++ b/native/__init__.py @@ -0,0 +1 @@ +"""Explicit Choice encoder fork research graph; no legacy native fallback.""" diff --git a/native/artifacts.py b/native/artifacts.py new file mode 100644 index 0000000000000000000000000000000000000000..f05d5b7ceaf7ef3fe43e8e3d714fd0cf7a03fa86 --- /dev/null +++ b/native/artifacts.py @@ -0,0 +1,126 @@ +"""Strict S0/S6/S22 Score-path native with explicit independent suffix payload.""" +from pathlib import Path +import shutil +import torch +try: + from .contract import (ARCHITECTURE, SCHEMA, POLICY, KINDS, TRANSFORMERS_VERSION, + PARENT_MANIFEST_SHA256, counts, head_shapes, encoder_shapes, suffix_shapes, + state_layout, validate_config, validate_encoder_config, native_config, inventory, score_suffix_shapes) + from .model import DecisionModel, runtime_gate, validate_tensor_map + from .packing import MarkerCollator + from .policy.model import DecisionModel as PolicyModel + from .policy.artifacts import RUNTIME_FILES as POLICY_RUNTIME_FILES + from .policy.reference.artifacts import file_ref, read_json, write_json, verify_export, install_layout +except ImportError: + from contract import (ARCHITECTURE, SCHEMA, POLICY, KINDS, TRANSFORMERS_VERSION, + PARENT_MANIFEST_SHA256, counts, head_shapes, encoder_shapes, suffix_shapes, + state_layout, validate_config, validate_encoder_config, native_config, inventory, score_suffix_shapes) + from model import DecisionModel, runtime_gate, validate_tensor_map + from packing import MarkerCollator + from policy.model import DecisionModel as PolicyModel + from policy.artifacts import RUNTIME_FILES as POLICY_RUNTIME_FILES + from policy.reference.artifacts import file_ref, read_json, write_json, verify_export, install_layout + +ROOT = Path(__file__).resolve().parent +RUNTIME_FILES = ('__init__.py','model.py','contract.py','packing.py','artifacts.py','infer.py','training_policy.py') + tuple('policy/'+n for n in POLICY_RUNTIME_FILES) + + +def save_weights(model, path): + from safetensors.torch import save_file + model.verify_inventory(check_values=True) + path = Path(path) + save_file({n:v.detach().cpu().contiguous() for n,v in model.state_dict().items()}, str(path)) + + +def restore_weights(model, path): + from safetensors.torch import load_file + try:from .contract import all_shapes + except ImportError:from contract import all_shapes + state = load_file(str(path), device='cpu') + validate_tensor_map(state, all_shapes(model.arm, model.training_arm)) + model.load_state_dict(state, strict=True) + model.verify_inventory(check_values=True) + + +def partition_state(model): + model.verify_inventory(check_values=True) + state = model.state_dict() + base = model.encoder.state_dict() + heads = {n:state[n] for n in head_shapes()} + suffix = {n:state[n] for n in suffix_shapes(model.arm)} + validate_tensor_map(base, encoder_shapes());validate_tensor_map(heads, head_shapes()) + validate_tensor_map(suffix, suffix_shapes(model.arm)) + score={n:state[n] for n in score_suffix_shapes(model.training_arm)} + validate_tensor_map(score,score_suffix_shapes(model.training_arm)) + return base,heads,suffix,score + + +def export_model(model, tokenizer, path, config, provenance): + from safetensors.torch import save_file + runtime_gate();validate_encoder_config(model.encoder.config) + config = dict(config);config['provenance'] = provenance;validate_config(config) + if config['arm'] != model.arm or config['training_arm'] != model.training_arm or config['head'] != model.head_config: + raise ValueError('Model/native identity mismatch') + base,heads,suffix,score = partition_state(model) + path = Path(path);path.mkdir(parents=True, exist_ok=False);(path/'encoder').mkdir() + ec = model.encoder.config.to_dict();ec.pop('_name_or_path',None);ec['architectures']=['ModernBertModel'];ec['reference_compile']=False + write_json(path/'encoder/config.json',ec) + for filename,state in (('encoder/model.safetensors',base),('decision_heads.safetensors',heads),('choice_encoder.safetensors',suffix)): + save_file({n:v.detach().cpu().contiguous() for n,v in state.items()},str(path/filename)) + if model.training_arm!='S0': + save_file({n:v.detach().cpu().contiguous() for n,v in score.items()},str(path/'score_encoder.safetensors')) + tokenizer.save_pretrained(path/'tokenizer') + for name in RUNTIME_FILES: + target=path/name;target.parent.mkdir(parents=True,exist_ok=True);shutil.copyfile(ROOT/name,target) + write_json(path/'decision_config.json',config) + write_json(path/'INVENTORY.json',inventory(model.arm, model.training_arm)) + write_json(path/'STATE_LAYOUT.json',state_layout(model.arm, model.training_arm)) + refs={p.relative_to(path).as_posix():file_ref(p) for p in sorted(path.rglob('*')) if p.is_file()} + write_json(path/'MANIFEST.json',{'schema':'decision.files.v1','files':refs}) + verify_native(path) + return file_ref(path/'MANIFEST.json') + + +def verify_native(path): + path=Path(path);manifest=verify_export(path) + cfg=validate_config(read_json(path/'decision_config.json')) + required=set(RUNTIME_FILES)|{'encoder/config.json','encoder/model.safetensors','decision_heads.safetensors', + 'choice_encoder.safetensors','decision_config.json','INVENTORY.json','STATE_LAYOUT.json'} + if cfg['training_arm']!='S0':required.add('score_encoder.safetensors') + if not required.issubset(manifest['files']):raise ValueError('Incomplete score-path native') + extras=set(manifest['files'])-required + if not extras or any(not n.startswith('tokenizer/') for n in extras): + raise ValueError('Native permits only required runtime/state and tokenizer files') + for name in RUNTIME_FILES: + if file_ref(ROOT/name)!=manifest['files'][name]:raise ValueError('Reviewed runtime mismatch: '+name) + if read_json(path/'INVENTORY.json')!=inventory(cfg['arm'], cfg['training_arm']) or read_json(path/'STATE_LAYOUT.json')!=state_layout(cfg['arm'], cfg['training_arm']): + raise ValueError('Native inventory/layout mismatch') + validate_encoder_config(read_json(path/'encoder/config.json')) + return manifest,cfg + + +def load_export(path, device='cpu'): + from transformers import AutoConfig, AutoModel, AutoTokenizer + from safetensors.torch import load_file + runtime_gate();path=Path(path);manifest,cfg=verify_native(path) + # Check persisted dtype, exact key/shape and finite values before loading; no casting acceptance. + base=load_file(str(path/'encoder/model.safetensors'),device='cpu');validate_tensor_map(base,encoder_shapes()) + heads=load_file(str(path/'decision_heads.safetensors'),device='cpu');validate_tensor_map(heads,head_shapes()) + suffix=load_file(str(path/'choice_encoder.safetensors'),device='cpu');validate_tensor_map(suffix,suffix_shapes(cfg['arm'])) + score={} if cfg['training_arm']=='S0' else load_file(str(path/'score_encoder.safetensors'),device='cpu') + validate_tensor_map(score,score_suffix_shapes(cfg['training_arm'])) + ec=AutoConfig.from_pretrained(path/'encoder',local_files_only=True,trust_remote_code=False) + validate_encoder_config(ec, require_sdpa=False) + encoder=AutoModel.from_config(ec,trust_remote_code=False,torch_dtype=torch.float32,attn_implementation='sdpa') + validate_encoder_config(encoder.config) + encoder.load_state_dict(base,strict=True);del base;install_layout(encoder,True) + policy_model=PolicyModel(encoder,cfg['head']) + policy_model.load_state_dict({**{'encoder.'+n:v for n,v in encoder.state_dict().items()},**heads},strict=True) + del heads + model=DecisionModel(policy_model,cfg['arm'],cfg['training_arm']) + # Constructor copies are temporary; strict persisted Choice and independent Score tensors replace them. + model.load_state_dict({**model.state_dict(),**suffix,**score},strict=True);del suffix,score + model.verify_inventory(check_values=True);model.to(device).eval() + tokenizer=AutoTokenizer.from_pretrained(path/'tokenizer',local_files_only=True,trust_remote_code=False) + if verify_native(path)!=(manifest,cfg):raise ValueError('Native changed while loading') + return model,MarkerCollator(tokenizer,**cfg['packing']),cfg diff --git a/native/choice_encoder.safetensors b/native/choice_encoder.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..304a24010b0ce16ca6e8490549384c29ea69b0aa --- /dev/null +++ b/native/choice_encoder.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9516cc841c485c98b27b4f63d2ea8e604fe8064121173064b113da8a6bf57ef6 +size 441337216 diff --git a/native/contract.py b/native/contract.py new file mode 100644 index 0000000000000000000000000000000000000000..1f6f411bcaaffb999b1014ce53ff9e2ab2af01b2 --- /dev/null +++ b/native/contract.py @@ -0,0 +1,221 @@ +"""Score-only S0/S6/S22 native and symbolic inventory; no model imports.""" + +KINDS = ('choice', 'noul', 'score') + +PARENT_MANIFEST_SHA256 = 'ab4d18e3f7aa2185907c796ff3cb37848033a641376daee2bdecbbf709829831' + +EXPECTED_HEAD_CONFIG = {'head_layers': 2, 'head_heads': 12, 'head_dropout': 0.1, + 'attention_mode': 'sdpa', 'gradient_checkpointing': True} + +def parent_head_shapes(d=768): + if type(d) is not int or d < 1: + raise ValueError('Positive integer hidden size') + layer = {'self_attn.in_proj_weight': (3*d, d), 'self_attn.in_proj_bias': (3*d,), + 'self_attn.out_proj.weight': (d, d), 'self_attn.out_proj.bias': (d,), + 'linear1.weight': (4*d, d), 'linear1.bias': (4*d,), + 'linear2.weight': (d, 4*d), 'linear2.bias': (d,), + 'norm1.weight': (d,), 'norm1.bias': (d,), 'norm2.weight': (d,), 'norm2.bias': (d,)} + result = {f'head.{i}.{n}': shape for i in range(2) for n, shape in layer.items()} + result.update({'type_embedding.weight': (3, d), 'scorer.0.weight': (d,), 'scorer.0.bias': (d,), + 'scorer.1.weight': (d, d), 'scorer.1.bias': (d,), + 'scorer.3.weight': (1, d), 'scorer.3.bias': (1,)}) + return result + +def head_mapping(): + result = {'type_embedding.weight': 'type_embedding.weight'} + for name in parent_head_shapes(): + if name.startswith('head.'): + result.update({f'heads.{kind}.{name[5:]}': name for kind in KINDS}) + elif name.startswith('scorer.'): + result.update({f'scorers.{kind}.{name[7:]}': name for kind in KINDS}) + return result + +def head_shapes(d=768): + parent = parent_head_shapes(d) + return {dst: parent[src] for dst, src in head_mapping().items()} + +def numel(shape): + out = 1 + for n in shape: out *= n + return out + +TRANSFORMERS_VERSION = '4.57.6' + +ARMS = {'all22': 0} + +ENCODER_CONFIG = { + 'model_type': 'modernbert', 'vocab_size': 256000, 'hidden_size': 768, + 'intermediate_size': 1152, 'num_hidden_layers': 22, 'num_attention_heads': 12, + 'attention_bias': False, 'mlp_bias': False, 'norm_bias': False, + 'attention_dropout': 0.0, 'embedding_dropout': 0.0, 'mlp_dropout': 0.0, + 'norm_eps': 1e-5, 'hidden_activation': 'gelu', 'local_attention': 128, + 'global_attn_every_n_layers': 3, 'global_rope_theta': 160000, + 'local_rope_theta': 160000, 'max_position_embeddings': 32768, + 'reference_compile': False, 'pad_token_id': 0, + 'rope_scaling': {'beta_fast': 32.0, 'beta_slow': 1.0, 'factor': 4.0, + 'original_max_position_embeddings': 8192, 'rope_type': 'yarn', 'truncate': True}, +} + +def split_index(arm): + if arm not in ARMS: + raise ValueError('Integrated all22 architecture required') + return ARMS[arm] + +def encoder_shapes(): + out = {'embeddings.tok_embeddings.weight': (256000, 768), 'embeddings.norm.weight': (768,)} + for i in range(22): + if i: + out[f'layers.{i}.attn_norm.weight'] = (768,) + for name, shape in {'attn.Wqkv.weight': (2304, 768), 'attn.Wo.weight': (768, 768), + 'mlp_norm.weight': (768,), 'mlp.Wi.weight': (2304, 768), + 'mlp.Wo.weight': (768, 1152)}.items(): + out[f'layers.{i}.{name}'] = shape + out['final_norm.weight'] = (768,) + return out + +def suffix_mapping(arm): + split = split_index(arm) + out = {} + for name in encoder_shapes(): + if name.startswith('layers.'): + _, index, rest = name.split('.', 2) + if int(index) >= split: + out[f'choice_blocks.{int(index)-split}.{rest}'] = 'encoder.'+name + out['choice_final_norm.weight'] = 'encoder.final_norm.weight' + return out + +def suffix_shapes(arm): + enc = {'encoder.'+n:s for n,s in encoder_shapes().items()} + return {dst: enc[src] for dst,src in suffix_mapping(arm).items()} + +def validate_encoder_config(config, require_sdpa=True): + value = config if isinstance(config, dict) else dict(config.to_dict()) + if not isinstance(config, dict): + # ModernBertConfig.to_dict deliberately removes this runtime-only flag. + value['reference_compile'] = getattr(config, 'reference_compile', None) + if value.get('reference_compile') is not False: + raise ValueError('Explicit reference_compile=False required') + for key, expected in ENCODER_CONFIG.items(): + if value.get(key) != expected: + raise ValueError('Pinned ModernBERT configuration mismatch: '+key) + if require_sdpa and not isinstance(config, dict) and config._attn_implementation != 'sdpa': + raise ValueError('Only pinned SDPA encoder forward is supported') + return config + +ARCHITECTURE = 'vela_decision_score_path_capacity_v1' +SCHEMA = 'decision.nano.score_path_capacity.v1' +POLICY = 'frozen_integrated_choice_noul_score_private_suffix_full_batch_v1' +TRAINING_ARMS = {'S0':'score_head_only', 'S6':'score_private_last6_and_head', + 'S22':'score_private_all22_and_head'} +SCORE_SPLITS = {'S0':22, 'S6':16, 'S22':0} + + +def training_arm(value): + if value in TRAINING_ARMS:return value + inverse={v:k for k,v in TRAINING_ARMS.items()} + if value not in inverse:raise ValueError('Explicit S0, S6 or S22 score policy required') + return inverse[value] + + +def training_policy(arm): + if arm not in TRAINING_ARMS:raise ValueError('Explicit S0, S6 or S22 arm required') + return TRAINING_ARMS[arm] + + +def score_split(arm): + training_policy(arm) + return SCORE_SPLITS[arm] + + +def score_suffix_mapping(arm): + split=score_split(arm) + if arm=='S0':return {} + out={} + for name in encoder_shapes(): + if name.startswith('layers.'): + _,index,rest=name.split('.',2) + if int(index)>=split:out[f'score_blocks.{int(index)-split}.{rest}']='encoder.'+name + out['score_final_norm.weight']='encoder.final_norm.weight' + return out + + +def score_suffix_shapes(arm): + enc={'encoder.'+n:s for n,s in encoder_shapes().items()} + return {dst:enc[src] for dst,src in score_suffix_mapping(arm).items()} + + +def all_shapes(arm, training_arm): + split_index(arm);training_policy(training_arm) + return {**{'encoder.'+n:s for n,s in encoder_shapes().items()}, **head_shapes(), + **suffix_shapes(arm), **score_suffix_shapes(training_arm)} + + +def trainable_name(name, policy): + arm=training_arm(policy) + return name.startswith(('heads.score.','scorers.score.')) or (arm!='S0' and name.startswith(('score_blocks.','score_final_norm.'))) + + +def parameter_group(name, arm): + if not trainable_name(name,arm):return None + return 'score_head' if name.startswith(('heads.score.','scorers.score.')) else 'score_encoder' + + +def counts(arm, training_arm): + shapes=all_shapes(arm,training_arm);active={n:s for n,s in shapes.items() if trainable_name(n,training_arm)} + return {'unique_parameters':sum(numel(s) for s in shapes.values()),'unique_tensors':len(shapes), + 'trainable_parameters':sum(numel(s) for s in active.values()),'trainable_tensors':len(active), + 'frozen_parameters':sum(numel(s) for n,s in shapes.items() if n not in active),'frozen_tensors':len(shapes)-len(active), + 'score_suffix_tensors':len(score_suffix_shapes(training_arm)), + 'score_suffix_parameters':sum(numel(s) for s in score_suffix_shapes(training_arm).values()), + 'shared_encoder_tensors':134,'integrated_choice_suffix_tensors':132,'type_and_head_tensors':91} + + +def state_layout(arm, training_arm): + split_index(arm);split=score_split(training_arm) + return {'arm':arm,'training_arm':training_arm,'trainability_policy':training_policy(training_arm), + 'unique_registered_state':True,'embedding_storage':'encoder.embeddings; exactly one frozen vocabulary', + 'shared_encoder_storage':'encoder.layers + encoder.final_norm; all frozen', + 'integrated_choice_storage':'choice_blocks + choice_final_norm; original trained values, all frozen', + 'type_embedding_frozen':True,'choice_and_noul_heads_frozen':True, + 'score_shared_prefix_layers':split,'score_suffix_copy_source':'original shared encoder, never trained Choice suffix', + 'score_suffix_copy_map':score_suffix_mapping(training_arm),'counts':counts(arm,training_arm), + 'forward_batch_policy':'one embedding pass; all encoder paths keep full B x L; only heads gather type rows', + 'cross_record_or_request_cache':False,'merged_weights':False} + + +def inventory(arm, training_arm): + return {'counts':counts(arm,training_arm), + 'encoder_shapes':{n:list(s) for n,s in encoder_shapes().items()}, + 'head_shapes':{n:list(s) for n,s in head_shapes().items()}, + 'choice_suffix_shapes':{n:list(s) for n,s in suffix_shapes(arm).items()}, + 'score_suffix_shapes':{n:list(s) for n,s in score_suffix_shapes(training_arm).items()}} + + +def validate_config(cfg): + arm=cfg.get('arm');split_index(arm);sa=cfg.get('training_arm');mode=training_policy(sa) + expected={'architecture':ARCHITECTURE,'schema':SCHEMA,'branch_policy':POLICY, + 'type_order':list(KINDS),'head':EXPECTED_HEAD_CONFIG, + 'packing':{'max_length':8192,'state_truncation':'error'}, + 'base_model':'llm-semantic-router/Vela-1.0-Encoder-307M', + 'base_revision':'fe9ccc074b781bc0e2e13c2c8d26f2640410636a', + 'initializer_manifest_sha256':PARENT_MANIFEST_SHA256,'transformers_version':TRANSFORMERS_VERSION, + 'trainability_mode':mode,'parameters':counts(arm,sa)['unique_parameters'], + 'tensor_count':counts(arm,sa)['unique_tensors'],'weight_dtype':'float32', + 'inference_precision':'fp32','state_cache':False,'rocm_contiguous_layout':True, + 'state_layout':state_layout(arm,sa)} + for key,value in expected.items(): + if cfg.get(key)!=value:raise ValueError('Score-path native identity mismatch: '+key) + return cfg + + +def native_config(parent_cfg, training_arm, provenance=None): + arm='all22' + cfg={key:parent_cfg[key] for key in ('base_model','base_revision','head','packing','rocm_contiguous_layout')} + cfg.update({'architecture':ARCHITECTURE,'schema':SCHEMA,'branch_policy':POLICY, + 'arm':arm,'training_arm':training_arm,'type_order':list(KINDS), + 'initializer_manifest_sha256':PARENT_MANIFEST_SHA256,'transformers_version':TRANSFORMERS_VERSION, + 'trainability_mode':training_policy(training_arm),'parameters':counts(arm,training_arm)['unique_parameters'], + 'tensor_count':counts(arm,training_arm)['unique_tensors'],'state_layout':state_layout(arm,training_arm), + 'weight_dtype':'float32','inference_precision':'fp32','state_cache':False, + 'calibration':'none; raw probabilities','provenance':provenance or {}}) + return validate_config(cfg) diff --git a/native/decision_config.json b/native/decision_config.json new file mode 100644 index 0000000000000000000000000000000000000000..d5c36e120143b512f64e99644f46672751cb25c0 --- /dev/null +++ b/native/decision_config.json @@ -0,0 +1,255 @@ +{ + "architecture": "vela_decision_score_path_capacity_v1", + "arm": "all22", + "base_model": "llm-semantic-router/Vela-1.0-Encoder-307M", + "base_revision": "fe9ccc074b781bc0e2e13c2c8d26f2640410636a", + "branch_policy": "frozen_integrated_choice_noul_score_private_suffix_full_batch_v1", + "calibration": "none; raw probabilities", + "head": { + "attention_mode": "sdpa", + "gradient_checkpointing": true, + "head_dropout": 0.1, + "head_heads": 12, + "head_layers": 2 + }, + "inference_precision": "fp32", + "initializer_manifest_sha256": "ab4d18e3f7aa2185907c796ff3cb37848033a641376daee2bdecbbf709829831", + "packing": { + "max_length": 8192, + "state_truncation": "error" + }, + "parameters": 571909635, + "provenance": { + "TEST_read": false, + "actual_training": { + "active_parameters": 375298563, + "active_tensors": 486, + "actual_training_policy": "typed_all_three_paths_and_heads", + "frozen_parameters": 196611072, + "helper_sha256": "78bf1f6bccb2b74970f19cfae2e50ff0aa4c4f7747c3fde184042f678d8f3843", + "native_policy_is_not_actual_training_provenance": true, + "restored_native_trainability_mode": "score_private_all22_and_head" + }, + "checkpoint_selection": false, + "configuration": { + "checkpoint_selection": false, + "cosine_eta_min": 1e-06, + "dev_rows": 0, + "epochs": 8, + "eval_batch_size": 8, + "eval_precision": "fp32_sdpa", + "gradient_clip": 1.0, + "logical_batch_size": 64, + "loss": "blend50_original_soft_and_hard_CE", + "loss_denominator": 64, + "max_input_tokens": 1024, + "micro_batch_size": 8, + "new_seeds_or_losses": true, + "seed": 20260921, + "selected0_export": false, + "steps": 750, + "temperature_fitting": false, + "train_precision": "bf16", + "train_rows": 6000, + "warmup_steps": 0, + "weight_decay": 0.01 + }, + "decision_exposures": 48000, + "epochs": 8, + "experiment": "lex-final-refit-v1", + "fixed_export_step": 750, + "initial_native_manifest_sha256": "da603662bc57e89ccfb51c972ed9c1f2825f267597353cf1337df9117a3dfabe", + "original_TRAIN_rows": 6000, + "plan_sha256": "d15c3cafd15d0ae14cf45e196d987016b304702f6f3930081b2ef30bbeabc638", + "quality_qualified": false, + "steps": 750, + "target_transform": { + "TEST_rows": 0, + "beta": 0.5, + "canonical_data_written": false, + "former_DEV_now_training": true, + "loss": "sum(.5*CE(original_normalized_soft,p)+.5*CE(onehot(original_label),p))/64", + "targets_changed": { + "dev": 1194, + "train": 4783 + } + }, + "workspace": "/work/decision-1.0" + }, + "rocm_contiguous_layout": true, + "schema": "decision.nano.score_path_capacity.v1", + "state_cache": false, + "state_layout": { + "arm": "all22", + "choice_and_noul_heads_frozen": true, + "counts": { + "frozen_parameters": 446810114, + "frozen_tensors": 327, + "integrated_choice_suffix_tensors": 132, + "score_suffix_parameters": 110330880, + "score_suffix_tensors": 132, + "shared_encoder_tensors": 134, + "trainable_parameters": 125099521, + "trainable_tensors": 162, + "type_and_head_tensors": 91, + "unique_parameters": 571909635, + "unique_tensors": 489 + }, + "cross_record_or_request_cache": false, + "embedding_storage": "encoder.embeddings; exactly one frozen vocabulary", + "forward_batch_policy": "one embedding pass; all encoder paths keep full B x L; only heads gather type rows", + "integrated_choice_storage": "choice_blocks + choice_final_norm; original trained values, all frozen", + "merged_weights": false, + "score_shared_prefix_layers": 0, + "score_suffix_copy_map": { + "score_blocks.0.attn.Wo.weight": "encoder.layers.0.attn.Wo.weight", + "score_blocks.0.attn.Wqkv.weight": "encoder.layers.0.attn.Wqkv.weight", + "score_blocks.0.mlp.Wi.weight": "encoder.layers.0.mlp.Wi.weight", + "score_blocks.0.mlp.Wo.weight": "encoder.layers.0.mlp.Wo.weight", + "score_blocks.0.mlp_norm.weight": "encoder.layers.0.mlp_norm.weight", + "score_blocks.1.attn.Wo.weight": "encoder.layers.1.attn.Wo.weight", + "score_blocks.1.attn.Wqkv.weight": "encoder.layers.1.attn.Wqkv.weight", + "score_blocks.1.attn_norm.weight": "encoder.layers.1.attn_norm.weight", + "score_blocks.1.mlp.Wi.weight": "encoder.layers.1.mlp.Wi.weight", + "score_blocks.1.mlp.Wo.weight": "encoder.layers.1.mlp.Wo.weight", + "score_blocks.1.mlp_norm.weight": "encoder.layers.1.mlp_norm.weight", + "score_blocks.10.attn.Wo.weight": "encoder.layers.10.attn.Wo.weight", + "score_blocks.10.attn.Wqkv.weight": "encoder.layers.10.attn.Wqkv.weight", + "score_blocks.10.attn_norm.weight": "encoder.layers.10.attn_norm.weight", + "score_blocks.10.mlp.Wi.weight": "encoder.layers.10.mlp.Wi.weight", + "score_blocks.10.mlp.Wo.weight": "encoder.layers.10.mlp.Wo.weight", + "score_blocks.10.mlp_norm.weight": "encoder.layers.10.mlp_norm.weight", + "score_blocks.11.attn.Wo.weight": "encoder.layers.11.attn.Wo.weight", + "score_blocks.11.attn.Wqkv.weight": "encoder.layers.11.attn.Wqkv.weight", + "score_blocks.11.attn_norm.weight": "encoder.layers.11.attn_norm.weight", + "score_blocks.11.mlp.Wi.weight": "encoder.layers.11.mlp.Wi.weight", + "score_blocks.11.mlp.Wo.weight": "encoder.layers.11.mlp.Wo.weight", + "score_blocks.11.mlp_norm.weight": "encoder.layers.11.mlp_norm.weight", + "score_blocks.12.attn.Wo.weight": "encoder.layers.12.attn.Wo.weight", + "score_blocks.12.attn.Wqkv.weight": "encoder.layers.12.attn.Wqkv.weight", + "score_blocks.12.attn_norm.weight": "encoder.layers.12.attn_norm.weight", + "score_blocks.12.mlp.Wi.weight": "encoder.layers.12.mlp.Wi.weight", + "score_blocks.12.mlp.Wo.weight": "encoder.layers.12.mlp.Wo.weight", + "score_blocks.12.mlp_norm.weight": "encoder.layers.12.mlp_norm.weight", + "score_blocks.13.attn.Wo.weight": "encoder.layers.13.attn.Wo.weight", + "score_blocks.13.attn.Wqkv.weight": "encoder.layers.13.attn.Wqkv.weight", + "score_blocks.13.attn_norm.weight": "encoder.layers.13.attn_norm.weight", + "score_blocks.13.mlp.Wi.weight": "encoder.layers.13.mlp.Wi.weight", + "score_blocks.13.mlp.Wo.weight": "encoder.layers.13.mlp.Wo.weight", + "score_blocks.13.mlp_norm.weight": "encoder.layers.13.mlp_norm.weight", + "score_blocks.14.attn.Wo.weight": "encoder.layers.14.attn.Wo.weight", + "score_blocks.14.attn.Wqkv.weight": "encoder.layers.14.attn.Wqkv.weight", + "score_blocks.14.attn_norm.weight": "encoder.layers.14.attn_norm.weight", + "score_blocks.14.mlp.Wi.weight": "encoder.layers.14.mlp.Wi.weight", + "score_blocks.14.mlp.Wo.weight": "encoder.layers.14.mlp.Wo.weight", + "score_blocks.14.mlp_norm.weight": "encoder.layers.14.mlp_norm.weight", + "score_blocks.15.attn.Wo.weight": "encoder.layers.15.attn.Wo.weight", + "score_blocks.15.attn.Wqkv.weight": "encoder.layers.15.attn.Wqkv.weight", + "score_blocks.15.attn_norm.weight": "encoder.layers.15.attn_norm.weight", + "score_blocks.15.mlp.Wi.weight": "encoder.layers.15.mlp.Wi.weight", + "score_blocks.15.mlp.Wo.weight": "encoder.layers.15.mlp.Wo.weight", + "score_blocks.15.mlp_norm.weight": "encoder.layers.15.mlp_norm.weight", + "score_blocks.16.attn.Wo.weight": "encoder.layers.16.attn.Wo.weight", + "score_blocks.16.attn.Wqkv.weight": "encoder.layers.16.attn.Wqkv.weight", + "score_blocks.16.attn_norm.weight": "encoder.layers.16.attn_norm.weight", + "score_blocks.16.mlp.Wi.weight": "encoder.layers.16.mlp.Wi.weight", + "score_blocks.16.mlp.Wo.weight": "encoder.layers.16.mlp.Wo.weight", + "score_blocks.16.mlp_norm.weight": "encoder.layers.16.mlp_norm.weight", + "score_blocks.17.attn.Wo.weight": "encoder.layers.17.attn.Wo.weight", + "score_blocks.17.attn.Wqkv.weight": "encoder.layers.17.attn.Wqkv.weight", + "score_blocks.17.attn_norm.weight": "encoder.layers.17.attn_norm.weight", + "score_blocks.17.mlp.Wi.weight": "encoder.layers.17.mlp.Wi.weight", + "score_blocks.17.mlp.Wo.weight": "encoder.layers.17.mlp.Wo.weight", + "score_blocks.17.mlp_norm.weight": "encoder.layers.17.mlp_norm.weight", + "score_blocks.18.attn.Wo.weight": "encoder.layers.18.attn.Wo.weight", + "score_blocks.18.attn.Wqkv.weight": "encoder.layers.18.attn.Wqkv.weight", + "score_blocks.18.attn_norm.weight": "encoder.layers.18.attn_norm.weight", + "score_blocks.18.mlp.Wi.weight": "encoder.layers.18.mlp.Wi.weight", + "score_blocks.18.mlp.Wo.weight": "encoder.layers.18.mlp.Wo.weight", + "score_blocks.18.mlp_norm.weight": "encoder.layers.18.mlp_norm.weight", + "score_blocks.19.attn.Wo.weight": "encoder.layers.19.attn.Wo.weight", + "score_blocks.19.attn.Wqkv.weight": "encoder.layers.19.attn.Wqkv.weight", + "score_blocks.19.attn_norm.weight": "encoder.layers.19.attn_norm.weight", + "score_blocks.19.mlp.Wi.weight": "encoder.layers.19.mlp.Wi.weight", + "score_blocks.19.mlp.Wo.weight": "encoder.layers.19.mlp.Wo.weight", + "score_blocks.19.mlp_norm.weight": "encoder.layers.19.mlp_norm.weight", + "score_blocks.2.attn.Wo.weight": "encoder.layers.2.attn.Wo.weight", + "score_blocks.2.attn.Wqkv.weight": "encoder.layers.2.attn.Wqkv.weight", + "score_blocks.2.attn_norm.weight": "encoder.layers.2.attn_norm.weight", + "score_blocks.2.mlp.Wi.weight": "encoder.layers.2.mlp.Wi.weight", + "score_blocks.2.mlp.Wo.weight": "encoder.layers.2.mlp.Wo.weight", + "score_blocks.2.mlp_norm.weight": "encoder.layers.2.mlp_norm.weight", + "score_blocks.20.attn.Wo.weight": "encoder.layers.20.attn.Wo.weight", + "score_blocks.20.attn.Wqkv.weight": "encoder.layers.20.attn.Wqkv.weight", + "score_blocks.20.attn_norm.weight": "encoder.layers.20.attn_norm.weight", + "score_blocks.20.mlp.Wi.weight": "encoder.layers.20.mlp.Wi.weight", + "score_blocks.20.mlp.Wo.weight": "encoder.layers.20.mlp.Wo.weight", + "score_blocks.20.mlp_norm.weight": "encoder.layers.20.mlp_norm.weight", + "score_blocks.21.attn.Wo.weight": "encoder.layers.21.attn.Wo.weight", + "score_blocks.21.attn.Wqkv.weight": "encoder.layers.21.attn.Wqkv.weight", + "score_blocks.21.attn_norm.weight": "encoder.layers.21.attn_norm.weight", + "score_blocks.21.mlp.Wi.weight": "encoder.layers.21.mlp.Wi.weight", + "score_blocks.21.mlp.Wo.weight": "encoder.layers.21.mlp.Wo.weight", + "score_blocks.21.mlp_norm.weight": "encoder.layers.21.mlp_norm.weight", + "score_blocks.3.attn.Wo.weight": "encoder.layers.3.attn.Wo.weight", + "score_blocks.3.attn.Wqkv.weight": "encoder.layers.3.attn.Wqkv.weight", + "score_blocks.3.attn_norm.weight": "encoder.layers.3.attn_norm.weight", + "score_blocks.3.mlp.Wi.weight": "encoder.layers.3.mlp.Wi.weight", + "score_blocks.3.mlp.Wo.weight": "encoder.layers.3.mlp.Wo.weight", + "score_blocks.3.mlp_norm.weight": "encoder.layers.3.mlp_norm.weight", + "score_blocks.4.attn.Wo.weight": "encoder.layers.4.attn.Wo.weight", + "score_blocks.4.attn.Wqkv.weight": "encoder.layers.4.attn.Wqkv.weight", + "score_blocks.4.attn_norm.weight": "encoder.layers.4.attn_norm.weight", + "score_blocks.4.mlp.Wi.weight": "encoder.layers.4.mlp.Wi.weight", + "score_blocks.4.mlp.Wo.weight": "encoder.layers.4.mlp.Wo.weight", + "score_blocks.4.mlp_norm.weight": "encoder.layers.4.mlp_norm.weight", + "score_blocks.5.attn.Wo.weight": "encoder.layers.5.attn.Wo.weight", + "score_blocks.5.attn.Wqkv.weight": "encoder.layers.5.attn.Wqkv.weight", + "score_blocks.5.attn_norm.weight": "encoder.layers.5.attn_norm.weight", + "score_blocks.5.mlp.Wi.weight": "encoder.layers.5.mlp.Wi.weight", + "score_blocks.5.mlp.Wo.weight": "encoder.layers.5.mlp.Wo.weight", + "score_blocks.5.mlp_norm.weight": "encoder.layers.5.mlp_norm.weight", + "score_blocks.6.attn.Wo.weight": "encoder.layers.6.attn.Wo.weight", + "score_blocks.6.attn.Wqkv.weight": "encoder.layers.6.attn.Wqkv.weight", + "score_blocks.6.attn_norm.weight": "encoder.layers.6.attn_norm.weight", + "score_blocks.6.mlp.Wi.weight": "encoder.layers.6.mlp.Wi.weight", + "score_blocks.6.mlp.Wo.weight": "encoder.layers.6.mlp.Wo.weight", + "score_blocks.6.mlp_norm.weight": "encoder.layers.6.mlp_norm.weight", + "score_blocks.7.attn.Wo.weight": "encoder.layers.7.attn.Wo.weight", + "score_blocks.7.attn.Wqkv.weight": "encoder.layers.7.attn.Wqkv.weight", + "score_blocks.7.attn_norm.weight": "encoder.layers.7.attn_norm.weight", + "score_blocks.7.mlp.Wi.weight": "encoder.layers.7.mlp.Wi.weight", + "score_blocks.7.mlp.Wo.weight": "encoder.layers.7.mlp.Wo.weight", + "score_blocks.7.mlp_norm.weight": "encoder.layers.7.mlp_norm.weight", + "score_blocks.8.attn.Wo.weight": "encoder.layers.8.attn.Wo.weight", + "score_blocks.8.attn.Wqkv.weight": "encoder.layers.8.attn.Wqkv.weight", + "score_blocks.8.attn_norm.weight": "encoder.layers.8.attn_norm.weight", + "score_blocks.8.mlp.Wi.weight": "encoder.layers.8.mlp.Wi.weight", + "score_blocks.8.mlp.Wo.weight": "encoder.layers.8.mlp.Wo.weight", + "score_blocks.8.mlp_norm.weight": "encoder.layers.8.mlp_norm.weight", + "score_blocks.9.attn.Wo.weight": "encoder.layers.9.attn.Wo.weight", + "score_blocks.9.attn.Wqkv.weight": "encoder.layers.9.attn.Wqkv.weight", + "score_blocks.9.attn_norm.weight": "encoder.layers.9.attn_norm.weight", + "score_blocks.9.mlp.Wi.weight": "encoder.layers.9.mlp.Wi.weight", + "score_blocks.9.mlp.Wo.weight": "encoder.layers.9.mlp.Wo.weight", + "score_blocks.9.mlp_norm.weight": "encoder.layers.9.mlp_norm.weight", + "score_final_norm.weight": "encoder.final_norm.weight" + }, + "score_suffix_copy_source": "original shared encoder, never trained Choice suffix", + "shared_encoder_storage": "encoder.layers + encoder.final_norm; all frozen", + "trainability_policy": "score_private_all22_and_head", + "training_arm": "S22", + "type_embedding_frozen": true, + "unique_registered_state": true + }, + "tensor_count": 489, + "trainability_mode": "score_private_all22_and_head", + "training_arm": "S22", + "transformers_version": "4.57.6", + "type_order": [ + "choice", + "noul", + "score" + ], + "weight_dtype": "float32" +} diff --git a/native/decision_heads.safetensors b/native/decision_heads.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..08a036bc5a4b06041755e00cfd77fbc7908a79cb --- /dev/null +++ b/native/decision_heads.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:bce3ee658a978a19c48c605b921cff994f892e7fc17abd6f8645f07428fbc36f +size 177241884 diff --git a/native/encoder/config.json b/native/encoder/config.json new file mode 100644 index 0000000000000000000000000000000000000000..748c13deadf6e0a945b1b025bbc1e65d98701e69 --- /dev/null +++ b/native/encoder/config.json @@ -0,0 +1,108 @@ +{ + "add_cross_attention": false, + "architectures": [ + "ModernBertModel" + ], + "attention_bias": false, + "attention_dropout": 0.0, + "bad_words_ids": null, + "begin_suppress_tokens": null, + "bos_token_id": 2, + "chunk_size_feed_forward": 0, + "classifier_activation": "gelu", + "classifier_bias": false, + "classifier_dropout": 0.0, + "classifier_pooling": "mean", + "cls_token_id": 1, + "cross_attention_hidden_size": null, + "decoder_bias": true, + "decoder_start_token_id": null, + "deterministic_flash_attn": false, + "diversity_penalty": 0.0, + "do_sample": false, + "dtype": "float32", + "early_stopping": false, + "embedding_dropout": 0.0, + "encoder_no_repeat_ngram_size": 0, + "eos_token_id": 1, + "exponential_decay_length_penalty": null, + "finetuning_task": null, + "forced_bos_token_id": null, + "forced_eos_token_id": null, + "global_attn_every_n_layers": 3, + "global_rope_theta": 160000, + "gradient_checkpointing": false, + "hidden_activation": "gelu", + "hidden_size": 768, + "id2label": { + "0": "LABEL_0", + "1": "LABEL_1" + }, + "initializer_cutoff_factor": 2.0, + "initializer_range": 0.02, + "intermediate_size": 1152, + "is_decoder": false, + "is_encoder_decoder": false, + "label2id": { + "LABEL_0": 0, + "LABEL_1": 1 + }, + "layer_norm_eps": 1e-05, + "length_penalty": 1.0, + "local_attention": 128, + "local_rope_theta": 160000, + "mask_token_id": 4, + "max_length": 20, + "max_position_embeddings": 32768, + "min_length": 0, + "mlp_bias": false, + "mlp_dropout": 0.0, + "model_type": "modernbert", + "no_repeat_ngram_size": 0, + "norm_bias": false, + "norm_eps": 1e-05, + "num_attention_heads": 12, + "num_beam_groups": 1, + "num_beams": 1, + "num_hidden_layers": 22, + "num_return_sequences": 1, + "output_attentions": false, + "output_hidden_states": false, + "output_scores": false, + "pad_token_id": 0, + "position_embedding_type": "sans_pos", + "prefix": null, + "problem_type": null, + "pruned_heads": {}, + "reference_compile": false, + "remove_invalid_values": false, + "repad_logits_with_grad": false, + "repetition_penalty": 1.0, + "return_dict": true, + "return_dict_in_generate": false, + "rope_scaling": { + "beta_fast": 32.0, + "beta_slow": 1.0, + "factor": 4.0, + "original_max_position_embeddings": 8192, + "rope_type": "yarn", + "truncate": true + }, + "sep_token_id": 1, + "sparse_pred_ignore_index": -100, + "sparse_prediction": false, + "suppress_tokens": null, + "task_specific_params": null, + "temperature": 1.0, + "tf_legacy_loss": false, + "tie_encoder_decoder": false, + "tie_word_embeddings": true, + "tokenizer_class": null, + "top_k": 50, + "top_p": 1.0, + "torchscript": false, + "transformers_version": "4.57.6", + "typical_p": 1.0, + "use_bfloat16": false, + "vocab_size": 256000 +} diff --git a/native/encoder/model.safetensors b/native/encoder/model.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..f251fd508e446abda3a9d5baf5b54def76ae96c8 --- /dev/null +++ b/native/encoder/model.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:daaafd81c4ed767d203e24226d78e792800068a7dac344aa043844b82a6306c7 +size 1227771752 diff --git a/native/infer.py b/native/infer.py new file mode 100644 index 0000000000000000000000000000000000000000..d3cb4b179f6b54b0319c6b3398741ad9fc832cf0 --- /dev/null +++ b/native/infer.py @@ -0,0 +1,25 @@ +"""Load only this explicit fork-native format; output semantics are unchanged.""" +import argparse +import json +from pathlib import Path +try: + from .artifacts import load_export + from .model import predict +except ImportError: + from artifacts import load_export + from model import predict + + +def main(): + p=argparse.ArgumentParser() + p.add_argument('--model',required=True);p.add_argument('--input',required=True);p.add_argument('--output',required=True) + p.add_argument('--device',default='cuda');p.add_argument('--batch-size',type=int,default=8) + a=p.parse_args() + if not a.device.startswith('cuda'):raise ValueError('Research execution requires separately authorized AMD GPU') + model,collator,cfg=load_export(a.model,device=a.device) + rows=[json.loads(s) for s in Path(a.input).read_text().splitlines() if s] + result=predict(model,collator,rows,batch_size=a.batch_size,device=a.device,precision='fp32') + with open(a.output,'x') as f: + for row in result:f.write(json.dumps(row,ensure_ascii=False,allow_nan=False)+'\n') + +if __name__=='__main__':main() diff --git a/native/model.py b/native/model.py new file mode 100644 index 0000000000000000000000000000000000000000..a2938d2854aefad6a36c68a6b26c7b9331b3198a --- /dev/null +++ b/native/model.py @@ -0,0 +1,158 @@ +"""Frozen integrated C/N routes plus an explicit trainable Score-only path.""" +from contextlib import nullcontext +from copy import deepcopy +import torch +from torch import nn +from torch.utils.checkpoint import checkpoint +try: + from .contract import (ARCHITECTURE, KINDS, EXPECTED_HEAD_CONFIG, TRANSFORMERS_VERSION, + split_index, counts, all_shapes, suffix_mapping, score_suffix_mapping, + score_split, training_policy, trainable_name, validate_encoder_config) + from .policy.model import DecisionModel as PolicyModel, route_indices, scatter_rows, validate_tensor_map + from .policy.reference.model import typed_loss, predict + from .policy.reference.modernbert_sdpa_layout import LAYOUT_POLICY +except ImportError: + from contract import (ARCHITECTURE, KINDS, EXPECTED_HEAD_CONFIG, TRANSFORMERS_VERSION, + split_index, counts, all_shapes, suffix_mapping, score_suffix_mapping, + score_split, training_policy, trainable_name, validate_encoder_config) + from policy.model import DecisionModel as PolicyModel, route_indices, scatter_rows, validate_tensor_map + from policy.reference.model import typed_loss, predict + from policy.reference.modernbert_sdpa_layout import LAYOUT_POLICY + + +def runtime_gate(): + import transformers + if transformers.__version__ != TRANSFORMERS_VERSION: + raise ValueError('Manual split forward is pinned to Transformers '+TRANSFORMERS_VERSION) + +def layer_gate(layer, original_index): + attention = layer.attn + if attention.layer_id != original_index or attention.local_attention != ((-1, -1) if original_index % 3 == 0 else (64, 64)): + raise ValueError('Layer index or local/global attention phase changed') + if not isinstance(layer.attn_norm, nn.Identity if original_index == 0 else nn.LayerNorm): + raise ValueError('First-layer norm policy mismatch') + if getattr(attention, '_vela_sdpa_layout_policy', None) != LAYOUT_POLICY: + raise ValueError('Missing model-local ROCm contiguous QKV guard') + if getattr(attention.forward, '__self__', None) is not attention or getattr(attention._vela_original_attention_forward, '__self__', None) is not attention: + raise ValueError('Deep-copied attention method refers to another module') + if not layer.gradient_checkpointing or getattr(layer._gradient_checkpointing_func, 'keywords', {}).get('use_reentrant') is not False: + raise ValueError('Non-reentrant gradient checkpointing must be enabled') + + +class DecisionModel(nn.Module): + def __init__(self, policy_model, arm, training_arm): + super().__init__();runtime_gate();split_index(arm) + if not isinstance(policy_model,PolicyModel) or policy_model.head_config!=EXPECTED_HEAD_CONFIG: + raise ValueError('Explicit three-head parent geometry required') + validate_encoder_config(policy_model.encoder.config) + from transformers.models.modernbert.modeling_modernbert import ModernBertModel + if type(policy_model.encoder) is not ModernBertModel:raise ValueError('Exact pinned ModernBertModel required') + self.arm=arm;self.training_arm=training_arm;self.score_split_index=score_split(training_arm) + self.trainability_mode=training_policy(training_arm) + self.encoder=policy_model.encoder;self.type_embedding=policy_model.type_embedding + self.heads=policy_model.heads;self.scorers=policy_model.scorers + self.head_config=dict(policy_model.head_config);self.hidden_size=768 + self.encoder.gradient_checkpointing_enable(gradient_checkpointing_kwargs={'use_reentrant':False}) + # The persisted integrated Choice suffix replaces these temporary constructor copies. + self.choice_blocks=nn.ModuleList([deepcopy(layer) for layer in self.encoder.layers]) + self.choice_final_norm=deepcopy(self.encoder.final_norm) + # Score copies ONLY the original shared encoder, never the trained Choice suffix. + self.score_blocks=nn.ModuleList([deepcopy(layer) for layer in self.encoder.layers[self.score_split_index:]]) + self.score_final_norm=deepcopy(self.encoder.final_norm) if training_arm!='S0' else None + for i,layer in enumerate(self.encoder.layers):layer_gate(layer,i) + for i,layer in enumerate(self.choice_blocks):layer_gate(layer,i) + for j,layer in enumerate(self.score_blocks):layer_gate(layer,self.score_split_index+j) + for name,p in self.named_parameters():p.requires_grad_(trainable_name(name,self.trainability_mode));p.grad=None + state=self.state_dict() + for dst,src in {**suffix_mapping(arm),**score_suffix_mapping(training_arm)}.items(): + if not torch.equal(state[dst],state[src]) or state[dst].data_ptr()==state[src].data_ptr(): + raise ValueError('Suffix copy value/storage mismatch: '+dst) + self.verify_inventory(check_values=False);self.eval() + + def verify_inventory(self,check_values=True): + expected=all_shapes(self.arm,self.training_arm);state=self.state_dict() + if set(state)!=set(expected):raise ValueError('Exact unique score-path state names required') + if check_values:validate_tensor_map(state,expected) + elif any(tuple(state[n].shape)!=s or state[n].dtype!=torch.float32 for n,s in expected.items()): + raise ValueError('State geometry/dtype mismatch') + raw=list(self.named_parameters(remove_duplicate=False)) + if len(raw)!=len(expected) or len({id(p) for _,p in raw})!=len(raw) or len({p.untyped_storage().data_ptr() for _,p in raw})!=len(raw): + raise ValueError('Duplicate registered parameter or storage alias') + if any(p.requires_grad!=trainable_name(n,self.trainability_mode) for n,p in raw):raise ValueError('Score-only trainability drift') + actual={'unique_parameters':sum(p.numel() for _,p in raw),'unique_tensors':len(raw), + 'trainable_parameters':sum(p.numel() for _,p in raw if p.requires_grad), + 'trainable_tensors':sum(p.requires_grad for _,p in raw)} + if any(actual[k]!=counts(self.arm,self.training_arm)[k] for k in actual):raise ValueError('Unique inventory mismatch') + return actual + + def train(self,mode=True): + super().train(mode) + self.encoder.eval();self.type_embedding.eval();self.choice_blocks.eval();self.choice_final_norm.eval() + for kind in ('choice','noul'):self.heads[kind].eval();self.scorers[kind].eval() + return self + + @staticmethod + def _suffix(hidden,layers,final_norm,layer_kwargs): + for layer in layers:hidden=layer(hidden,**layer_kwargs)[0] + return final_norm(hidden) + + def forward(self,batch): + routes=route_indices(batch['kind_ids']) + ids,mask=batch['input_ids'],batch['attention_mask'] + if ids.ndim!=2 or mask.shape!=ids.shape or ids.shape[0]!=batch['kind_ids'].numel() or ids.shape[1]>8192: + raise ValueError('Original complete B x L input and 8192 cap required') + self.encoder._maybe_set_compile() + position_ids=torch.arange(ids.shape[1],device=ids.device).unsqueeze(0) + global_mask,local_mask=self.encoder._update_attention_mask(mask,output_attentions=False) + kwargs=dict(attention_mask=global_mask,sliding_window_mask=local_mask,position_ids=position_ids, + cu_seqlens=None,max_seqlen=None,output_attentions=False) + need_choice=routes[0][1].numel()>0;need_noul=routes[1][1].numel()>0;need_score=routes[2][1].numel()>0 + need_shared_full=need_noul or (need_score and self.training_arm=='S0') + need_score_private=need_score and self.training_arm!='S0' + hidden_by_path={} + with torch.no_grad(): + embedded=self.encoder.embeddings(input_ids=ids,inputs_embeds=None) + # Preserve embedding autocast dtype and original layer indices/local-global masks. + score_prefix=embedded + shared=embedded + stop=22 if need_shared_full else self.score_split_index if need_score_private else 0 + for i,layer in enumerate(self.encoder.layers[:stop]): + shared=layer(shared,**kwargs)[0] + if i+1==self.score_split_index:score_prefix=shared + if need_shared_full: + h=self.encoder.final_norm(shared) + hidden_by_path['frozen']=h+self.type_embedding(batch['kind_ids'])[:,None,:].to(h.dtype) + if need_choice: + h=self._suffix(embedded,self.choice_blocks,self.choice_final_norm,kwargs) + hidden_by_path['choice']=h+self.type_embedding(batch['kind_ids'])[:,None,:].to(h.dtype) + if need_score_private: + # Prefix is frozen, but the private suffix must remain outside no_grad for training. + h=self._suffix(score_prefix,self.score_blocks,self.score_final_norm,kwargs) + hidden_by_path['score']=h+self.type_embedding(batch['kind_ids'])[:,None,:].to(h.dtype) + pad=~mask.bool();output=torch.empty(batch['marker_positions'].shape,device=ids.device,dtype=torch.float32) + for kind,indices in routes: + if indices.numel()==0:continue + path='choice' if kind=='choice' else 'score' if kind=='score' and need_score_private else 'frozen' + with nullcontext() if kind=='score' else torch.no_grad(): + hidden=hidden_by_path[path].index_select(0,indices);branch_pad=pad.index_select(0,indices) + for layer in self.heads[kind]: + if layer.training and self.head_config['gradient_checkpointing']: + hidden=checkpoint(layer,hidden,src_key_padding_mask=branch_pad,use_reentrant=False) + else:hidden=layer(hidden,src_key_padding_mask=branch_pad) + positions=batch['marker_positions'].index_select(0,indices) + markers=torch.gather(hidden,1,positions[:,:,None].expand(-1,-1,hidden.shape[-1])) + logits=self.scorers[kind](markers).squeeze(-1).float() + output=scatter_rows(output,indices,logits) + return output.masked_fill(~batch['valid_candidates'],torch.finfo(torch.float32).min) + + def compute_ledger(self,batch): + b,l=batch['input_ids'].shape;valid=int(batch['attention_mask'].sum().item()) + kinds=set(batch['kind_ids'].detach().cpu().tolist());has_score=2 in kinds + shared=22 if 1 in kinds or (has_score and self.training_arm=='S0') else self.score_split_index if has_score else 0 + choice=22 if 0 in kinds else 0;score=22-self.score_split_index if has_score and self.training_arm!='S0' else 0 + return {'original_records':b,'padded_sequence_length':l,'embedding_passes':1, + 'embedding_valid_tokens':valid,'embedding_padded_tokens':b*l, + 'shared_encoder_layers':shared,'frozen_choice_layers':choice,'private_score_layers':score, + 'encoder_layer_valid_tokens':valid*(shared+choice+score), + 'encoder_layer_padded_tokens':b*l*(shared+choice+score), + 'full_batch_suffix_policy':True,'is_FLOPs_or_causal_timing':False} diff --git a/native/packing.py b/native/packing.py new file mode 100644 index 0000000000000000000000000000000000000000..d8a9fb92bb2eb561d54dc17a312575cb368506d7 --- /dev/null +++ b/native/packing.py @@ -0,0 +1,129 @@ +"""One joint row per question. Metadata IDs and gold labels never enter tokens.""" +import math +import torch + +KINDS = ("choice", "noul", "score") +DEFAULT_NO = "No. The statement or question is not satisfied." +DEFAULT_YES = "Yes. The statement or question is satisfied." + + +def nonempty(value, name): + if not isinstance(value, str) or not value.strip(): + raise ValueError(f"{name} must be a nonempty string") + return value + + +def view(record, labeled=False): + q = record["question"] + kind = str(q["type"]).lower() + if kind not in KINDS: + raise ValueError("Unsupported question type") + nonempty(record["state_text"], "state_text") + nonempty(q["text"], "question.text") + if kind == "noul": + options = [{"id": "no", "text": q.get("false_criterion", DEFAULT_NO)}, + {"id": "yes", "text": q.get("true_criterion", DEFAULT_YES)}] + else: + options = q["options" if kind == "choice" else "levels"] + if not isinstance(options, list) or not 2 <= len(options) <= 255: + raise ValueError("Require 2..255 complete candidate descriptions") + for o in options: + nonempty(o["id"], "candidate.id") + nonempty(o["text"], "candidate.text") + if len({o["id"] for o in options}) != len(options): + raise ValueError("Duplicate candidate IDs") + values = [float(o["value"]) for o in options] if kind == "score" else list(range(len(options))) + if kind == "score" and (not all(math.isfinite(v) for v in values) + or any(a >= b for a, b in zip(values, values[1:]))): + raise ValueError("Score level values must be finite and strictly increasing") + target = None + if labeled: + gold = record["target"] + if kind == "noul": + yes = float(gold["probability"]) + target = [1 - yes, yes] + elif "probabilities" in gold: + target = [float(x) for x in gold["probabilities"]] + elif kind == "choice" and "choice_id" in gold: + if gold["choice_id"] not in {o["id"] for o in options}: + raise ValueError("Unknown gold choice_id") + target = [float(o["id"] == gold["choice_id"]) for o in options] + else: + raise ValueError("Score requires an explicit target probability distribution") + if (len(target) != len(options) or not all(math.isfinite(x) and 0 <= x <= 1 for x in target) + or abs(sum(target) - 1) > 1e-6): + raise ValueError("Invalid target probability distribution") + return kind, options, values, target + + +class MarkerCollator: + def __init__(self, tokenizer, max_length=8192, state_truncation="error"): + if isinstance(max_length, bool) or not isinstance(max_length, int) or max_length < 8: + raise ValueError("Invalid max_length") + if state_truncation not in ("error", "right"): + raise ValueError("state_truncation must be error or right") + self.tokenizer, self.max_length = tokenizer, max_length + self.state_truncation = state_truncation + self.bos = tokenizer.cls_token_id if tokenizer.cls_token_id is not None else tokenizer.bos_token_id + self.sep = tokenizer.sep_token_id if tokenizer.sep_token_id is not None else tokenizer.eos_token_id + self.pad, self.marker = tokenizer.pad_token_id, tokenizer.mask_token_id + if any(v is None for v in (self.bos, self.sep, self.pad, self.marker)): + raise ValueError("Tokenizer must provide existing BOS/CLS, SEP/EOS, PAD and MASK IDs") + + def tokens(self, text): + ids = self.tokenizer(text, add_special_tokens=False, truncation=False)["input_ids"] + if not ids: + raise ValueError("Empty token span") + return ids + + def encode(self, record, labeled=False): + kind, options, values, target = view(record, labeled) + question = self.tokens(f"{kind} question: {record['question']['text']}") + ids = [self.bos] + question + [self.sep] + positions, spans = [], [] + for j, option in enumerate(options): + # Ordinal rank is meaningful only for Score; opaque external IDs never enter text. + text = f"level {j}: {option['text']}" if kind == "score" else option["text"] + desc = self.tokens(text) + positions.append(len(ids)) + ids.append(self.marker) + spans.append([len(ids), len(ids) + len(desc)]) + ids.extend(desc + [self.sep]) + state = self.tokens(record["state_text"]) + room = self.max_length - len(ids) - 1 + if room < 1: + raise ValueError("Complete question and candidates leave no room for state") + if len(state) > room and self.state_truncation == "error": + raise ValueError(f"Record {record.get('id')} exceeds {self.max_length} tokens; no implicit truncation") + kept = state[:room] + state_span = [len(ids), len(ids) + len(kept)] + ids.extend(kept + [self.sep]) + return {"ids": ids, "positions": positions, "candidate_spans": spans, + "state_span": state_span, "kind": kind, "target": target, "values": values, + "candidate_ids": [o["id"] for o in options], "id": record.get("id"), + "question_id": record["question"].get("id"), "state_tokens_original": len(state), + "state_tokens_kept": len(kept), "input_tokens": len(ids)} + + def __call__(self, records, labeled=False, device="cpu"): + encoded = [self.encode(r, labeled) for r in records] + if not encoded: + raise ValueError("Empty batch") + b, length, k = len(encoded), max(len(e["ids"]) for e in encoded), max(len(e["positions"]) for e in encoded) + ids = torch.full((b, length), self.pad, dtype=torch.long) + mask = torch.zeros((b, length), dtype=torch.bool) + positions = torch.zeros((b, k), dtype=torch.long) + valid = torch.zeros((b, k), dtype=torch.bool) + target, values = torch.zeros((b, k)), torch.zeros((b, k)) + for i, e in enumerate(encoded): + n, count = len(e["ids"]), len(e["positions"]) + ids[i, :n] = torch.tensor(e["ids"]) + mask[i, :n] = True + positions[i, :count] = torch.tensor(e["positions"]) + valid[i, :count] = True + values[i, :count] = torch.tensor(e["values"]) + if labeled: + target[i, :count] = torch.tensor(e["target"]) + batch = {"input_ids": ids, "attention_mask": mask, "marker_positions": positions, + "valid_candidates": valid, "targets": target, "values": values, + "kind_ids": torch.tensor([KINDS.index(e["kind"]) for e in encoded])} + return {name: value.to(device) for name, value in batch.items()}, encoded diff --git a/native/policy/__init__.py b/native/policy/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..20fb499056ab826586f2af4dde8c5eb6e420be5a --- /dev/null +++ b/native/policy/__init__.py @@ -0,0 +1,2 @@ +"""Type-specific interaction heads; source-only prototype.""" +from .artifacts import from_shared_export, load_export, export_model diff --git a/native/policy/artifacts.py b/native/policy/artifacts.py new file mode 100644 index 0000000000000000000000000000000000000000..f5022c4af36beaa158588d05698fcb5fc01d09e8 --- /dev/null +++ b/native/policy/artifacts.py @@ -0,0 +1,133 @@ +"""Explicit71d expansion and strict distinct native format. No implicit old loader.""" +from pathlib import Path +import shutil +import torch +try: + from .contract import ARCHITECTURE, SCHEMA, POLICY, KINDS, PARENT_MANIFEST_SHA256, counts, head_shapes, validate_config + from .model import DecisionModel, from_shared_model, validate_tensor_map + from .packing import MarkerCollator + from .reference.artifacts import file_ref, read_json, write_json, verify_export, install_layout + from .reference.artifacts import load_export as load_shared_export +except ImportError: + from contract import ARCHITECTURE, SCHEMA, POLICY, KINDS, PARENT_MANIFEST_SHA256, counts, head_shapes, validate_config + from model import DecisionModel, from_shared_model, validate_tensor_map + from packing import MarkerCollator + from reference.artifacts import file_ref, read_json, write_json, verify_export, install_layout + from reference.artifacts import load_export as load_shared_export +ROOT = Path(__file__).resolve().parent +REFERENCE_FILES = ('__init__.py','model.py','packing.py','artifacts.py','infer.py','modernbert_sdpa_layout.py') +RUNTIME_FILES = ('__init__.py','model.py','contract.py','packing.py','artifacts.py','infer.py') + tuple('reference/'+n for n in REFERENCE_FILES) + + +def native_config(parent_cfg, trainability_mode='all', provenance=None): + cfg = {'architecture': ARCHITECTURE, 'schema': SCHEMA, 'branch_policy': POLICY, + 'type_order': list(KINDS), 'head': parent_cfg['head'], 'packing': parent_cfg['packing'], + 'base_model': parent_cfg['base_model'], 'base_revision': parent_cfg['base_revision'], + 'rocm_contiguous_layout': parent_cfg['rocm_contiguous_layout'], + 'parameters': counts()['total_parameters'], 'tensor_count': 225, + 'initializer_manifest_sha256': PARENT_MANIFEST_SHA256, 'trainability_mode': trainability_mode, + 'state_cache': False, 'weight_dtype': 'float32', 'inference_precision': 'fp32', + 'calibration': 'none; raw probabilities', 'provenance': provenance or {}} + return validate_config(cfg) + + +def from_shared_export(path, device='cpu', return_parent=False): + """Explicit conversion; one encoder allocation. Optional reference shares it.""" + path = Path(path) + if file_ref(path/'MANIFEST.json')['sha256'] != PARENT_MANIFEST_SHA256: + raise ValueError('Only fixed71d may initialize this prototype') + manifest = verify_export(path) + for name in REFERENCE_FILES: + if file_ref(ROOT/'reference'/name) != manifest['files'][name]: + raise ValueError('Frozen reference loader/runtime mismatch: '+name) + parent, collator, parent_cfg = load_shared_export(path, device='cpu') + model = from_shared_model(parent) + cfg = native_config(parent_cfg, provenance={'kind':'exact_shared_head_copies', + 'parent_manifest':file_ref(path/'MANIFEST.json'),'training_steps':0, + 'encoder_copies':1,'eval_parity_measured':False}) + model.to(device).eval() + if return_parent: + parent.to(device).eval() + if verify_export(path) != manifest: + raise ValueError('Parent changed during explicit conversion') + result = (model, collator, cfg) + return (*result, parent) if return_parent else result + + +def _verify_inventory(model): + if len(model.encoder.state_dict()) != 134 or sum(p.numel() for p in model.encoder.parameters()) != 306939648: + raise ValueError('Expected one134-tensor encoder') + if len(model.state_dict()) != 225 or sum(p.numel() for p in model.parameters()) != counts()['total_parameters']: + raise ValueError('Complete type-specific inventory mismatch') + for n, v in model.state_dict().items(): + if v.dtype != torch.float32 or not torch.isfinite(v).all(): + raise ValueError('Nonfinite/non-FP32 export tensor: '+n) + head = {n:v for n,v in model.state_dict().items() if not n.startswith('encoder.')} + validate_tensor_map(head, head_shapes()) + return head + + +def export_model(model, tokenizer, path, config, provenance): + from safetensors.torch import save_file + config = dict(config) + config['trainability_mode'] = model.trainability_mode + config['provenance'] = provenance + validate_config(config) + if model.head_config != config['head']: + raise ValueError('Model head/config mismatch') + head = _verify_inventory(model) + path = Path(path);path.mkdir(parents=True, exist_ok=False) + (path/'encoder').mkdir() + ec = model.encoder.config.to_dict();ec.pop('_name_or_path', None) + ec['architectures'] = [type(model.encoder).__name__] + write_json(path/'encoder/config.json', ec) + save_file({n:v.detach().cpu().contiguous() for n,v in model.encoder.state_dict().items()},str(path/'encoder/model.safetensors')) + save_file({n:v.detach().cpu().contiguous() for n,v in head.items()},str(path/'decision_heads.safetensors')) + tokenizer.save_pretrained(path/'tokenizer') + for name in RUNTIME_FILES: + dest = path/name;dest.parent.mkdir(parents=True, exist_ok=True) + shutil.copyfile(ROOT/name, dest) + write_json(path/'decision_config.json', config) + write_json(path/'INVENTORY.json', {'counts':counts(),'head_shapes':head_shapes(), + 'trainability_mode':model.trainability_mode}) + refs = {p.relative_to(path).as_posix():file_ref(p) for p in sorted(path.rglob('*')) if p.is_file()} + write_json(path/'MANIFEST.json', {'schema':'decision.files.v1','files':refs}) + verify_export(path) + return file_ref(path/'MANIFEST.json') + + +def load_export(path, device='cpu'): + from transformers import AutoConfig, AutoModel, AutoTokenizer + from safetensors.torch import load_file + path = Path(path);manifest = verify_export(path) + cfg = validate_config(read_json(path/'decision_config.json')) + # Every runtime file is bound to this reviewed adapter, not arbitrary export code. + for name in RUNTIME_FILES: + if name not in manifest['files'] or file_ref(ROOT/name) != manifest['files'][name]: + raise ValueError('Native runtime mismatch: '+name) + inventory = read_json(path/'INVENTORY.json') + if inventory != {'counts':counts(),'head_shapes':{n:list(s) for n,s in head_shapes().items()}, + 'trainability_mode':cfg['trainability_mode']}: + raise ValueError('Native declared inventory mismatch') + ec = AutoConfig.from_pretrained(path/'encoder', local_files_only=True, trust_remote_code=False) + if ec.model_type!='modernbert' or ec.hidden_size!=768 or ec.num_hidden_layers!=22 or ec.max_position_embeddings<8192: + raise ValueError('Native encoder geometry') + ec.reference_compile=False + encoder = AutoModel.from_config(ec, trust_remote_code=False, torch_dtype=torch.float32, attn_implementation='sdpa') + state = load_file(str(path/'encoder/model.safetensors'), device='cpu') + if len(state)!=134 or any(v.dtype!=torch.float32 or not torch.isfinite(v).all() for v in state.values()): + raise ValueError('Native encoder tensor format') + encoder.load_state_dict(state, strict=True);del state + install_layout(encoder, True) + model = DecisionModel(encoder, cfg['head']) + head = load_file(str(path/'decision_heads.safetensors'), device='cpu') + validate_tensor_map(head, head_shapes()) + # load_state_dict requires a full exact set; no missing/unexpected fallback. + model.load_state_dict({**{'encoder.'+n:v for n,v in encoder.state_dict().items()},**head},strict=True) + del head + _verify_inventory(model) + model.set_trainability(cfg['trainability_mode']);model.to(device).eval() + tokenizer = AutoTokenizer.from_pretrained(path/'tokenizer',local_files_only=True,trust_remote_code=False) + if verify_export(path) != manifest: + raise ValueError('Native changed while loading') + return model, MarkerCollator(tokenizer, **cfg['packing']), cfg diff --git a/native/policy/contract.py b/native/policy/contract.py new file mode 100644 index 0000000000000000000000000000000000000000..dd8d367b1b5ada4478bbc2840db1d24644e1a2e4 --- /dev/null +++ b/native/policy/contract.py @@ -0,0 +1,78 @@ +"""Pure metadata and tensor-name contract; no Torch/model import.""" +ARCHITECTURE = 'vela_decision_type_specific_interaction_v1' +SCHEMA = 'decision.nano.type_specific_heads.v1' +POLICY = 'choice_noul_score_independent_interaction_scorer_shared_encoder_type_embedding_v1' +KINDS = ('choice', 'noul', 'score') +PARENT_MANIFEST_SHA256 = '71d961913984a2fdf311ba6cb732ae9a745238d3e5347ad35be00f64a5024219' +ENCODER_PARAMETERS = 306939648 +EXPECTED_HEAD_CONFIG = {'head_layers': 2, 'head_heads': 12, 'head_dropout': 0.1, + 'attention_mode': 'sdpa', 'gradient_checkpointing': True} + +def parent_head_shapes(d=768): + if type(d) is not int or d < 1: + raise ValueError('Positive integer hidden size') + layer = {'self_attn.in_proj_weight': (3*d, d), 'self_attn.in_proj_bias': (3*d,), + 'self_attn.out_proj.weight': (d, d), 'self_attn.out_proj.bias': (d,), + 'linear1.weight': (4*d, d), 'linear1.bias': (4*d,), + 'linear2.weight': (d, 4*d), 'linear2.bias': (d,), + 'norm1.weight': (d,), 'norm1.bias': (d,), 'norm2.weight': (d,), 'norm2.bias': (d,)} + result = {f'head.{i}.{n}': shape for i in range(2) for n, shape in layer.items()} + result.update({'type_embedding.weight': (3, d), 'scorer.0.weight': (d,), 'scorer.0.bias': (d,), + 'scorer.1.weight': (d, d), 'scorer.1.bias': (d,), + 'scorer.3.weight': (1, d), 'scorer.3.bias': (1,)}) + return result + +def head_mapping(): + result = {'type_embedding.weight': 'type_embedding.weight'} + for name in parent_head_shapes(): + if name.startswith('head.'): + result.update({f'heads.{kind}.{name[5:]}': name for kind in KINDS}) + elif name.startswith('scorer.'): + result.update({f'scorers.{kind}.{name[7:]}': name for kind in KINDS}) + return result + +def head_shapes(d=768): + parent = parent_head_shapes(d) + return {dst: parent[src] for dst, src in head_mapping().items()} + +def numel(shape): + out = 1 + for n in shape: out *= n + return out + +def counts(d=768, encoder_parameters=ENCODER_PARAMETERS, encoder_tensors=134): + parent = parent_head_shapes(d) + branch = sum(numel(s) for n, s in parent.items() if n != 'type_embedding.weight') + type_embedding = 3*d + return {'encoder_parameters': encoder_parameters, 'encoder_tensors': encoder_tensors, + 'shared_type_embedding_parameters': type_embedding, 'shared_type_embedding_tensors': 1, + 'branch_parameters': branch, 'branch_tensors': 30, 'branches': 3, + 'nonencoder_parameters': 3*branch+type_embedding, 'nonencoder_tensors': 91, + 'total_parameters': encoder_parameters+3*branch+type_embedding, + 'total_tensors': encoder_tensors+91, 'score_only_trainable_parameters': branch, + 'score_only_trainable_tensors': 30, + 'extra_parameters_over_parent': 2*branch, + 'fp32_parameter_bytes': 4*(encoder_parameters+3*branch+type_embedding)} + +def trainable_name(name, mode): + if mode not in ('all', 'choice_only', 'noul_only', 'score_only'): + raise ValueError('Unknown explicit trainability mode') + return mode == 'all' or name.startswith(('heads.'+mode[:-5]+'.', 'scorers.'+mode[:-5]+'.')) + +def validate_config(cfg): + expected = {'architecture': ARCHITECTURE, 'schema': SCHEMA, 'branch_policy': POLICY, + 'type_order': list(KINDS), 'parameters': counts()['total_parameters'], + 'tensor_count': 225, 'state_cache': False, 'weight_dtype': 'float32', + 'base_model': 'llm-semantic-router/Vela-1.0-Encoder-307M', + 'base_revision': 'fe9ccc074b781bc0e2e13c2c8d26f2640410636a', + 'rocm_contiguous_layout': True, 'head': EXPECTED_HEAD_CONFIG, + 'packing': {'max_length': 8192, 'state_truncation': 'error'}, + 'inference_precision': 'fp32'} + for key, value in expected.items(): + if cfg.get(key) != value: + raise ValueError('Type-specific native identity mismatch: '+key) + if cfg.get('trainability_mode') not in ('all', 'choice_only', 'noul_only', 'score_only'): + raise ValueError('Missing explicit trainability mode') + if cfg.get('initializer_manifest_sha256') != PARENT_MANIFEST_SHA256: + raise ValueError('Unknown prototype initializer') + return cfg diff --git a/native/policy/infer.py b/native/policy/infer.py new file mode 100644 index 0000000000000000000000000000000000000000..324ff02c94ee68f800b5e66993b2240176a909d9 --- /dev/null +++ b/native/policy/infer.py @@ -0,0 +1,43 @@ +"""Local export inference. Each record is an independent question row.""" +import argparse +import json +try: + from .artifacts import load_export + from .model import predict +except ImportError: + from artifacts import load_export + from model import predict + + +class DecisionClient: + def __init__(self, model, collator, config, device): + self.model, self.collator, self.config, self.device = model, collator, config, device + + @classmethod + def from_pretrained(cls, path, device="cpu"): + return cls(*load_export(path, device), device) + + def predict(self, records, batch_size=1, precision=None): + return predict(self.model, self.collator, records, batch_size, self.device, + precision or self.config.get("inference_precision", "fp32")) + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--model", required=True) + parser.add_argument("--input", required=True) + parser.add_argument("--output", required=True) + parser.add_argument("--device", default="cpu") + parser.add_argument("--batch-size", type=int, default=1) + args = parser.parse_args() + with open(args.input) as f: + rows = [json.loads(line) for line in f if line.strip()] + client = DecisionClient.from_pretrained(args.model, args.device) + outputs = client.predict(rows, args.batch_size) + with open(args.output, "x") as f: + for row in outputs: + f.write(json.dumps(row, ensure_ascii=False, allow_nan=False) + "\n") + + +if __name__ == "__main__": + main() diff --git a/native/policy/model.py b/native/policy/model.py new file mode 100644 index 0000000000000000000000000000000000000000..904a5f40a921c5fb90ad026e060141ccbf61b772 --- /dev/null +++ b/native/policy/model.py @@ -0,0 +1,135 @@ +"""One shared joint encoder; independent interaction/scorer branches per type.""" +from contextlib import nullcontext +from copy import deepcopy +import torch +from torch import nn +from torch.utils.checkpoint import checkpoint +try: + from .contract import ARCHITECTURE, KINDS, EXPECTED_HEAD_CONFIG, counts, head_mapping, parent_head_shapes, head_shapes, trainable_name + from .reference.model import typed_loss, predict +except ImportError: + from contract import ARCHITECTURE, KINDS, EXPECTED_HEAD_CONFIG, counts, head_mapping, parent_head_shapes, head_shapes, trainable_name + from reference.model import typed_loss, predict + + +def validate_tensor_map(state, expected): + if set(state) != set(expected): + raise ValueError('Exact tensor-name set required') + for name, shape in expected.items(): + v = state[name] + if tuple(v.shape) != tuple(shape) or v.dtype != torch.float32 or not torch.isfinite(v).all(): + raise ValueError('Invalid tensor: '+name) + + +def expand_parent_head(state, hidden_size=768): + """Exact independent clones, used for explicit copy verification and static tests.""" + validate_tensor_map(state, parent_head_shapes(hidden_size)) + return {dest: state[src].detach().clone().contiguous() for dest, src in head_mapping().items()} + + +def route_indices(kind_ids): + if kind_ids.ndim != 1 or kind_ids.dtype != torch.long or kind_ids.numel() < 1: + raise ValueError('Nonempty int64 kind vector required') + if ((kind_ids < 0) | (kind_ids >= 3)).any(): + raise ValueError('Unknown kind; no default branch') + return [(kind, torch.nonzero(kind_ids == i, as_tuple=False).flatten()) for i, kind in enumerate(KINDS)] + + +def scatter_rows(output, indices, branch_logits): + """Functional index_copy retains autograd and original record/candidate order.""" + return output.index_copy(0, indices, branch_logits) + + +class DecisionModel(nn.Module): + def __init__(self, encoder, head_config, shared_template=None): + super().__init__() + if head_config != EXPECTED_HEAD_CONFIG or encoder.config.hidden_size != 768: + raise ValueError('Prototype fixes the existing768/2layer/12head geometry') + self.encoder = encoder + self.hidden_size = d = encoder.config.hidden_size + self.head_config = dict(head_config) + if shared_template is None: + self.type_embedding = nn.Embedding(3, d) + self.heads = nn.ModuleDict({kind: nn.ModuleList([ + nn.TransformerEncoderLayer(d, 12, 4*d, .1, activation='relu', + batch_first=True, norm_first=True) for _ in range(2)]) for kind in KINDS}) + self.scorers = nn.ModuleDict({kind: nn.Sequential(nn.LayerNorm(d), nn.Linear(d, d), + nn.GELU(), nn.Linear(d, 1)) for kind in KINDS}) + else: + if shared_template.encoder is not encoder or shared_template.head_config != head_config: + raise ValueError('Explicit same encoder/template required') + self.type_embedding = deepcopy(shared_template.type_embedding) + self.heads = nn.ModuleDict({kind: deepcopy(shared_template.head) for kind in KINDS}) + self.scorers = nn.ModuleDict({kind: deepcopy(shared_template.scorer) for kind in KINDS}) + parent = {n: v for n, v in shared_template.state_dict().items() if not n.startswith('encoder.')} + validate_tensor_map(parent, parent_head_shapes()) + copied = {n: v for n, v in self.state_dict().items() if not n.startswith('encoder.')} + validate_tensor_map(copied, head_shapes()) + for dest, source in head_mapping().items(): + if not torch.equal(copied[dest], parent[source]) or copied[dest].data_ptr() == parent[source].data_ptr(): + raise ValueError('Branch copy value/storage mismatch: '+dest) + if len({v.data_ptr() for v in copied.values()}) != len(copied): + raise ValueError('Aliased type/branch parameter storage') + if self.head_config['gradient_checkpointing'] and hasattr(encoder, 'gradient_checkpointing_enable'): + encoder.gradient_checkpointing_enable(gradient_checkpointing_kwargs={'use_reentrant': False}) + self.trainability_mode = 'all' + self.set_trainability('all') + + def set_trainability(self, mode): + for name, parameter in self.named_parameters(): + parameter.requires_grad_(trainable_name(name, mode)) + parameter.grad = None # Fresh optimizer required after policy changes. + self.trainability_mode = mode + self.train(self.training) + return {'mode': mode, 'trainable_parameters': sum(p.numel() for p in self.parameters() if p.requires_grad), + 'trainable_names': [n for n, p in self.named_parameters() if p.requires_grad]} + + def train(self, mode=True): + super().train(mode) + if getattr(self, 'trainability_mode', 'all') != 'all': + self.encoder.eval() + self.type_embedding.eval() + for kind in KINDS: + if kind != self.trainability_mode[:-5]: + self.heads[kind].eval() + self.scorers[kind].eval() + return self + + def forward(self, batch): + from torch.nn.attention import sdpa_kernel, SDPBackend + routes = route_indices(batch['kind_ids']) + if batch['input_ids'].shape[0] != batch['kind_ids'].numel(): + raise ValueError('Type routing and original rows differ') + context = sdpa_kernel(SDPBackend.MATH) if self.head_config['attention_mode'] == 'math' else nullcontext() + with context: + shared_context = torch.no_grad() if self.trainability_mode != 'all' else nullcontext() + with shared_context: + h = self.encoder(input_ids=batch['input_ids'], attention_mask=batch['attention_mask'], + return_dict=True).last_hidden_state + h = h + self.type_embedding(batch['kind_ids'])[:, None, :].to(h.dtype) + pad = ~batch['attention_mask'].bool() + output = torch.empty(batch['marker_positions'].shape, device=h.device, dtype=torch.float32) + for kind, indices in routes: + if indices.numel() == 0: + continue + hidden = h.index_select(0, indices) + branch_pad = pad.index_select(0, indices) # Keep original shared L, never repack/retrim. + for layer in self.heads[kind]: + if layer.training and self.head_config['gradient_checkpointing']: + hidden = checkpoint(layer, hidden, src_key_padding_mask=branch_pad, use_reentrant=False) + else: + hidden = layer(hidden, src_key_padding_mask=branch_pad) + positions = batch['marker_positions'].index_select(0, indices) + markers = torch.gather(hidden, 1, positions[:, :, None].expand(-1, -1, hidden.shape[-1])) + logits = self.scorers[kind](markers).squeeze(-1).float() + output = scatter_rows(output, indices, logits) + return output.masked_fill(~batch['valid_candidates'], torch.finfo(torch.float32).min) + + +def from_shared_model(parent): + """Borrows exactly the same encoder object; never deep-copies the backbone.""" + model = DecisionModel(parent.encoder, parent.head_config, shared_template=parent) + if sum(p.numel() for p in model.parameters()) != counts()['total_parameters'] or len(model.state_dict()) != 225: + raise ValueError('Unexpected complete parameter inventory') + model.eval() + return model diff --git a/native/policy/packing.py b/native/policy/packing.py new file mode 100644 index 0000000000000000000000000000000000000000..d8a9fb92bb2eb561d54dc17a312575cb368506d7 --- /dev/null +++ b/native/policy/packing.py @@ -0,0 +1,129 @@ +"""One joint row per question. Metadata IDs and gold labels never enter tokens.""" +import math +import torch + +KINDS = ("choice", "noul", "score") +DEFAULT_NO = "No. The statement or question is not satisfied." +DEFAULT_YES = "Yes. The statement or question is satisfied." + + +def nonempty(value, name): + if not isinstance(value, str) or not value.strip(): + raise ValueError(f"{name} must be a nonempty string") + return value + + +def view(record, labeled=False): + q = record["question"] + kind = str(q["type"]).lower() + if kind not in KINDS: + raise ValueError("Unsupported question type") + nonempty(record["state_text"], "state_text") + nonempty(q["text"], "question.text") + if kind == "noul": + options = [{"id": "no", "text": q.get("false_criterion", DEFAULT_NO)}, + {"id": "yes", "text": q.get("true_criterion", DEFAULT_YES)}] + else: + options = q["options" if kind == "choice" else "levels"] + if not isinstance(options, list) or not 2 <= len(options) <= 255: + raise ValueError("Require 2..255 complete candidate descriptions") + for o in options: + nonempty(o["id"], "candidate.id") + nonempty(o["text"], "candidate.text") + if len({o["id"] for o in options}) != len(options): + raise ValueError("Duplicate candidate IDs") + values = [float(o["value"]) for o in options] if kind == "score" else list(range(len(options))) + if kind == "score" and (not all(math.isfinite(v) for v in values) + or any(a >= b for a, b in zip(values, values[1:]))): + raise ValueError("Score level values must be finite and strictly increasing") + target = None + if labeled: + gold = record["target"] + if kind == "noul": + yes = float(gold["probability"]) + target = [1 - yes, yes] + elif "probabilities" in gold: + target = [float(x) for x in gold["probabilities"]] + elif kind == "choice" and "choice_id" in gold: + if gold["choice_id"] not in {o["id"] for o in options}: + raise ValueError("Unknown gold choice_id") + target = [float(o["id"] == gold["choice_id"]) for o in options] + else: + raise ValueError("Score requires an explicit target probability distribution") + if (len(target) != len(options) or not all(math.isfinite(x) and 0 <= x <= 1 for x in target) + or abs(sum(target) - 1) > 1e-6): + raise ValueError("Invalid target probability distribution") + return kind, options, values, target + + +class MarkerCollator: + def __init__(self, tokenizer, max_length=8192, state_truncation="error"): + if isinstance(max_length, bool) or not isinstance(max_length, int) or max_length < 8: + raise ValueError("Invalid max_length") + if state_truncation not in ("error", "right"): + raise ValueError("state_truncation must be error or right") + self.tokenizer, self.max_length = tokenizer, max_length + self.state_truncation = state_truncation + self.bos = tokenizer.cls_token_id if tokenizer.cls_token_id is not None else tokenizer.bos_token_id + self.sep = tokenizer.sep_token_id if tokenizer.sep_token_id is not None else tokenizer.eos_token_id + self.pad, self.marker = tokenizer.pad_token_id, tokenizer.mask_token_id + if any(v is None for v in (self.bos, self.sep, self.pad, self.marker)): + raise ValueError("Tokenizer must provide existing BOS/CLS, SEP/EOS, PAD and MASK IDs") + + def tokens(self, text): + ids = self.tokenizer(text, add_special_tokens=False, truncation=False)["input_ids"] + if not ids: + raise ValueError("Empty token span") + return ids + + def encode(self, record, labeled=False): + kind, options, values, target = view(record, labeled) + question = self.tokens(f"{kind} question: {record['question']['text']}") + ids = [self.bos] + question + [self.sep] + positions, spans = [], [] + for j, option in enumerate(options): + # Ordinal rank is meaningful only for Score; opaque external IDs never enter text. + text = f"level {j}: {option['text']}" if kind == "score" else option["text"] + desc = self.tokens(text) + positions.append(len(ids)) + ids.append(self.marker) + spans.append([len(ids), len(ids) + len(desc)]) + ids.extend(desc + [self.sep]) + state = self.tokens(record["state_text"]) + room = self.max_length - len(ids) - 1 + if room < 1: + raise ValueError("Complete question and candidates leave no room for state") + if len(state) > room and self.state_truncation == "error": + raise ValueError(f"Record {record.get('id')} exceeds {self.max_length} tokens; no implicit truncation") + kept = state[:room] + state_span = [len(ids), len(ids) + len(kept)] + ids.extend(kept + [self.sep]) + return {"ids": ids, "positions": positions, "candidate_spans": spans, + "state_span": state_span, "kind": kind, "target": target, "values": values, + "candidate_ids": [o["id"] for o in options], "id": record.get("id"), + "question_id": record["question"].get("id"), "state_tokens_original": len(state), + "state_tokens_kept": len(kept), "input_tokens": len(ids)} + + def __call__(self, records, labeled=False, device="cpu"): + encoded = [self.encode(r, labeled) for r in records] + if not encoded: + raise ValueError("Empty batch") + b, length, k = len(encoded), max(len(e["ids"]) for e in encoded), max(len(e["positions"]) for e in encoded) + ids = torch.full((b, length), self.pad, dtype=torch.long) + mask = torch.zeros((b, length), dtype=torch.bool) + positions = torch.zeros((b, k), dtype=torch.long) + valid = torch.zeros((b, k), dtype=torch.bool) + target, values = torch.zeros((b, k)), torch.zeros((b, k)) + for i, e in enumerate(encoded): + n, count = len(e["ids"]), len(e["positions"]) + ids[i, :n] = torch.tensor(e["ids"]) + mask[i, :n] = True + positions[i, :count] = torch.tensor(e["positions"]) + valid[i, :count] = True + values[i, :count] = torch.tensor(e["values"]) + if labeled: + target[i, :count] = torch.tensor(e["target"]) + batch = {"input_ids": ids, "attention_mask": mask, "marker_positions": positions, + "valid_candidates": valid, "targets": target, "values": values, + "kind_ids": torch.tensor([KINDS.index(e["kind"]) for e in encoded])} + return {name: value.to(device) for name, value in batch.items()}, encoded diff --git a/native/policy/reference/__init__.py b/native/policy/reference/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..b06135a8780dbf6d18a66d0313e18a6bad8ab88c --- /dev/null +++ b/native/policy/reference/__init__.py @@ -0,0 +1,5 @@ +"""Vela Decision Nano: dynamic typed decisions over complete descriptions.""" +from .artifacts import load_export +from .infer import DecisionClient + +__all__ = ["DecisionClient", "load_export"] diff --git a/native/policy/reference/artifacts.py b/native/policy/reference/artifacts.py new file mode 100644 index 0000000000000000000000000000000000000000..ff370e1ed06f07d3dcffab80390f4cde0f0f2514 --- /dev/null +++ b/native/policy/reference/artifacts.py @@ -0,0 +1,190 @@ +"""Pinned local base loading and self-contained, hash-checked exports.""" +from pathlib import Path +import hashlib +import json +import shutil +import torch +try: + from .model import DecisionModel, ARCHITECTURE + from .packing import MarkerCollator +except ImportError: + from model import DecisionModel, ARCHITECTURE + from packing import MarkerCollator + +ROOT = Path(__file__).resolve().parent +RUNTIME_FILES = ("__init__.py", "model.py", "packing.py", "artifacts.py", "infer.py", "modernbert_sdpa_layout.py") + + +def sha(path): + h = hashlib.sha256() + with open(path, "rb") as stream: + for chunk in iter(lambda: stream.read(1 << 20), b""): + h.update(chunk) + return h.hexdigest() + + +def write_json(path, value): + Path(path).write_text(json.dumps(value, ensure_ascii=False, indent=2, sort_keys=True, allow_nan=False) + "\n") + + +def read_json(path): + return json.loads(Path(path).read_text()) + + +def file_ref(path): + path = Path(path) + return {"sha256": sha(path), "bytes": path.stat().st_size} + + +def source_refs(): + return {p.name: file_ref(p) for p in sorted(ROOT.glob("*.py"))} + + +def install_layout(encoder, enabled): + if enabled and encoder.config.model_type == "modernbert": + try: + from .modernbert_sdpa_layout import install_rocm_sdpa_layout_guard + except ImportError: + from modernbert_sdpa_layout import install_rocm_sdpa_layout_guard + install_rocm_sdpa_layout_guard(encoder) + + +def create_model(base_path, config): + """No network fallback and no permissive MLM-state loading.""" + from transformers import AutoConfig, AutoModel, AutoTokenizer + from safetensors.torch import load_file + base_path = Path(base_path) + pin = read_json(ROOT / "BASE_PIN.json") + if config["base_model"] != pin["repo"] or config["base_revision"] != pin["revision"]: + raise ValueError("This implementation pins the declared Vela Base revision") + for item in pin["files"]: + if file_ref(base_path / item["name"]) != {k: item[k] for k in ("sha256", "bytes")}: + raise ValueError("Base identity mismatch: " + item["name"]) + ec = AutoConfig.from_pretrained(base_path, local_files_only=True, trust_remote_code=False) + ec.reference_compile = False + if ec.model_type != "modernbert" or ec.hidden_size != 768 or ec.num_hidden_layers != 22: + raise ValueError("Unexpected Vela backbone geometry") + if config["packing"]["max_length"] > ec.max_position_embeddings: + raise ValueError("Packing limit exceeds backbone configuration") + encoder = AutoModel.from_config(ec, trust_remote_code=False, torch_dtype=torch.float32, attn_implementation="sdpa") + raw = load_file(str(base_path / "model.safetensors"), device="cpu") + expected = encoder.state_dict() + extras = {"head.dense.weight": (768, 768), "head.norm.weight": (768,), "decoder.bias": (256000,)} + if set(raw) != {"model." + n for n in expected} | set(extras): + raise ValueError("Unexpected masked-LM tensor names") + for n, shape in extras.items(): + if tuple(raw[n].shape) != shape: + raise ValueError("Unexpected MLM head geometry") + for n, value in raw.items(): + if value.dtype != torch.float32 or not torch.isfinite(value).all(): + raise ValueError("Invalid base tensor: " + n) + encoder.load_state_dict({n: raw["model." + n] for n in expected}, strict=True) + del raw + if sum(p.numel() for p in encoder.parameters()) != 306939648: + raise ValueError("Unexpected encoder parameter count") + install_layout(encoder, config.get("rocm_contiguous_layout", True)) + torch.manual_seed(config["head_seed"]) + model = DecisionModel(encoder, **config["head"]) + tokenizer = AutoTokenizer.from_pretrained(base_path, local_files_only=True, trust_remote_code=False) + return model, tokenizer, {"encoder_parameters": sum(p.numel() for p in encoder.parameters()), + "total_parameters": sum(p.numel() for p in model.parameters()), + "trainable_parameters": sum(p.numel() for p in model.parameters() if p.requires_grad)} + + +def save_weights(model, path): + from safetensors.torch import save_file + state = {n: v.detach().cpu().contiguous() for n, v in model.state_dict().items()} + for n, value in state.items(): + if value.is_floating_point() and not torch.isfinite(value).all(): + raise ValueError("Cannot save nonfinite tensor: " + n) + save_file(state, str(path)) + + +def restore_weights(model, path): + from safetensors.torch import load_file + model.load_state_dict(load_file(str(path), device="cpu"), strict=True) + + +def export_model(model, tokenizer, path, config, provenance): + """All weights are full fine-tuned tensors; no adapters or merge operation.""" + from safetensors.torch import save_file + path = Path(path) + path.mkdir(parents=True, exist_ok=False) + ec = model.encoder.config.to_dict() + ec.pop("_name_or_path", None) + ec["architectures"] = [type(model.encoder).__name__] + (path / "encoder").mkdir() + write_json(path / "encoder/config.json", ec) + save_file({n: v.detach().cpu().contiguous() for n, v in model.encoder.state_dict().items()}, str(path / "encoder/model.safetensors")) + head = {n: v.detach().cpu().contiguous() for n, v in model.state_dict().items() if not n.startswith("encoder.")} + save_file(head, str(path / "decision_head.safetensors")) + tokenizer.save_pretrained(path / "tokenizer") + for filename in RUNTIME_FILES: + shutil.copyfile(ROOT / filename, path / filename) + cfg = {"architecture": ARCHITECTURE, "schema": "decision.nano.v1", "head": model.head_config, + "packing": config["packing"], "rocm_contiguous_layout": config.get("rocm_contiguous_layout", True), + "base_model": config["base_model"], "base_revision": config["base_revision"], + "parameters": sum(p.numel() for p in model.parameters()), "provenance": provenance, + "calibration": "none; raw probabilities", "state_cache": False, + "inference_precision": config.get("precision", "fp32"), "weight_dtype": "float32"} + write_json(path / "decision_config.json", cfg) + refs = {p.relative_to(path).as_posix(): file_ref(p) for p in sorted(path.rglob("*")) if p.is_file()} + write_json(path / "MANIFEST.json", {"schema": "decision.files.v1", "files": refs}) + return file_ref(path / "MANIFEST.json") + + +def verify_export(path): + path = Path(path) + manifest = read_json(path / "MANIFEST.json") + if manifest.get("schema") != "decision.files.v1": + raise ValueError("Unknown export schema") + actual = set() + for p in path.rglob("*"): + if p.is_symlink(): + raise ValueError("Export contains a symlink") + rel = p.relative_to(path) + if p.is_file() and rel.as_posix() != "MANIFEST.json": + if "__pycache__" in rel.parts and p.suffix == ".pyc": + continue + actual.add(rel.as_posix()) + if actual != set(manifest["files"]): + raise ValueError("Export file set mismatch") + for name, ref in manifest["files"].items(): + if Path(name).is_absolute() or ".." in Path(name).parts or file_ref(path / name) != ref: + raise ValueError("Export integrity mismatch: " + name) + return manifest + + +def load_export(path, device="cpu"): + from transformers import AutoConfig, AutoModel, AutoTokenizer + from safetensors.torch import load_file + path = Path(path) + manifest = verify_export(path) + cfg = read_json(path / "decision_config.json") + if cfg.get("architecture") != ARCHITECTURE or cfg.get("schema") != "decision.nano.v1": + raise ValueError("Unknown model architecture") + ec = AutoConfig.from_pretrained(path / "encoder", local_files_only=True, trust_remote_code=False) + ec.reference_compile = False + encoder = AutoModel.from_config(ec, trust_remote_code=False, torch_dtype=torch.float32, attn_implementation="sdpa") + enc_state = load_file(str(path / "encoder/model.safetensors")) + for name, value in enc_state.items(): + if value.dtype != torch.float32 or not torch.isfinite(value).all(): + raise ValueError("Invalid exported encoder tensor: " + name) + encoder.load_state_dict(enc_state, strict=True) + del enc_state + install_layout(encoder, cfg["rocm_contiguous_layout"]) + model = DecisionModel(encoder, **cfg["head"]) + head = load_file(str(path / "decision_head.safetensors")) + expected = {n for n in model.state_dict() if not n.startswith("encoder.")} + if set(head) != expected: + raise ValueError("Interaction-head tensor set mismatch") + if any(v.dtype != torch.float32 or not torch.isfinite(v).all() for v in head.values()): + raise ValueError("Invalid exported interaction-head tensor") + model.load_state_dict({**{"encoder."+n: v for n, v in encoder.state_dict().items()}, **head}, strict=True) + if sum(p.numel() for p in model.parameters()) != cfg["parameters"]: + raise ValueError("Export parameter count mismatch") + model.to(device).eval() + tok = AutoTokenizer.from_pretrained(path / "tokenizer", local_files_only=True, trust_remote_code=False) + if verify_export(path) != manifest: + raise ValueError("Export changed while loading") + return model, MarkerCollator(tok, **cfg["packing"]), cfg diff --git a/native/policy/reference/infer.py b/native/policy/reference/infer.py new file mode 100644 index 0000000000000000000000000000000000000000..324ff02c94ee68f800b5e66993b2240176a909d9 --- /dev/null +++ b/native/policy/reference/infer.py @@ -0,0 +1,43 @@ +"""Local export inference. Each record is an independent question row.""" +import argparse +import json +try: + from .artifacts import load_export + from .model import predict +except ImportError: + from artifacts import load_export + from model import predict + + +class DecisionClient: + def __init__(self, model, collator, config, device): + self.model, self.collator, self.config, self.device = model, collator, config, device + + @classmethod + def from_pretrained(cls, path, device="cpu"): + return cls(*load_export(path, device), device) + + def predict(self, records, batch_size=1, precision=None): + return predict(self.model, self.collator, records, batch_size, self.device, + precision or self.config.get("inference_precision", "fp32")) + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--model", required=True) + parser.add_argument("--input", required=True) + parser.add_argument("--output", required=True) + parser.add_argument("--device", default="cpu") + parser.add_argument("--batch-size", type=int, default=1) + args = parser.parse_args() + with open(args.input) as f: + rows = [json.loads(line) for line in f if line.strip()] + client = DecisionClient.from_pretrained(args.model, args.device) + outputs = client.predict(rows, args.batch_size) + with open(args.output, "x") as f: + for row in outputs: + f.write(json.dumps(row, ensure_ascii=False, allow_nan=False) + "\n") + + +if __name__ == "__main__": + main() diff --git a/native/policy/reference/model.py b/native/policy/reference/model.py new file mode 100644 index 0000000000000000000000000000000000000000..ae1fe7363218105f3aa8f3872c464d83444b9d50 --- /dev/null +++ b/native/policy/reference/model.py @@ -0,0 +1,90 @@ +"""Bidirectional backbone plus two full-token typed interaction layers.""" +from contextlib import nullcontext +import torch +from torch import nn +from torch.nn import functional as F +from torch.utils.checkpoint import checkpoint + +ARCHITECTURE = "vela_decision_candidate_markers_v1" + + +class DecisionModel(nn.Module): + def __init__(self, encoder, head_layers=2, head_heads=12, head_dropout=0.1, + attention_mode="sdpa", gradient_checkpointing=False): + super().__init__() + if head_layers < 1 or encoder.config.hidden_size % head_heads: + raise ValueError("Invalid interaction-head geometry") + if attention_mode not in ("math", "sdpa"): + raise ValueError("Unknown attention mode") + self.encoder = encoder + self.hidden_size = d = encoder.config.hidden_size + self.head_config = dict(head_layers=head_layers, head_heads=head_heads, head_dropout=head_dropout, + attention_mode=attention_mode, gradient_checkpointing=gradient_checkpointing) + # Independent initialization per layer, rather than cloning a single layer's weights. + self.head = nn.ModuleList([nn.TransformerEncoderLayer(d, head_heads, 4*d, head_dropout, + activation="relu", batch_first=True, norm_first=True) for _ in range(head_layers)]) + self.type_embedding = nn.Embedding(3, d) + self.scorer = nn.Sequential(nn.LayerNorm(d), nn.Linear(d, d), nn.GELU(), nn.Linear(d, 1)) + if gradient_checkpointing and hasattr(encoder, "gradient_checkpointing_enable"): + encoder.gradient_checkpointing_enable(gradient_checkpointing_kwargs={"use_reentrant": False}) + + def forward(self, batch): + from torch.nn.attention import sdpa_kernel, SDPBackend + context = sdpa_kernel(SDPBackend.MATH) if self.head_config["attention_mode"] == "math" else nullcontext() + with context: + h = self.encoder(input_ids=batch["input_ids"], attention_mask=batch["attention_mask"], + return_dict=True).last_hidden_state + h = h + self.type_embedding(batch["kind_ids"])[:, None, :].to(h.dtype) + pad = ~batch["attention_mask"].bool() + for layer in self.head: + if self.training and self.head_config["gradient_checkpointing"]: + h = checkpoint(layer, h, src_key_padding_mask=pad, use_reentrant=False) + else: + h = layer(h, src_key_padding_mask=pad) + markers = torch.gather(h, 1, batch["marker_positions"][:, :, None].expand(-1, -1, h.shape[-1])) + logits = self.scorer(markers).squeeze(-1).float() + return logits.masked_fill(~batch["valid_candidates"], torch.finfo(torch.float32).min) + + +def typed_loss(logits, batch, score_rps_weight=1.0): + """Soft-label CE, binary CE for Noul, and optional normalized ordinal RPS for Score.""" + logp = F.log_softmax(logits.float(), dim=-1) + ce = -(batch["targets"] * logp).sum(-1) + count = batch["valid_candidates"].sum(-1) + cdf_error = (logp.exp().cumsum(-1) - batch["targets"].cumsum(-1)).square() + # Exclude the terminal CDF (always one), and any padded candidates. + thresholds = torch.arange(logits.shape[1], device=logits.device)[None] < (count - 1)[:, None] + rps = (cdf_error * thresholds).sum(-1) / (count - 1) + return ce + score_rps_weight * rps * (batch["kind_ids"] == 2), ce, rps + + +@torch.inference_mode() +def predict(model, collator, records, batch_size=1, device="cpu", precision="fp32"): + if batch_size < 1: + raise ValueError("Positive batch_size required") + model.eval() + result = [] + for start in range(0, len(records), batch_size): + batch, rows = collator(records[start:start+batch_size], device=device) + if precision not in ("fp32", "bf16"): + raise ValueError("Unknown inference precision") + context = torch.autocast(torch.device(device).type, dtype=torch.bfloat16) if precision == "bf16" else nullcontext() + with context: + z = model(batch) + if not torch.isfinite(z).all(): + raise FloatingPointError("Nonfinite prediction") + for i, row in enumerate(rows): + logits = z[i, :len(row["positions"])] + p = logits.softmax(-1) + out = {k: row[k] for k in ("id", "question_id", "candidate_ids", "state_tokens_original", "state_tokens_kept", "input_tokens")} + out.update(type=row["kind"].capitalize(), logits=logits.cpu().tolist(), + probabilities=p.cpu().tolist(), confidence=float(p.max())) + if row["kind"] == "noul": + out["probability"] = float(p[1]) + else: + out["choice_id"] = row["candidate_ids"][int(p.argmax())] + if row["kind"] == "score": + out["score"] = float((p * torch.arange(len(p), device=p.device)).sum()) + out["expected_value"] = float((p * batch["values"][i, :len(p)]).sum()) + result.append(out) + return result diff --git a/native/policy/reference/modernbert_sdpa_layout.py b/native/policy/reference/modernbert_sdpa_layout.py new file mode 100644 index 0000000000000000000000000000000000000000..9d2812fcdd3b4ef65b703bc5c72cc2aae83b58b8 --- /dev/null +++ b/native/policy/reference/modernbert_sdpa_layout.py @@ -0,0 +1,93 @@ +"""Keep ModernBERT's ROCm efficient-SDPA inputs contiguous. + +The native 4.57 ModernBERT readout leaves Q/K transposed and V as a strided +view of a packed QKV projection. ROCm's efficient backend can miscompute that +layout. This model-local guard changes storage only; CPU and forced-MATH calls +continue through the original attention implementation. +""" + +from __future__ import annotations + +from types import MethodType + +import torch +from torch.nn import functional +from transformers.models.modernbert.modeling_modernbert import ( + ModernBertAttention, + ModernBertModel, + apply_rotary_pos_emb, +) + +LAYOUT_POLICY = "rocm_efficient_contiguous_qkv" + + +def contiguous_sdpa_forward( + attention, + hidden_states, + *, + attention_mask, + sliding_window_mask, + position_ids, +): + """Native SDPA arithmetic with contiguous post-RoPE Q/K/V storage.""" + batch_size = hidden_states.shape[0] + qkv = attention.Wqkv(hidden_states).view( + batch_size, -1, 3, attention.num_heads, attention.head_dim + ) + cos, sin = attention.rotary_emb(qkv, position_ids=position_ids) + query, key, value = qkv.transpose(3, 1).unbind(dim=2) + query, key = apply_rotary_pos_emb(query, key, cos, sin) + mask = ( + attention_mask if attention.local_attention == (-1, -1) else sliding_window_mask + ) + output = functional.scaled_dot_product_attention( + query.contiguous(), + key.contiguous(), + value.contiguous(), + attn_mask=mask, + dropout_p=attention.attention_dropout if attention.training else 0.0, + ) + output = ( + output.transpose(1, 2) + .contiguous() + .view(batch_size, -1, attention.all_head_size) + ) + return (attention.out_drop(attention.Wo(output)),) + + +def _guarded_forward(self, hidden_states, output_attentions=False, **kwargs): + if ( + torch.version.hip is None + or hidden_states.device.type != "cuda" + or not torch.backends.cuda.mem_efficient_sdp_enabled() + or self.config._attn_implementation != "sdpa" + or output_attentions + ): + return self._vela_original_attention_forward( + hidden_states, output_attentions=output_attentions, **kwargs + ) + return contiguous_sdpa_forward( + self, + hidden_states, + attention_mask=kwargs["attention_mask"], + sliding_window_mask=kwargs["sliding_window_mask"], + position_ids=kwargs["position_ids"], + ) + + +def install_rocm_sdpa_layout_guard(encoder): + """Install once on this model's attention modules; no global monkey patch.""" + if not isinstance(encoder, ModernBertModel): + raise ValueError("The SDPA layout guard requires native ModernBERT") + for layer in encoder.layers: + attention = layer.attn + if not isinstance(attention, ModernBertAttention): + raise ValueError("Unexpected attention module in native ModernBERT") + if getattr(attention, "_vela_sdpa_layout_policy", None) == LAYOUT_POLICY: + continue + if hasattr(attention, "_vela_original_attention_forward"): + raise ValueError("An unknown attention guard is already installed") + attention._vela_original_attention_forward = attention.forward + attention.forward = MethodType(_guarded_forward, attention) + attention._vela_sdpa_layout_policy = LAYOUT_POLICY + return encoder diff --git a/native/policy/reference/packing.py b/native/policy/reference/packing.py new file mode 100644 index 0000000000000000000000000000000000000000..d8a9fb92bb2eb561d54dc17a312575cb368506d7 --- /dev/null +++ b/native/policy/reference/packing.py @@ -0,0 +1,129 @@ +"""One joint row per question. Metadata IDs and gold labels never enter tokens.""" +import math +import torch + +KINDS = ("choice", "noul", "score") +DEFAULT_NO = "No. The statement or question is not satisfied." +DEFAULT_YES = "Yes. The statement or question is satisfied." + + +def nonempty(value, name): + if not isinstance(value, str) or not value.strip(): + raise ValueError(f"{name} must be a nonempty string") + return value + + +def view(record, labeled=False): + q = record["question"] + kind = str(q["type"]).lower() + if kind not in KINDS: + raise ValueError("Unsupported question type") + nonempty(record["state_text"], "state_text") + nonempty(q["text"], "question.text") + if kind == "noul": + options = [{"id": "no", "text": q.get("false_criterion", DEFAULT_NO)}, + {"id": "yes", "text": q.get("true_criterion", DEFAULT_YES)}] + else: + options = q["options" if kind == "choice" else "levels"] + if not isinstance(options, list) or not 2 <= len(options) <= 255: + raise ValueError("Require 2..255 complete candidate descriptions") + for o in options: + nonempty(o["id"], "candidate.id") + nonempty(o["text"], "candidate.text") + if len({o["id"] for o in options}) != len(options): + raise ValueError("Duplicate candidate IDs") + values = [float(o["value"]) for o in options] if kind == "score" else list(range(len(options))) + if kind == "score" and (not all(math.isfinite(v) for v in values) + or any(a >= b for a, b in zip(values, values[1:]))): + raise ValueError("Score level values must be finite and strictly increasing") + target = None + if labeled: + gold = record["target"] + if kind == "noul": + yes = float(gold["probability"]) + target = [1 - yes, yes] + elif "probabilities" in gold: + target = [float(x) for x in gold["probabilities"]] + elif kind == "choice" and "choice_id" in gold: + if gold["choice_id"] not in {o["id"] for o in options}: + raise ValueError("Unknown gold choice_id") + target = [float(o["id"] == gold["choice_id"]) for o in options] + else: + raise ValueError("Score requires an explicit target probability distribution") + if (len(target) != len(options) or not all(math.isfinite(x) and 0 <= x <= 1 for x in target) + or abs(sum(target) - 1) > 1e-6): + raise ValueError("Invalid target probability distribution") + return kind, options, values, target + + +class MarkerCollator: + def __init__(self, tokenizer, max_length=8192, state_truncation="error"): + if isinstance(max_length, bool) or not isinstance(max_length, int) or max_length < 8: + raise ValueError("Invalid max_length") + if state_truncation not in ("error", "right"): + raise ValueError("state_truncation must be error or right") + self.tokenizer, self.max_length = tokenizer, max_length + self.state_truncation = state_truncation + self.bos = tokenizer.cls_token_id if tokenizer.cls_token_id is not None else tokenizer.bos_token_id + self.sep = tokenizer.sep_token_id if tokenizer.sep_token_id is not None else tokenizer.eos_token_id + self.pad, self.marker = tokenizer.pad_token_id, tokenizer.mask_token_id + if any(v is None for v in (self.bos, self.sep, self.pad, self.marker)): + raise ValueError("Tokenizer must provide existing BOS/CLS, SEP/EOS, PAD and MASK IDs") + + def tokens(self, text): + ids = self.tokenizer(text, add_special_tokens=False, truncation=False)["input_ids"] + if not ids: + raise ValueError("Empty token span") + return ids + + def encode(self, record, labeled=False): + kind, options, values, target = view(record, labeled) + question = self.tokens(f"{kind} question: {record['question']['text']}") + ids = [self.bos] + question + [self.sep] + positions, spans = [], [] + for j, option in enumerate(options): + # Ordinal rank is meaningful only for Score; opaque external IDs never enter text. + text = f"level {j}: {option['text']}" if kind == "score" else option["text"] + desc = self.tokens(text) + positions.append(len(ids)) + ids.append(self.marker) + spans.append([len(ids), len(ids) + len(desc)]) + ids.extend(desc + [self.sep]) + state = self.tokens(record["state_text"]) + room = self.max_length - len(ids) - 1 + if room < 1: + raise ValueError("Complete question and candidates leave no room for state") + if len(state) > room and self.state_truncation == "error": + raise ValueError(f"Record {record.get('id')} exceeds {self.max_length} tokens; no implicit truncation") + kept = state[:room] + state_span = [len(ids), len(ids) + len(kept)] + ids.extend(kept + [self.sep]) + return {"ids": ids, "positions": positions, "candidate_spans": spans, + "state_span": state_span, "kind": kind, "target": target, "values": values, + "candidate_ids": [o["id"] for o in options], "id": record.get("id"), + "question_id": record["question"].get("id"), "state_tokens_original": len(state), + "state_tokens_kept": len(kept), "input_tokens": len(ids)} + + def __call__(self, records, labeled=False, device="cpu"): + encoded = [self.encode(r, labeled) for r in records] + if not encoded: + raise ValueError("Empty batch") + b, length, k = len(encoded), max(len(e["ids"]) for e in encoded), max(len(e["positions"]) for e in encoded) + ids = torch.full((b, length), self.pad, dtype=torch.long) + mask = torch.zeros((b, length), dtype=torch.bool) + positions = torch.zeros((b, k), dtype=torch.long) + valid = torch.zeros((b, k), dtype=torch.bool) + target, values = torch.zeros((b, k)), torch.zeros((b, k)) + for i, e in enumerate(encoded): + n, count = len(e["ids"]), len(e["positions"]) + ids[i, :n] = torch.tensor(e["ids"]) + mask[i, :n] = True + positions[i, :count] = torch.tensor(e["positions"]) + valid[i, :count] = True + values[i, :count] = torch.tensor(e["values"]) + if labeled: + target[i, :count] = torch.tensor(e["target"]) + batch = {"input_ids": ids, "attention_mask": mask, "marker_positions": positions, + "valid_candidates": valid, "targets": target, "values": values, + "kind_ids": torch.tensor([KINDS.index(e["kind"]) for e in encoded])} + return {name: value.to(device) for name, value in batch.items()}, encoded diff --git a/native/score_encoder.safetensors b/native/score_encoder.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..1d23d017fc14eb7614368c7459f5e54296918284 --- /dev/null +++ b/native/score_encoder.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4f45795977846ef4c31b34e4f35bf95dabf5951d71358bab32c9a94814a84a53 +size 441337080 diff --git a/native/tokenizer/special_tokens_map.json b/native/tokenizer/special_tokens_map.json new file mode 100644 index 0000000000000000000000000000000000000000..be4ad795f49075ac5e56d8108ef63be314a1e967 --- /dev/null +++ b/native/tokenizer/special_tokens_map.json @@ -0,0 +1,55 @@ +{ + "additional_special_tokens": [ + "", + "" + ], + "bos_token": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false + }, + "cls_token": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false + }, + "eos_token": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false + }, + "mask_token": { + "content": "", + "lstrip": true, + "normalized": false, + "rstrip": false, + "single_word": false + }, + "pad_token": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false + }, + "sep_token": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false + }, + "unk_token": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false + } +} diff --git a/native/tokenizer/tokenizer.json b/native/tokenizer/tokenizer.json new file mode 100644 index 0000000000000000000000000000000000000000..56d765858ed38eaabd35cd90759cbb1f89b0baa4 --- /dev/null +++ b/native/tokenizer/tokenizer.json @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:609d8f4c067cd3950f88594c5a802616cea245823836ef5848ee4fc40aab5b6f +size 34363188 diff --git a/native/tokenizer/tokenizer_config.json b/native/tokenizer/tokenizer_config.json new file mode 100644 index 0000000000000000000000000000000000000000..8b7a2254239805140f894102e5bdc199186057b3 --- /dev/null +++ b/native/tokenizer/tokenizer_config.json @@ -0,0 +1,2018 @@ +{ + "add_bos_token": true, + "added_tokens_decoder": { + "0": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": true + }, + "1": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": true + }, + "2": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": true + }, + "3": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": true + }, + "4": { + "content": "", + "lstrip": true, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": true + }, + "5": { + "content": "<2mass>", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "6": { + "content": "[@BOS@]", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "7": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "8": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "9": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "10": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "11": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "12": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "13": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "14": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "15": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "16": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "17": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "18": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "19": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "20": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "21": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "22": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "23": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "24": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "25": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "26": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "27": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "28": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "29": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "30": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "31": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "32": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "33": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "34": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "35": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "36": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "37": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "38": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "39": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "40": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "41": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "42": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "43": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "44": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "45": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "46": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "47": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "48": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "49": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "50": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "51": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "52": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "53": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "54": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "55": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "56": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "57": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "58": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "59": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "60": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "61": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "62": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "63": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "64": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "65": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "66": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "67": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "68": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "69": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "70": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "71": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "72": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "73": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "74": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "75": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "76": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "77": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "78": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "79": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "80": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "81": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "82": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "83": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "84": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "85": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "86": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "87": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "88": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "89": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "90": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "91": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "92": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "93": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "94": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "95": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "96": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "97": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "98": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "99": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "100": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "101": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "102": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "103": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "104": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "105": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "106": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": true + }, + "107": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": true + }, + "108": { + "content": "\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "109": { + "content": "\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "110": { + "content": "\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "111": { + "content": "\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "112": { + "content": "\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "113": { + "content": "\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "114": { + "content": "\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "115": { + "content": "\n\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "116": { + "content": "\n\n\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "117": { + "content": "\n\n\n\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "118": { + "content": "\n\n\n\n\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "119": { + "content": "\n\n\n\n\n\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "120": { + "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "121": { + "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "122": { + "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "123": { + "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "124": { + "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "125": { + "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "126": { + "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "127": { + "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "128": { + "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "129": { + "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "130": { + "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "131": { + "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "132": { + "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "133": { + "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "134": { + "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "135": { + "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "136": { + "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "137": { + "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "138": { + "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "139": { + "content": "▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "140": { + "content": "▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "141": { + "content": "▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "142": { + "content": "▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "143": { + "content": "▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "144": { + "content": "▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "145": { + "content": "▁▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "146": { + "content": "▁▁▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "147": { + "content": "▁▁▁▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "148": { + "content": "▁▁▁▁▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "149": { + "content": "▁▁▁▁▁▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "150": { + "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "151": { + "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "152": { + "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "153": { + "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "154": { + "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "155": { + "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "156": { + "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "157": { + "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "158": { + "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "159": { + "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "160": { + "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "161": { + "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "162": { + "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "163": { + "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "164": { + "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "165": { + "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "166": { + "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "167": { + "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "168": { + "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "169": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "170": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "172": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "173": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "174": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "175": { + "content": "
", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "171": { + "content": "
", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "176": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "177": { + "content": "
", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "178": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "179": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "180": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "181": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "182": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "183": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "184": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "185": { + "content": "

", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "186": { + "content": "

", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "187": { + "content": "

", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "188": { + "content": "

", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "189": { + "content": "

", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "190": { + "content": "
", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "191": { + "content": "
", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "192": { + "content": "
", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "193": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "194": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "195": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "196": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "197": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "198": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "199": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "200": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "201": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "202": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "203": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "204": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "205": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "206": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "207": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "208": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "209": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "210": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "211": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "212": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "213": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "214": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "215": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "216": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255968": { + "content": "[toxicity=0]", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255969": { + "content": "\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255970": { + "content": "\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255971": { + "content": "\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255972": { + "content": "\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255973": { + "content": "\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255974": { + "content": "\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255975": { + "content": "\t\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255976": { + "content": "\t\t\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255977": { + "content": "\t\t\t\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255978": { + "content": "\t\t\t\t\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255979": { + "content": "\t\t\t\t\t\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255980": { + "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255981": { + "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255982": { + "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255983": { + "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255984": { + "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255985": { + "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255986": { + "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255987": { + "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255988": { + "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255989": { + "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255990": { + "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255991": { + "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255992": { + "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255993": { + "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255994": { + "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255995": { + "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255996": { + "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255997": { + "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255998": { + "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + }, + "255999": { + "content": "", + "lstrip": false, + "normalized": false, + "rstrip": false, + "single_word": false, + "special": false + } + }, + "additional_special_tokens": [ + "", + "" + ], + "bos_token": "", + "clean_up_tokenization_spaces": false, + "cls_token": "", + "eos_token": "", + "extra_special_tokens": {}, + "mask_token": "", + "model_input_names": [ + "input_ids", + "attention_mask" + ], + "model_max_length": 32768, + "pad_token": "", + "padding_side": "right", + "sep_token": "", + "spaces_between_special_tokens": false, + "tokenizer_class": "PreTrainedTokenizerFast", + "unk_token": "" +} diff --git a/native/training_policy.py b/native/training_policy.py new file mode 100644 index 0000000000000000000000000000000000000000..871277967f5d75e827702764bbcd203328e65664 --- /dev/null +++ b/native/training_policy.py @@ -0,0 +1,77 @@ +"""Score-only unique optimizer ownership and frozen-state proof helpers; no forward.""" +import hashlib +try: + from .contract import all_shapes, counts, trainable_name, training_policy, parameter_group +except ImportError: + from contract import all_shapes, counts, trainable_name, training_policy, parameter_group + + +def policy_inventory(model): + policy=training_policy(model.training_arm) + if model.arm!='all22' or model.trainability_mode!=policy:raise ValueError('Explicit score-path policy required') + raw=list(model.named_parameters(remove_duplicate=False));params=dict(raw);shapes=all_shapes(model.arm,model.training_arm) + if len(raw)!=len(params) or len({id(p) for _,p in raw})!=len(raw) or set(params)!=set(shapes): + raise ValueError('Exact unique parameter ownership required') + for n,p in raw: + if tuple(p.shape)!=shapes[n] or p.requires_grad!=trainable_name(n,policy):raise ValueError('Shape/trainability mismatch: '+n) + active=sorted(n for n in params if trainable_name(n,policy));frozen=sorted(set(params)-set(active)) + groups={g:sorted(n for n in active if parameter_group(n,model.training_arm)==g) for g in ('score_encoder','score_head')} + actual={'unique_parameters':sum(p.numel() for _,p in raw),'unique_tensors':len(raw), + 'trainable_parameters':sum(params[n].numel() for n in active),'trainable_tensors':len(active)} + expected=counts(model.arm,model.training_arm) + if any(expected[k]!=v for k,v in actual.items()):raise ValueError('Score-path policy count mismatch') + return {'training_arm':model.training_arm,'policy':policy,'counts':expected, + 'trainable_names':active,'frozen_names':frozen,'groups':groups} + + +def named_trainables(model): + inv=policy_inventory(model);params=dict(model.named_parameters()) + return [(n,params[n]) for n in inv['trainable_names']] + + +def optimizer_groups(model,encoder_lr=5e-6,head_lr=2.5e-5): + if encoder_lr!=5e-6 or head_lr!=2.5e-5:raise ValueError('Fixed encoder/head LRs required') + inv=policy_inventory(model);params=dict(model.named_parameters());groups=[];seen=[] + for key,lr in (('score_encoder',encoder_lr),('score_head',head_lr)): + names=inv['groups'][key] + if not names:continue # S0 has exactly one nonempty optimizer group, never a fake encoder parameter. + groups.append({'name':key,'params':[params[n] for n in names],'lr':lr,'initial_lr':lr});seen.extend(names) + if len(seen)!=len(set(seen)) or set(seen)!=set(inv['trainable_names']):raise ValueError('Disjoint complete optimizer partition required') + return groups + + +def assert_modes(model): + training=bool(model.training) + frozen_modules=[model.encoder,model.type_embedding,model.choice_blocks,model.choice_final_norm, + model.heads['choice'],model.scorers['choice'],model.heads['noul'],model.scorers['noul']] + if any(m.training for module in frozen_modules for m in module.modules()):raise ValueError('Frozen module mode drift') + active=[model.heads['score'],model.scorers['score'],model.score_blocks] + if model.training_arm!='S0':active.append(model.score_final_norm) + elif model.score_final_norm is not None or len(model.score_blocks)!=0:raise ValueError('S0 must not contain Score suffix') + if any(m.training!=training for module in active for m in module.modules()):raise ValueError('Score module mode drift') + return True + + +def frozen_hashes(model): + inv=policy_inventory(model);params=dict(model.named_parameters()) + return {n:hashlib.sha256(params[n].detach().cpu().contiguous().numpy().tobytes()).hexdigest() for n in inv['frozen_names']} + + +def snapshot_frozen(model,hash_values=False): + inv=policy_inventory(model);params=dict(model.named_parameters());assert_modes(model) + out={'training_arm':model.training_arm,'policy':inv['policy'],'names':inv['frozen_names'], + 'versions':{n:int(params[n]._version) for n in inv['frozen_names']}} + if hash_values:out['sha256']=frozen_hashes(model) + return out + + +def assert_frozen(model,snapshot,check_versions=True,check_hashes=False): + inv=policy_inventory(model);params=dict(model.named_parameters()) + if snapshot['training_arm']!=model.training_arm or snapshot['policy']!=inv['policy'] or snapshot['names']!=inv['frozen_names']: + raise ValueError('Frozen identity mismatch') + assert_modes(model) + for n in inv['frozen_names']: + if params[n].grad is not None:raise ValueError('Frozen gradient present: '+n) + if check_versions and int(params[n]._version)!=snapshot['versions'][n]:raise ValueError('Frozen version changed: '+n) + if check_hashes and ('sha256' not in snapshot or frozen_hashes(model)!=snapshot['sha256']):raise ValueError('Frozen content changed') + return True diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000000000000000000000000000000000000..83172881e9de8d6ab5f26cede29614994d9e4f95 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,7 @@ +# Install a compatible AMD ROCm PyTorch environment separately. +# Validation: Python 3.12.13, torch 2.12.0+git6bbd260 (ROCm development build). +# This file does not claim a generic PyPI torch wheel matches that environment. +transformers==4.57.6 +tokenizers==0.22.2 +safetensors==0.8.0 +numpy==2.5.3 diff --git a/source-metadata/ENCODER_README.md b/source-metadata/ENCODER_README.md new file mode 100644 index 0000000000000000000000000000000000000000..2564790aafccb6e0d3bb50b0a6465cad9fdea04a --- /dev/null +++ b/source-metadata/ENCODER_README.md @@ -0,0 +1,62 @@ +--- +language: +- en +- zh +- de +- fr +- ja +- ar +license: mit +library_name: transformers +pipeline_tag: fill-mask +base_model: jhu-clsp/mmBERT-base +base_model_relation: finetune +tags: +- vela +- modernbert +- multilingual +- long-context +--- + +
+ vLLM Semantic Router +

+ Docs | + Blog | + Slack | + GitHub +

+
+ +# Vela Base + +Vela Base is a multilingual encoder foundation for specialized routing models. + +**307M parameters · Input capacity: 32,768 tokens, including special tokens.** + +## Evaluation + +Masked-token negative log-likelihood on the same multilingual development set, compared with [the original mmBERT32K Base](https://huggingface.co/llm-semantic-router/mmbert-32k-yarn). Lower is better. + +| Context length | Original mmBERT | Vela | +|---|---:|---:| +| 512 tokens · 156 windows | 1.441 | **1.387** | +| 8K tokens · 54 windows | 1.226 | **1.212** | +| 16K tokens · 54 windows | 1.206 | **1.168** | +| 32K tokens · 54 windows | 1.075 | **1.018** | + +The 318 windows cover English, Chinese, German, French, Japanese and Arabic. Both models use FP32, identical fixed 15% masked-token targets, and complete inputs without truncation. This development set informed Vela checkpoint selection; it is not an independent test set. + +## Quick start + +With PyTorch and Transformers 4.57.6 or 5.17.0: + +```python +from transformers import pipeline + +model_id = "llm-semantic-router/Vela-1.0-Encoder-307M" +model = pipeline("fill-mask", model=model_id, device=-1) +print(model(f"The capital of France is {model.tokenizer.mask_token}.")) +``` + +[Explore the Vela model collection](https://huggingface.co/collections/llm-semantic-router/vela-10-router-models-6aa555ba70cc6997d6d67798) diff --git a/source-metadata/ENCODER_SOURCE.json b/source-metadata/ENCODER_SOURCE.json new file mode 100644 index 0000000000000000000000000000000000000000..326372cf7c8e15f37d9c91e1eb730422a0c7db93 --- /dev/null +++ b/source-metadata/ENCODER_SOURCE.json @@ -0,0 +1,8 @@ +{ + "repo": "llm-semantic-router/Vela-1.0-Encoder-307M", + "revision": "fe9ccc074b781bc0e2e13c2c8d26f2640410636a", + "url": "https://huggingface.co/llm-semantic-router/Vela-1.0-Encoder-307M/resolve/fe9ccc074b781bc0e2e13c2c8d26f2640410636a/README.md", + "file": "release/nano-package-preview/source-metadata/ENCODER_README.md", + "sha256": "817df5015d443e6a9e6bbc870d8bd4f0f5c0d48e29b8f50f665319609838f7ef", + "bytes": 2046 +} diff --git a/source-metadata/MMBERT_LICENSE_SOURCE_README.md b/source-metadata/MMBERT_LICENSE_SOURCE_README.md new file mode 100644 index 0000000000000000000000000000000000000000..29d50f7aa2c43c82c408fdc1877d3e9fa325618f --- /dev/null +++ b/source-metadata/MMBERT_LICENSE_SOURCE_README.md @@ -0,0 +1,2295 @@ +--- +license: mit +datasets: +- jhu-clsp/mmbert-decay +- jhu-clsp/mmbert-midtraining +- jhu-clsp/mmbert-pretrain-p1-fineweb2-langs +- jhu-clsp/mmbert-pretrain-p2-fineweb2-remaining +- jhu-clsp/mmbert-pretrain-p3-others +pipeline_tag: fill-mask +library_name: transformers +language: + - aai + - aak + - aau + - aaz + - aba + - abi + - abk + - abn + - abq + - abs + - abt + - abx + - aby + - abz + - aca + - acd + - ace + - acf + - ach + - acm + - acn + - acr + - acu + - ada + - ade + - adh + - adi + - adj + - adl + - ady + - adz + - aeb + - aer + - aeu + - aey + - afr + - agd + - agg + - agm + - agn + - agr + - agt + - agu + - agw + - agx + - aha + - ahk + - aia + - aii + - aim + - ain + - ajg + - aji + - ajz + - akb + - ake + - akh + - akp + - alj + - aln + - alp + - alq + - als + - alt + - aly + - alz + - ame + - amf + - amh + - ami + - amk + - amm + - amn + - amp + - amr + - amu + - amx + - ang + - anm + - ann + - anp + - anv + - any + - aoi + - aoj + - aom + - aoz + - apb + - apc + - ape + - apn + - apr + - apt + - apu + - apw + - apy + - apz + - arb + - are + - arg + - arl + - arn + - arp + - arq + - ars + - ary + - arz + - asg + - asm + - aso + - ast + - ata + - atb + - atd + - atg + - ati + - atj + - atq + - att + - auc + - aui + - auy + - ava + - avk + - avn + - avt + - avu + - awa + - awb + - awx + - ayo + - ayp + - ayr + - azb + - azg + - azj + - azz + - bak + - bam + - ban + - bao + - bar + - bas + - bav + - bba + - bbb + - bbc + - bbj + - bbk + - bbo + - bbr + - bch + - bci + - bcl + - bco + - bcw + - bdd + - bdh + - bdq + - bea + - bef + - bel + - bem + - ben + - beq + - bew + - bex + - bfd + - bfo + - bgr + - bgs + - bgt + - bgz + - bhg + - bhl + - bho + - bhp + - bhw + - bhz + - bib + - big + - bim + - bin + - bis + - biu + - biv + - bjn + - bjp + - bjr + - bjv + - bkd + - bkl + - bkq + - bku + - bkv + - bla + - blh + - blk + - blw + - blz + - bmh + - bmk + - bmq + - bmr + - bmu + - bmv + - bno + - bnp + - boa + - bod + - boj + - bom + - bon + - bos + - bov + - box + - bpr + - bps + - bpy + - bqc + - bqj + - bqp + - bre + - brh + - bru + - brx + - bsc + - bsn + - bsp + - bsq + - bss + - btd + - bth + - bts + - btt + - btx + - bud + - bug + - buk + - bul + - bum + - bus + - bvc + - bvd + - bvr + - bvz + - bwd + - bwi + - bwq + - bwu + - bxh + - bxr + - byr + - byv + - byx + - bzd + - bzh + - bzi + - bzj + - caa + - cab + - cac + - caf + - cag + - cak + - cao + - cap + - caq + - car + - cas + - cat + - cav + - cax + - cbc + - cbi + - cbk + - cbr + - cbs + - cbt + - cbu + - cbv + - cce + - cco + - ccp + - ceb + - ceg + - cek + - ces + - cfm + - cgc + - cgg + - cha + - chd + - che + - chf + - chj + - chk + - cho + - chq + - chr + - chu + - chv + - chw + - chz + - cjk + - cjo + - cjp + - cjs + - cjv + - ckb + - cko + - ckt + - cle + - clu + - cly + - cme + - cmn + - cmo + - cmr + - cnh + - cni + - cnk + - cnl + - cnt + - cnw + - coe + - cof + - cok + - con + - cop + - cor + - cos + - cot + - cou + - cpa + - cpb + - cpc + - cpu + - cpy + - crh + - crj + - crk + - crl + - crm + - crn + - crs + - crt + - crx + - csb + - csk + - cso + - csw + - csy + - cta + - ctd + - cto + - ctp + - ctu + - cub + - cuc + - cui + - cuk + - cul + - cut + - cux + - cwe + - cwt + - cya + - cym + - czt + - daa + - dad + - daf + - dag + - dah + - dak + - dan + - dar + - ddg + - ddn + - ded + - des + - deu + - dga + - dgc + - dgi + - dgr + - dgz + - dhg + - dhm + - dhv + - did + - dig + - dik + - diq + - dis + - diu + - div + - dje + - djk + - djr + - dks + - dln + - dng + - dnj + - dnw + - dob + - doi + - dop + - dos + - dow + - drg + - dru + - dsb + - dtb + - dtp + - dts + - dty + - dua + - due + - dug + - duo + - dur + - dwr + - dww + - dyi + - dyo + - dyu + - dzo + - ebk + - efi + - eka + - ekk + - eko + - ell + - emi + - eml + - emp + - enb + - enl + - enm + - enq + - enx + - epo + - eri + - ese + - esi + - esk + - ess + - esu + - eto + - etr + - etu + - eus + - eve + - ewe + - ewo + - ext + - eza + - faa + - fad + - fai + - fal + - fan + - fao + - far + - fas + - fat + - ffm + - fij + - fil + - fin + - fit + - fkv + - fmu + - fon + - for + - fra + - frd + - fro + - frp + - frr + - fry + - fub + - fud + - fue + - fuf + - fuh + - fuq + - fur + - fuv + - gaa + - gag + - gah + - gai + - gam + - gaw + - gaz + - gbi + - gbo + - gbr + - gcf + - gcr + - gde + - gdg + - gdn + - gdr + - geb + - gej + - gfk + - ghs + - gid + - gil + - giz + - gjn + - gkn + - gla + - gle + - glg + - glk + - glv + - gmh + - gmv + - gna + - gnb + - gnd + - gng + - gnn + - gnw + - goa + - gof + - gog + - goh + - gom + - gor + - gos + - got + - gqr + - grc + - grt + - gso + - gsw + - gub + - guc + - gud + - gug + - guh + - gui + - guj + - guk + - gul + - gum + - gun + - guo + - guq + - gur + - guu + - guw + - gux + - guz + - gvc + - gvf + - gvl + - gvn + - gwi + - gwr + - gya + - gym + - gyr + - hac + - hae + - hag + - hak + - hat + - hav + - haw + - hay + - hbo + - hch + - heb + - heg + - heh + - her + - hif + - hig + - hil + - hin + - hix + - hla + - hmo + - hmr + - hne + - hnj + - hnn + - hns + - hop + - hot + - hra + - hrv + - hrx + - hsb + - hto + - hub + - hui + - hun + - hus + - huu + - huv + - hvn + - hwc + - hye + - hyw + - ian + - iba + - ibg + - ibo + - icr + - ido + - idu + - ifa + - ifb + - ife + - ifk + - ifu + - ify + - ige + - ign + - ike + - ikk + - ikt + - ikw + - ilb + - ile + - ilo + - imo + - ina + - inb + - ind + - inh + - ino + - iou + - ipi + - iqw + - iri + - irk + - iry + - isd + - ish + - isl + - iso + - ita + - itv + - ium + - ivb + - ivv + - iws + - ixl + - izr + - izz + - jaa + - jac + - jae + - jam + - jav + - jbo + - jbu + - jic + - jiv + - jmc + - jpn + - jra + - jun + - jvn + - kaa + - kab + - kac + - kak + - kal + - kam + - kan + - kao + - kaq + - kas + - kat + - kaz + - kbc + - kbd + - kbh + - kbm + - kbo + - kbp + - kbq + - kbr + - kby + - kca + - kcg + - kck + - kdc + - kde + - kdh + - kdi + - kdj + - kdl + - kdr + - kea + - kei + - kek + - ken + - keo + - ker + - kew + - kez + - kff + - kgf + - kgk + - kgp + - kgr + - kha + - khk + - khm + - khs + - khz + - kia + - kij + - kik + - kin + - kir + - kiu + - kix + - kjb + - kje + - kjh + - kjs + - kkc + - kki + - kkj + - kkl + - kle + - klt + - klv + - kmb + - kmg + - kmh + - kmk + - kmm + - kmo + - kmr + - kms + - kmu + - kmy + - knc + - kne + - knf + - kng + - knj + - knk + - kno + - knv + - knx + - kny + - kog + - koi + - koo + - kor + - kos + - kpe + - kpf + - kpg + - kpj + - kpq + - kpr + - kpv + - kpw + - kpx + - kpz + - kqc + - kqe + - kqf + - kql + - kqn + - kqo + - kqp + - kqs + - kqw + - kqy + - krc + - kri + - krj + - krl + - kru + - krx + - ksb + - ksc + - ksd + - ksf + - ksh + - ksj + - ksp + - ksr + - kss + - ksw + - ktb + - ktj + - ktm + - kto + - ktu + - ktz + - kua + - kub + - kud + - kue + - kuj + - kum + - kup + - kus + - kvg + - kvj + - kvn + - kwd + - kwf + - kwi + - kwj + - kwn + - kwy + - kxc + - kxm + - kxw + - kyc + - kyf + - kyg + - kyq + - kyu + - kyz + - kze + - kzf + - kzj + - lac + - lad + - lai + - laj + - lam + - lao + - lap + - lat + - lbb + - lbe + - lbj + - lbk + - lcm + - lcp + - ldi + - ldn + - lee + - lef + - leh + - lem + - leu + - lew + - lex + - lez + - lfn + - lgg + - lgl + - lgm + - lhi + - lhu + - lia + - lid + - lif + - lij + - lim + - lin + - lip + - lis + - lit + - liv + - ljp + - lki + - llb + - lld + - llg + - lln + - lmk + - lmo + - lmp + - lnd + - lob + - loe + - log + - lok + - lol + - lom + - loq + - loz + - lrc + - lsi + - lsm + - ltg + - ltz + - lua + - lub + - luc + - lud + - lue + - lug + - lun + - luo + - lus + - lvs + - lwg + - lwo + - lww + - lzh + - maa + - mad + - maf + - mag + - mah + - mai + - maj + - mak + - mal + - mam + - maq + - mar + - mas + - mau + - mav + - maw + - maz + - mbb + - mbc + - mbd + - mbf + - mbh + - mbi + - mbj + - mbl + - mbs + - mbt + - mca + - mcb + - mcd + - mcf + - mck + - mcn + - mco + - mcp + - mcq + - mcu + - mda + - mdf + - mdy + - med + - mee + - mej + - mek + - men + - meq + - mer + - met + - meu + - mev + - mfe + - mfg + - mfh + - mfi + - mfk + - mfq + - mfy + - mfz + - mgc + - mgh + - mgo + - mgr + - mhi + - mhl + - mhr + - mhw + - mhx + - mhy + - mib + - mic + - mie + - mif + - mig + - mih + - mil + - mim + - min + - mio + - mip + - miq + - mir + - mit + - miy + - miz + - mjc + - mjw + - mkd + - mkl + - mkn + - mks + - mkz + - mlh + - mlp + - mlt + - mlu + - mmn + - mmo + - mmx + - mna + - mnb + - mnf + - mni + - mnk + - mns + - mnw + - mnx + - mny + - moa + - moc + - mog + - moh + - mop + - mor + - mos + - mox + - mpg + - mph + - mpm + - mpp + - mps + - mpt + - mpx + - mqb + - mqj + - mqy + - mrg + - mri + - mrj + - mrq + - mrv + - mrw + - msb + - msc + - mse + - msk + - msy + - mta + - mtg + - mti + - mto + - mtp + - mua + - mug + - muh + - mui + - mup + - mur + - mus + - mux + - muy + - mva + - mvn + - mvp + - mwc + - mwf + - mwl + - mwm + - mwn + - mwp + - mwq + - mwv + - mww + - mxb + - mxp + - mxq + - mxt + - mxv + - mya + - myb + - myk + - myu + - myv + - myw + - myx + - myy + - mza + - mzh + - mzk + - mzl + - mzm + - mzn + - mzw + - mzz + - nab + - naf + - nah + - nak + - nap + - naq + - nas + - nav + - naw + - nba + - nbc + - nbe + - nbl + - nbq + - nbu + - nca + - nch + - ncj + - ncl + - ncq + - nct + - ncu + - ncx + - ndc + - nde + - ndh + - ndi + - ndj + - ndo + - nds + - ndz + - neb + - new + - nfa + - nfr + - ngb + - ngc + - ngl + - ngp + - ngu + - nhd + - nhe + - nhg + - nhi + - nhk + - nho + - nhr + - nhu + - nhw + - nhx + - nhy + - nia + - nif + - nii + - nij + - nim + - nin + - nio + - niu + - niy + - njb + - njm + - njn + - njo + - njz + - nkf + - nko + - nld + - nlg + - nma + - nmf + - nmh + - nmo + - nmw + - nmz + - nnb + - nng + - nnh + - nnl + - nno + - nnp + - nnq + - nnw + - noa + - nob + - nod + - nog + - non + - nop + - not + - nou + - nov + - nph + - npi + - npl + - npo + - npy + - nqo + - nre + - nrf + - nri + - nrm + - nsa + - nse + - nsm + - nsn + - nso + - nss + - nst + - nsu + - ntp + - ntr + - ntu + - nuj + - nus + - nuy + - nvm + - nwb + - nwi + - nwx + - nxd + - nya + - nyf + - nyk + - nyn + - nyo + - nyu + - nyy + - nza + - nzi + - nzm + - obo + - oci + - ogo + - ojb + - oke + - oku + - okv + - old + - olo + - omb + - omw + - ong + - ons + - ood + - opm + - orv + - ory + - oss + - ota + - otd + - ote + - otm + - otn + - oto + - otq + - ots + - otw + - oym + - ozm + - pab + - pad + - pag + - pah + - pam + - pan + - pao + - pap + - pau + - pbb + - pbc + - pbi + - pbt + - pcd + - pck + - pcm + - pdc + - pdt + - pem + - pfe + - pfl + - phm + - pib + - pio + - pir + - pis + - pjt + - pkb + - plg + - pls + - plt + - plu + - plw + - pma + - pmf + - pmq + - pms + - pmx + - pnb + - pne + - pnt + - pny + - poe + - poh + - poi + - pol + - pon + - por + - pos + - pot + - pov + - poy + - ppk + - ppo + - pps + - prf + - prg + - pri + - prq + - pse + - pss + - ptp + - ptu + - pui + - pwg + - pwn + - pww + - pxm + - qub + - quc + - quf + - qug + - quh + - qul + - qup + - qus + - quw + - quy + - quz + - qva + - qvc + - qve + - qvh + - qvi + - qvm + - qvn + - qvo + - qvs + - qvw + - qvz + - qwh + - qxh + - qxl + - qxn + - qxo + - qxr + - rad + - rai + - rap + - rar + - rav + - raw + - rcf + - rej + - rel + - rgu + - rhg + - ria + - rim + - rjs + - rkb + - rmc + - rme + - rml + - rmn + - rmo + - rmq + - rmy + - rnd + - rng + - rnl + - roh + - ron + - roo + - rop + - row + - rro + - rtm + - rub + - rue + - ruf + - rug + - run + - rup + - rus + - rwo + - sab + - sag + - sah + - san + - sas + - sat + - sba + - sbd + - sbe + - sbl + - sbs + - sby + - sck + - scn + - sco + - sda + - sdc + - sdh + - sdo + - sdq + - seh + - ses + - sey + - sfw + - sgb + - sgc + - sgh + - sgs + - sgw + - sgz + - shi + - shk + - shn + - shp + - shu + - sid + - sig + - sil + - sim + - sin + - sja + - sjo + - sju + - skg + - skr + - sld + - slk + - sll + - slv + - sma + - sme + - smj + - smk + - sml + - smn + - smo + - sms + - smt + - sna + - snc + - snd + - snf + - snn + - snp + - snw + - sny + - soe + - som + - sop + - soq + - sot + - soy + - spa + - spl + - spm + - spp + - sps + - spy + - srd + - sri + - srm + - srn + - srp + - srq + - srr + - ssd + - ssg + - ssw + - ssx + - stn + - stp + - stq + - sua + - suc + - sue + - suk + - sun + - sur + - sus + - suz + - swb + - swc + - swe + - swg + - swh + - swk + - swp + - sxb + - sxn + - syb + - syc + - syl + - szl + - szy + - tab + - tac + - tah + - taj + - tam + - tap + - taq + - tar + - tat + - tav + - taw + - tay + - tbc + - tbg + - tbk + - tbl + - tbo + - tbw + - tby + - tbz + - tca + - tcc + - tcf + - tcs + - tcy + - tcz + - ted + - tee + - tel + - tem + - teo + - ter + - tet + - tew + - tfr + - tgk + - tgo + - tgp + - tha + - thk + - thl + - tif + - tig + - tih + - tik + - tim + - tir + - tiv + - tiy + - tke + - tkl + - tkr + - tku + - tlb + - tlf + - tlh + - tlj + - tll + - tly + - tmc + - tmd + - tna + - tnc + - tnk + - tnn + - tnp + - tnr + - tob + - toc + - tod + - tog + - toh + - toi + - toj + - tok + - ton + - too + - top + - tos + - tpa + - tpi + - tpm + - tpp + - tpt + - tpw + - tpz + - tqo + - trc + - trn + - tro + - trp + - trq + - trs + - trv + - tsc + - tsg + - tsn + - tso + - tsw + - tsz + - ttc + - tte + - ttj + - ttq + - tuc + - tue + - tuf + - tui + - tuk + - tul + - tum + - tuo + - tur + - tuv + - tvk + - tvl + - twi + - twu + - twx + - txq + - txu + - tyv + - tzh + - tzj + - tzl + - tzm + - tzo + - ubr + - ubu + - udm + - udu + - uig + - ukr + - umb + - upv + - ura + - urb + - urd + - urh + - uri + - urk + - urt + - urw + - ury + - usa + - usp + - uth + - uvh + - uvl + - uzn + - uzs + - vag + - vap + - var + - vec + - ven + - vep + - vid + - vie + - viv + - vls + - vmk + - vmw + - vmy + - vol + - vot + - vro + - vun + - vut + - waj + - wal + - wap + - war + - wat + - way + - wba + - wbm + - wbp + - wed + - wer + - wes + - wew + - whg + - whk + - wib + - wim + - wiu + - wln + - wls + - wlv + - wlx + - wmt + - wmw + - wnc + - wnu + - wob + - wol + - wos + - wrk + - wrs + - wsg + - wsk + - wuu + - wuv + - wwa + - xal + - xav + - xbi + - xbr + - xed + - xho + - xla + - xmf + - xmm + - xmv + - xnn + - xog + - xon + - xrb + - xsb + - xsi + - xsm + - xsr + - xsu + - xtd + - xtm + - xtn + - xuo + - yaa + - yad + - yal + - yam + - yan + - yao + - yap + - yaq + - yat + - yaz + - ybb + - yby + - ycn + - ydd + - yim + - yka + - yle + - yli + - yml + - yom + - yon + - yor + - yrb + - yre + - yrk + - yrl + - yss + - yua + - yue + - yuj + - yup + - yut + - yuw + - yuz + - yva + - zaa + - zab + - zac + - zad + - zae + - zai + - zam + - zao + - zar + - zas + - zat + - zav + - zaw + - zca + - zdj + - zea + - zgh + - zia + - ziw + - zne + - zom + - zos + - zpa + - zpc + - zpg + - zpi + - zpj + - zpl + - zpm + - zpo + - zpq + - zpt + - zpu + - zpv + - zpz + - zsm + - zsr + - ztq + - zty + - zul + - zyb + - zyp +--- + +# mmBERT: A Modern Multilingual Encoder + +[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) +[![Paper](https://img.shields.io/badge/Paper-Arxiv-red)](https://arxiv.org/abs/2509.06888) +[![Model](https://img.shields.io/badge/🤗%20Hugging%20Face-Model-blue)](https://huggingface.co/jhu-clsp/mmBERT-base) +[![Collection](https://img.shields.io/badge/🤗%20Model%20Collection-blue)](https://huggingface.co/collections/jhu-clsp/mmbert-a-modern-multilingual-encoder-68b725831d7c6e3acc435ed4) +[![GitHub](https://img.shields.io/badge/GitHub-Code-black)](https://github.com/jhu-clsp/mmBERT) + +> TL;DR: A state-of-the-art multilingual encoder trained on 3T+ tokens across 1800+ languages, introducing novel techniques for learning low-resource languages during the decay phase. + +mmBERT is a modern multilingual encoder that significantly outperforms previous generation models like XLM-R on classification, embedding, and retrieval tasks. Built on the ModernBERT architecture with novel multilingual training innovations, mmBERT demonstrates that low-resource languages can be effectively learned during the decay phase of training. It is also significantly faster than any previous multilingual encoder. + +## Table of Contents +- [Highlights](#highlights) +- [Quick Start](#quick-start) +- [Model Description](#model-description) +- [Novel Training Innovations](#novel-training-innovations) +- [Model Family](#model-family) +- [Training Data](#training-data) +- [Usage Examples](#usage-examples) +- [Fine-tuning Examples](#fine-tuning-examples) +- [Model Architecture](#model-architecture) +- [Citation](#citation) + + +## Quick Start + +### Installation +```bash +pip install torch>=1.9.0 +pip install transformers>=4.21.0 +``` + +### Usage + +```python +from transformers import AutoTokenizer, AutoModel + +tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmBERT-base") +model = AutoModel.from_pretrained("jhu-clsp/mmBERT-base") + +inputs = tokenizer("Hello world", return_tensors="pt") +outputs = model(**inputs) +``` + +## Model Description + +mmBERT represents the first significant advancement over XLM-R for massively multilingual encoder models. Key features include: + +1. **Massive Language Coverage** - Trained on over 1800 languages with progressive inclusion strategy +2. **Modern Architecture** - Built on ModernBERT foundation with Flash Attention 2 and unpadding techniques +3. **Novel Training Recipe** - Introduces inverse mask scheduling and temperature sampling +4. **Open Training Data** - Complete 3T+ token dataset publicly available +5. **Decay Phase Innovation** - Demonstrates effective learning of low-resource languages in final training phase + +The model uses bidirectional attention with masked language modeling objectives, optimized specifically for multilingual understanding and cross-lingual transfer. + +## Novel Training Innovations + +**Progressive Language Addition**: Start with 60 high-resource languages, expand to 110 mid-resource languages, then include all 1833 languages in decay phase. + +**Inverse Mask Schedule**: Reduce mask ratio from 30% → 15% → 5% across training phases for progressively refined learning. + +**Inverse Temperature Sampling**: Adjust multilingual sampling from high-resource bias (τ=0.7) to uniform sampling (τ=0.3). + +**Model Merging**: Combine English-focused, high-resource, and all-language decay variants using TIES merging. + +## Model Family + +| Model | Total Params | Non-embed Params | Languages | Download | +|:------|:-------------|:------------------|:----------|:---------| +| [mmBERT-small](https://huggingface.co/jhu-clsp/mmBERT-small) | 140M | 42M | 1800+ | [![Download](https://img.shields.io/badge/🤗-Download-blue)](https://huggingface.co/jhu-clsp/mmBERT-small) | +| [mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base) | 307M | 110M | 1800+ | [![Download](https://img.shields.io/badge/🤗-Download-blue)](https://huggingface.co/jhu-clsp/mmBERT-base) | + +## Training Data + +mmBERT training data is publicly available across different phases: + +| Phase | Dataset | Tokens | Description | +|:------|:--------|:-------|:------------| +| Pre-training P1 | [mmbert-pretrain-p1](https://huggingface.co/datasets/jhu-clsp/mmbert-pretrain-p1-fineweb2-langs) | 2.3T | 60 languages, foundational training | +| Pre-training P2 | [mmbert-pretrain-p2](https://huggingface.co/datasets/jhu-clsp/mmBERT-pretrain-p2-fineweb2-remaining) | - | Extension data for pre-training phase | +| Pre-training P3 | [mmbert-pretrain-p3](https://huggingface.co/datasets/jhu-clsp/mmBERT-pretrain-p3-others) | - | Final pre-training data | +| Mid-training | [mmbert-midtraining](https://huggingface.co/datasets/jhu-clsp/mmbert-midtraining-data) | 600B | 110 languages, context extension to 8K | +| Decay Phase | [mmbert-decay](https://huggingface.co/datasets/jhu-clsp/mmbert-decay-data) | 100B | 1833 languages, premium quality | + +**Data Sources**: Filtered DCLM (English), FineWeb2 (multilingual), FineWeb2-HQ (20 high-resource languages), Wikipedia (MegaWika), code repositories (StarCoder, ProLong), academic papers (ArXiv, PeS2o), and community discussions (StackExchange). + +## Model Architecture + +| Parameter | mmBERT-small | mmBERT-base | +|:----------|:-------------|:------------| +| Layers | 22 | 22 | +| Hidden Size | 384 | 768 | +| Intermediate Size | 1152 | 1152 | +| Attention Heads | 6 | 12 | +| Total Parameters | 140M | 307M | +| Non-embedding Parameters | 42M | 110M | +| Max Sequence Length | 8192 | 8192 | +| Vocabulary Size | 256,000 | 256,000 | +| Tokenizer | Gemma 2 | Gemma 2 | + +## Usage Examples + +### Masked Language Modeling + +```python +from transformers import AutoTokenizer, AutoModelForMaskedLM +import torch + +tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmBERT-base") +model = AutoModelForMaskedLM.from_pretrained("jhu-clsp/mmBERT-base") + +def predict_masked_token(text): + inputs = tokenizer(text, return_tensors="pt") + with torch.no_grad(): + outputs = model(**inputs) + + mask_indices = torch.where(inputs["input_ids"] == tokenizer.mask_token_id) + predictions = outputs.logits[mask_indices] + top_tokens = torch.topk(predictions, 5, dim=-1) + + return [tokenizer.decode(token) for token in top_tokens.indices[0]] + +# Works across languages +texts = [ + "The capital of France is .", + "La capital de España es .", + "Die Hauptstadt von Deutschland ist ." +] + +for text in texts: + predictions = predict_masked_token(text) + print(f"Text: {text}") + print(f"Predictions: {predictions}") +``` + +### Cross-lingual Embeddings + +```python +from transformers import AutoTokenizer, AutoModel +import torch +from sklearn.metrics.pairwise import cosine_similarity + +tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmBERT-base") +model = AutoModel.from_pretrained("jhu-clsp/mmBERT-base") + +def get_embeddings(texts): + inputs = tokenizer(texts, padding=True, truncation=True, return_tensors="pt") + + with torch.no_grad(): + outputs = model(**inputs) + embeddings = outputs.last_hidden_state.mean(dim=1) + + return embeddings.numpy() + +multilingual_texts = [ + "Artificial intelligence is transforming technology", + "La inteligencia artificial está transformando la tecnología", + "L'intelligence artificielle transforme la technologie", + "人工智能正在改变技术" +] + +embeddings = get_embeddings(multilingual_texts) +similarities = cosine_similarity(embeddings) +print("Cross-lingual similarity matrix:") +print(similarities) +``` + +## Fine-tuning Examples + +### Dense Retrieval with Sentence Transformers + +
+Click to expand dense retrieval fine-tuning example + +```python +import argparse +from datasets import load_dataset +from sentence_transformers import ( + SentenceTransformer, + SentenceTransformerTrainer, + SentenceTransformerTrainingArguments, +) +from sentence_transformers.evaluation import TripletEvaluator +from sentence_transformers.losses import CachedMultipleNegativesRankingLoss +from sentence_transformers.training_args import BatchSamplers + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--lr", type=float, default=8e-5) + parser.add_argument("--model_name", type=str, default="jhu-clsp/mmBERT-base") + args = parser.parse_args() + + lr = args.lr + model_name = args.model_name + model_shortname = model_name.split("/")[-1] + + model = SentenceTransformer(model_name) + + dataset = load_dataset( + "sentence-transformers/msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1", + "triplet-hard", + split="train", + ) + dataset_dict = dataset.train_test_split(test_size=1_000, seed=12) + train_dataset = dataset_dict["train"].select(range(1_250_000)) + eval_dataset = dataset_dict["test"] + + loss = CachedMultipleNegativesRankingLoss(model, mini_batch_size=16) + run_name = f"{model_shortname}-DPR-{lr}" + + training_args = SentenceTransformerTrainingArguments( + output_dir=f"output/{model_shortname}/{run_name}", + num_train_epochs=1, + per_device_train_batch_size=512, + per_device_eval_batch_size=512, + warmup_ratio=0.05, + fp16=False, + bf16=True, + batch_sampler=BatchSamplers.NO_DUPLICATES, + learning_rate=lr, + save_strategy="steps", + save_steps=500, + save_total_limit=2, + logging_steps=500, + run_name=run_name, + ) + + dev_evaluator = TripletEvaluator( + anchors=eval_dataset["query"], + positives=eval_dataset["positive"], + negatives=eval_dataset["negative"], + name="msmarco-co-condenser-dev", + ) + dev_evaluator(model) + + trainer = SentenceTransformerTrainer( + model=model, + args=training_args, + train_dataset=train_dataset, + eval_dataset=eval_dataset, + loss=loss, + evaluator=dev_evaluator, + ) + trainer.train() + + model.save_pretrained(f"output/{model_shortname}/{run_name}/final") + model.push_to_hub(run_name, private=False) + +if __name__ == "__main__": + main() +``` + +
+ +### Cross-lingual Classification + +
+Click to expand multilingual classification fine-tuning example + +```python +from transformers import ( + AutoTokenizer, + AutoModelForSequenceClassification, + TrainingArguments, + Trainer +) +from datasets import load_dataset +import numpy as np +from sklearn.metrics import accuracy_score, f1_score + +def compute_metrics(eval_pred): + predictions, labels = eval_pred + predictions = np.argmax(predictions, axis=1) + return { + 'accuracy': accuracy_score(labels, predictions), + 'f1': f1_score(labels, predictions, average='weighted') + } + +def main(): + model_name = "jhu-clsp/mmBERT-base" + + tokenizer = AutoTokenizer.from_pretrained(model_name) + model = AutoModelForSequenceClassification.from_pretrained( + model_name, + num_labels=3 + ) + + dataset = load_dataset("xnli", "all_languages") + + def tokenize_function(examples): + texts = [f"{p} {tokenizer.sep_token} {h}" + for p, h in zip(examples["premise"], examples["hypothesis"])] + + return tokenizer( + texts, + truncation=True, + padding=True, + max_length=512 + ) + + train_dataset = dataset["train"].map(tokenize_function, batched=True) + eval_dataset = dataset["validation"].map(tokenize_function, batched=True) + + training_args = TrainingArguments( + output_dir="./mmbert-xnli", + learning_rate=3e-5, + per_device_train_batch_size=32, + per_device_eval_batch_size=32, + num_train_epochs=3, + weight_decay=0.01, + evaluation_strategy="epoch", + save_strategy="epoch", + load_best_model_at_end=True, + metric_for_best_model="f1", + greater_is_better=True, + ) + + trainer = Trainer( + model=model, + args=training_args, + train_dataset=train_dataset, + eval_dataset=eval_dataset, + compute_metrics=compute_metrics, + ) + + trainer.train() + +if __name__ == "__main__": + main() +``` + +
+ +### Multilingual Reranking + +
+Click to expand multilingual reranking fine-tuning example + +```python +import logging +from datasets import load_dataset +from sentence_transformers.cross_encoder import ( + CrossEncoder, + CrossEncoderModelCardData, + CrossEncoderTrainer, + CrossEncoderTrainingArguments, +) +from sentence_transformers.cross_encoder.evaluation import CrossEncoderNanoBEIREvaluator +from sentence_transformers.cross_encoder.losses import BinaryCrossEntropyLoss +from sentence_transformers.util import mine_hard_negatives +from sentence_transformers import SentenceTransformer +import torch + +def main(): + model_name = "jhu-clsp/mmBERT-base" + train_batch_size = 32 + num_epochs = 2 + num_hard_negatives = 7 + + model = CrossEncoder( + model_name, + model_card_data=CrossEncoderModelCardData( + language="multilingual", + license="mit", + ), + ) + + full_dataset = load_dataset("sentence-transformers/gooaq", split="train").select(range(50_000)) + dataset_dict = full_dataset.train_test_split(test_size=1_000, seed=42) + train_dataset = dataset_dict["train"] + eval_dataset = dataset_dict["test"] + + embedding_model = SentenceTransformer("sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2", device="cpu") + hard_train_dataset = mine_hard_negatives( + train_dataset, + embedding_model, + num_negatives=num_hard_negatives, + margin=0, + range_min=0, + range_max=100, + sampling_strategy="top", + batch_size=2048, + output_format="labeled-pair", + use_faiss=True, + ) + + loss = BinaryCrossEntropyLoss(model=model, pos_weight=torch.tensor(num_hard_negatives)) + + nano_beir_evaluator = CrossEncoderNanoBEIREvaluator( + dataset_names=["msmarco", "nfcorpus", "nq"], + batch_size=train_batch_size, + ) + + args = CrossEncoderTrainingArguments( + output_dir="./mmbert-reranker", + num_train_epochs=num_epochs, + per_device_train_batch_size=train_batch_size, + per_device_eval_batch_size=train_batch_size, + learning_rate=2e-5, + warmup_ratio=0.1, + fp16=False, + bf16=True, + dataloader_num_workers=4, + load_best_model_at_end=True, + metric_for_best_model="eval_msmarco_ndcg@10", + eval_strategy="steps", + eval_steps=1000, + save_strategy="steps", + save_steps=1000, + save_total_limit=2, + logging_steps=200, + seed=42, + ) + + trainer = CrossEncoderTrainer( + model=model, + args=args, + train_dataset=hard_train_dataset, + loss=loss, + evaluator=nano_beir_evaluator, + ) + trainer.train() + + model.save_pretrained("./mmbert-reranker/final") + +if __name__ == "__main__": + main() +``` + +
+ +## Training Data + +mmBERT was trained on a carefully curated 3T+ token multilingual dataset: + +| Phase | Dataset | Description | +|:------|:--------|:------------| +| [Pre-training P1](https://huggingface.co/datasets/jhu-clsp/mmbert-pretrain-p1-fineweb2-langs) | 2.3T tokens | 60 languages, diverse data mixture | +| [Pre-training P2](https://huggingface.co/datasets/jhu-clsp/mmbert-pretrain-p2-fineweb2-langs) | - | Extension data for pre-training | +| [Pre-training P3](https://huggingface.co/datasets/jhu-clsp/mmbert-pretrain-p3-fineweb2-langs) | - | Final pre-training data | +| [Mid-training](https://huggingface.co/datasets/jhu-clsp/mmbert-midtraining-data) | 600B tokens | 110 languages, context extension | +| [Decay Phase](https://huggingface.co/datasets/jhu-clsp/mmbert-decay-data) | 100B tokens | 1833 languages, premium quality | + +**Primary Sources:** +- **Filtered DCLM**: High-quality English content +- **FineWeb2**: Broad multilingual web coverage (1800+ languages) +- **FineWeb2-HQ**: Filtered subset of 20 high-resource languages +- **Code**: StarCoder and ProLong repositories +- **Academic**: ArXiv papers and PeS2o scientific content +- **Reference**: Wikipedia (MegaWika) and textbooks +- **Community**: StackExchange discussions + + +## Citation + +If you use mmBERT in your research, please cite our work: + +```bibtex +@misc{marone2025mmbertmodernmultilingualencoder, + title={mmBERT: A Modern Multilingual Encoder with Annealed Language Learning}, + author={Marc Marone and Orion Weller and William Fleshman and Eugene Yang and Dawn Lawrie and Benjamin Van Durme}, + year={2025}, + eprint={2509.06888}, + archivePrefix={arXiv}, + primaryClass={cs.CL}, + url={https://arxiv.org/abs/2509.06888}, +} +``` +""" \ No newline at end of file diff --git a/source-metadata/MMBERT_TOKENIZER_LINEAGE.yaml b/source-metadata/MMBERT_TOKENIZER_LINEAGE.yaml new file mode 100644 index 0000000000000000000000000000000000000000..3b45b3b6b1891ed41f953831df35df44150f92bf --- /dev/null +++ b/source-metadata/MMBERT_TOKENIZER_LINEAGE.yaml @@ -0,0 +1,124 @@ +seed: 21 +precision: amp_bf16 +autoresume: true + +save_folder: + +save_interval: 10000000000tok +save_num_checkpoints_to_keep: 10 +max_duration: 400000000000tok +eval_interval: 100000000000ba +log_to_console: true +console_log_interval: 25ba + +model: + name: flex_bert + pretrained_model_name: bert-base-uncased # unused + tokenizer_name: google/gemma-2-9b + disable_train_metrics: true + model_config: + num_hidden_layers: 22 + hidden_size: 768 + intermediate_size: 1152 + num_attention_heads: 12 + vocab_size: 256000 + padding: unpadded + mlp_layer: glu + bert_layer: prenorm + embed_norm: true + final_norm: true + hidden_act: gelu + init_method: full_megatron + mlp_in_bias: false + mlp_out_bias: false + attn_out_bias: false + attn_qkv_bias: false + compile_model: true + head_pred_act: gelu + loss_function: fa_cross_entropy + normalization: layernorm + rotary_emb_dim: null + sliding_window: 128 + attention_layer: rope + embedding_layer: sans_pos + rotary_emb_base: 10000 + mlp_dropout_prob: 0 + unpad_embeddings: true + masked_prediction: true + embed_dropout_prob: 0 + skip_first_prenorm: true + activation_function: gelu + attn_out_dropout_prob: 0.1 + rotary_emb_scale_base: null + rotary_emb_interleaved: false + allow_embedding_resizing: true + global_attn_every_n_layers: 3 + local_attn_rotary_emb_base: 10000 + attention_probs_dropout_prob: 0 + norm_kwargs: + eps: 1.0e-05 + bias: false + loss_kwargs: + reduction: mean + +optimizer: + name: decoupled_stableadamw + lr: 0.0008 + eps: 1.0e-08 + betas: [0.9, 0.95] + weight_decay: 8.0e-05 + log_grad_norm: true + filter_bias_norm_wd: true + +scheduler: + name: warmup_stable_decay + alpha_f: 0.02 + t_warmup: 3000000000tok + t_decay: 0tok + +train_loader: + name: text + dataset: + local: + split: train + remote: null + shuffle: true + streaming: false + max_seq_len: 1024 + shuffle_seed: 21 + tokenizer_name: google/gemma-2-9b + mlm_probability: 0.25 + drop_last: true + num_workers: 6 + bos_token_id: 2 + eos_token_id: 1 + pad_token_id: 0 + sep_token_id: 1 + mask_token_id: 4 + sequence_packing: true + batch_size_warmup_tokens: 60000000000tok + batch_size_warmup_min_size: 48 + +device_train_microbatch_size: 48 +global_train_batch_size: 4608 +device_eval_batch_size: 36 +global_eval_batch_size: 1024 + +# loggers: +# wandb: +# name: +# group: +# entity: +# project: modernBERTv2 +# tags: ["tags go here in a list"] + +callbacks: + lr_monitor: {} + scheduled_gc: {} + log_grad_norm: + batch_log_interval: 100 + speed_monitor: + window_size: 100 + runtime_estimator: {} + packing_efficiency: + log_interval: 100 diff --git a/source-metadata/MODEL_LICENSE_METADATA.json b/source-metadata/MODEL_LICENSE_METADATA.json new file mode 100644 index 0000000000000000000000000000000000000000..2ebbdf7c82825a7893ff0acb6079e246a0f89348 --- /dev/null +++ b/source-metadata/MODEL_LICENSE_METADATA.json @@ -0,0 +1,44 @@ +{ + "vela": { + "api_url": "https://huggingface.co/api/models/llm-semantic-router/Vela-1.0-Encoder-307M/revision/fe9ccc074b781bc0e2e13c2c8d26f2640410636a", + "revision": "fe9ccc074b781bc0e2e13c2c8d26f2640410636a", + "license": "mit", + "base_model": "jhu-clsp/mmBERT-base", + "files": [ + ".gitattributes", + "README.md", + "config.json", + "model.safetensors", + "special_tokens_map.json", + "tokenizer.json", + "tokenizer_config.json" + ], + "readme": { + "url": "https://huggingface.co/llm-semantic-router/Vela-1.0-Encoder-307M/resolve/fe9ccc074b781bc0e2e13c2c8d26f2640410636a/README.md", + "sha256": "817df5015d443e6a9e6bbc870d8bd4f0f5c0d48e29b8f50f665319609838f7ef", + "bytes": 2046, + "local_file": "source-metadata/VELA_LICENSE_SOURCE_README.md" + } + }, + "mmbert": { + "api_url": "https://huggingface.co/api/models/jhu-clsp/mmBERT-base", + "revision": "c5955035435e2bf121cde7f3c8863ef52ff35d82", + "license": "mit", + "base_model": null, + "files": [ + ".gitattributes", + "README.md", + "config.json", + "pytorch_model.bin", + "special_tokens_map.json", + "tokenizer.json", + "tokenizer_config.json" + ], + "readme": { + "url": "https://huggingface.co/jhu-clsp/mmBERT-base/resolve/c5955035435e2bf121cde7f3c8863ef52ff35d82/README.md", + "sha256": "35724415037ee3159aa733c16050620cc19a76219112a6ceeb3949572e0b37e9", + "bytes": 29650, + "local_file": "source-metadata/MMBERT_LICENSE_SOURCE_README.md" + } + } +} diff --git a/source-metadata/TRANSFORMERS_LICENSE_SOURCE.json b/source-metadata/TRANSFORMERS_LICENSE_SOURCE.json new file mode 100644 index 0000000000000000000000000000000000000000..e34db862e8af9d3a2a5aa21f7d2a9af3251aff27 --- /dev/null +++ b/source-metadata/TRANSFORMERS_LICENSE_SOURCE.json @@ -0,0 +1,24 @@ +[ + { + "url": "https://raw.githubusercontent.com/huggingface/transformers/753d61104116eefc8ffc977327b441ee0c8d599f/LICENSE", + "repository": "huggingface/transformers", + "revision": "753d61104116eefc8ffc977327b441ee0c8d599f", + "tag": "v4.57.6", + "upstream_file_sha256": "77fd4710def9ec3c0f6225800e0235f15a425abd4a8b03559127fcd782612049", + "saved_file": "LICENSES/Transformers-Apache-2.0.txt", + "saved_sha256": "77fd4710def9ec3c0f6225800e0235f15a425abd4a8b03559127fcd782612049", + "saved_bytes": 11418, + "full_license": true + }, + { + "url": "https://raw.githubusercontent.com/huggingface/transformers/753d61104116eefc8ffc977327b441ee0c8d599f/src/transformers/models/modernbert/modeling_modernbert.py", + "repository": "huggingface/transformers", + "revision": "753d61104116eefc8ffc977327b441ee0c8d599f", + "tag": "v4.57.6", + "upstream_file_sha256": "1b23db2f59d3f80bd9af89fb6df3ce15affd90b5ffc09925e732450e694868ff", + "saved_file": "source-metadata/TRANSFORMERS_MODERNBERT_SOURCE_HEADER.txt", + "saved_sha256": "4577f1803f9dbe8b43f5da709f8750c798def2c33f7eb055d62baf286ed472c8", + "saved_bytes": 1497, + "full_license": false + } +] diff --git a/source-metadata/TRANSFORMERS_MODERNBERT_SOURCE_HEADER.txt b/source-metadata/TRANSFORMERS_MODERNBERT_SOURCE_HEADER.txt new file mode 100644 index 0000000000000000000000000000000000000000..a73cbcb967e4f6ef582423e24dbfdb3dc0424f6b --- /dev/null +++ b/source-metadata/TRANSFORMERS_MODERNBERT_SOURCE_HEADER.txt @@ -0,0 +1,22 @@ +# 🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨 +# This file was automatically generated from src/transformers/models/modernbert/modular_modernbert.py. +# Do NOT edit this file manually as any edits will be overwritten by the generation of +# the file from the modular. If any change should be done, please apply the change to the +# modular_modernbert.py file directly. One of our CI enforces this. +# 🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨 +# Copyright 2024 Answer.AI, LightOn, and contributors, and the HuggingFace Inc. team. All rights reserved. +# +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import copy diff --git a/source-metadata/VELA_LICENSE_SOURCE_README.md b/source-metadata/VELA_LICENSE_SOURCE_README.md new file mode 100644 index 0000000000000000000000000000000000000000..2564790aafccb6e0d3bb50b0a6465cad9fdea04a --- /dev/null +++ b/source-metadata/VELA_LICENSE_SOURCE_README.md @@ -0,0 +1,62 @@ +--- +language: +- en +- zh +- de +- fr +- ja +- ar +license: mit +library_name: transformers +pipeline_tag: fill-mask +base_model: jhu-clsp/mmBERT-base +base_model_relation: finetune +tags: +- vela +- modernbert +- multilingual +- long-context +--- + +
+ vLLM Semantic Router +

+ Docs | + Blog | + Slack | + GitHub +

+
+ +# Vela Base + +Vela Base is a multilingual encoder foundation for specialized routing models. + +**307M parameters · Input capacity: 32,768 tokens, including special tokens.** + +## Evaluation + +Masked-token negative log-likelihood on the same multilingual development set, compared with [the original mmBERT32K Base](https://huggingface.co/llm-semantic-router/mmbert-32k-yarn). Lower is better. + +| Context length | Original mmBERT | Vela | +|---|---:|---:| +| 512 tokens · 156 windows | 1.441 | **1.387** | +| 8K tokens · 54 windows | 1.226 | **1.212** | +| 16K tokens · 54 windows | 1.206 | **1.168** | +| 32K tokens · 54 windows | 1.075 | **1.018** | + +The 318 windows cover English, Chinese, German, French, Japanese and Arabic. Both models use FP32, identical fixed 15% masked-token targets, and complete inputs without truncation. This development set informed Vela checkpoint selection; it is not an independent test set. + +## Quick start + +With PyTorch and Transformers 4.57.6 or 5.17.0: + +```python +from transformers import pipeline + +model_id = "llm-semantic-router/Vela-1.0-Encoder-307M" +model = pipeline("fill-mask", model=model_id, device=-1) +print(model(f"The capital of France is {model.tokenizer.mask_token}.")) +``` + +[Explore the Vela model collection](https://huggingface.co/collections/llm-semantic-router/vela-10-router-models-6aa555ba70cc6997d6d67798)