"""Configuration : constantes d'environnement, endpoints par défaut et vocabulaires métier partagés par tous les modules du POC.""" from __future__ import annotations import os import threading from pathlib import Path from dotenv import load_dotenv load_dotenv() _REPO_DIR = Path(__file__).parent def _runtime_store_dir() -> Path: """Where the app's runtime JSONL journals live by default. On Hugging Face Spaces the container filesystem is ephemeral (wiped on every rebuild/restart), but an attached Persistent Storage volume is mounted at /data and survives. Prefer it when it exists and is writable; otherwise fall back to the repo directory (local dev, or a Space without persistent storage). An explicit STORE_DIR env var overrides the choice. Per-file *_PATH env vars still win over this, since they're read as full paths below.""" override = os.getenv("STORE_DIR", "").strip() if override: return Path(override) data_dir = Path("/data") if data_dir.is_dir() and os.access(data_dir, os.W_OK): return data_dir return _REPO_DIR _STORE_DIR = _runtime_store_dir() DEFAULT_VLM_URL = os.getenv("VLM_API_URL", "https://vlm.smartbiblia.fr/v1/chat/completions") DEFAULT_VLM_MODEL = os.getenv("VLM_MODEL", "Qwen3-VL-8B-Instruct-GGUF") #DEFAULT_VLM_URL = os.getenv("VLM_API_URL", "https://mqt7w4m4abb63whh.eu-west-1.aws.endpoints.huggingface.cloud/v1/chat/completions") #DEFAULT_VLM_MODEL = os.getenv("VLM_MODEL", "unsloth/Qwen3-VL-8B-Instruct-GGUF") DEFAULT_OPENAI_URL = os.getenv("OPENAI_API_URL", "https://api.openai.com/v1/chat/completions") DEFAULT_OPENAI_MODEL = os.getenv("OPENAI_MODEL", "gpt-4o") DEFAULT_HF_URL = os.getenv("HF_API_URL", "https://router.huggingface.co/v1/chat/completions") #suffixe ":fastest" = routage automatique vers le provider le plus rapide HF_MODEL_CHOICES = [ "google/gemma-3-12b-it:fastest", "Qwen/Qwen3.6-35B-A3B:fastest", "moonshotai/Kimi-K2.6:fastest", ] DEFAULT_HF_MODEL = os.getenv("HF_MODEL", HF_MODEL_CHOICES[0]) DEFAULT_ALBERT_URL = os.getenv("ALBERT_API_URL", "https://albert.api.etalab.gouv.fr/v1/chat/completions") DEFAULT_ALBERT_MODEL = os.getenv("ALBERT_MODEL", "mistralai/Ministral-3-8B-Instruct-2512") # Qwen/Qwen3.5-9B via les HF Inference Providers, suffixe ":fastest" = routage # automatique vers le provider le plus rapide. Auth : HF_TOKEN en bearer. DEFAULT_QWEN35_URL = os.getenv("QWEN35_API_URL", "https://router.huggingface.co/v1/chat/completions") DEFAULT_QWEN35_MODEL = os.getenv("QWEN35_MODEL", "Qwen/Qwen3.5-9B:together") # Extras de requête pour Qwen3.5 : thinking désactivé (Qwen ignore les réglages # serveur, le toggle du chat template est le seul fiable — même levier que dans # humatheque-vlm-evaluation) + paramètres d'échantillonnage recommandés par Qwen # en mode non-thinking. Fusionnés au niveau racine du corps JSON de la requête. QWEN35_REQUEST_EXTRAS = { "temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0, "max_tokens": 4096, "chat_template_kwargs": {"enable_thinking": False}, } # datalab-to/lift, quantized Q8 (GGUF), served by llama.cpp on a HF dedicated endpoint. # Strict JSON-schema model: uses its own prompt template (build_lift_prompt); # multivalued fields are arrays, comme partout ailleurs depuis le schéma v2. DEFAULT_LIFT_URL = os.getenv("LIFT_API_URL", "https://jpectntdw48b1jip.eu-west-1.aws.endpoints.huggingface.cloud/v1/chat/completions") DEFAULT_LIFT_MODEL = os.getenv("LIFT_MODEL", "prithivMLmods/lift-GGUF") # Endpoint HF dédié qui sert lift : configuré en scale-to-zero (minReplica 0, # arrêt après 30 min d'inactivité). Un appel sur un endpoint endormi renvoie 503 # et déclenche son redémarrage (~2-5 min). L'API de gestion permet d'afficher le # cycle de vie dans l'UI et de réveiller/relancer l'endpoint explicitement. LIFT_ENDPOINT_NAME = os.getenv("LIFT_ENDPOINT_NAME", "lift-gguf-ykc") LIFT_ENDPOINT_NAMESPACE = os.getenv("LIFT_ENDPOINT_NAMESPACE", "Geraldine") HF_ENDPOINTS_API_BASE = os.getenv("HF_ENDPOINTS_API_BASE", "https://api.endpoints.huggingface.cloud/v2/endpoint") DEFAULT_SUDOC_URL = os.getenv("SUDOC_CHECKER_API_URL", "https://sudoc-checker.smartbiblia.fr") DEFAULT_IDREF_URL = os.getenv("IDREF_QUALINKA_API_URL", "https://idref-linker.smartbiblia.fr") DEFAULT_IDREF_API_KEY = os.getenv("IDREF_QUALINKA_API_KEY", "") DEFAULT_DEWEY_URL = os.getenv("DEWEY_CLASSIFICATION_API_URL", "https://dewey-classifier.smartbiblia.fr") DEFAULT_DEWEY_API_KEY = os.getenv("CLASSIFICATION_API_KEY", "") # Cross-session collaborative annotation store. Unlike the per-session gr.State # (which follows the empty_state/add_event contract), this is a shared, durable, # append-only event log: every notation/validation is one JSON line, folded back # into per-image state on read. Kept deliberately separate from the pipeline state. ANNOTATIONS_PATH = Path(os.getenv("ANNOTATIONS_PATH", str(_STORE_DIR / "annotations.jsonl"))) MINIO_URLS_PATH = Path(os.getenv("MINIO_URLS_PATH", str(Path(__file__).parent / "minio_urls.txt"))) _ANNOTATIONS_LOCK = threading.Lock() # Journal des résultats d'étapes (cache de restauration, voir step_results.py) : # dernière réponse Sudoc/IdRef/Dewey/brouillon par page, PPN validés compris. # Séparé d'annotations.jsonl (données d'entraînement) — exclu de l'export HF. # Donnée runtime — à gitignorer. STEP_RESULTS_PATH = Path(os.getenv("STEP_RESULTS_PATH", str(_STORE_DIR / "step_results.jsonl"))) _STEP_RESULTS_LOCK = threading.Lock() DEFAULT_ANNOTATOR = os.getenv("ANNOTATOR", "") EMBEDDING_MODEL = "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2" # Poids de scoring du service idref-linker (voir README de humatheque-idref-qualinka-api). DEFAULT_WEIGHT_NAME = float(os.getenv("IDREF_WEIGHT_NAME", "0.40")) DEFAULT_WEIGHT_ATTRRA_SOURCE = float(os.getenv("IDREF_WEIGHT_ATTRRA_SOURCE", "0.25")) DEFAULT_WEIGHT_ATTRRA_NOTE = float(os.getenv("IDREF_WEIGHT_ATTRRA_NOTE", "0.15")) DEFAULT_WEIGHT_REFERENCES = float(os.getenv("IDREF_WEIGHT_REFERENCES", "0.15")) DEFAULT_WEIGHT_INSTITUTION_YEAR = float(os.getenv("IDREF_WEIGHT_INSTITUTION_YEAR", "0.05")) # Paramètres avancés du service idref-linker (génération des candidats + seuils # de décision) ; mêmes défauts que l'API humatheque-idref-qualinka-api. DEFAULT_IDREF_MAX_CANDIDATES = int(os.getenv("IDREF_MAX_CANDIDATES", "20")) DEFAULT_IDREF_MAX_DOCS_PER_ROLE = int(os.getenv("IDREF_MAX_DOCS_PER_ROLE", "20")) DEFAULT_IDREF_REFERENCE_TOP_K = int(os.getenv("IDREF_REFERENCE_TOP_K", "3")) DEFAULT_IDREF_ACCEPT_THRESHOLD = float(os.getenv("IDREF_ACCEPT_THRESHOLD", "0.65")) DEFAULT_IDREF_MARGIN_THRESHOLD = float(os.getenv("IDREF_MARGIN_THRESHOLD", "0.08")) # Title-page scans can be huge (e.g. 2597x3670), which makes base64 payloads slow # to upload and slow for the on-premise llama.cpp VLM to process. Downscale the # longest side and re-encode as JPEG before sending. VLM_IMAGE_MAX_SIZE = int(os.getenv("VLM_IMAGE_MAX_SIZE", "768")) #512, 1024,1248 VLM_IMAGE_JPEG_QUALITY = int(os.getenv("VLM_IMAGE_JPEG_QUALITY", "85")) # Image size embedded into exported HF datasets (larger than the VLM inference # downscale: the dataset should keep enough detail for future fine-tuning). EXPORT_IMAGE_MAX_SIZE = int(os.getenv("EXPORT_IMAGE_MAX_SIZE", "1024")) DEFAULT_HF_DATASET_REPO = os.getenv("HF_DATASET_REPO", "Geraldine/humatheque-vlm-sft") THESIS_DEGREE_TYPE_VALUES = [ "Thèse d'État", "Thèse de doctorat", "Thèse de 3e cycle", "Thèse d'université", "Thèse de docteur-ingénieur", "Thèse d'exercice", ] DISSERTATION_DEGREE_TYPE_VALUES = [ "Habilitation à diriger des recherches", "Mémoire de DEA", "Mémoire de DES", "Mémoire de DESS", "Mémoire de DU", "Mémoire de DIU", "Mémoire de DUT", "Mémoire de maîtrise", "Mémoire de master professionnel 1re année", "Mémoire de master professionnel 2e année", "Mémoire de master recherche 1re année", "Mémoire de master recherche 2e année", ] # Schéma d'extraction v2 (harmonisé avec humatheque-vlm-evaluation) : 16 champs # + confidence. `volume` = numéro de tome en chiffres arabes si la page le # mentionne explicitement, null sinon. METADATA_FIELDS = [ "title", "subtitle", "author", "degree_type", "discipline", "volume", "granting_institution", "co_tutelle_institutions", "doctoral_school", "defense_year", "advisor", "jury_president", "reviewers", "committee_members", "language", "confidence", ] PERSON_FIELDS = ["author", "advisor", "jury_president", "reviewers", "committee_members"] # Convention v2 : les champs multivalués sont TOUJOURS des tableaux JSON ([] si # vide) ; les champs scalaires utilisent null. jury_president reste scalaire. MULTIVALUED_FIELDS = ["co_tutelle_institutions", "advisor", "reviewers", "committee_members"] ROLE_LABELS = { "author": "Auteur", "advisor": "Directeur", "jury_president": "President du jury", "reviewers": "Rapporteur", "committee_members": "Membre du jury", } PERSON_ROLE_CODES = { "author": "070", "advisor_thesis": "727", "advisor_dissertation": "003", "jury_president": "956", "reviewers": "958", "committee_members": "555", } CORPORATE_ROLE_CODES = { "granting_institution": "295", "co_tutelle_institutions": "995", "doctoral_school": "996", "partner_institutions": "985", } STRONG_IDREF_STATUSES = {"accepted"} EXAMPLE_IMAGE_URL = "https://minio.smartbiblia.fr/images/theses/theses.fr/2015EPHE4076/p1.png" # Spécifications collaboratives (onglet 7). Le mécanisme est générique (specs.py) : # un fichier seed définit les champs de la spec (ici UNIMARC), un journal # append-only JSONL porte les révisions des annotateurs. Pointer SPEC_SEED_PATH # vers un autre seed (MARC21, Dublin Core…) suffit pour réutiliser le système # dans un autre projet. SPEC_EVENTS_PATH est une donnée runtime — à gitignorer. SPEC_SEED_PATH = Path(os.getenv("SPEC_SEED_PATH", str(Path(__file__).parent / "unimarc_spec_seed.json"))) SPEC_EVENTS_PATH = Path(os.getenv("SPEC_EVENTS_PATH", str(_STORE_DIR / "spec_events.jsonl"))) _SPEC_EVENTS_LOCK = threading.Lock()