silv4souz4/mediapipe-iris-landmark-coreai-fp16

Model Card for MediaPipe Iris Landmark — Apple Core AI (fp16)

Op-by-op conversion of Google's MediaPipe model iris_landmark.tflite from the official MediaPipe repository, tag v0.8.10 from TFLite to Apple Core AI (.aimodel) in fp16, for on-device inference on Apple silicon. Verified against the official TFLite interpreter on real images (see Evaluation). Tested on macOS 27 (Apple M5); not tested on an iPhone.

Model Details

Model Description

The model predicts, from a 64×64 crop around one eye, 71 points of the eye contour and eyebrow and 5 iris points (center + 4 on the contour), each with x, y and a relative depth z. It is the second stage of MediaPipe Iris: the eye crop is computed from face landmarks. This repository contains the network as one Core AI function; cropping and mapping back to the image run in the host.

• Original model developed by: Google (MediaPipe).
• Conversion by: silv4souz4 (independent; not affiliated with or endorsed by Google or Apple).
• Model type: eye and iris landmark regression, input 64×64.
• Base model: iris_landmark.tflite from the official MediaPipe repository, tag v0.8.10 (link, SHA-256 of the .tflite d1744d2a09c25f501d39eba4faff47e53ecca8852c5ce19bce8eeac39357521f); not hosted on the Hugging Face Hub, so no base_model link is declared.
• Relation to base: format conversion (TFLite → PyTorch op by op with the original weights → Core AI fp16); no quantization.
• License: Apache License 2.0 (MediaPipe repository).

Model Sources

• Original: https://github.com/google-ai-edge/mediapipe
• Sibling conversion: silv4souz4/mediapipe-face-landmarks-detector-coreai-fp16 — MediaPipe Face Landmarks Detector (Face Mesh V2, 478 points)

Uses

Direct Use

On-device eye tracking and gaze-related AR effects, iris size measurement (for example as a physical scale reference), virtual try-on of eyewear or colored lenses.

Downstream Use

Pipelines that combine a face detector, this model and the face landmarks model on Apple devices.

Out-of-Scope Use

• Face recognition or identification: these landmarks do not identify people or store a unique face representation (as stated for MediaPipe face landmarks in the Face Mesh V2 model card); do not use them for surveillance or biometrics.
• Medical diagnosis or other life-critical decisions.
• Any use that violates applicable law.

Bias, Risks, and Limitations

• Inherits the limitations in the official MediaPipe documentation: accuracy depends on the crop, head pose, occlusion (glasses, hair), lighting and image quality.
• fp16 moves points by a few hundredths of a pixel against the official model on our real crops (see below); the Neural Engine adds a little more.
• Our real-image check is small (26 eye crops computed (like MediaPipe) from the official TFLite face landmarks on 13 COCO val2017 portraits, plus 3 noise inputs); it measures fidelity to the official model, not accuracy against human labels.
• Not tested on iPhone. Debug information embedded in the asset contains build-time file paths of the converting machine; it has no effect on inference.

Recommendations

Use the crop convention below exactly; filter by the presence score; validate on your own data and device.

How to Get Started with the Model

Download the whole repository (asset, config.json and checksums):

hf download silv4souz4/mediapipe-iris-landmark-coreai-fp16 --local-dir mediapipe-iris-landmark-coreai-fp16

config.json is a machine-readable summary of the function, tensor shapes, output meanings and crop convention below (it is also the file the Hub uses to count downloads of this repository).

Function	Input	Outputs
main	image float16 [1,3,64,64] RGB in [0,1]	eye_contours [1,213] = 71×(x,y,z); iris [1,15] = 5×(x,y,z), 0 = center, 1–4 = contour; px of the 64 crop

File: IrisLandmark_fp16.aimodel (1.4 MB).

• Eye crop: take the eye corners from face landmarks (33→133 for the eye on the left of the image, 362→263 for the right one); center = midpoint, rotate so the corner line is horizontal, square side = 2.3 × the corner distance, resample to 64×64 (bilinear), RGB in [0,1]. Mirror the right eye of the image horizontally (MediaPipe's is_right_eye) and mirror the outputs back.
• Iris size: the iris diameter in the crop is twice the mean distance from point 0 to points 1–4; scale it back to the image with the crop side / 64.

from coreai.runtime import AIModel, NDArray, SpecializationOptions, ComputeUnitKind
opts = SpecializationOptions.from_preferred_compute_unit_kind(ComputeUnitKind.neural_engine())
f = (await AIModel.load("IrisLandmark_fp16.aimodel", opts)).load_function("main")
out = await f({"image": NDArray(crop)})   # crop: float16 [1,3,64,64], RGB in [0,1]

Ahead-of-time compilation (.aimodelc) — tutorial

The asset here is a portable .aimodel. Apple's documentation, Compiling Core AI models ahead of time, explains that every Core AI model must be specialized to the device before inference; the coreai-build tool moves the most expensive part of that specialization, model compilation, to your build machine, so the model loads faster in your app. It produces one .aimodelc per device architecture; you ship (or download) only the one that matches the device. According to Apple, AOT compilation targets devices that support Apple Intelligence (iPhone/iPad with A17 Pro or later, Mac with M1 or later, Apple Vision Pro); it is not supported on tvOS and watchOS.

1. Install the Metal Toolchain (Xcode > Settings > Components > Metal Toolchain, or):

xcodebuild -downloadComponent MetalToolchain

2. Compile (all architectures of a platform, or one with --architecture):

xcrun coreai-build compile IrisLandmark_fp16.aimodel --platform macOS --min-deployment-version 27.0 --output compiled/
xcrun coreai-build compile IrisLandmark_fp16.aimodel --platform iOS --min-deployment-version 27.0 --output compiled/

Output files are named <asset>.<arch>.aimodelc (for example .h17g for the Apple M5 used here); each works on any OS at or above --min-deployment-version. Run coreai-build compile --help for --preferred-compute and the other options.

3. Load the matching bundle — the same API as for .aimodel:

let arch = AIModel.deviceArchitectureName
let url = compiledDir.appending(path: "IrisLandmark_fp16.\(arch).aimodelc")
let model = try await AIModel(contentsOf: url, options: .default)

Apple recommends hosting the per-architecture bundles and downloading only the matching one (for example with the BackgroundAssets framework). Some on-device specialization still remains after AOT, depending on the model and compute units.

What we measured (Apple M5, macOS 27.2, Swift runtime, first load with an empty specialization cache, two rounds):

Step	Portable .aimodel	AOT .aimodelc
Compile on the Mac (coreai-build, h17g)	—	0.6 s
First load (model + function), cold cache	0.1–0.2 s	0.1 s
Later loads (cache warm)	< 0.05 s	< 0.05 s

For a model this small the gain on the Mac is negligible; it may matter more on iPhone, where on-device specialization is slower (not measured). AOT is not an inference speed-up.

Training Details

Training Data

No training or fine-tuning. Weights are Google's, as described in the MediaPipe documentation.

Training Procedure (conversion)

1. The TFLite flatbuffer was read and executed op by op in PyTorch with the original weights (CONV_2D, DEPTHWISE_CONV_2D, PRELU, ADD, MAX_POOL_2D, channel PAD, RESHAPE, LOGISTIC), reproducing TF's asymmetric SAME padding; input range [0, 1] as in MediaPipe's graphs.
2. torch.export → decompositions → fp16 cast → coreai-torch TorchConverter → portable .aimodel.

Evaluation

Testing Data, Factors & Metrics

Reference: the official TFLite interpreter (ai-edge-litert 2.3.0) running the original .tflite (SHA-256 verified). Inputs: 26 eye crops computed (like MediaPipe) from the official TFLite face landmarks on 13 COCO val2017 portraits, plus 3 noise inputs. Same inputs to the Core AI asset (default and Neural Engine preferred). Metrics: point error in pixels of the crop, PSNR of the outputs, relative error of the iris diameter. Apple M5, macOS 27.2, 12 Oct 2026.

Results

Check	Result
This fp16 asset vs official TFLite, 26 real eye crops: iris points, mean / max error	0.009 / 0.023 px (of 64)
… eye contour points, mean / max error	0.010 / 0.032 px
… iris diameter, mean / max relative error	0.06 % / 0.13 % (diameter ≈ 10.8 px)
Same, Neural Engine preferred	iris 0.010 / 0.028 px; diameter 0.07 % / 0.22 %
PSNR vs TFLite (min over noise + real crops), fp16	iris 68.9 dB, contour 75.6 dB
fp32 conversion vs TFLite (not published, proves the port)	129–133 dB, 0.000 px

Visual check: the landmarks drawn from this asset fall on the faces and eyes of the real crops.

Summary

Equivalent to the official MediaPipe TFLite model (errors of a few hundredths of a pixel), 1.4 MB.

Environmental Impact

No training. Conversion and evaluation ran locally on one Apple M5 laptop; negligible.

Technical Specifications

Compute Infrastructure

• Hardware: Apple M5 (Mac17,3), 24 GB unified memory.
• Software: conversion with macOS 27, PyTorch 2.11, coreai-torch 0.4.2; verification with macOS 27.2, coreai-core 1.0.0b3, ai-edge-litert 2.3.0; coreai-build 3605.5.4.

Citation

If you use these assets, please cite:

@article{ablavatski2020iris,
  title={Real-time Pupil Tracking from Monocular Video for Digital Puppetry},
  author={Ablavatski, Artsiom and Vakunov, Andrey and Grishchenko, Ivan and Raveendran, Karthik and Zhdanovich, Matsvei},
  journal={arXiv preprint arXiv:2006.11341},
  year={2020}
}

The evaluation used COCO val2017 images (subject to the Flickr Terms of Use).

License and Attribution

• License: Apache License 2.0 — full text in LICENSE.
• Original model: © Google LLC (MediaPipe).
• Modifications (Apache 2.0 §4(b)): op-by-op re-implementation of the TFLite graph in PyTorch with the original weights and fp16 conversion to the Core AI format. No retraining. See NOTICE.
• "Apple", "Core AI", "Google" and "MediaPipe" are trademarks of their respective owners; their use here is descriptive only.

Model Card Authors

silv4souz4

Model Card Contact

Please open a discussion in the Community tab of this repository.