|
|
--- |
|
|
license: mit |
|
|
language: |
|
|
- en |
|
|
tags: |
|
|
- seq2seq |
|
|
- UnrealEngine |
|
|
- LiveLink |
|
|
- FeatureMapping |
|
|
- PyTorch |
|
|
--- |
|
|
|
|
|
# NeuroSync |
|
|
|
|
|
[View Demo Video](https://www.youtube.com/watch?v=Da-Vt3C3aWQ) |
|
|
|
|
|
|
|
|
## Model Overview |
|
|
|
|
|
The **NeuroSync transformer seq2seq model** converts sequences of audio features into corresponding facial blendshape coefficients, enabling real-time character animation. It integrates seamlessly with Unreal Engine via LiveLink. |
|
|
|
|
|
--- |
|
|
|
|
|
## Features |
|
|
|
|
|
- **Audio-to-Face Transformation**: Converts raw audio features into facial blendshape coefficients. |
|
|
- **Transformer Seq2Seq Architecture**: Utilizes encoder-decoder layers to capture complex dependencies between audio and facial expressions. |
|
|
- **Unreal Engine Integration (LiveLink)**: Stream facial blendshapes in real time with the [NeuroSync Player](https://github.com/AnimaVR/NeuroSync_Player). |
|
|
|
|
|
--- |
|
|
|
|
|
## Usage |
|
|
|
|
|
### Local API |
|
|
Set up your local API using the [NeuroSync Local API repository](https://github.com/AnimaVR/NeuroSync_Local_API) to process audio files and stream generated blendshapes. |
|
|
|
|
|
--- |
|
|
|
|
|
## Model Architecture |
|
|
|
|
|
- **Encoder**: Processes audio features with a transformer encoder using positional encodings. |
|
|
- **Decoder**: Uses cross-attention in a transformer decoder to generate blendshape coefficients. |
|
|
- **Output**: Produces 68 blendshape coefficients, 61 relate to ARKit and 7 "emotions". |
|
|
|
|
|
 |
|
|
|
|
|
### Blendshape Coefficients |
|
|
|
|
|
- **Included**: Eye movements (e.g., *EyeBlinkLeft*, *EyeSquintRight*), jaw movements (e.g., *JawOpen*, *JawRight*), mouth movements (e.g., *MouthSmileLeft*, *MouthPucker*), brow movements (e.g., *BrowInnerUp*, *BrowDownLeft*), and cheek/nose movements (e.g., *CheekPuff*, *NoseSneerRight*). |
|
|
- **Note**: Coefficients 62–68 (related to emotional states) should be ignored or used for additive sliders since they are not streamed into LiveLink. |
|
|
|
|
|
--- |
|
|
|
|
|
## Community & Resources |
|
|
|
|
|
### Live Demo |
|
|
|
|
|
**Twitch**: [Talk to a NeuroSync prototype live on Twitch](https://www.twitch.tv/mai_anima_ai) |
|
|
|
|
|
### YouTube Channel |
|
|
|
|
|
For tutorials, updates, and more, visit our [YouTube channel](https://www.youtube.com/@animaai_mai). |
|
|
|
|
|
--- |
|
|
|
|
|
## NeuroSync License |
|
|
|
|
|
This software uses a **dual-license model**: |
|
|
|
|
|
### 1. Free License (MIT License) |
|
|
|
|
|
For individuals and businesses earning **under $1M per year**: |
|
|
|
|
|
MIT License |
|
|
|
|
|
Copyright (c) 2025 NeuroSync |
|
|
|
|
|
Permission is hereby granted, free of charge, to any person obtaining a copy |
|
|
of this software and associated documentation files (the "Software"), to deal |
|
|
in the Software without restriction, including without limitation the rights |
|
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell |
|
|
copies of the Software, and to permit persons to whom the Software is |
|
|
furnished to do so, subject to the following conditions: |
|
|
|
|
|
The above copyright notice and this permission notice shall be included in all |
|
|
copies or substantial portions of the Software. |
|
|
|
|
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR |
|
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, |
|
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE |
|
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER |
|
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, |
|
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE |
|
|
SOFTWARE. |
|
|
|
|
|
|
|
|
## 2. Commercial License (For Businesses Earning $1M+ Per Year) |
|
|
|
|
|
Businesses or organizations with **annual revenue of $1,000,000 or more** must obtain a **commercial license** to use this software. |
|
|
|
|
|
- To acquire a commercial license, please contact us. |
|
|
|
|
|
## Compliance |
|
|
|
|
|
By using this software, you agree to these licensing terms. If your business exceeds the revenue threshold, you must transition to a commercial license or **cease using the software**. |
|
|
|
|
|
© 2025 NeuroSync |
|
|
|
|
|
## References |
|
|
|
|
|
- [NeuroSync Local API](https://github.com/AnimaVR/NeuroSync_Local_API) |
|
|
- [NeuroSync Player (Unreal Engine LiveLink)](https://github.com/AnimaVR/NeuroSync_Player) |
|
|
- [Website](https://neurosync.info) |
|
|
|
|
|
For any questions or further support, please feel free to contribute to the repository or raise an issue. |
