---
tags:
- multimodal
- multilingual
- pdf
- embeddings
- rag
- google-cloud
- vertex-ai
- gemma
- python
datasets:
  - no_dataset
license: mit
---

# Multimodal & Multilingual PDF Embedding Pipeline with Gemma and Vertex AI

This repository hosts a Python pipeline that extracts text, tables, and images from PDF documents, generates multimodal descriptions for visual content (tables and images) using **Google's Gemma model (running locally)**, and then creates multilingual text embeddings for all extracted information using **Google Cloud Vertex AI's `text-multilingual-embedding-002` model**. The generated embeddings are stored in a JSON file, ready for use in Retrieval Augmented Generation (RAG) systems or other downstream applications.

**Key Features:**
- **Multimodal Descriptions (via Gemma):** Processes tables and images from PDFs, generating rich descriptive text in French using the open-source Gemma 3.4B-IT model, which runs locally on your machine/Colab GPU.
- **Multilingual Text Embeddings (via Vertex AI):** Leverages Google Cloud's `text-multilingual-embedding-002` model for embeddings, supporting a wide range of languages.
- **Structured Output:** Stores embeddings and metadata (PDF source, page number, content type, links to extracted assets) in a comprehensive JSON format.

## How it Works

1.  **PDF Parsing:** Utilizes `PyMuPDF` to extract text blocks and images, and `Camelot` to accurately extract tabular data.
2.  **Content Separation:** Distinguishes between plain text, tables, and non-table images.
3.  **Multimodal Description (for Tables & Images using Gemma):**
    - For tables, the pipeline captures an image of the table and also uses its text representation.
    - For standalone images (e.g., graphs, charts), it captures the image.
    - These images (and optionally table text) are then passed to the **Gemma 3.4B-IT model** (via the `gemma` Python library) with specific prompts to generate rich, descriptive text in French. **This step runs locally and does not incur direct API costs.**
4.  **Multilingual Text Embedding (via Vertex AI):**
    - The cleaned text content (original text chunks, or generated descriptions for tables/images) is then passed to the `text-multilingual-embedding-002` model (via Vertex AI).
    - This model generates a high-dimensional embedding vector (768 dimensions) for each piece of content. **This step connects to Google Cloud Vertex AI and will incur costs.**
5.  **JSON Output:** All generated embeddings, along with rich metadata (original PDF, page, content type, links to extracted assets), are compiled into a single JSON file.

## Requirements & Setup

This pipeline uses a combination of local models (Gemma) and **Google Cloud Platform** services (Vertex AI).

1.  **Google Cloud Project with Billing Enabled (for Text Embeddings):**
    -   **CRITICAL:** The text embedding generation step uses Google Cloud Vertex AI. This **will incur costs** on your Google Cloud Platform account. Ensure you have an [active billing account](https://cloud.google.com/billing/docs/how-to/create-billing-account) linked to your project.
    -   Enable the **Vertex AI API**.
2.  **Authentication for Google Cloud (for Text Embeddings):**
    -   The easiest way to run this in a Colab environment is using `google.colab.auth.authenticate_user()`.
    -   For local execution, ensure your Google Cloud SDK is configured and authenticated (`gcloud auth application-default login`).
3.  **Hardware Requirements (for Gemma):**
    -   Running the Gemma 3.4B-IT model requires a **GPU with sufficient VRAM** (e.g., a Colab T4 or V100 GPU, or a local GPU with at least ~8-10GB VRAM is recommended). If a GPU is not available, Gemma will likely run on CPU but will be significantly slower.

### Local Setup

1.  **Clone the repository:**
    ```bash
    git clone https://huggingface.co/Anonymous1223334444/pdf-multimodal-multilingual-embedding-pipeline
    cd pdf-multimodal-multilingual-embedding-pipeline
    ```
2.  **Install Python dependencies:**
    ```bash
    pip install -r requirements.txt
    ```
    **System-level dependencies for Camelot/PyMuPDF (Linux/Colab):**
    You might need to install these system packages for `PyMuPDF` and `Camelot` to function correctly.
    ```bash
    # Update package list
    sudo apt-get update
    # Install Ghostscript (required by Camelot)
    sudo apt-get install -y ghostscript
    # Install python3-tk (required by some PyMuPDF functionalities)
    sudo apt-get install -y python3-tk
    # Install OpenCV (via apt, for camelot-py[cv])
    sudo apt-get install -y libopencv-dev python3-opencv
    ```
    *Note: If you are running on Windows or macOS, the installation steps for `camelot-py` might differ. Refer to the [Camelot documentation](https://camelot-py.readthedocs.io/en/master/user/install-deps.html) for more details.*

3.  **Set up Environment Variables (for Vertex AI Text Embeddings):**
    ```bash
    export GOOGLE_CLOUD_PROJECT="your-gcp-project-id"
    export VERTEX_AI_LOCATION="us-central1" # Or your preferred Vertex AI region (e.g., us-east4)
    ```
    Replace `your-gcp-project-id` and `us-central1` with your actual Google Cloud Project ID and Vertex AI region.

4.  **Place your PDF files:**
    Create a `docs` directory in the root of the repository and place your PDF documents inside it.
    ```
    pdf-multimodal-multilingual-embedding-pipeline/
    ├── docs/
    │   └── your_document.pdf
    └── another_document.pdf
    ```

5.  **Run the pipeline:**
    ```bash
    python run_pipeline.py
    ```
    The generated embedding file (`embeddings_statistiques_multimodal.json`) and extracted assets will be saved in the `output/` directory.

### Google Colab Usage

A Colab notebook version of this pipeline is ideal for quick experimentation due to pre-configured environments and GPU access.

1.  **Open a new Google Colab notebook.**
2.  **Change runtime to GPU:** Go to `Runtime > Change runtime type` and select `T4 GPU` or `V100 GPU`.
3.  **Install system and Python dependencies:**
    ```python
    !pip uninstall -y camelot camelot-py # Ensure clean install
    !pip install PyMuPDF
    !apt-get update
    !apt-get install -y ghostscript python3-tk libopencv-dev python3-opencv
    !pip install camelot-py[cv] google-cloud-aiplatform tiktoken pandas beautifulsoup4 Pillow gemma jax jaxlib numpy
    ```
4.  **Authenticate to Google Cloud (for Vertex AI):**
    ```python
    from google.colab import auth
    auth.authenticate_user()
    ```
5.  **Set your Google Cloud Project ID and Location:**
    ```python
    import os
    # Replace with your actual Google Cloud Project ID
    os.environ["GOOGLE_CLOUD_PROJECT"] = "YOUR_GCP_PROJECT_ID_HERE"
    # Set your preferred Vertex AI location (e.g., "us-central1", "us-east4")
    os.environ["VERTEX_AI_LOCATION"] = "us-central1"
    
    # Critical: Adjust JAX memory allocation for Gemma
    os.environ["XLA_PYTHON_CLIENT_MEM_FRACTION"]="1.00"
    ```
6.  **Upload your PDF files:**
    You can use the Colab file upload feature or mount Google Drive. Ensure your PDFs are in a directory named `docs` within `/content/`.
    ```python
    # Example for uploading
    from google.colab import files
    import os
    from pathlib import Path

    PDF_DIRECTORY = Path("/content/docs")
    PDF_DIRECTORY.mkdir(parents=True, exist_ok=True)
    uploaded = files.upload()
    for filename in uploaded.keys():
        os.rename(filename, PDF_DIRECTORY / filename)
    ```
7.  **Copy and paste the code from `src/pdf_processor.py`, `src/embedding_utils.py` and `run_pipeline.py` into Colab cells and execute.** Make sure to execute `embedding_utils.py` content first, then `pdf_processor.py` content, then `run_pipeline.py` content, or combine them logically into your notebook.

## Output

The pipeline will generate:
- `embeddings_statistiques_multimodal.json`: A JSON file containing all generated embeddings and their metadata.
- `output/extracted_graphs/`: Directory containing extracted images (PNG format).
- `output/extracted_tables/`: Directory containing HTML representations of extracted tables.

## Example `embeddings_statistiques_multimodal.json` Entry

```json
[
  {
    "pdf_file": "sample.pdf",
    "page_number": 1,
    "chunk_id": "text_0",
    "content_type": "text",
    "text_content": "This is a chunk of text extracted from the first page of the document...",
    "embedding": [0.123, -0.456, ..., 0.789],
    "pdf_title": "Sample Document",
    "pdf_subject": "Data Analysis",
    "pdf_keywords": "statistics, report"
  },
  {
    "pdf_file": "sample.pdf",
    "page_number": 2,
    "chunk_id": "table_0",
    "content_type": "table",
    "text_content": "Description en français du tableau: Ce tableau présente les ventes mensuelles par région. Il inclut les colonnes Mois, Région, et Ventes. La région Nord a la plus forte croissance...",
    "embedding": [-0.987, 0.654, ..., 0.321],
    "table_html_url": "/static/extracted_tables/sample_p2_table0.html",
    "image_url": "/static/extracted_graphs/sample_p2_table0.png",
    "pdf_title": "Sample Document",
    "pdf_subject": "Data Analysis",
    "pdf_keywords": "statistics, report"
  },
  {
    "pdf_file": "sample.pdf",
    "page_number": 3,
    "chunk_id": "image_0",
    "content_type": "image",
    "text_content": "Description en français de l'image: Ce graphique est un histogramme montrant la répartition des âges dans la population. L'axe des X représente les tranches d'âge et l'axe des Y la fréquence. La majorité de la population se situe entre 25 et 40 ans.",
    "embedding": [0.456, -0.789, ..., 0.123],
    "image_url": "/static/extracted_graphs/sample_p3_img0.png",
    "pdf_title": "Sample Document",
    "pdf_subject": "Data Analysis",
    "pdf_keywords": "statistics, report"
  }
]
```

# Acknowledgments
This pipeline leverages the power of:
- Gemma AI
- Google AI Gemini Models
- PyMuPDF
- Camelot
- Tiktoken
- Pandas
- BeautifulSoup