updated README

README.md

@@ -87,20 +87,119 @@ All communication between modules happens over HTTP:
 - **Orchestrator ↔ Generator**: HTTP streaming (SSE for real-time responses)
 - **ChatUI ↔ Orchestrator**: LangServe streaming endpoints
 
-### Workflow Logic
+## Workflow Logic and File Processing
 
-The orchestrator implements two distinct workflows:
+The orchestrator implements a dual-mode workflow designed to handle non-standard ingesion operations (e.g., Whisp GeoJSON API calls - as these need to be returned directly without going through the generator) while maintaining conversational context across multiple turns. This also addresses an issue with ChatUI in that it resends uploaded files on each turn in the conversation (e.g. follow-up queries).
 
-**Direct Output Workflow** (when `DIRECT_OUTPUT=True` and file is new):
+### Processing Modes Overview
+
+#### Mode 1: Direct Output (DIRECT_OUTPUT = True)
+
+**Purpose:** Immediately return long-running ingestor results to the user without LLM processing, then use those results as context for follow-up questions.
+
+**First File Upload:**
+```
+File Upload → Detect Type → Direct Output Ingest → Return Raw Results
+                                         ↓
+                                    Cache Result (by file hash)
+```
+
+**Subsequent Conversation Turns:**
 ```
-File Upload → Detect Type → Ingest → Direct Output → Return Results
+Follow-up Query → Detect Cached File → Retrieved Context → Combined Context → Generator
+                         ↓
+                  Use Cached Ingestor Output as Context
 ```
 
-**Standard RAG Workflow** (default or cached files):
+**Key Behaviors:**
+- First upload returns raw ingestor output immediately (no LLM generation)
+- File content is hashed (SHA256) for deduplication
+- Ingestor results are cached with the file hash
+- All follow-up queries in the conversation use the cached ingestor output as retrieval context
+
+**Noteable Unintuitive Behavior:** Once the file is cached on the Orchestrator, re-uploading the same file (even with different filename in a different chat) skips re-processing.
+
+**Example Conversation Flow:**
 ```
-Query → Retrieve Context → Generate Response → Stream to User
+User: [Uploads plot_boundaries.geojson]
+System: [Returns API analysis results directly - no LLM processing]
+
+User: "What deforestation risks were identified?"
+System: [Uses cached GeoJSON results + retrieval → LLM generation]
+
+User: "How does this compare to EUDR requirements?"
+System: [Uses same cached results + conversation history + retrieval → LLM generation]
 ```
 
+#### Mode 2: Standard RAG (DIRECT_OUTPUT = False)
+
+**Purpose:** Traditional RAG pipeline where uploaded files are treated as additional context for generation from first instance.
+
+**Every Query (with or without file):**
+```
+Query + Optional File → Detect Type → Ingest → Retrieved Context → Combined Context → Generator
+                                         ↓
+                                    Add to Context
+```
+
+**Key Behaviors:**
+- Files are processed through ingestor when uploaded
+- Ingestor output is added to the retrieval context (not returned directly)
+- Generator always processes the combined context (ingestor + retriever)
+- No special caching or deduplication logic
+
+**Example Conversation Flow:**
+```
+User: "What are EUDR requirements?" + policy_document.pdf
+System: [PDF → Retrieval → Combined Context → Generator]
+
+User: "Summarize section 3"
+System: [Retrieval → Combined Context → Generator]
+```
+
+### File Hash Caching Mechanism
+
+The orchestrator uses SHA256 hashing to detect duplicate file uploads:
+
+**Cache Structure:**
+```python
+{
+  "a3f5c91...": {
+    "ingestor_context": "API results...",
+    "timestamp": "2025-10-02T14:30:00",
+    "filename": "boundaries.geojson",
+    "file_type": "geojson"
+  }
+}
+```
+
+**Detection Logic:**
+1. File is uploaded
+2. Compute SHA256 hash of file content
+3. Check if hash exists in cache
+4. If not found: Process through ingestor, cache results
+5. If found: Use cached results (Skip Ingestion → Retrieved Context → Combined Context → Generator)
+
+
+### Conversation Context Management
+
+The system maintains conversation history separately from file processing with a simple management approach:
+
+**Context Building (`build_conversation_context()`):**
+- Always includes first user/assistant exchange
+- Includes last N complete turns (default: 3)
+- Respects character limits (default: 12,000 chars)
+
+**Retrieval Strategy:**
+- Uses **only the latest user query** for semantic search
+- Does NOT send entire conversation history to retriever
+- Ensures relevant document retrieval based on current question
+
+**Generation Context:**
+- Combines: Conversation history + Retrieved context + Cached file results
+- Generator uses full context to produce coherent, contextually-aware responses
+
+
 ## Components
 
 ### 1. Main Application (`main.py`)
@@ -207,7 +306,7 @@ MAX_CONTEXT_CHARS = 15000
 Create a `.env` file with:
 
 ```bash
-# Required for private HuggingFace Spaces
+# Required for accessing private HuggingFace Spaces modules
 HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxx
 
 ```
@@ -267,7 +366,7 @@ LLM_SUMMARIZATION=false
 
 Key things to ensure here:
 - multimodalAcceptedMimetypes: file types to accept for upload via ChatUI
-- endpoints: orchestrator url + endpoints
+- endpoints: orchestrator url + endpoints (note these are the HF API urls, not the HF UI urls)
 
 ## Deployment Guide
 
@@ -487,6 +586,8 @@ Clears the direct output file cache.
 
 Gradio's default API endpoint for UI interactions. If running on huggingface spaces, access via: https://[ORG_NAME]-[SPACE_NAME].hf.space/gradio/
 
+*NOTE - for HF deployment we have to access the Gradio test UI in this way, because it is not possible to expose multiple ports on HF Spaces. For the other modules we directly expose Gradio, but with the Orchestrator we need to run FastAPI to support the LangServe endpoints.*
+
 
 ## Troubleshooting