Writer/palmyra-mini-thinking-b
Text Generation
•
2B
•
Updated
•
10
•
27
Sam Julien
samjulien
AI & ML interests
None yet
Recent Activity
liked
a model
about 2 months ago
Writer/palmyra-mini-thinking-b
liked
a model
about 2 months ago
Writer/palmyra-mini-thinking-a
liked
a model
about 2 months ago
Writer/palmyra-mini
Organizations
liked
3
models
about 2 months ago
upvoted
an
article
about 2 months ago
Article
Introducing the Palmyra-mini family: Powerful, lightweight, and ready to reason!
By
and 1 other
•
•
58
upvoted
a
paper
5 months ago
liked
a
Space
8 months ago
liked
a
Space
9 months ago
upvoted
a
paper
9 months ago
liked
a
Space
10 months ago
updated
a
Space
11 months ago
posted
an
update
11 months ago
Post
1572
🔥 RAG in just a few lines of code?!
Try out our Hacker News Listener with new built-in RAG capabilities and Palmyra X 004 from the team at Writer!
This Writer Framework app:
- Scrapes up to 500 HN stories and comments
- Uploads them to a Knowledge Graph
- Enables interactive chat with the content using graph-based RAG
- Provides source attribution with every response
The best part? Setting up RAG is now incredibly simple - just a few lines of code to connect your Knowledge Graph as a tool with Palmyra X 004.
🤗 Space: samjulien/hacker-news-listener
💻 Code: https://github.com/writer/framework-tutorials/tree/main/hacker-news-social-listener
Try out our Hacker News Listener with new built-in RAG capabilities and Palmyra X 004 from the team at Writer!
This Writer Framework app:
- Scrapes up to 500 HN stories and comments
- Uploads them to a Knowledge Graph
- Enables interactive chat with the content using graph-based RAG
- Provides source attribution with every response
The best part? Setting up RAG is now incredibly simple - just a few lines of code to connect your Knowledge Graph as a tool with Palmyra X 004.
🤗 Space: samjulien/hacker-news-listener
💻 Code: https://github.com/writer/framework-tutorials/tree/main/hacker-news-social-listener
updated
a
Space
11 months ago
commented
a
paper
about 1 year ago
Congratulations, team! Amazing work! 👏
reacted to
melisa's
post with 🔥
about 1 year ago
Post
3260
🔥 Introducing "Writing in the Margins (WiM)" - better inference pattern for long context LLMs that solves the Lost-in-the-Middle problem 🔥
Paper page: Writing in the Margins: Better Inference Pattern for Long Context Retrieval (2408.14906)
TL;DR
Make your model write "margin notes" as you chunk prefill the KV cache. Then ask it reread all notes before it speaks up.
Works with humans, works with AI 🤖
WiM leverages the chunked prefill of the key-value cache, which concurrently generates query-based extractive summaries at each step of the prefill that are subsequently reintegrated at the end of the computation. We term these intermediate outputs “margins”, drawing inspiration from the practice of making margin notes for improved comprehension of long contexts in human reading. We show that this technique, which adds only minimal additional computation, significantly improves LLMs long context reasoning capabilities.
Think: Every chunk has a chance to be attended to/ be at the end of the context at least once. 🎉
📊 Results:
- An average accuracy boost of 7.5% in multi-hop reasoning tasks like HotpotQA and MultiHop-RAG.
- Even a 30% increase in F1-score for summarisation-like tasks (CWE).
Plus, WiM fits seamlessly into interactive applications (think: progress bar!). It can provide real-time progress updates during data retrieval and integration, making it user-friendly and transparent - a stark contrast to feeding 1mln tokens to an LLMs and waiting 6 min for the first token. 🤯
👩💻🧑💻 Check it out and contribute to our open-source project here: https://github.com/writer/writing-in-the-margins
🧠 More about chunked prefill: https://docs.vllm.ai/en/latest/models/performance.html#chunked-prefill
Paper page: Writing in the Margins: Better Inference Pattern for Long Context Retrieval (2408.14906)
TL;DR
Make your model write "margin notes" as you chunk prefill the KV cache. Then ask it reread all notes before it speaks up.
Works with humans, works with AI 🤖
WiM leverages the chunked prefill of the key-value cache, which concurrently generates query-based extractive summaries at each step of the prefill that are subsequently reintegrated at the end of the computation. We term these intermediate outputs “margins”, drawing inspiration from the practice of making margin notes for improved comprehension of long contexts in human reading. We show that this technique, which adds only minimal additional computation, significantly improves LLMs long context reasoning capabilities.
Think: Every chunk has a chance to be attended to/ be at the end of the context at least once. 🎉
📊 Results:
- An average accuracy boost of 7.5% in multi-hop reasoning tasks like HotpotQA and MultiHop-RAG.
- Even a 30% increase in F1-score for summarisation-like tasks (CWE).
Plus, WiM fits seamlessly into interactive applications (think: progress bar!). It can provide real-time progress updates during data retrieval and integration, making it user-friendly and transparent - a stark contrast to feeding 1mln tokens to an LLMs and waiting 6 min for the first token. 🤯
👩💻🧑💻 Check it out and contribute to our open-source project here: https://github.com/writer/writing-in-the-margins
🧠 More about chunked prefill: https://docs.vllm.ai/en/latest/models/performance.html#chunked-prefill
-
- 2 replies
upvoted
a
paper
about 1 year ago
published
an
article
about 1 year ago
Article
Using Writer Framework with Hugging Face Spaces
By
•
•
30
updated
2
Spaces
about 1 year ago
replied to
their
post
about 1 year ago
Thank you for all of your support!