Instructions to use togethercomputer/GPT-JT-6B-v1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use togethercomputer/GPT-JT-6B-v1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="togethercomputer/GPT-JT-6B-v1")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("togethercomputer/GPT-JT-6B-v1")
model = AutoModelForCausalLM.from_pretrained("togethercomputer/GPT-JT-6B-v1")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use togethercomputer/GPT-JT-6B-v1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "togethercomputer/GPT-JT-6B-v1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "togethercomputer/GPT-JT-6B-v1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/togethercomputer/GPT-JT-6B-v1

SGLang

How to use togethercomputer/GPT-JT-6B-v1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "togethercomputer/GPT-JT-6B-v1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "togethercomputer/GPT-JT-6B-v1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "togethercomputer/GPT-JT-6B-v1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "togethercomputer/GPT-JT-6B-v1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use togethercomputer/GPT-JT-6B-v1 with Docker Model Runner:
```
docker model run hf.co/togethercomputer/GPT-JT-6B-v1
```

Question-Answering over documents

#19

by tmishinev - opened Jan 9, 2023

Discussion

tmishinev

Jan 9, 2023

Hello,

Does anyone use this model as a Reader in a QnA pipeline over document collection? I see I can use it with a
'''
Context:
XXXXX

Q:XXXXX
A:
'''
This kind of promt works and I can fetch context via embedding retrieval. Is there any other way I can train it with the document corpus ?

Best Regards,
Todor

juewang

Together org Jan 23, 2023

Hi, reading comprehension task should be doable, but the result should depend on the length type of the context. To improve accuracy, it is recommended to include a small number of examples :)

AayushShah

Mar 9, 2023

Hey, @juewang you really are amazing and helpful in helping out, I have been following you in the discussions here.
I need guidance in my project if you could help.

I want to tune the model (if it is possible either with the Trainer or other way) with some dataset of specific domain and then I should be able to ask the questions. Now, here I can see that we also need to pass the context from which we need to generate the answer.

Is there a way that doesn't require the context to be supplied? Just ask question and the model can answer? Like the close book answering procedure? If yes, could you direct me in that direction please.

Thanks,
Aayush

juewang

Together org Mar 11, 2023

Hey, @juewang you really are amazing and helpful in helping out, I have been following you in the discussions here.
I need guidance in my project if you could help.

I want to tune the model (if it is possible either with the Trainer or other way) with some dataset of specific domain and then I should be able to ask the questions. Now, here I can see that we also need to pass the context from which we need to generate the answer.

Is there a way that doesn't require the context to be supplied? Just ask question and the model can answer? Like the close book answering procedure? If yes, could you direct me in that direction please.

Thanks,
Aayush

Hello! Sure, closed-book answering is possible! We have recently released OpenChatKit, along with its training code base, which is designed for training chatbot models but can also be used to fine-tune GPT-JT (with dist_prefixlm_train.py). You shall prepare your dataset in jsonl format and perform fine-tuning.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment