Beetle-FineWeb-100M
/

camera-ready-ablation-onset-35pct-total-l2-held-33pct-b3-sequential-nld-eng-fineweb-100m

Instructions to use Beetle-FineWeb-100M/camera-ready-ablation-onset-35pct-total-l2-held-33pct-b3-sequential-nld-eng-fineweb-100m with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Beetle-FineWeb-100M/camera-ready-ablation-onset-35pct-total-l2-held-33pct-b3-sequential-nld-eng-fineweb-100m with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Beetle-FineWeb-100M/camera-ready-ablation-onset-35pct-total-l2-held-33pct-b3-sequential-nld-eng-fineweb-100m", trust_remote_code=True)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("Beetle-FineWeb-100M/camera-ready-ablation-onset-35pct-total-l2-held-33pct-b3-sequential-nld-eng-fineweb-100m", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Beetle-FineWeb-100M/camera-ready-ablation-onset-35pct-total-l2-held-33pct-b3-sequential-nld-eng-fineweb-100m with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Beetle-FineWeb-100M/camera-ready-ablation-onset-35pct-total-l2-held-33pct-b3-sequential-nld-eng-fineweb-100m"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Beetle-FineWeb-100M/camera-ready-ablation-onset-35pct-total-l2-held-33pct-b3-sequential-nld-eng-fineweb-100m",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Beetle-FineWeb-100M/camera-ready-ablation-onset-35pct-total-l2-held-33pct-b3-sequential-nld-eng-fineweb-100m

SGLang

How to use Beetle-FineWeb-100M/camera-ready-ablation-onset-35pct-total-l2-held-33pct-b3-sequential-nld-eng-fineweb-100m with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Beetle-FineWeb-100M/camera-ready-ablation-onset-35pct-total-l2-held-33pct-b3-sequential-nld-eng-fineweb-100m" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Beetle-FineWeb-100M/camera-ready-ablation-onset-35pct-total-l2-held-33pct-b3-sequential-nld-eng-fineweb-100m",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Beetle-FineWeb-100M/camera-ready-ablation-onset-35pct-total-l2-held-33pct-b3-sequential-nld-eng-fineweb-100m" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Beetle-FineWeb-100M/camera-ready-ablation-onset-35pct-total-l2-held-33pct-b3-sequential-nld-eng-fineweb-100m",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use Beetle-FineWeb-100M/camera-ready-ablation-onset-35pct-total-l2-held-33pct-b3-sequential-nld-eng-fineweb-100m with Docker Model Runner:
```
docker model run hf.co/Beetle-FineWeb-100M/camera-ready-ablation-onset-35pct-total-l2-held-33pct-b3-sequential-nld-eng-fineweb-100m
```

camera-ready-ablation-onset-35pct-total-l2-held-33pct-b3-sequential-nld-eng-fineweb-100m / training_config.yaml

suchirsalhan

Saving Training Config -- Step 0

fd79bbc verified about 2 months ago

Raw History Blame Contribute Delete

2.13 kB

run_name: camera-ready-ablation-onset-35pct-total-l2-held-33pct-b3-sequential-nld-eng-fineweb-100m
hf_user: Beetle-FineWeb-100M
model:
  architecture: pico
  n_layers: 14
  d_model: 768
  vocab_size: 50000
  attention_n_heads: 12
  attention_n_kv_heads: 1
  max_seq_len: 512
  batch_size: 64
  position_emb_theta: 10000.0
  activation_hidden_dim: 3072
  norm_eps: 1.0e-05
  dropout: 0.1
data:
  data_mix: B3
  lang_sources:
  - Beetle-Data/nl-100M-pretok
  l2_streams:
  - Beetle-Data/en-for-nl-100M-pretok
  pretokenized: true
  pretokenized_cache_dir: ./pretok_cache/humanscale-100m/nld_L1-eng_L2
tokenizer:
  pretrained_tokenizer_path: Beetle-Data/tokenizer-nl-en
  dataset_source: fineweb
  vocab_size: 50000
  vocab_sharing: shared
  type: bpe
  target_tokens: 100000000
  output_dir: ./tokenizers/fineweb_humanscale_nld_L1-eng_L2_shared
  overwrite: false
  push_to_hub: true
training:
  fabric:
    accelerator: cuda
    devices: 1
    strategy: ddp
    precision: bf16-mixed
  optimization:
    optimizer: adamw
    lr: 0.0005
    lr_warmup_steps: 500
    lr_scheduler: linear_with_warmup
  max_steps: 10000
checkpointing:
  runs_dir: ./runs
  run_name: camera-ready-ablation-onset-35pct-total-l2-held-33pct-b3-sequential-nld-eng-fineweb-100m
  checkpoints_dir: checkpoints
  fabric_checkpoint_dir: fabric_checkpoint
  fabric_checkpoint_filename: checkpoint.pt
  logs_dir: logs
  save_every_n_steps: 1000
  keep_local_checkpoints: 1
  save_to_hf: true
  learning_dynamics: {}
  compute_gradients: false
curriculum:
  type: smooth
  mode: tokens
  schedule:
  - kind: sigmoid_ramp
    from_weights:
      l1: 1.0
      l2: 0.0
    to_weights:
      l1: 0.4846
      l2: 0.5154
    midpoint: 0.35
    steepness: 12.0
ewc:
  enabled: false
maml:
  enabled: false
generative_replay:
  enabled: false
polca:
  enabled: false
  every_n_steps: 500
  k_eigenvectors: 32
  basis_rank: 32
  holdout_batch_size: 8
  holdout_seq_len: 512
  artifact_dir: polca
  push_to_hf: false
  precision: fp32
monitoring:
  logging:
    log_level: INFO
    log_every_n_steps: 100
  save_to_wandb: false
  wandb:
    project: ''
    entity: ''

	run_name: camera-ready-ablation-onset-35pct-total-l2-held-33pct-b3-sequential-nld-eng-fineweb-100m
	hf_user: Beetle-FineWeb-100M
	model:
	architecture: pico
	n_layers: 14
	d_model: 768
	vocab_size: 50000
	attention_n_heads: 12
	attention_n_kv_heads: 1
	max_seq_len: 512
	batch_size: 64
	position_emb_theta: 10000.0
	activation_hidden_dim: 3072
	norm_eps: 1.0e-05
	dropout: 0.1
	data:
	data_mix: B3
	lang_sources:
	- Beetle-Data/nl-100M-pretok
	l2_streams:
	- Beetle-Data/en-for-nl-100M-pretok
	pretokenized: true
	pretokenized_cache_dir: ./pretok_cache/humanscale-100m/nld_L1-eng_L2
	tokenizer:
	pretrained_tokenizer_path: Beetle-Data/tokenizer-nl-en
	dataset_source: fineweb
	vocab_size: 50000
	vocab_sharing: shared
	type: bpe
	target_tokens: 100000000
	output_dir: ./tokenizers/fineweb_humanscale_nld_L1-eng_L2_shared
	overwrite: false
	push_to_hub: true
	training:
	fabric:
	accelerator: cuda
	devices: 1
	strategy: ddp
	precision: bf16-mixed
	optimization:
	optimizer: adamw
	lr: 0.0005
	lr_warmup_steps: 500
	lr_scheduler: linear_with_warmup
	max_steps: 10000
	checkpointing:
	runs_dir: ./runs
	run_name: camera-ready-ablation-onset-35pct-total-l2-held-33pct-b3-sequential-nld-eng-fineweb-100m
	checkpoints_dir: checkpoints
	fabric_checkpoint_dir: fabric_checkpoint
	fabric_checkpoint_filename: checkpoint.pt
	logs_dir: logs
	save_every_n_steps: 1000
	keep_local_checkpoints: 1
	save_to_hf: true
	learning_dynamics: {}
	compute_gradients: false
	curriculum:
	type: smooth
	mode: tokens
	schedule:
	- kind: sigmoid_ramp
	from_weights:
	l1: 1.0
	l2: 0.0
	to_weights:
	l1: 0.4846
	l2: 0.5154
	midpoint: 0.35
	steepness: 12.0
	ewc:
	enabled: false
	maml:
	enabled: false
	generative_replay:
	enabled: false
	polca:
	enabled: false
	every_n_steps: 500
	k_eigenvectors: 32
	basis_rank: 32
	holdout_batch_size: 8
	holdout_seq_len: 512
	artifact_dir: polca
	push_to_hf: false
	precision: fp32
	monitoring:
	logging:
	log_level: INFO
	log_every_n_steps: 100
	save_to_wandb: false
	wandb:
	project: ''
	entity: ''