Ollpheist-12B

Probably a good generalist model.

Merge Details

Merge Method

This model was merged using the Karcher Mean merge method.

Models Merged

The following models were included in the merge:

• yamatazen/LorablatedStock-12B
• yamatazen/EtherealAurora-12B
• DreadPoor/Irix-12B-Model_Stock
• yamatazen/BlueLight-12B

Reproduction steps

Spoiler warning

merge_method: passthrough
slices:
- sources: # Best Willingness 
  - model: yamatazen/LorablatedStock-12B
    layer_range: [0, 6]

- sources: # Best NatInt
  - model: yamatazen/EtherealAurora-12B
    layer_range: [6, 34]

- sources: # Best UGI 
  - model: DreadPoor/Irix-12B-Model_Stock
    layer_range: [34, 40]    
dtype: bfloat16

mergekit-yaml ollpheist_base.yml ./Ollpheist-base-12B

merge_method: arcee_fusion
base_model: ./Ollpheist-12B
models:
  - model: ./Ollpheist-12B
  - model: yamatazen/BlueLight-12B # Best Coding 
dtype: bfloat16
tokenizer_source: ./Ollpheist-12B

mergekit-yaml ollpheist_fusion.yml ./Ollpheist-fusion-12B

merge_method: karcher
models:
  - model: ./Ollpheist-base-12B
  - model: ./Ollpheist-fusion-12B
parameters:
  max_iter: 100000
  tol: 1e-9
dtype: bfloat16

mergekit-yaml ollpheist_karcher.yml ./Ollpheist-k-12B

merge_method: karcher
models:
  - model: ./Ollpheist-base-12B
  - model: ./Ollpheist-k-12B
parameters:
  max_iter: 100000
  tol: 1e-9
dtype: bfloat16

mergekit-yaml ollpheist_karcher2.yml ./Ollpheist-12B

Acknowledgments

• Team mradermacher: for awesome quants