Update README.md

README.md

@@ -73,7 +73,16 @@ TODO:
 - Experiment with new `-mla 3` (recent **ik_llama** patches enable new MLA implementation on CUDA)
 - Re-evaluate `-rtr` (in case Q8_0 can be repacked as Q8_0_R8 after some of the recent patches)
 
-## Quantization Approach
+### Inference Performance vs VRAM Considerations
+You can try the following to squeeze out more context on your system:
+- Running with `-ctk q8_0` can save some VRAM, but is a little slower on the target system
+- Reducing buffers can free up a bit more VRAM at a very minor cost to performance (`-amb 512` and `-b 1024 -ub 1024`)
+- Switching to an IQ quant will save some memory at the cost of performance (*very very roughly* 10% memory savings at the cost of 10% in inference performance)
+
+## Optimizing For coding
+Smaller quants, like `UD-Q2_K_XL` are much faster when generating tokens, but often produce code that fails to run or contains bugs. Based on empirical observations, coding seems to be strongly affected by the model quantization. So we use larger quantization where it matters to reduce perplexity while remaining within the target system constraints of 24GB-32GB VRAM, 512GB RAM.
+
+### Quantization Approach
 When running with **Flash MLA** optimization enabled, **ik_llama** will unpack **attention** layers into `Q8_0`, so we match that in our model (similar to ubergarm's ik_llama.cpp quants). We also keep all the other small layers as `Q8_0` while also leaving any `F32` layers untouched. The MoE layers make up the bulk of the model. The **ffn_down_exps** layers are especially sensitive to quantization (we borrow this idea from `unsloth` quants), so we quantize them as `Q6_K_R4`. Finally, all the other large MoE layers (ffn_up_exps, ffn_gate_exps) are quantized as `Q4_K_R4`
 
 Quantization Summary:
@@ -84,19 +93,11 @@ Quantization Summary:
 
 The **attn_kv_b** layers are included in the original model, but they contain the same information as **attn_k_b** and **attn_v_b** layers. Some quants, like `unsloth`, remove **attn_kv_b** layers altogether. We keep these layers for completeness, but push them out of VRAM with `attn_kv_b=CPU` when running the model.
 
-## Inference Performance vs VRAM Considerations
-You can try the following to squeeze out more context on your system:
-- Running with `-ctk q8_0` can save some VRAM, but is a little slower on the target system
-- Reducing buffers can free up a bit more VRAM at a very minor cost to performance (`-amb 512` and `-b 1024 -ub 1024`)
-- Switching to an IQ quant will save some memory at the cost of performance (*very very roughly* 10% memory savings at the cost of 10% in inference performance)
-
-## No imatrix
+### No imatrix
 Generally, imatrix is not recommended for Q4 and larger quants. The problem with imatrix is that it will guide what model remembers, while anything not covered by the text sample used to generate the imartrix is more likely to be forgotten. For example, an imatrix derived from wikipedia sample is likely to negatively affect tasks like coding. In other words, while imatrix can improve specific benchmarks, that are similar to the imatrix input sample, it will also skew the model performance towards tasks similar to the imatrix sample at the expense of other tasks.
 
 ## Benchmarks
-Smaller quants, like `UD-Q2_K_XL` are much faster when generating tokens, but often produce code that fails to run or contains bugs. Based on empirical observations, coding seems to be strongly affected by the model quantization. So we use larger quantization where it matters to reduce perplexity while remaining within the target system constraints of 24GB-32GB VRAM, 512GB RAM.
-
-**System:** Threadripper Pro 7975WX, 768GB DDR5@5600MHz, RTX 5090 32GB
+**Benchmark System:** Threadripper Pro 7975WX, 768GB DDR5@5600MHz, RTX 5090 32GB
 
 The following quants were tested:
 - **Q2_K_R4** (attention - `Q8_0`, all MoE - `Q2_K_R4`)