head_dim in config.json is incorrect?
#36
by
zackangelo
- opened
Shouldn't head_dim be equal to hidden_size / num_attention_heads?
Maybe I'm missing something but that would mean head_dim == 64, right?
zackangelo
changed discussion title from
`head_dim` in config.json is incorrect?
to head_dim in config.json is incorrect?