jinaai
/

jina-reranker-v2-base-multilingual

Transformers.js

sentence-transformers

text-classification

🇪🇺 Region: EU

Model card Files Files and versions

numb3r3 commited on 27 days ago

Commit

9cfeff2

·

verified ·

1 Parent(s): 2f894e6

fix: empty passage

Files changed (1) hide show

modeling_xlm_roberta.py +2 -2

modeling_xlm_roberta.py CHANGED Viewed

@@ -1056,9 +1056,9 @@ def reranker_tokenize_preproc(
         chunk1['input_ids'].append(sep_id)
         chunk1['input_ids'].extend(chunk2['input_ids'])
         chunk1['input_ids'].append(sep_id)
-        chunk1['attention_mask'].append(chunk2['attention_mask'][0])
         chunk1['attention_mask'].extend(chunk2['attention_mask'])
-        chunk1['attention_mask'].append(chunk2['attention_mask'][-1])
         if 'token_type_ids' in chunk1:
             token_type_ids = [1 for _ in range(len(chunk2['token_type_ids']) + 2)]
             chunk1['token_type_ids'].extend(token_type_ids)

         chunk1['input_ids'].append(sep_id)
         chunk1['input_ids'].extend(chunk2['input_ids'])
         chunk1['input_ids'].append(sep_id)
+        chunk1['attention_mask'].append(1)
         chunk1['attention_mask'].extend(chunk2['attention_mask'])
+        chunk1['attention_mask'].append(1)
         if 'token_type_ids' in chunk1:
             token_type_ids = [1 for _ in range(len(chunk2['token_type_ids']) + 2)]
             chunk1['token_type_ids'].extend(token_type_ids)