nguyen599/ViDistilBERT-ESG-base

ESG analysis can help investors determine a business' long-term sustainability and identify associated risks. ViDistilBERT-ESG-base is a distilbert/distilbert-base-multilingual-cased model fine-tuned on ViEn-ESG-100 dataset, include 100,000 annotated sentences from Vietnam, English news and ESG reports.

Input: A financial text.

Output: Environmental, Social, Governance or None.

Language support: English, Vietnamese

How to use

You can use this model with Transformers pipeline for ESG classification.

# tested in transformers==4.51.0 
from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline

esgbert = AutoModelForSequenceClassification.from_pretrained('nguyen599/ViDistilBERT-ESG-base',num_labels=4)
tokenizer = AutoTokenizer.from_pretrained('nguyen599/ViDistilBERT-ESG-base')
nlp = pipeline("text-classification", model=esgbert, tokenizer=tokenizer)
results = nlp('Over three chapters, it covers a range of topics from energy efficiency and renewable energy to the circular economy and sustainable transportation.')
print(results) # [{'label': 'Environment', 'score': 0.9206041026115417}]

Benchmark

F1 scores of models on each ESG category in the English ViEn-ESG-100 dataset.

Model	Backbone	Param	E	S	G	N
SEC-BERT-ft	SEC-BERT-base	109M	83.12	66.77	66.53	60.30
FinBERT-ESG	FinBERT	109M	92.67	84.90	86.25	87.26
FinBERT-ESG-9-class	FinBERT	109M	92.16	89.01	91.35	86.89
ESGify	MPNet-base	109M	67.72	30.20	50.76	43.44
EnvironmentBERT	DistilRoBERTa	82M	92.15	-	-	92.76
SocialBERT	DistilRoBERTa	82M	-	76.81	-	81.23
GovernanceBERT	DistilRoBERTa	82M	-	-	64.46	80.06
ViBERT-ESG(Our)	BERT-base-cased	168M	93.76	94.53	94.98	94.15
ViRoBERTa-ESG(Our)	RoBERTa-base	124M	95.43	94.06	95.01	91.32
ViXLMRoBERTa-ESG(Our)	XLM-RoBERTa-base	278M	95.00	95.00	95.47	92.19
ViDeBERTa-ESG(Our)	DeBERTa-v3-base	184M	95.50	94.49	94.81	91.48
ViDeBERTa-small-ESG(Our)	DeBERTa-v3-small	141M	94.55	94.85	94.58	90.19
ViDistilBERT-ESG(Our)	DistilBERT-base-cased	135M	95.15	95.19	94.33	91.75
ViBERT-Env(Our)	BERT-base-cased	168M	94.62	-	-	92.13
ViBERT-Soc(Our)	BERT-base-cased	168M	-	94.86	-	92.22
ViBERT-Gov(Our)	BERT-base-cased	168M	-	-	93.47	93.82

F1 scores of models on each ESG category in the Vietnamese ViEn-ESG-100 dataset.

Model	Backbone	Param	E	S	G	N
ViBERT-ESG	BERT-base-cased	168M	93.50	89.73	91.77	91.78
ViRoBERTa-ESG	RoBERTa-base	124M	93.41	91.49	89.93	84.32
ViXLMRoBERTa-ESG	XLM-RoBERTa-base	278M	93.45	91.02	91.69	90.41
ViDeBERTa-ESG	DeBERTa-v3-base	184M	95.24	89.36	93.18	85.23
ViDeBERTa-small-ESG	DeBERTa-v3-small	141M	92.90	87.79	90.63	81.48
ViDistilBERT-ESG	DistilBERT-base-cased	135M	93.87	91.98	90.63	87.17
ViBERT-Env	BERT-base-cased	168M	94.87	-	-	91.15
ViBERT-Soc	BERT-base-cased	168M	-	91.07	-	90.29
ViBERT-Gov	BERT-base-cased	168M	-	-	92.62	90.11