Davlan/bert-base-multilingual-cased-ner-hrlNER命名实体识别模型

共8个文件

json：3个

bin：2个

gitattributes：1个

python

nlp

深度学习

需积分: 9 95 浏览量 2022-07-25 15:13:42 上传评论 2 收藏 627.59MB ZIP 举报

《Davlan/bert-base-multilingual-cased-ner-hrl：多语言命名实体识别的深度学习模型》在自然语言处理（NLP）领域，命名实体识别（NER）是一项重要的任务，它涉及到从文本中识别出具有特定意义的实体，如人名、地名和组织名等。Davlan/bert-base-multilingual-cased-ner-hrl模型是基于Hugging Face平台的一个高效解决方案，特别针对多语言环境，涵盖了中文和英文在内的16种语言的NER任务。此模型的核心是BERT（Bidirectional Encoder Representations from Transformers），由Google于2018年提出。BERT是一种预训练的Transformer架构，通过自我监督学习方式，如Masked Language Modeling和Next Sentence Prediction，从大量无标注文本中学习到丰富的语言表示。在预训练完成后，BERT可以被微调到各种下游任务，如NER。 Davlan/bert-base-multilingual-cased-ner-hrl模型的特点在于“cased”，这意味着它区分大小写，对于许多语言来说，大小写往往含有重要的语义信息。此外，模型支持多语言，这使得它能够处理跨语言的NER任务，无需为每种语言单独训练模型，极大地扩展了其应用范围。在实际使用中，开发者可以通过Python接口与Hugging Face的Transformers库进行交互，加载并微调该模型以适应特定任务的需求。以下是一个简单的示例代码： ```python from transformers import BertForTokenClassification, BertTokenizer model = BertForTokenClassification.from_pretrained("Davlan/bert-base-multilingual-cased-ner-hrl") tokenizer = BertTokenizer.from_pretrained("Davlan/bert-base-multilingual-cased-ner-hrl") inputs = tokenizer.encode("这是一个多语言NER模型的示例", return_tensors="pt") outputs = model(**inputs) # 解码并分析结果 ``` 在模型的微调过程中，通常会利用标注好的数据集，通过交叉熵损失函数进行优化，调整模型参数以提高对特定任务的性能。在评估阶段，常见的评估指标包括精确率（Precision）、召回率（Recall）和F1分数。值得注意的是，尽管Davlan/bert-base-multilingual-cased-ner-hrl模型已经具备良好的多语言识别能力，但针对某些特定语言或领域，可能还需要进行额外的定制化训练，以达到最优效果。此外，由于NER任务的复杂性，模型的推理速度和资源消耗也是需要考虑的因素，尤其是在资源有限的环境中。 Davlan/bert-base-multilingual-cased-ner-hrl模型是NLP领域的一个强大工具，它结合了BERT的深度学习能力与多语言识别的灵活性，为处理全球多语言的命名实体识别问题提供了有效途径。通过持续的优化和微调，我们可以进一步提升其在实际应用中的表现，推动多语言NLP技术的发展。

资源详情

资源评论

资源推荐

收起资源包目录

Davlan_bert-base-multilingual-cased-ner-hrl.zip （8个子文件）

Davlan_bert-base-multilingual-cased-ner-hrl

tokenizer_config.json 264B

special_tokens_map.json 112B

training_args.bin 1KB

vocab.txt 972KB

pytorch_model.bin 676.31MB

README.md 3KB

.gitattributes 737B

config.json 1KB

Hugging Face's logo --- language: - ar - de - en - es - fr - it - lv - nl - pt - zh - multilingual --- # bert-base-multilingual-cased-ner-hrl ## Model description **bert-base-multilingual-cased-ner-hrl** is a **Named Entity Recognition** model for 10 high resourced languages (Arabic, German, English, Spanish, French, Italian, Latvian, Dutch, Portuguese and Chinese) based on a fine-tuned mBERT base model. It has been trained to recognize three types of entities: location (LOC), organizations (ORG), and person (PER). Specifically, this model is a *bert-base-multilingual-cased* model that was fine-tuned on an aggregation of 10 high-resourced languages ## Intended uses & limitations #### How to use You can use this model with Transformers *pipeline* for NER. ```python from transformers import AutoTokenizer, AutoModelForTokenClassification from transformers import pipeline tokenizer = AutoTokenizer.from_pretrained("Davlan/bert-base-multilingual-cased-ner-hrl") model = AutoModelForTokenClassification.from_pretrained("Davlan/bert-base-multilingual-cased-ner-hrl") nlp = pipeline("ner", model=model, tokenizer=tokenizer) example = "Nader Jokhadar had given Syria the lead with a well-struck header in the seventh minute." ner_results = nlp(example) print(ner_results) ``` #### Limitations and bias This model is limited by its training dataset of entity-annotated news articles from a specific span of time. This may not generalize well for all use cases in different domains. ## Training data The training data for the 10 languages are from: Language|Dataset -|- Arabic | [ANERcorp](https://camel.abudhabi.nyu.edu/anercorp/) German | [conll 2003](https://www.clips.uantwerpen.be/conll2003/ner/) English | [conll 2003](https://www.clips.uantwerpen.be/conll2003/ner/) Spanish | [conll 2002](https://www.clips.uantwerpen.be/conll2002/ner/) French | [Europeana Newspapers](https://github.com/EuropeanaNewspapers/ner-corpora/tree/master/enp_FR.bnf.bio) Italian | [Italian I-CAB](https://ontotext.fbk.eu/icab.html) Latvian | [Latvian NER](https://github.com/LUMII-AILab/FullStack/tree/master/NamedEntities) Dutch | [conll 2002](https://www.clips.uantwerpen.be/conll2002/ner/) Portuguese |[Paramopama + Second Harem](https://github.com/davidsbatista/NER-datasets/tree/master/Portuguese) Chinese | [MSRA](https://huggingface.co/datasets/msra_ner) The training dataset distinguishes between the beginning and continuation of an entity so that if there are back-to-back entities of the same type, the model can output where the second entity begins. As in the dataset, each token will be classified as one of the following classes: Abbreviation|Description -|- O|Outside of a named entity B-PER |Beginning of a person’s name right after another person’s name I-PER |Person’s name B-ORG |Beginning of an organisation right after another organisation I-ORG |Organisation B-LOC |Beginning of a location right after another location I-LOC |Location ## Training procedure This model was trained on NVIDIA V100 GPU with recommended hyperparameters from HuggingFace code.