Pretrained BERT base language model for Arabic
bert-base-arabic model was pretrained on ~8.2 Billion words:
- Arabic version of OSCAR - filtered from Common Crawl
- Recent dump of Arabic Wikipedia
and other Arabic resources which sum up to ~95GB of text.
Notes on training data:
- Our final version of corpus contains some non-Arabic words inlines, which we did not remove from sentences since that would affect some tasks like NER.
- Although non-Arabic characters were lowered as a preprocessing step, since Arabic characters does not have upper or lower case, there is no cased and uncased version of the model.
- The corpus and vocabulary set are not restricted to Modern Standard Arabic, they contain some dialectical Arabic too.
- This model was trained using Google BERT's github repository on a single TPU v3-8 provided for free from TFRC.
- Our pretraining procedure follows training settings of bert with some changes: trained for 3M training steps with batchsize of 128, instead of 1M with batchsize of 256.
- You can find the outputs of the training process on tensorboard: Arabic-BERT
You can use this model by installing torch or tensorflow and Huggingface library transformers. And you can use it directly by initializing it like this:
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("asafaya/bert-base-arabic")
model = AutoModel.from_pretrained("asafaya/bert-base-arabic")| Dataset | Details | ML-BERT | hULMona | Arabic-BERT |
|---|---|---|---|---|
| HARD | 2 Classes, Mixed dialects | 0.957 | 0.957 | - |
| ArSenLev | 5 Classes, Levantine dialect | 0.510 | 0.511 | 0.552 |
| ASTD | 4 Classes, MSA and Egyptian dialects | 0.670 | 0.677 | 0.714 |
To be added.
Note: More results on other downstream NLP tasks will be added soon. if you use this model, I would appreciate your feedback.
Thanks to Google for providing free TPU for the training process and for Huggingface for hosting this model on their servers 😊